Member-only story
Stream Processing Introduction
Easy to understand what is streaming architecture
The purpose of this article is to introduce stream processing in an easy-to-understand way, so it does not dive too deep into the technical details, nor does it mention specific frameworks. However, a few popular frameworks will be used as examples to illustrate.
Before we start to introduce stream processing, let’s talk about batch processing, i.e., the opposing concept of stream processing.
Batch processing
I believe most of you are familiar with batch processing, in which a large scale data processing is performed after a given period of data and the final result is produced. This is the actual operation of batch processing, and the expertise area of batch processing is fixed amount of data processing.
However, in an event-driven architecture, the data, i.e., the events, are endless, that is to say, it is difficult to define a fixed amount. Therefore, the compromise is to change the fixed amount to a fixed time, so that the amount of data can be expected after a given time interval, and batch processing can be applied.
In other words, batch processing can be summarized in several characteristics.
- Fixed. Whether it is fixed time or fixed volume…
