A data stream is a continuous flow of data that comes from various systems. The difference to traditional batch processing is that streams can be processed immediately. This holds many benefits such as:

• More up-to-date insights
  Data can be processed faster and in turn speed up the way from data to value.
   
• Enabling new use cases
  Stream processing character enables new use cases like real time monitoring and other operational scenarios.
   
• Increase flexibility in architectures
  Building streaming-first architectures introduces a new way of thinking architectures which makes it easier to integrate various systems and thereby increase the flexibility.

Streaming Pipelines

The most straight-forward way to implement a real time system is the use of a streaming pipeline that usually consists out of the following components:

• Stream Sourcing
  The source of a data stream can be any system from actual data source, like a sensor in a car, over more sophisticated edge processing components to batch-oriented systems, where streaming data can be extracted via change data capture.
   
• Stream Transportation
  Somehow the data has to be transported from one system to another. Here, the ways differ in their complexity from direct messaging up to more complex event broker or event logs.
   
• Stream Processing
  This is the part where the actual work happens. Data is transformed and processed with methods like complex event processing or other analytics tools. The results are then passed to other subsequent systems or constitute a new stream for another pipeline.

Event-driven Architectures

In more sophisticated scenarios event-driven architectures can help to cope with complexity and acelerate data processing. The idea is that a central event-hub serves as a router for various event provider and event consumer. Events are then saved in message queues (topics) that can be subscribed and read by various consumer in a asynchronous way. The majore benefits of such an approach are:

• Flexibility
  It is easy to integrate new event provider and consumer. Moreover, a topic can be subscribed by arbitrary consumers without interference of other systems.
   
• Fault-Tolerance
  The asynchronous idea of an event-driven architecture makes it very fault-tolerance as a failure in one node does not affect the system at all. Moreover, event loks, like Apache Kafka, persist the messages queues, which shifts the persistence responsibility from the consumer to the event log and enables new use cases.
   
• Scalability
  Modern event-logs, like Apache Kafka or Azure Event Hubs, allow distributed operation and thereby are easily scalable to process large data volumes.

Conclusion

Streaming becomes an increasingly important part of modern software systems. Depending on the use case a streamlined pipeline or a more sophisticated event-driven architecture can provide an adequate solution. Due to the manifold technologies and products it is important to analyze the requirements and build a future-proof software landscape.