Streaming-First Architectures - Real Time and Event-Driven

Having the right data at the right time

Having the right data at the right time is essential for organizations that need to compete. Especially in today’s fast-moving and data-driven world!
Accordingly, software systems need to process data in (near) real time and provide a flexibility that allows to quickly integrate new systems and data sources.

A data stream is a continuous flow of data that comes from various systems. The difference to traditional batch processing is that streams can be processed immediately. This holds many benefits such as:

  • More up-to-date insights
    Data can be processed faster and in turn speed up the way from data to value.
     
  • Enabling new use cases
    Stream processing character enables new use cases like real time monitoring and other operational scenarios.
     
  • Increase flexibility in architectures
    Building streaming-first architectures introduces a new way of thinking architectures which makes it easier to integrate various systems and thereby increase the flexibility.

Streaming Pipelines

The most straight-forward way to implement a real time system is the use of a streaming pipeline that usually consists out of the following components:

  • Stream Sourcing
    The source of a data stream can be any system from actual data source, like a sensor in a car, over more sophisticated edge processing components to batch-oriented systems, where streaming data can be extracted via change data capture.
     
  • Stream Transportation
    Somehow the data has to be transported from one system to another. Here, the ways differ in their complexity from direct messaging up to more complex event broker or event logs.
     
  • Stream Processing
    This is the part where the actual work happens. Data is transformed and processed with methods like complex event processing or other analytics tools. The results are then passed to other subsequent systems or constitute a new stream for another pipeline.

Erfahren Sie mehr in unserem Whitepaper

Die richtigen Informationen zum richtigen Zeitpunkt zu haben ist ein wichtiger Erfolgsfaktor in der heutigen Welt. Entsprechend müssen Daten mit Hilfe von Streaming Systemen in Echtzeit verarbeitet werden und Software-Architekturen so strukturiert werden, dass schnell auf unvorhergesehene Anforderungen reagiert werden kann. Unser Whitepaper zum Thema „Streaming-First Architectures“ stellt aktuelle Herausforderungen für Software-Architekturen dar und diskutiert wie Streaming Pipelines und Event-getriebene Architekturen in diesem Kontext eingesetzt werden können.

Event-driven Architectures

In more sophisticated scenarios event-driven architectures can help to cope with complexity and acelerate data processing. The idea is that a central event-hub serves as a router for various event provider and event consumer. Events are then saved in message queues (topics) that can be subscribed and read by various consumer in a asynchronous way. The majore benefits of such an approach are:

  • Flexibility
    It is easy to integrate new event provider and consumer. Moreover, a topic can be subscribed by arbitrary consumers without interference of other systems.
     
  • Fault-Tolerance
    The asynchronous idea of an event-driven architecture makes it very fault-tolerance as a failure in one node does not affect the system at all. Moreover, event loks, like Apache Kafka, persist the messages queues, which shifts the persistence responsibility from the consumer to the event log and enables new use cases.
     
  • Scalability
    Modern event-logs, like Apache Kafka or Azure Event Hubs, allow distributed operation and thereby are easily scalable to process large data volumes.

Conclusion

Streaming becomes an increasingly important part of modern software systems. Depending on the use case a streamlined pipeline or a more sophisticated event-driven architecture can provide an adequate solution. Due to the manifold technologies and products it is important to analyze the requirements and build a future-proof software landscape.

Sprechen Sie mit unserem Experten

Maximilian Heim

UI/UX Designer