{"id":24830,"date":"2020-08-03T06:00:00","date_gmt":"2020-08-03T13:00:00","guid":{"rendered":"https:\/\/insidebigdata.com\/?p=24830"},"modified":"2020-08-04T09:19:55","modified_gmt":"2020-08-04T16:19:55","slug":"analyze-then-store-the-journey-to-continuous-intelligence-2","status":"publish","type":"post","link":"https:\/\/insidebigdata.com\/2020\/08\/03\/analyze-then-store-the-journey-to-continuous-intelligence-2\/","title":{"rendered":"Analyze-then-Store: The Journey to Continuous Intelligence – Part 2"},"content":{"rendered":"\n

A Technical Article Series for Data Architects<\/p>\n\n\n\n

This multi-part article series is intended for data architects and anyone else interested in learning how to design modern real-time data analytics solutions. It explores key principles and implications of event streaming and streaming analytics, and concludes that the biggest opportunity to derive meaningful value from data \u2013 and gain continuous intelligence about the state of things – lies in the ability to analyze, learn and predict from real-time events in concert with contextual, static and dynamic data. This article series places continuous intelligence in an architectural context, with reference to established technologies and use cases in place today.<\/p>\n\n\n\n

Part 2: Event Streaming Applications<\/strong><\/p>\n\n\n\n

Brokers don\u2019t execute applications – they simply act as a buffer between the real world and applications. The broker keeps an index per application and topic so if an application restarts, it can resume processing from the next event in each queue – if it is stateful. If not, it must start from the head of each queue again. But re-playing events for a long period of (real-world) time is wasteful and leaves the application even further behind the real-world, which never stops!<\/p>\n\n\n\n

Every application that delivers useful insights needs a model of the system generating events. The stream may be used to deliver data to a machine learning pipeline, or modify a relational or graph database, but ultimately the reasoning about the meaning<\/em> of each event and the state changes it communicates both to an entity and to related entities in the context of the system overall requires a stateful mod<\/em>el which is typically saved in a database.  The unfortunate consequence is that now applications need to deal with two sources of delay – topic queues at the broker and a database at the application layer.  Neither is predictable in its delay characteristics, making it impossible to bound response times.<\/p>\n\n\n\n

If the application state is organized in a database, event streaming fits nicely into a microservices architecture. Applications can scale by adding stateless microservice instances to consume events.  Each instance independently asks the broker for the next event – avoiding the need for a load balancer, and if an instance fails it can simply restart and carry on. <\/p>\n\n\n\n

But applications are always stateful, and pushing the hard problems of distribution, consistency, availability, load balancing and resilience onto the database layer impacts performance and ultimately cannot deliver continuous intelligence: A database is after all simply a repository that reflects an application\u2019s current model of the system – it doesn\u2019t compute based on that. Context is problematic too: Processing a single event can cause scores of database accesses as its knock-on effects change the states of many additional entities. Each access adds to the end-to-end processing latency for each event.<\/p>\n\n\n\n

Brokers Buffer Data<\/strong><\/p>\n\n\n\n

Historically databases attempted to represent the current state of the system, on disk.  But the demand for data-driven computation of insights or responses means that the idea of a database as an organized repository for state is no longer sufficient. The application logic \u2013 analysis, learning and prediction \u2013 need to be carried out in real-time, continuously, as data flows into the application. Time is of the essence: If the app is too slow it will lose track and deliver useless insights, and \u201cstore-then-analyze\u201d architectures fall into this trap. <\/p>\n\n\n\n

Unfortunately, topic queues managed by the broker introduce a serious problem:<\/p>\n\n\n\n