{"id":24889,"date":"2020-08-17T06:00:00","date_gmt":"2020-08-17T13:00:00","guid":{"rendered":"https:\/\/insidebigdata.com\/?p=24889"},"modified":"2020-08-18T09:30:47","modified_gmt":"2020-08-18T16:30:47","slug":"analyze-then-store-the-journey-to-continuous-intelligence-part-3","status":"publish","type":"post","link":"https:\/\/insidebigdata.com\/2020\/08\/17\/analyze-then-store-the-journey-to-continuous-intelligence-part-3\/","title":{"rendered":"Analyze-then-Store: The Journey to Continuous Intelligence \u2013 Part 3"},"content":{"rendered":"\n

A Technical Article Series for Data Architects<\/p>\n\n\n\n

This multi-part article series is intended for data architects and anyone else interested in learning how to design modern real-time data analytics solutions. It explores key principles and implications of event streaming and streaming analytics, and concludes that the biggest opportunity to derive meaningful value from data \u2013 and gain continuous intelligence about the state of things – lies in the ability to analyze, learn and predict from real-time events in concert with contextual, static and dynamic data. This article series places continuous intelligence in an architectural context, with reference to established technologies and use cases in place today.<\/p>\n\n\n\n

Part 3: Smarter Databases Can\u2019t Help<\/strong><\/p>\n\n\n\n

The rise of event streaming is due to the importance of the change in system behavior over time<\/em>.  The challenge at the application layer is to deliver intelligence gleaned from streamed events, continuously, over time<\/em>.  But a major drawback of event streaming architectures is that they store events in topic queues ordered by event arrival time<\/em>, forcing applications to only consume from the head of the queue. What we\u2019d really like is a stream of intelligence – delivered like events – that results from the continuously concurrent interpretation of the effects of all events on a stateful model of the system. Instead, today\u2019s application developers must use some sort of database to represent the state of the system. And that just isn\u2019t enough: modifying a representation of the system state in response to changes communicated in events is one thing but delivering a continuous stream of intelligence that results from those changes is quite another. Databases can help with the first – but lead to performance impacts. They do nothing for the second (see figure 4).  <\/p>\n\n\n\n

\"\"<\/figure><\/div>\n\n\n\n

There is a vast number of databases and cloud database services available.  Most can store streaming data, and many have evolved powerful features to confirm their role as masters of application state. Sophisticated data management capabilities are migrating into the database engine to deal with latency challenges. Leading the feature development race are the hosted database services from the major cloud providers.<\/p>\n\n\n\n

But there are hundreds of others. Broadly the trend is toward large in-memory stores, grids and caches that attempt to reduce latency. All of today\u2019s database engines can ingest events at huge rates.  But that\u2019s not the problem. No database, in-memory or other, can understand the meaning <\/em>of data, or deliver real-time, situationally relevant responses. Applications interpret events from the real-world to change a model of the state of the system, but a single event may cause state changes to multiple related entities.  By way of example: A truck needing maintenance enters a geo-fence meaning that the truck is near an inspector, so the inspector is alerted. A single event with the GPS coordinates of the truck might change the states of the geo-fence and the inspector. Every time the states or relationships between entities change, the application may need to evaluate sophisticated logical or mathematical predicates, joins, maps or other aggregations, and <\/em>execute business logic. Each of these might require scores of round-trips to the database. For every truck, and every inspector, in real-time.<\/p>\n\n\n\n

For an application at scale, this leads rapidly to a situation where the database is the bottleneck. For distributed applications, the round-trip latency for database access can quickly dominate performance. For an application processing hundreds of thousands of events per second, the only way to reduce latency is to execute application logic in the memory context of each impacted entity, avoiding database latency entirely. That is exactly what Swim does.<\/p>\n\n\n\n

There\u2019s another reason that smarter databases can\u2019t help with continuous intelligence: They don\u2019t drive computation<\/em> of insights or \u201cpush\u201d them to users. The inversion of the control loop is fundamental: In most applications that claim to be real-time, the query to a database drives computation, and the results are delivered to the user.  But that\u2019s not enough for today\u2019s continuous intelligence use cases.  Users want to deliver real-time responses to analysis, learning and prediction, as it occurs, concurrently for all entities in the system. They want the application to always have an answer. Databases don\u2019t do that. <\/p>\n\n\n\n

To read parts 1 and 2 of this guest article series, please visit the Swim blog<\/a>.<\/em><\/p>\n\n\n\n

About the Author<\/strong><\/p>\n\n\n\n

Simon Crosby is CTO at Swim<\/a>. Swim offers the first open core, enterprise-grade platform for continuous intelligence at scale, providing businesses with complete situational awareness and operational decision support at every moment. Simon co-founded Bromium (now HP SureClick) in 2010 and currently serves as a strategic advisor. Previously, he was the CTO of the Data Center and Cloud Division at Citrix Systems; founder, CTO, and vice president of strategy and corporate development at XenSource; and a principal engineer at Intel, as well as a faculty member at Cambridge University, where he led the research on network performance and control and multimedia operating systems.<\/em><\/p>\n\n\n\n

Simon is an equity partner at DCVC, serves on the board of Cambridge in America, and is an investor in and advisor to numerous startups. He is the author of 35 research papers and patents on a number of data center and networking topics, including security, network and server virtualization, and resource optimization and performance. He holds a PhD in computer science from the University of Cambridge, an MSc from the University of Stellenbosch, South Africa, and a BSc (with honors) in computer science and mathematics from the University of Cape Town, South Africa.<\/em><\/em><\/p>\n","protected":false},"excerpt":{"rendered":"

This multi-part article series by our friends at Swim is intended for data architects and anyone else interested in learning how to design modern real-time data analytics solutions. It explores key principles and implications of event streaming and streaming analytics, and concludes that the biggest opportunity to derive meaningful value from data \u2013 and gain continuous intelligence about the state of things \u2013 lies in the ability to analyze, learn and predict from real-time events in concert with contextual, static and dynamic data. This article series places continuous intelligence in an architectural context, with reference to established technologies and use cases in place today.<\/p>\n","protected":false},"author":10513,"featured_media":24770,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"footnotes":""},"categories":[65,115,87,180,56,97,311,1],"tags":[335,729,317,488,95],"acf":[],"yoast_head":"\nAnalyze-then-Store: The Journey to Continuous Intelligence \u2013 Part 3 - insideBIGDATA<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/insidebigdata.com\/2020\/08\/17\/analyze-then-store-the-journey-to-continuous-intelligence-part-3\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Analyze-then-Store: The Journey to Continuous Intelligence \u2013 Part 3 - insideBIGDATA\" \/>\n<meta property=\"og:description\" content=\"This multi-part article series by our friends at Swim is intended for data architects and anyone else interested in learning how to design modern real-time data analytics solutions. It explores key principles and implications of event streaming and streaming analytics, and concludes that the biggest opportunity to derive meaningful value from data \u2013 and gain continuous intelligence about the state of things \u2013 lies in the ability to analyze, learn and predict from real-time events in concert with contextual, static and dynamic data. This article series places continuous intelligence in an architectural context, with reference to established technologies and use cases in place today.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/insidebigdata.com\/2020\/08\/17\/analyze-then-store-the-journey-to-continuous-intelligence-part-3\/\" \/>\n<meta property=\"og:site_name\" content=\"insideBIGDATA\" \/>\n<meta property=\"article:publisher\" content=\"http:\/\/www.facebook.com\/insidebigdata\" \/>\n<meta property=\"article:published_time\" content=\"2020-08-17T13:00:00+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2020-08-18T16:30:47+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/insidebigdata.com\/wp-content\/uploads\/2020\/07\/Data_architecture_shutterstock_562411702.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"300\" \/>\n\t<meta property=\"og:image:height\" content=\"212\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Editorial Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@insideBigData\" \/>\n<meta name=\"twitter:site\" content=\"@insideBigData\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Editorial Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"5 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/insidebigdata.com\/2020\/08\/17\/analyze-then-store-the-journey-to-continuous-intelligence-part-3\/\",\"url\":\"https:\/\/insidebigdata.com\/2020\/08\/17\/analyze-then-store-the-journey-to-continuous-intelligence-part-3\/\",\"name\":\"Analyze-then-Store: The Journey to Continuous Intelligence \u2013 Part 3 - insideBIGDATA\",\"isPartOf\":{\"@id\":\"https:\/\/insidebigdata.com\/#website\"},\"datePublished\":\"2020-08-17T13:00:00+00:00\",\"dateModified\":\"2020-08-18T16:30:47+00:00\",\"author\":{\"@id\":\"https:\/\/insidebigdata.com\/#\/schema\/person\/2949e412c144601cdbcc803bd234e1b9\"},\"breadcrumb\":{\"@id\":\"https:\/\/insidebigdata.com\/2020\/08\/17\/analyze-then-store-the-journey-to-continuous-intelligence-part-3\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/insidebigdata.com\/2020\/08\/17\/analyze-then-store-the-journey-to-continuous-intelligence-part-3\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/insidebigdata.com\/2020\/08\/17\/analyze-then-store-the-journey-to-continuous-intelligence-part-3\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/insidebigdata.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Analyze-then-Store: The Journey to Continuous Intelligence \u2013 Part 3\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/insidebigdata.com\/#website\",\"url\":\"https:\/\/insidebigdata.com\/\",\"name\":\"insideBIGDATA\",\"description\":\"Your Source for AI, Data Science, Deep Learning & Machine Learning Strategies\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/insidebigdata.com\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/insidebigdata.com\/#\/schema\/person\/2949e412c144601cdbcc803bd234e1b9\",\"name\":\"Editorial Team\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/insidebigdata.com\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/e137ce7ea40e38bd4d25bb7860cfe3e4?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/e137ce7ea40e38bd4d25bb7860cfe3e4?s=96&d=mm&r=g\",\"caption\":\"Editorial Team\"},\"sameAs\":[\"http:\/\/www.insidebigdata.com\"],\"url\":\"https:\/\/insidebigdata.com\/author\/editorial\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Analyze-then-Store: The Journey to Continuous Intelligence \u2013 Part 3 - insideBIGDATA","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/insidebigdata.com\/2020\/08\/17\/analyze-then-store-the-journey-to-continuous-intelligence-part-3\/","og_locale":"en_US","og_type":"article","og_title":"Analyze-then-Store: The Journey to Continuous Intelligence \u2013 Part 3 - insideBIGDATA","og_description":"This multi-part article series by our friends at Swim is intended for data architects and anyone else interested in learning how to design modern real-time data analytics solutions. It explores key principles and implications of event streaming and streaming analytics, and concludes that the biggest opportunity to derive meaningful value from data \u2013 and gain continuous intelligence about the state of things \u2013 lies in the ability to analyze, learn and predict from real-time events in concert with contextual, static and dynamic data. This article series places continuous intelligence in an architectural context, with reference to established technologies and use cases in place today.","og_url":"https:\/\/insidebigdata.com\/2020\/08\/17\/analyze-then-store-the-journey-to-continuous-intelligence-part-3\/","og_site_name":"insideBIGDATA","article_publisher":"http:\/\/www.facebook.com\/insidebigdata","article_published_time":"2020-08-17T13:00:00+00:00","article_modified_time":"2020-08-18T16:30:47+00:00","og_image":[{"width":300,"height":212,"url":"https:\/\/insidebigdata.com\/wp-content\/uploads\/2020\/07\/Data_architecture_shutterstock_562411702.jpg","type":"image\/jpeg"}],"author":"Editorial Team","twitter_card":"summary_large_image","twitter_creator":"@insideBigData","twitter_site":"@insideBigData","twitter_misc":{"Written by":"Editorial Team","Est. reading time":"5 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/insidebigdata.com\/2020\/08\/17\/analyze-then-store-the-journey-to-continuous-intelligence-part-3\/","url":"https:\/\/insidebigdata.com\/2020\/08\/17\/analyze-then-store-the-journey-to-continuous-intelligence-part-3\/","name":"Analyze-then-Store: The Journey to Continuous Intelligence \u2013 Part 3 - insideBIGDATA","isPartOf":{"@id":"https:\/\/insidebigdata.com\/#website"},"datePublished":"2020-08-17T13:00:00+00:00","dateModified":"2020-08-18T16:30:47+00:00","author":{"@id":"https:\/\/insidebigdata.com\/#\/schema\/person\/2949e412c144601cdbcc803bd234e1b9"},"breadcrumb":{"@id":"https:\/\/insidebigdata.com\/2020\/08\/17\/analyze-then-store-the-journey-to-continuous-intelligence-part-3\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/insidebigdata.com\/2020\/08\/17\/analyze-then-store-the-journey-to-continuous-intelligence-part-3\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/insidebigdata.com\/2020\/08\/17\/analyze-then-store-the-journey-to-continuous-intelligence-part-3\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/insidebigdata.com\/"},{"@type":"ListItem","position":2,"name":"Analyze-then-Store: The Journey to Continuous Intelligence \u2013 Part 3"}]},{"@type":"WebSite","@id":"https:\/\/insidebigdata.com\/#website","url":"https:\/\/insidebigdata.com\/","name":"insideBIGDATA","description":"Your Source for AI, Data Science, Deep Learning & Machine Learning Strategies","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/insidebigdata.com\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/insidebigdata.com\/#\/schema\/person\/2949e412c144601cdbcc803bd234e1b9","name":"Editorial Team","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/insidebigdata.com\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/e137ce7ea40e38bd4d25bb7860cfe3e4?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/e137ce7ea40e38bd4d25bb7860cfe3e4?s=96&d=mm&r=g","caption":"Editorial Team"},"sameAs":["http:\/\/www.insidebigdata.com"],"url":"https:\/\/insidebigdata.com\/author\/editorial\/"}]}},"jetpack_featured_media_url":"https:\/\/insidebigdata.com\/wp-content\/uploads\/2020\/07\/Data_architecture_shutterstock_562411702.jpg","jetpack_shortlink":"https:\/\/wp.me\/p9eA3j-6tr","jetpack-related-posts":[{"id":25070,"url":"https:\/\/insidebigdata.com\/2020\/10\/05\/analyze-then-store-the-journey-to-continuous-intelligence-part-6\/","url_meta":{"origin":24889,"position":0},"title":"Analyze-then-Store: The Journey to Continuous Intelligence \u2013 Part 6","date":"October 5, 2020","format":false,"excerpt":"This multi-part article series by our friends at Swim is intended for data architects and anyone else interested in learning how to design modern real-time data analytics solutions. It explores key principles and implications of event streaming and streaming analytics, and concludes that the biggest opportunity to derive meaningful value\u2026","rel":"","context":"In "Analytics"","img":{"alt_text":"","src":"https:\/\/i0.wp.com\/insidebigdata.com\/wp-content\/uploads\/2020\/09\/Figure-part-6.png?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":25011,"url":"https:\/\/insidebigdata.com\/2020\/09\/21\/analyze-then-store-the-journey-to-continuous-intelligence-part-5\/","url_meta":{"origin":24889,"position":1},"title":"Analyze-then-Store: The Journey to Continuous Intelligence – Part 5","date":"September 21, 2020","format":false,"excerpt":"This multi-part article series by our friends at Swim is intended for data architects and anyone else interested in learning how to design modern real-time data analytics solutions. It explores key principles and implications of event streaming and streaming analytics, and concludes that the biggest opportunity to derive meaningful value\u2026","rel":"","context":"In "Analytics"","img":{"alt_text":"","src":"https:\/\/i0.wp.com\/insidebigdata.com\/wp-content\/uploads\/2020\/09\/Part5_fig1.png?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":24766,"url":"https:\/\/insidebigdata.com\/2020\/07\/20\/analyze-then-store-the-journey-to-continuous-intelligence\/","url_meta":{"origin":24889,"position":2},"title":"Analyze-then-Store: The Journey to Continuous Intelligence","date":"July 20, 2020","format":false,"excerpt":"In this technical blog for data architects by our friends over at Swim, we learn how to design modern real-time data analytics solutions. It explores key principles and implications of event streaming and streaming analytics, and concludes that the biggest opportunity to derive meaningful value from data \u2013 and gain\u2026","rel":"","context":"In "Analytics"","img":{"alt_text":"","src":"https:\/\/i0.wp.com\/insidebigdata.com\/wp-content\/uploads\/2020\/07\/Fig1.png?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":24936,"url":"https:\/\/insidebigdata.com\/2020\/08\/31\/analyze-then-store-the-journey-to-continuous-intelligence-part-4\/","url_meta":{"origin":24889,"position":3},"title":"Analyze-then-Store: The Journey to Continuous Intelligence \u2013 Part 4","date":"August 31, 2020","format":false,"excerpt":"This multi-part article series by our friends at Swim is intended for data architects and anyone else interested in learning how to design modern real-time data analytics solutions. It explores key principles and implications of event streaming and streaming analytics, and concludes that the biggest opportunity to derive meaningful value\u2026","rel":"","context":"In "Analytics"","img":{"alt_text":"","src":"https:\/\/i0.wp.com\/insidebigdata.com\/wp-content\/uploads\/2020\/07\/Data_architecture_shutterstock_562411702.jpg?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":24830,"url":"https:\/\/insidebigdata.com\/2020\/08\/03\/analyze-then-store-the-journey-to-continuous-intelligence-2\/","url_meta":{"origin":24889,"position":4},"title":"Analyze-then-Store: The Journey to Continuous Intelligence – Part 2","date":"August 3, 2020","format":false,"excerpt":"This multi-part article series by our friends at Swim is intended for data architects and anyone else interested in learning how to design modern real-time data analytics solutions. It explores key principles and implications of event streaming and streaming analytics, and concludes that the biggest opportunity to derive meaningful value\u2026","rel":"","context":"In "Analytics"","img":{"alt_text":"","src":"https:\/\/i0.wp.com\/insidebigdata.com\/wp-content\/uploads\/2020\/07\/Data_architecture_shutterstock_562411702.jpg?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":25075,"url":"https:\/\/insidebigdata.com\/2020\/10\/07\/swim-releases-new-platform-for-managing-continuous-intelligence-at-scale\/","url_meta":{"origin":24889,"position":5},"title":"Swim Releases New Platform for Managing Continuous Intelligence at Scale","date":"October 7, 2020","format":false,"excerpt":"Swim, the developer of the industry\u2019s first open core platform for continuous intelligence at scale, today announced Swim Continuum 4.0, the newest release of its flagship product. Providing enterprises with a live window into the current state of their business by concurrently processing and analyzing streaming and contextual data, Swim\u2026","rel":"","context":"In "Analytics"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"_links":{"self":[{"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/posts\/24889"}],"collection":[{"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/users\/10513"}],"replies":[{"embeddable":true,"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/comments?post=24889"}],"version-history":[{"count":0,"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/posts\/24889\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/media\/24770"}],"wp:attachment":[{"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/media?parent=24889"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/categories?post=24889"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/tags?post=24889"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}