{"id":25877,"date":"2021-04-03T06:00:00","date_gmt":"2021-04-03T13:00:00","guid":{"rendered":"https:\/\/insidebigdata.com\/?p=25877"},"modified":"2021-04-03T10:09:03","modified_gmt":"2021-04-03T17:09:03","slug":"accelerated-integration-of-eventador-with-cloudera-sql-stream-builder","status":"publish","type":"post","link":"https:\/\/insidebigdata.com\/2021\/04\/03\/accelerated-integration-of-eventador-with-cloudera-sql-stream-builder\/","title":{"rendered":"Accelerated Integration of Eventador with Cloudera – SQL Stream Builder"},"content":{"rendered":"\n
\"\"<\/figure><\/div>\n\n\n\n

In October 2020, Cloudera made a strategic acquisition of a company called Eventador<\/a>. This was primarily to augment our streaming capabilities within Cloudera DataFlow. Eventador was adept at simplifying the process of building streaming applications. Their flagship product, SQL Stream Builder, made access to real-time data streams easily possible with just SQL (Structured Query Language). Cloudera\u2019s customers were struggling to solve the same challenge – to query high-volumes of real-time data streams with something as simple as SQL.<\/p>\n\n\n\n

Within 5 months after the acquisition of Eventador, Cloudera announced that SQL Stream Builder is now being re-launched as Cloudera SQL Stream Builder. This is being done after it was fully integrated with\u00a0Cloudera Data Platform<\/a>\u2019s (CDP)\u00a0Shared Data Experience<\/a>\u00a0(SDX). This means that SQL Stream Builder can take advantage of the same unified security and governance as the rest of the platform does, using SDX.<\/p>\n\n\n\n

What is SQL Stream Builder?<\/strong>
Cloudera SQL Stream Builder now augments the powerful stream processing capabilities<\/a> of the Cloudera DataFlow<\/a> (CDF) streaming platform. It offers a slick user interface for writing SQL queries to run against real-time data streams in Kafka or Flink. This enables developers, data analysts and data scientists to write streaming applications using just SQL. They no longer have to depend on any skilled Java or Scala developers to write special programs to gain access to such data streams.<\/p>\n\n\n\n

SQL Stream Builder<\/a> continuously runs SQL via Apache Flink. It offers syntax checking, error reporting, schema detection, query creation, sampling results and creating outputs with its simple yet intuitive user interface. It also provides an advanced materialized view engine to enable live aggregated datasets to be accessible by other applications via a simple REST API.<\/p>\n\n\n\n

Data decays<\/strong>
Yes, data has a shelf life. And in today\u2019s business environments, the data you receive must be instantaneously processed to understand the business impact and act upon it. A streaming analytics solution is no good if you can just ingest all the data in real-time but are unable to harness the value of what the data means to you. Imagine a manufacturer receiving streams of data with millions of messages a day from their dozen or more manufacturing plants. If they need to understand where a specific surge of streams is coming from or if they need to detect a specific anomaly in the streams, they should be able to query the streams in real-time. They cannot afford to send it all into a data store and then analyze it the next day to find such actionable insights. The data is of no value the next day for this purpose. To execute such real-time queries, the skills are typically in the hands of a select few in the organization who possess unique skills like Scala or Java and can write code to get such insights. This is not a scalable model.<\/p>\n\n\n\n

SQL is a universal language<\/strong><\/p>\n\n\n\n

For more than three decades, SQL has been an accepted way to conduct queries across a range of database systems. SQL is also one of the most popular skillsets amongst the key enterprise data personas. With data analysts and data scientists struggling to gain access to real-time data streams easily, SQL becomes an easy choice for the task. However, there is a key challenge. Unlike database tables which typically have a fixed number of rows at any given point in time, streams are unbounded. This means that they are continuous by nature and have no limit. They also don\u2019t come in sequentially. Some messages can come in late or out of order too. This makes it challenging to adopt SQL as-is to query data streams.<\/p>\n\n\n\n

Streaming SQL<\/strong><\/p>\n\n\n\n

Data streams have to be treated with tiny slices of time called \u201cwindows\u201d e.g. last 5 seconds. Each message on the stream also has a timestamp that can be used to detect the order in which it should be processed. So, using SQL as the base construct, a few additional keywords were added to handle data streams in the context of time windows. And thus was born Streaming SQL or Continuous SQL. They look and function like regular SQL but you also have a lot of additional constructs to allow you to group the streams over a specific time window. It also supports a range of aggregation functions so that you can perform various enrichment tasks on the streams like finding averages, sums, counts etc. This immediately allows the data analysts and data scientists in the organization to query data streams using SQL! This is what Cloudera calls the democratization of real-time data<\/a> within the organization.<\/p>\n\n\n\n

Why should you be excited about SQL Stream Builder?<\/strong><\/p>\n\n\n\n