{"id":29842,"date":"2022-07-14T06:00:00","date_gmt":"2022-07-14T13:00:00","guid":{"rendered":"https:\/\/insidebigdata.com\/?p=29842"},"modified":"2022-07-15T08:46:30","modified_gmt":"2022-07-15T15:46:30","slug":"maximizing-data-lake-utility-with-query-optimization","status":"publish","type":"post","link":"https:\/\/insidebigdata.com\/2022\/07\/14\/maximizing-data-lake-utility-with-query-optimization\/","title":{"rendered":"Maximizing Data Lake Utility with Query Optimization\u00a0"},"content":{"rendered":"\n

Of all the user personas across the data landscape, the data consumer is arguably the most difficult to appease. Oftentimes, such users are heedless of the backend processes required to service the data they need to do their jobs better.<\/p>\n\n\n\n

They simply want to probe their data at will, get rapid responses to questions, and apply them to better fulfill their business objectives.<\/p>\n\n\n\n

Starburst\u2019s recent acquisition of Varada was calculated to do just that, particularly in data lake settings in which organizations have tremendous quantities of data. According to Russell Christopher, Director of Product Strategy at Starburst<\/a>, the company\u2019s annexation of Varada behooves data consumers like \u201cthe analyst who now can ask more questions, and more complete questions, because all the data on the lake is now accessible in a performant way.\u201d<\/p>\n\n\n\n

Varada provides Starburst\u2019s platform two pivotal benefits. On the one hand, it employs cognitive computing to intelligently index data at scale. On the other, it has caching capabilities that make queries even more responsive for swiftly retrieving answers for informed decision-making, analytics, and applications.<\/p>\n\n\n\n

Such query acceleration methods can mean the difference between simply accumulating data and actually using data.<\/p>\n\n\n\n

\u201cIn my experience, because I\u2019ve always been in analytics, [if] you make users wait too long or make it more cumbersome, they just stop asking questions, and there\u2019s huge risks in that for the organization,\u201d Christopher cautioned.<\/p>\n\n\n\n

Intelligent Indexing<\/strong><\/p>\n\n\n\n

Varada equips Starburst\u2019s compute engine with a primarily automated form of indexing that removes much of the work from this task to expedite query responses. It utilizes statistical Artificial Intelligence<\/a> technologies to assess what data should be indexed, then implements indexes accordingly. A similar approach determines which data to cache.<\/p>\n\n\n\n

\u201cVerada has, essentially, a machine learning looping mechanism that is watching all the queries that are executed on the lake,\u201d Christopher explained. \u201cBased on the columns that are being accessed and the tables that are being accessed the most frequently, it actually generates instructions for what should be cached and indexed, and how.\u201d<\/p>\n\n\n\n

The underlying system relies on a variety of indexing schemes, including trees. Moreover, it enables organizations to eschew manual approaches to indexing, which are typically time-consuming. \u201cThe cost in people time, to bring people in and say what are the important data, how do you use it, and show me how, and then using that information to try and index and cache, you don\u2019t have to do that anymore,\u201d Christopher revealed.<\/p>\n\n\n\n

Cache Rules<\/strong><\/p>\n\n\n\n

It\u2019s not uncommon for Varada to store both its indexes and caches in SSD<\/a>. The latter involves what Christopher characterized as a \u201cproprietary format, columnar format.\u201d Thus, instead of having to constantly rescan a data lake to retrieve information from it to answer queries for a particular use case, firms can access that data via the cache to accelerate the time, resources, and cost of employing data for business insights.<\/p>\n\n\n\n

The performance benefits of automatically indexing the most widely used data and caching them to hasten query times are formidable. Giving data consumers faster query results decreases the amount of compute resources for such information retrieval. Consequently, this approach can substantially \u201creduce cloud compute costs,\u201d mentioned Matt Fuller, Starburst co-founder and VP of Product. \u201cFor the machines that are running, in our experience, we see about 40 percent in terms of cost savings. In terms of productivity, we\u2019re seeing response times around 7 times [faster].\u201d<\/p>\n\n\n\n

Customization <\/strong><\/p>\n\n\n\n

Another boon of this query optimization method is firms can tailor it to meet the specific needs of individual users, departments, and deployments. It\u2019s possible to prioritize queries according to the above considerations and others so the C-suite\u2019s queries about monthly reports, for example, are answered before those of other users. Additionally, users can ascribe what\u2019s essentially metadata<\/a> to queries to make them more utilitarian to the enterprise.<\/p>\n\n\n\n

\u201cYou can create groups of queries based on the users that are executing them or even, this again is one of the things that I think is kind of fun, [based on] free text that just happens to be sitting in the query,\u201d Christopher divulged. \u201cLike maybe someone puts a comment in the query saying this is a great query for the marketing team.\u201d This functionality surpasses the capability to simply speed up queries, but makes them more helpful as well, thereby multiplying the value from data-centric processes altogether.<\/p>\n\n\n\n

Human-Curated Automation <\/strong><\/p>\n\n\n\n

The configurable nature of Starburst\u2019s pairing with Varada also provides a degree of human control over the underlying automation. With machine learning automating the indexing and caching to accelerate queries, while people compartmentalize queries into groups, prioritize them, and annotate them with metadata, humans are overseeing the impact of these AI models.<\/p>\n\n\n\n

The resulting combination is beneficial for optimizing queries to consistently deliver the best results. \u201cThe exciting part is the technology is \u2018set it and forget it\u2019, and it does what you need it to do without pulling in all the data and subject matters experts in to be the brains behind it,\u201d Christopher concluded.<\/p>\n\n\n\n

About the Author<\/strong><\/p>\n\n\n

\n
\"\"<\/figure><\/div>\n\n\n

Jelani Harper is an editorial consultant servicing the information technology market. He specializes in data-driven applications focused on semantic technologies, data governance and analytics.<\/em><\/p>\n\n\n\n

Sign up for the free insideBIGDATA newsletter<\/a>.<\/em><\/p>\n\n\n\n

Join us on Twitter: @InsideBigData1 \u2013 https:\/\/twitter.com\/InsideBigData1<\/a><\/em><\/p>\n","protected":false},"excerpt":{"rendered":"

In this contributed article, editorial consultant Jelani Harper highlights the recent acquisition of Varada by Starburst in terms how the deal provides Starburst\u2019s platform two pivotal benefits. On the one hand, it employs cognitive computing to intelligently index data at scale. On the other, it has caching capabilities that make queries even more responsive for swiftly retrieving answers for informed decision-making, analytics, and applications.<\/p>\n","protected":false},"author":10513,"featured_media":23458,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"footnotes":""},"categories":[115,68,1054,87,180,56,97,1],"tags":[336,95],"acf":[],"yoast_head":"\nMaximizing Data Lake Utility with Query Optimization\u00a0 - insideBIGDATA<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/insidebigdata.com\/2022\/07\/14\/maximizing-data-lake-utility-with-query-optimization\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Maximizing Data Lake Utility with Query Optimization\u00a0 - insideBIGDATA\" \/>\n<meta property=\"og:description\" content=\"In this contributed article, editorial consultant Jelani Harper highlights the recent acquisition of Varada by Starburst in terms how the deal provides Starburst\u2019s platform two pivotal benefits. On the one hand, it employs cognitive computing to intelligently index data at scale. On the other, it has caching capabilities that make queries even more responsive for swiftly retrieving answers for informed decision-making, analytics, and applications.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/insidebigdata.com\/2022\/07\/14\/maximizing-data-lake-utility-with-query-optimization\/\" \/>\n<meta property=\"og:site_name\" content=\"insideBIGDATA\" \/>\n<meta property=\"article:publisher\" content=\"http:\/\/www.facebook.com\/insidebigdata\" \/>\n<meta property=\"article:published_time\" content=\"2022-07-14T13:00:00+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2022-07-15T15:46:30+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/insidebigdata.com\/wp-content\/uploads\/2019\/10\/data-lakes_SHUTTERSTOCK.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"200\" \/>\n\t<meta property=\"og:image:height\" content=\"200\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Editorial Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@insideBigData\" \/>\n<meta name=\"twitter:site\" content=\"@insideBigData\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Editorial Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"5 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/insidebigdata.com\/2022\/07\/14\/maximizing-data-lake-utility-with-query-optimization\/\",\"url\":\"https:\/\/insidebigdata.com\/2022\/07\/14\/maximizing-data-lake-utility-with-query-optimization\/\",\"name\":\"Maximizing Data Lake Utility with Query Optimization\u00a0 - insideBIGDATA\",\"isPartOf\":{\"@id\":\"https:\/\/insidebigdata.com\/#website\"},\"datePublished\":\"2022-07-14T13:00:00+00:00\",\"dateModified\":\"2022-07-15T15:46:30+00:00\",\"author\":{\"@id\":\"https:\/\/insidebigdata.com\/#\/schema\/person\/2949e412c144601cdbcc803bd234e1b9\"},\"breadcrumb\":{\"@id\":\"https:\/\/insidebigdata.com\/2022\/07\/14\/maximizing-data-lake-utility-with-query-optimization\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/insidebigdata.com\/2022\/07\/14\/maximizing-data-lake-utility-with-query-optimization\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/insidebigdata.com\/2022\/07\/14\/maximizing-data-lake-utility-with-query-optimization\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/insidebigdata.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Maximizing Data Lake Utility with Query Optimization\u00a0\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/insidebigdata.com\/#website\",\"url\":\"https:\/\/insidebigdata.com\/\",\"name\":\"insideBIGDATA\",\"description\":\"Your Source for AI, Data Science, Deep Learning & Machine Learning Strategies\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/insidebigdata.com\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/insidebigdata.com\/#\/schema\/person\/2949e412c144601cdbcc803bd234e1b9\",\"name\":\"Editorial Team\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/insidebigdata.com\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/e137ce7ea40e38bd4d25bb7860cfe3e4?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/e137ce7ea40e38bd4d25bb7860cfe3e4?s=96&d=mm&r=g\",\"caption\":\"Editorial Team\"},\"sameAs\":[\"http:\/\/www.insidebigdata.com\"],\"url\":\"https:\/\/insidebigdata.com\/author\/editorial\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Maximizing Data Lake Utility with Query Optimization\u00a0 - insideBIGDATA","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/insidebigdata.com\/2022\/07\/14\/maximizing-data-lake-utility-with-query-optimization\/","og_locale":"en_US","og_type":"article","og_title":"Maximizing Data Lake Utility with Query Optimization\u00a0 - insideBIGDATA","og_description":"In this contributed article, editorial consultant Jelani Harper highlights the recent acquisition of Varada by Starburst in terms how the deal provides Starburst\u2019s platform two pivotal benefits. On the one hand, it employs cognitive computing to intelligently index data at scale. On the other, it has caching capabilities that make queries even more responsive for swiftly retrieving answers for informed decision-making, analytics, and applications.","og_url":"https:\/\/insidebigdata.com\/2022\/07\/14\/maximizing-data-lake-utility-with-query-optimization\/","og_site_name":"insideBIGDATA","article_publisher":"http:\/\/www.facebook.com\/insidebigdata","article_published_time":"2022-07-14T13:00:00+00:00","article_modified_time":"2022-07-15T15:46:30+00:00","og_image":[{"width":200,"height":200,"url":"https:\/\/insidebigdata.com\/wp-content\/uploads\/2019\/10\/data-lakes_SHUTTERSTOCK.jpg","type":"image\/jpeg"}],"author":"Editorial Team","twitter_card":"summary_large_image","twitter_creator":"@insideBigData","twitter_site":"@insideBigData","twitter_misc":{"Written by":"Editorial Team","Est. reading time":"5 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/insidebigdata.com\/2022\/07\/14\/maximizing-data-lake-utility-with-query-optimization\/","url":"https:\/\/insidebigdata.com\/2022\/07\/14\/maximizing-data-lake-utility-with-query-optimization\/","name":"Maximizing Data Lake Utility with Query Optimization\u00a0 - insideBIGDATA","isPartOf":{"@id":"https:\/\/insidebigdata.com\/#website"},"datePublished":"2022-07-14T13:00:00+00:00","dateModified":"2022-07-15T15:46:30+00:00","author":{"@id":"https:\/\/insidebigdata.com\/#\/schema\/person\/2949e412c144601cdbcc803bd234e1b9"},"breadcrumb":{"@id":"https:\/\/insidebigdata.com\/2022\/07\/14\/maximizing-data-lake-utility-with-query-optimization\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/insidebigdata.com\/2022\/07\/14\/maximizing-data-lake-utility-with-query-optimization\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/insidebigdata.com\/2022\/07\/14\/maximizing-data-lake-utility-with-query-optimization\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/insidebigdata.com\/"},{"@type":"ListItem","position":2,"name":"Maximizing Data Lake Utility with Query Optimization\u00a0"}]},{"@type":"WebSite","@id":"https:\/\/insidebigdata.com\/#website","url":"https:\/\/insidebigdata.com\/","name":"insideBIGDATA","description":"Your Source for AI, Data Science, Deep Learning & Machine Learning Strategies","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/insidebigdata.com\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/insidebigdata.com\/#\/schema\/person\/2949e412c144601cdbcc803bd234e1b9","name":"Editorial Team","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/insidebigdata.com\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/e137ce7ea40e38bd4d25bb7860cfe3e4?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/e137ce7ea40e38bd4d25bb7860cfe3e4?s=96&d=mm&r=g","caption":"Editorial Team"},"sameAs":["http:\/\/www.insidebigdata.com"],"url":"https:\/\/insidebigdata.com\/author\/editorial\/"}]}},"jetpack_featured_media_url":"https:\/\/insidebigdata.com\/wp-content\/uploads\/2019\/10\/data-lakes_SHUTTERSTOCK.jpg","jetpack_shortlink":"https:\/\/wp.me\/p9eA3j-7Lk","jetpack-related-posts":[{"id":20806,"url":"https:\/\/insidebigdata.com\/2018\/07\/27\/5-reasons-data-lake-isnt-giving-good-bi\/","url_meta":{"origin":29842,"position":0},"title":"5 Reasons Your Data Lake Isn’t Giving Good BI","date":"July 27, 2018","format":false,"excerpt":"In this contributed article, technology writer and blogger Kayla Matthews take a look at the data lake The best fix to gather better business intelligence, or BI, is to make your own corporate, digital data lake. A lake with all the ground rules your company needs from the data is\u2026","rel":"","context":"In "Big Data"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":13050,"url":"https:\/\/insidebigdata.com\/2015\/04\/30\/the-rise-of-the-data-lake-in-support-of-the-industrial-internet-part-1\/","url_meta":{"origin":29842,"position":1},"title":"The Rise of the Data Lake in Support of the Industrial Internet (Part 1)","date":"April 30, 2015","format":false,"excerpt":"Data lakes are enterprise-wide data management platforms designed for storing and analyzing vast amounts of information from disparate data sources in their native format. The idea is to place data into a data lake in their native structure instead of a repository built for a specific purpose such as a\u2026","rel":"","context":"In "Big Data"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":15589,"url":"https:\/\/insidebigdata.com\/2016\/08\/03\/zaloni-launches-new-data-lake-360-solution-to-drive-higher-roi-from-big-data\/","url_meta":{"origin":29842,"position":2},"title":"Zaloni Launches New Data Lake 360\u00b0 Solution to Drive Higher ROI from Big Data","date":"August 3, 2016","format":false,"excerpt":"Zaloni, the data lake company, announced a new solution, Zaloni\u2019s Data Lake 360\u00b0, to meet the needs of a growing number of enterprises that understand that the data lake is key to the future of the enterprise data ecosystem.","rel":"","context":"In "Big Data"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":15696,"url":"https:\/\/insidebigdata.com\/2016\/08\/19\/15696\/","url_meta":{"origin":29842,"position":3},"title":"EMC Isilon Scale-Out Data Lake Foundation","date":"August 19, 2016","format":false,"excerpt":"In this special lab validation brief, EMC Isilon Scale-Out Data Lake Foundation, you\u2019ll learn how the EMC Isilon product family is an easy to operate, highly scalable and efficient Enterprise Data Lake Platform (EDLP).","rel":"","context":"In "Big Data"","img":{"alt_text":"EMC_benchmark","src":"https:\/\/i0.wp.com\/insidebigdata.com\/wp-content\/uploads\/2016\/08\/EMC_benchmark.png?resize=350%2C200","width":350,"height":200},"classes":[]},{"id":15580,"url":"https:\/\/insidebigdata.com\/2016\/08\/02\/scale-out-data-lake-solution-built-on-emc-isilon\/","url_meta":{"origin":29842,"position":4},"title":"Scale-out Data Lake Solution Built on EMC Isilon","date":"August 2, 2016","format":false,"excerpt":"In this special technology white paper, \"Scale-out Data Lake Solution Built on EMC Isilon,\" you\u2019ll learn all the essentials a data lake based on EMC Isilon provides- CAPEX savings, OPEX savings, elimination of storage silos, regulatory compliance, and multi-protocol access.","rel":"","context":"In "Big Data"","img":{"alt_text":"EMC_diagram","src":"https:\/\/i0.wp.com\/insidebigdata.com\/wp-content\/uploads\/2016\/07\/EMC_diagram.png?resize=350%2C200","width":350,"height":200},"classes":[]},{"id":23524,"url":"https:\/\/insidebigdata.com\/2019\/11\/06\/do-you-actually-need-a-data-lake\/","url_meta":{"origin":29842,"position":5},"title":"Do You Actually Need a Data Lake?","date":"November 6, 2019","format":false,"excerpt":"In this contributed article, Eran Levy, Director of Marketing at Upsolver, sets out to formally define \"data lake\" and then goes on to ask whether your organization needs a data lake by examining 5 key indicators. Data lakes have become the cornerstone of many big data initiatives, just as they\u2026","rel":"","context":"In "Big Data"","img":{"alt_text":"","src":"https:\/\/i0.wp.com\/insidebigdata.com\/wp-content\/uploads\/2019\/10\/data-lakes_SHUTTERSTOCK.jpg?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]}],"_links":{"self":[{"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/posts\/29842"}],"collection":[{"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/users\/10513"}],"replies":[{"embeddable":true,"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/comments?post=29842"}],"version-history":[{"count":0,"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/posts\/29842\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/media\/23458"}],"wp:attachment":[{"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/media?parent=29842"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/categories?post=29842"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/tags?post=29842"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}