{"id":28936,"date":"2022-04-06T06:00:00","date_gmt":"2022-04-06T13:00:00","guid":{"rendered":"https:\/\/insidebigdata.com\/?p=28936"},"modified":"2022-04-05T13:50:47","modified_gmt":"2022-04-05T20:50:47","slug":"databricks-announces-general-availability-of-delta-live-tables","status":"publish","type":"post","link":"https:\/\/insidebigdata.com\/2022\/04\/06\/databricks-announces-general-availability-of-delta-live-tables\/","title":{"rendered":"Databricks Announces General Availability of Delta Live Tables"},"content":{"rendered":"\n<p class=\"has-text-align-center\"><em>ETL framework is the first to both automatically manage infrastructure and bring modern software engineering practices to data engineering, allowing data engineers and analysts to focus on transforming data, not managing pipelines<\/em><\/p>\n\n\n\n<div class=\"wp-block-image is-style-default\"><figure class=\"alignright size-large\"><img decoding=\"async\" loading=\"lazy\" width=\"146\" height=\"128\" src=\"https:\/\/insidebigdata.com\/wp-content\/uploads\/2022\/01\/Databricks_logo_2022.png\" alt=\"\" class=\"wp-image-28259\"\/><\/figure><\/div>\n\n\n\n<p><a href=\"https:\/\/databricks.com\/\" target=\"_blank\" rel=\"noreferrer noopener\">Databricks<\/a>, the Data and AI company and pioneer of the data lakehouse paradigm, announced the general availability of Delta Live Tables (DLT), the first ETL framework to use a simple declarative approach to build reliable data pipelines and to automatically manage data infrastructure at scale. Turning SQL queries into production ETL pipelines often requires a lot of tedious, complicated operational work. By using modern software engineering practices to automate the most time consuming parts of data engineering, data engineers and analysts can concentrate on delivering data rather than on operating and maintaining pipelines.\u00a0<\/p>\n\n\n\n<p>As companies develop strategies to get the most value out of their data, many will hire expensive, highly-skilled data engineers &#8211; a resource that is already hard to come by &#8211; to avoid delays and failed projects. What is often not well understood is that many of the delays or failed projects are driven by a core issue: it is hard to build reliable data pipelines that work automatically without a lot of operational rigor to keep them up and running. As such, even at a small scale, the majority of a data practitioner&#8217;s time is spent on tooling and managing infrastructure to make sure these data pipelines don&#8217;t break.<\/p>\n\n\n\n<p>Delta Live Tables is the first and only ETL framework to solve this problem by combining both modern engineering practices <em>and<\/em> automatic management of infrastructure, whereas past efforts in the market have only tackled one aspect or the other.&nbsp; It simplifies ETL development by allowing engineers to simply describe the outcomes of data transformations. Delta Live Tables then understands dependencies of the full data pipeline live and automates away virtually all of the manual complexity. It also enables data engineers to treat their data as code and apply modern software engineering best practices like testing, error-handling, monitoring, and documentation to deploy reliable pipelines at scale more easily. Delta Live Tables fully supports both Python and SQL and is tailored to work with both streaming and batch workloads.<\/p>\n\n\n\n<p>Delta Live Tables is already powering production use cases at leading companies around the globe like JLL, Shell, Jumbo, Bread Finance, and ADP. &#8220;At ADP, we are migrating our human resource management data to an integrated data store on the lakehouse. Delta Live Tables has helped our team build in quality controls, and because of the declarative APIs, support for batch and real-time using only SQL, it has enabled our team to save time and effort in managing our data,&#8221; said Jack Berkowitz, Chief Data Officer, ADP.<\/p>\n\n\n\n<blockquote class=\"wp-block-quote\"><p>\u201cThe power of DLT comes from something no one else can do &#8211; combine modern software engineering practices and automatically manage infrastructure. It\u2019s game-changing technology that will allow data engineers and analysts to be more productive than ever,\u201d said Ali Ghodsi, CEO and Co-Founder at Databricks. \u201cIt also broadens Databricks\u2019 reach; DLT supports any type of data workload with a single API, eliminating the need for advanced data engineering skills.\u201d\u00a0<\/p><\/blockquote>\n\n\n\n<p><em>Sign up for the free insideBIGDATA&nbsp;<a href=\"http:\/\/inside-bigdata.com\/newsletter\/\" target=\"_blank\" rel=\"noreferrer noopener\">newsletter<\/a>.<\/em><\/p>\n\n\n\n<p><em>Join us on Twitter:&nbsp;@InsideBigData1 \u2013 <a href=\"https:\/\/twitter.com\/InsideBigData1\" target=\"_blank\" rel=\"noreferrer noopener\">https:\/\/twitter.com\/InsideBigData1<\/a><\/em><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Databricks, the Data and AI company and pioneer of the data lakehouse paradigm, announced the general availability of Delta Live Tables (DLT), the first ETL framework to use a simple declarative approach to build reliable data pipelines and to automatically manage data infrastructure at scale. Turning SQL queries into production ETL pipelines often requires a lot of tedious, complicated operational work. By using modern software engineering practices to automate the most time consuming parts of data engineering, data engineers and analysts can concentrate on delivering data rather than on operating and maintaining pipelines.<\/p>\n","protected":false},"author":10513,"featured_media":28259,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"footnotes":""},"categories":[65,115,64,1054,201,180,56,1],"tags":[437,780,432,744,1108,360,96],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v20.6 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Databricks Announces General Availability of Delta Live Tables - insideBIGDATA<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/insidebigdata.com\/2022\/04\/06\/databricks-announces-general-availability-of-delta-live-tables\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Databricks Announces General Availability of Delta Live Tables - insideBIGDATA\" \/>\n<meta property=\"og:description\" content=\"Databricks, the Data and AI company and pioneer of the data lakehouse paradigm, announced the general availability of Delta Live Tables (DLT), the first ETL framework to use a simple declarative approach to build reliable data pipelines and to automatically manage data infrastructure at scale. Turning SQL queries into production ETL pipelines often requires a lot of tedious, complicated operational work. By using modern software engineering practices to automate the most time consuming parts of data engineering, data engineers and analysts can concentrate on delivering data rather than on operating and maintaining pipelines.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/insidebigdata.com\/2022\/04\/06\/databricks-announces-general-availability-of-delta-live-tables\/\" \/>\n<meta property=\"og:site_name\" content=\"insideBIGDATA\" \/>\n<meta property=\"article:publisher\" content=\"http:\/\/www.facebook.com\/insidebigdata\" \/>\n<meta property=\"article:published_time\" content=\"2022-04-06T13:00:00+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2022-04-05T20:50:47+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/insidebigdata.com\/wp-content\/uploads\/2022\/01\/Databricks_logo_2022.png\" \/>\n\t<meta property=\"og:image:width\" content=\"146\" \/>\n\t<meta property=\"og:image:height\" content=\"128\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Editorial Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@insideBigData\" \/>\n<meta name=\"twitter:site\" content=\"@insideBigData\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Editorial Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/insidebigdata.com\/2022\/04\/06\/databricks-announces-general-availability-of-delta-live-tables\/\",\"url\":\"https:\/\/insidebigdata.com\/2022\/04\/06\/databricks-announces-general-availability-of-delta-live-tables\/\",\"name\":\"Databricks Announces General Availability of Delta Live Tables - insideBIGDATA\",\"isPartOf\":{\"@id\":\"https:\/\/insidebigdata.com\/#website\"},\"datePublished\":\"2022-04-06T13:00:00+00:00\",\"dateModified\":\"2022-04-05T20:50:47+00:00\",\"author\":{\"@id\":\"https:\/\/insidebigdata.com\/#\/schema\/person\/2949e412c144601cdbcc803bd234e1b9\"},\"breadcrumb\":{\"@id\":\"https:\/\/insidebigdata.com\/2022\/04\/06\/databricks-announces-general-availability-of-delta-live-tables\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/insidebigdata.com\/2022\/04\/06\/databricks-announces-general-availability-of-delta-live-tables\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/insidebigdata.com\/2022\/04\/06\/databricks-announces-general-availability-of-delta-live-tables\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/insidebigdata.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Databricks Announces General Availability of Delta Live Tables\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/insidebigdata.com\/#website\",\"url\":\"https:\/\/insidebigdata.com\/\",\"name\":\"insideBIGDATA\",\"description\":\"Your Source for AI, Data Science, Deep Learning &amp; Machine Learning Strategies\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/insidebigdata.com\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/insidebigdata.com\/#\/schema\/person\/2949e412c144601cdbcc803bd234e1b9\",\"name\":\"Editorial Team\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/insidebigdata.com\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/e137ce7ea40e38bd4d25bb7860cfe3e4?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/e137ce7ea40e38bd4d25bb7860cfe3e4?s=96&d=mm&r=g\",\"caption\":\"Editorial Team\"},\"sameAs\":[\"http:\/\/www.insidebigdata.com\"],\"url\":\"https:\/\/insidebigdata.com\/author\/editorial\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Databricks Announces General Availability of Delta Live Tables - insideBIGDATA","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/insidebigdata.com\/2022\/04\/06\/databricks-announces-general-availability-of-delta-live-tables\/","og_locale":"en_US","og_type":"article","og_title":"Databricks Announces General Availability of Delta Live Tables - insideBIGDATA","og_description":"Databricks, the Data and AI company and pioneer of the data lakehouse paradigm, announced the general availability of Delta Live Tables (DLT), the first ETL framework to use a simple declarative approach to build reliable data pipelines and to automatically manage data infrastructure at scale. Turning SQL queries into production ETL pipelines often requires a lot of tedious, complicated operational work. By using modern software engineering practices to automate the most time consuming parts of data engineering, data engineers and analysts can concentrate on delivering data rather than on operating and maintaining pipelines.","og_url":"https:\/\/insidebigdata.com\/2022\/04\/06\/databricks-announces-general-availability-of-delta-live-tables\/","og_site_name":"insideBIGDATA","article_publisher":"http:\/\/www.facebook.com\/insidebigdata","article_published_time":"2022-04-06T13:00:00+00:00","article_modified_time":"2022-04-05T20:50:47+00:00","og_image":[{"width":146,"height":128,"url":"https:\/\/insidebigdata.com\/wp-content\/uploads\/2022\/01\/Databricks_logo_2022.png","type":"image\/png"}],"author":"Editorial Team","twitter_card":"summary_large_image","twitter_creator":"@insideBigData","twitter_site":"@insideBigData","twitter_misc":{"Written by":"Editorial Team","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/insidebigdata.com\/2022\/04\/06\/databricks-announces-general-availability-of-delta-live-tables\/","url":"https:\/\/insidebigdata.com\/2022\/04\/06\/databricks-announces-general-availability-of-delta-live-tables\/","name":"Databricks Announces General Availability of Delta Live Tables - insideBIGDATA","isPartOf":{"@id":"https:\/\/insidebigdata.com\/#website"},"datePublished":"2022-04-06T13:00:00+00:00","dateModified":"2022-04-05T20:50:47+00:00","author":{"@id":"https:\/\/insidebigdata.com\/#\/schema\/person\/2949e412c144601cdbcc803bd234e1b9"},"breadcrumb":{"@id":"https:\/\/insidebigdata.com\/2022\/04\/06\/databricks-announces-general-availability-of-delta-live-tables\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/insidebigdata.com\/2022\/04\/06\/databricks-announces-general-availability-of-delta-live-tables\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/insidebigdata.com\/2022\/04\/06\/databricks-announces-general-availability-of-delta-live-tables\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/insidebigdata.com\/"},{"@type":"ListItem","position":2,"name":"Databricks Announces General Availability of Delta Live Tables"}]},{"@type":"WebSite","@id":"https:\/\/insidebigdata.com\/#website","url":"https:\/\/insidebigdata.com\/","name":"insideBIGDATA","description":"Your Source for AI, Data Science, Deep Learning &amp; Machine Learning Strategies","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/insidebigdata.com\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/insidebigdata.com\/#\/schema\/person\/2949e412c144601cdbcc803bd234e1b9","name":"Editorial Team","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/insidebigdata.com\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/e137ce7ea40e38bd4d25bb7860cfe3e4?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/e137ce7ea40e38bd4d25bb7860cfe3e4?s=96&d=mm&r=g","caption":"Editorial Team"},"sameAs":["http:\/\/www.insidebigdata.com"],"url":"https:\/\/insidebigdata.com\/author\/editorial\/"}]}},"jetpack_featured_media_url":"https:\/\/insidebigdata.com\/wp-content\/uploads\/2022\/01\/Databricks_logo_2022.png","jetpack_shortlink":"https:\/\/wp.me\/p9eA3j-7wI","jetpack-related-posts":[{"id":29715,"url":"https:\/\/insidebigdata.com\/2022\/07\/02\/databricks-announces-major-contributions-to-flagship-open-source-projects\/","url_meta":{"origin":28936,"position":0},"title":"Databricks Announces Major Contributions to Flagship Open Source Projects","date":"July 2, 2022","format":false,"excerpt":"Databricks announced that the company will contribute all features and enhancements it has made to Delta Lake to the Linux Foundation and open source all Delta Lake APIs as part of the Delta Lake 2.0 release. In addition, the company announced MLflow 2.0, which includes MLflow Pipelines, a new feature\u2026","rel":"","context":"In &quot;Big Data&quot;","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":30797,"url":"https:\/\/insidebigdata.com\/2022\/11\/03\/video-highlights-modernize-your-ibm-mainframe-netezza-with-databricks-lakehouse\/","url_meta":{"origin":28936,"position":1},"title":"Video Highlights: Modernize your IBM Mainframe &#038; Netezza With Databricks Lakehouse","date":"November 3, 2022","format":false,"excerpt":"In the video presentation below, learn from experts how to architect modern data pipelines to consolidate data from multiple IBM data sources into Databricks Lakehouse, using the state-of-the-art replication technique\u2014Change Data Capture (CDC).","rel":"","context":"In &quot;Big Data&quot;","img":{"alt_text":"","src":"https:\/\/i0.wp.com\/insidebigdata.com\/wp-content\/uploads\/2019\/10\/data-lakes_SHUTTERSTOCK.jpg?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":28258,"url":"https:\/\/insidebigdata.com\/2022\/01\/17\/databricks-launches-data-lakehouse-for-retail-and-consumer-goods-customers\/","url_meta":{"origin":28936,"position":2},"title":"Databricks Launches Data Lakehouse for Retail and Consumer Goods Customers","date":"January 17, 2022","format":false,"excerpt":"Databricks, the Data and AI company and pioneer of the data lakehouse architecture, announced the Databricks Lakehouse for Retail, the company\u2019s first industry-specific data lakehouse for retailers and consumer goods (CG) customers. With Databricks\u2019 Lakehouse for Retail, data teams are enabled with a centralized data and AI platform that is\u2026","rel":"","context":"In &quot;AI Deep Learning&quot;","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":31820,"url":"https:\/\/insidebigdata.com\/2023\/03\/07\/databricks-launches-simplified-real-time-machine-learning-for-the-lakehouse\/","url_meta":{"origin":28936,"position":3},"title":"Databricks Launches Simplified Real-Time Machine Learning for the Lakehouse","date":"March 7, 2023","format":false,"excerpt":"Databricks, the lakehouse company, announced the launch of Databricks Model Serving to provide simplified production machine learning (ML) natively within the Databricks Lakehouse Platform. Model Serving removes the complexity of building and maintaining complicated infrastructure for intelligent applications. Now, organizations can leverage the Databricks Lakehouse Platform to integrate real-time machine\u2026","rel":"","context":"In &quot;AI Deep Learning&quot;","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":25231,"url":"https:\/\/insidebigdata.com\/2020\/11\/14\/databricks-launches-sql-analytics-to-enable-cloud-data-warehousing-on-data-lakes\/","url_meta":{"origin":28936,"position":4},"title":"Databricks Launches SQL Analytics to Enable Cloud Data Warehousing on Data Lakes","date":"November 14, 2020","format":false,"excerpt":"Databricks, the data and AI company, announced the launch of SQL Analytics, which for the first time enables data analysts to perform workloads previously meant only for a data warehouse on a data lake. This expands the traditional scope of the data lake from data science and machine learning to\u2026","rel":"","context":"In &quot;Analytics&quot;","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":19213,"url":"https:\/\/insidebigdata.com\/2017\/10\/26\/databricks-launches-delta-combine-best-data-lakes-data-warehouses-streaming-systems\/","url_meta":{"origin":28936,"position":5},"title":"Databricks Launches Delta To Combine the Best of Data Lakes, Data Warehouses and Streaming Systems","date":"October 26, 2017","format":false,"excerpt":"Databricks, provider of the leading Unified Analytics Platform and founded by the team who created Apache Spark\u2122, announced Databricks Delta, the first unified data management system that provides the scale and cost-efficiency of a data lake, the query performance of a data warehouse, and the low latency of a streaming\u2026","rel":"","context":"In &quot;Big Data&quot;","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"_links":{"self":[{"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/posts\/28936"}],"collection":[{"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/users\/10513"}],"replies":[{"embeddable":true,"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/comments?post=28936"}],"version-history":[{"count":0,"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/posts\/28936\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/media\/28259"}],"wp:attachment":[{"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/media?parent=28936"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/categories?post=28936"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/tags?post=28936"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}