{"id":29701,"date":"2022-06-28T06:00:00","date_gmt":"2022-06-28T13:00:00","guid":{"rendered":"https:\/\/insidebigdata.com\/?p=29701"},"modified":"2022-06-29T10:18:40","modified_gmt":"2022-06-29T17:18:40","slug":"what-is-data-reliability-engineering","status":"publish","type":"post","link":"https:\/\/insidebigdata.com\/2022\/06\/28\/what-is-data-reliability-engineering\/","title":{"rendered":"What Is Data Reliability Engineering?"},"content":{"rendered":"\n

Data Reliability Engineering (DRE) is the work done to keep data pipelines delivering fresh and high-quality input data to the users and applications that depend on them. The goal of DRE is to allow for iteration on data infrastructure, the logical data model, etc. as quickly as possible, while\u2014and this is the key part! \u2014still guaranteeing that the data is usable for the applications that depend on it.<\/p>\n\n\n\n

End users\u2014data scientists examining a\/b-test results, executives looking at dashboards, customers seeing product recommendations, etc.\u2014don\u2019t care about data quality in the abstract. They care about whether the data they\u2019re seeing is useful for the task at hand. DRE focuses on quantifying and meeting those needs, without slowing down the organization\u2019s ability to grow and evolve its data architecture.<\/p>\n\n\n\n

It borrows the core concepts of Site Reliability Engineering<\/a>, which is used at companies like Google<\/a>, Meta<\/a>, Netflix<\/a>, and Stripe<\/a> to iterate quickly while keeping their products reliable 24\/7. These concepts bring a methodical and quantified approach to defining quality, gracefully handling problems, and aligning teams to balance speed and reliability.<\/p>\n\n\n\n

Why is data reliability engineering important now?<\/strong><\/p>\n\n\n\n

Nobody needs to be told how critical data is becoming to nearly every industry. As we move to a world where more roles\u2014not just data science and engineering professionals\u2014are interacting with data whether through self-service analytics or the outputs from machine learning models, there\u2019s more demand for it to \u201cjust work\u201d every hour of every day.<\/p>\n\n\n\n

But in addition to having more users and more use cases to serve, data teams are simultaneously dealing with larger and more diverse volumes of data. Thanks to Snowflake, Databricks, Airflow, dbt, and other modern data infra tools, it\u2019s never been easier to reach a scale where ad hoc approaches can\u2019t keep up.<\/p>\n\n\n\n

While the most obvious big-data companies like Uber<\/a>, AirBnB<\/a>, and Netflix<\/a> felt these pains sooner and led much of the foundational work in this discipline, it\u2019s rapidly catching on<\/a> more<\/a> broadly<\/a>.<\/p>\n\n\n\n

What are the fundamental principles of data reliability engineering? <\/strong><\/p>\n\n\n\n

The seven principles<\/a> from Google\u2019s SRE Handbook provide a great starting point for DRE, which can adapt them to deal with data warehouses and pipelines, instead of software applications.<\/p>\n\n\n\n

  1. Embrace risk<\/strong>: Because something<\/em> will eventually fail, data teams need a plan to detect, manage, and mitigate failures when (or before) they occur. <\/li>
  2. Monitor everything<\/strong>: Problems can\u2019t be mitigated if they can\u2019t be detected. Monitoring and alerting give teams the details they need to address data issues. <\/li>
  3. Set data-quality standards<\/strong>: Acceptable data quality standards need to be quantified and agreed upon for teams to do anything about it. SLIs, SLOs, and SLAs<\/a> are the standards-setting tools being adopted for DRE. <\/li>
  4. Reduce toil<\/strong>: Toil is the human-led, operational work required to improve a system. For efficient DRE, teams ideally should reduce toil to reduce overhead. <\/li>
  5. Use automation<\/strong>: Automating manual processes helps data teams scale reliability efforts and increase time for tackling higher-order problems. <\/li>
  6. Control releases<\/strong>: Making changes is how things improve and how they break. Pipeline code is still code, faulty or not, that needs to QA\u2019ed before deployment.  <\/li>
  7. Maintain simplicity<\/strong>: Minimizing and isolating the complexity in any one pipeline job goes a long way toward keeping it reliable<\/li><\/ol>\n\n\n\n

    Will data reliability engineering become an industry standard? <\/strong><\/p>\n\n\n\n

    While one could argue that data reliability engineering is still an emerging concept, modern companies (Uber, DoorDash, Instacart, etc.) that use data to operate and grow their businesses are leading the charge to establish DRE as a standard practice. And job postings for the role<\/a> are already starting to grow. Given the pace of business and the need for data to be trusted, expect to see DRE someday be as prevalent as SRE is now. <\/p>\n\n\n\n

    About the Author<\/strong><\/p>\n\n\n

    \n
    \"\"<\/figure><\/div>\n\n\n

    Kyle Kirwan, CEO and co-founder of Bigeye<\/a>. Bigeye is the data observability platform that helps data teams keep their pipelines fresh and high quality. Data teams at companies like Instacart, Zoom, and Udacity use Bigeye to automate their data monitoring, detect issues proactively, and keep data reliable for the data scientists, executives, and customers who depend on it.<\/em><\/p>\n\n\n\n

    Sign up for the free insideBIGDATA newsletter<\/a>.<\/em><\/p>\n\n\n\n

    Join us on Twitter: @InsideBigData1 \u2013 https:\/\/twitter.com\/InsideBigData1<\/a><\/em><\/p>\n","protected":false},"excerpt":{"rendered":"

    In this contributed article, Kyle Kirwan, CEO and co-founder of Bigeye, discusses Data Reliability Engineering (DRE), the work done to keep data pipelines delivering fresh and high-quality input data to the users and applications that depend on them. The goal of DRE is to allow for iteration on data infrastructure, the logical data model, etc. as quickly as possible, while\u2014and this is the key part! \u2014still guaranteeing that the data is usable for the applications that depend on it. <\/p>\n","protected":false},"author":10513,"featured_media":24971,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"footnotes":""},"categories":[115,64,182,87,180,67,56,97,1],"tags":[781,1067,95],"acf":[],"yoast_head":"\nWhat Is Data Reliability Engineering? - insideBIGDATA<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/insidebigdata.com\/2022\/06\/28\/what-is-data-reliability-engineering\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"What Is Data Reliability Engineering? - insideBIGDATA\" \/>\n<meta property=\"og:description\" content=\"In this contributed article, Kyle Kirwan, CEO and co-founder of Bigeye, discusses Data Reliability Engineering (DRE), the work done to keep data pipelines delivering fresh and high-quality input data to the users and applications that depend on them. The goal of DRE is to allow for iteration on data infrastructure, the logical data model, etc. as quickly as possible, while\u2014and this is the key part! \u2014still guaranteeing that the data is usable for the applications that depend on it.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/insidebigdata.com\/2022\/06\/28\/what-is-data-reliability-engineering\/\" \/>\n<meta property=\"og:site_name\" content=\"insideBIGDATA\" \/>\n<meta property=\"article:publisher\" content=\"http:\/\/www.facebook.com\/insidebigdata\" \/>\n<meta property=\"article:published_time\" content=\"2022-06-28T13:00:00+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2022-06-29T17:18:40+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/insidebigdata.com\/wp-content\/uploads\/2020\/09\/data_engineering_shutterstock_669226057.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"300\" \/>\n\t<meta property=\"og:image:height\" content=\"168\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Editorial Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@insideBigData\" \/>\n<meta name=\"twitter:site\" content=\"@insideBigData\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Editorial Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/insidebigdata.com\/2022\/06\/28\/what-is-data-reliability-engineering\/\",\"url\":\"https:\/\/insidebigdata.com\/2022\/06\/28\/what-is-data-reliability-engineering\/\",\"name\":\"What Is Data Reliability Engineering? - insideBIGDATA\",\"isPartOf\":{\"@id\":\"https:\/\/insidebigdata.com\/#website\"},\"datePublished\":\"2022-06-28T13:00:00+00:00\",\"dateModified\":\"2022-06-29T17:18:40+00:00\",\"author\":{\"@id\":\"https:\/\/insidebigdata.com\/#\/schema\/person\/2949e412c144601cdbcc803bd234e1b9\"},\"breadcrumb\":{\"@id\":\"https:\/\/insidebigdata.com\/2022\/06\/28\/what-is-data-reliability-engineering\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/insidebigdata.com\/2022\/06\/28\/what-is-data-reliability-engineering\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/insidebigdata.com\/2022\/06\/28\/what-is-data-reliability-engineering\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/insidebigdata.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"What Is Data Reliability Engineering?\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/insidebigdata.com\/#website\",\"url\":\"https:\/\/insidebigdata.com\/\",\"name\":\"insideBIGDATA\",\"description\":\"Your Source for AI, Data Science, Deep Learning & Machine Learning Strategies\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/insidebigdata.com\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/insidebigdata.com\/#\/schema\/person\/2949e412c144601cdbcc803bd234e1b9\",\"name\":\"Editorial Team\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/insidebigdata.com\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/e137ce7ea40e38bd4d25bb7860cfe3e4?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/e137ce7ea40e38bd4d25bb7860cfe3e4?s=96&d=mm&r=g\",\"caption\":\"Editorial Team\"},\"sameAs\":[\"http:\/\/www.insidebigdata.com\"],\"url\":\"https:\/\/insidebigdata.com\/author\/editorial\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"What Is Data Reliability Engineering? - insideBIGDATA","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/insidebigdata.com\/2022\/06\/28\/what-is-data-reliability-engineering\/","og_locale":"en_US","og_type":"article","og_title":"What Is Data Reliability Engineering? - insideBIGDATA","og_description":"In this contributed article, Kyle Kirwan, CEO and co-founder of Bigeye, discusses Data Reliability Engineering (DRE), the work done to keep data pipelines delivering fresh and high-quality input data to the users and applications that depend on them. The goal of DRE is to allow for iteration on data infrastructure, the logical data model, etc. as quickly as possible, while\u2014and this is the key part! \u2014still guaranteeing that the data is usable for the applications that depend on it.","og_url":"https:\/\/insidebigdata.com\/2022\/06\/28\/what-is-data-reliability-engineering\/","og_site_name":"insideBIGDATA","article_publisher":"http:\/\/www.facebook.com\/insidebigdata","article_published_time":"2022-06-28T13:00:00+00:00","article_modified_time":"2022-06-29T17:18:40+00:00","og_image":[{"width":300,"height":168,"url":"https:\/\/insidebigdata.com\/wp-content\/uploads\/2020\/09\/data_engineering_shutterstock_669226057.jpg","type":"image\/jpeg"}],"author":"Editorial Team","twitter_card":"summary_large_image","twitter_creator":"@insideBigData","twitter_site":"@insideBigData","twitter_misc":{"Written by":"Editorial Team","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/insidebigdata.com\/2022\/06\/28\/what-is-data-reliability-engineering\/","url":"https:\/\/insidebigdata.com\/2022\/06\/28\/what-is-data-reliability-engineering\/","name":"What Is Data Reliability Engineering? - insideBIGDATA","isPartOf":{"@id":"https:\/\/insidebigdata.com\/#website"},"datePublished":"2022-06-28T13:00:00+00:00","dateModified":"2022-06-29T17:18:40+00:00","author":{"@id":"https:\/\/insidebigdata.com\/#\/schema\/person\/2949e412c144601cdbcc803bd234e1b9"},"breadcrumb":{"@id":"https:\/\/insidebigdata.com\/2022\/06\/28\/what-is-data-reliability-engineering\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/insidebigdata.com\/2022\/06\/28\/what-is-data-reliability-engineering\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/insidebigdata.com\/2022\/06\/28\/what-is-data-reliability-engineering\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/insidebigdata.com\/"},{"@type":"ListItem","position":2,"name":"What Is Data Reliability Engineering?"}]},{"@type":"WebSite","@id":"https:\/\/insidebigdata.com\/#website","url":"https:\/\/insidebigdata.com\/","name":"insideBIGDATA","description":"Your Source for AI, Data Science, Deep Learning & Machine Learning Strategies","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/insidebigdata.com\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/insidebigdata.com\/#\/schema\/person\/2949e412c144601cdbcc803bd234e1b9","name":"Editorial Team","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/insidebigdata.com\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/e137ce7ea40e38bd4d25bb7860cfe3e4?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/e137ce7ea40e38bd4d25bb7860cfe3e4?s=96&d=mm&r=g","caption":"Editorial Team"},"sameAs":["http:\/\/www.insidebigdata.com"],"url":"https:\/\/insidebigdata.com\/author\/editorial\/"}]}},"jetpack_featured_media_url":"https:\/\/insidebigdata.com\/wp-content\/uploads\/2020\/09\/data_engineering_shutterstock_669226057.jpg","jetpack_shortlink":"https:\/\/wp.me\/p9eA3j-7J3","jetpack-related-posts":[{"id":17997,"url":"https:\/\/insidebigdata.com\/2017\/05\/27\/cloudera-launches-altus-simplify-big-data-workloads-cloud\/","url_meta":{"origin":29701,"position":0},"title":"Cloudera Launches Altus to Simplify Big Data Workloads in the Cloud","date":"May 27, 2017","format":false,"excerpt":"Cloudera, Inc, (NYSE:CLDR) the provider of a leading modern platform for machine learning and advanced analytics, announced the release of Cloudera Altus, a Platform-as-a-Service (PaaS) offering that makes it easier to run large-scale data processing applications on public cloud.","rel":"","context":"In "Big Data"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":32010,"url":"https:\/\/insidebigdata.com\/2023\/04\/04\/slidecast-ashwin-rajeev-co-founder-cto-of-acceldata-discusses-data-observability\/","url_meta":{"origin":29701,"position":1},"title":"Slidecast: Ashwin Rajeeva, Co-founder & CTO of Acceldata Discusses Data Observability","date":"April 4, 2023","format":false,"excerpt":"In this slidecast presentation, Ashwin Rajeev from Acceldata\u00a0describes the company\u2019s data observability solutions. Acceldata solutions allow you to gain comprehensive insights into your data stack to improve data and pipeline reliability, platform performance, and spend efficiency.","rel":"","context":"In "Big Data"","img":{"alt_text":"","src":"https:\/\/i0.wp.com\/insidebigdata.com\/wp-content\/uploads\/2023\/03\/Ashwin_Rajeev.jpg?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":27655,"url":"https:\/\/insidebigdata.com\/2021\/11\/18\/2022-state-of-data-engineering-emerging-challenges-with-data-security-quality\/","url_meta":{"origin":29701,"position":2},"title":"2022 State of Data Engineering: Emerging Challenges with Data Security & Quality","date":"November 18, 2021","format":false,"excerpt":"The 2022 Data Engineering Survey, from our friends over at Immuta, examined the changing landscape of data engineering and operations challenges, tools, and opportunities. The modern data engineering technology market is dynamic, driven by the tectonic shift from on-premise databases and BI tools to modern, cloud-based data platforms built on\u2026","rel":"","context":"In "Big Data"","img":{"alt_text":"","src":"https:\/\/i0.wp.com\/insidebigdata.com\/wp-content\/uploads\/2021\/11\/Immuta_whitepaper_fig.png?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":31906,"url":"https:\/\/insidebigdata.com\/2023\/03\/25\/acceldata-expands-data-observability-platform-with-new-data-reliability-capabilities\/","url_meta":{"origin":29701,"position":3},"title":"Acceldata Expands Data Observability Platform with New Data Reliability Capabilities","date":"March 25, 2023","format":false,"excerpt":"Acceldata, a market leader in data observability, announced significant enhancements to its data reliability solution, including no-code\/low-code options, intelligent alerting, targeted recommendations, and self-healing capabilities to solve the most complex data reliability challenges while improving operational efficiency and reducing costs.\u00a0","rel":"","context":"In "Analytics"","img":{"alt_text":"","src":"https:\/\/i0.wp.com\/insidebigdata.com\/wp-content\/uploads\/2022\/07\/Acceldata_logo.png?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":16029,"url":"https:\/\/insidebigdata.com\/2016\/09\/16\/unravel-data-launches-platform-to-dramatically-improve-big-data-application-performance\/","url_meta":{"origin":29701,"position":4},"title":"Unravel Data Launches Platform to Dramatically Improve Big Data Application Performance","date":"September 16, 2016","format":false,"excerpt":"Unravel Data, provider of a full-stack performance intelligence platform for optimizing Big Data operations (DataOps [1]), emerged from stealth, announcing a new platform to accelerate Big Data applications, optimize resource usage, and provide operations intelligence, all from a single platform.","rel":"","context":"In "Big Data"","img":{"alt_text":"unravel_graphic","src":"https:\/\/i0.wp.com\/insidebigdata.com\/wp-content\/uploads\/2016\/09\/Unravel_Graphic.png?resize=350%2C200","width":350,"height":200},"classes":[]},{"id":19428,"url":"https:\/\/insidebigdata.com\/2017\/11\/20\/sentenai-delivers-automated-data-engineering-data-science-machine-learning-applications\/","url_meta":{"origin":29701,"position":5},"title":"Sentenai Delivers Automated Data Engineering for Data Science and Machine Learning Applications","date":"November 20, 2017","format":false,"excerpt":"Sentenai, an emerging sensor data technology company, announced the availability of its flagship product, the Sentenai Sensor Data Cloud. By going beyond the initial harnessing of machine-based data and understanding the information that data provides, organizations can streamline their operational processes and develop predictive maintenance solutions that decrease unplanned downtime.","rel":"","context":"In "Big Data"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"_links":{"self":[{"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/posts\/29701"}],"collection":[{"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/users\/10513"}],"replies":[{"embeddable":true,"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/comments?post=29701"}],"version-history":[{"count":0,"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/posts\/29701\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/media\/24971"}],"wp:attachment":[{"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/media?parent=29701"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/categories?post=29701"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/tags?post=29701"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}