{"id":27655,"date":"2021-11-18T06:00:00","date_gmt":"2021-11-18T14:00:00","guid":{"rendered":"https:\/\/insidebigdata.com\/?p=27655"},"modified":"2021-11-19T09:20:48","modified_gmt":"2021-11-19T17:20:48","slug":"2022-state-of-data-engineering-emerging-challenges-with-data-security-quality","status":"publish","type":"post","link":"https:\/\/insidebigdata.com\/2021\/11\/18\/2022-state-of-data-engineering-emerging-challenges-with-data-security-quality\/","title":{"rendered":"2022 State of Data Engineering: Emerging Challenges with Data Security &#038; Quality"},"content":{"rendered":"\n<div class=\"wp-block-image is-style-default\"><figure class=\"alignright size-large\"><img decoding=\"async\" loading=\"lazy\" width=\"152\" height=\"99\" src=\"https:\/\/insidebigdata.com\/wp-content\/uploads\/2021\/01\/Immuta_logo.png\" alt=\"\" class=\"wp-image-25555\" srcset=\"https:\/\/insidebigdata.com\/wp-content\/uploads\/2021\/01\/Immuta_logo.png 152w, https:\/\/insidebigdata.com\/wp-content\/uploads\/2021\/01\/Immuta_logo-150x98.png 150w\" sizes=\"(max-width: 152px) 100vw, 152px\" \/><\/figure><\/div>\n\n\n\n<p>The <a href=\"https:\/\/insidebigdata.com\/white-paper\/2022-state-of-data-engineering-emerging-challenges-with-data-security-quality\/\" target=\"_blank\" rel=\"noreferrer noopener\">2022 Data Engineering Survey<\/a>, from our friends over at <a href=\"https:\/\/www.immuta.com\/\" target=\"_blank\" rel=\"noreferrer noopener\">Immuta<\/a>, examined the changing landscape of data engineering and operations challenges, tools, and opportunities. The modern data engineering technology market is dynamic, driven by the tectonic shift from on-premise databases and BI tools to modern, cloud-based data platforms built on lakehouse architectures.<\/p>\n\n\n\n<p>More than the on-premises market that preceded it, the cloud data technology market is evolving rapidly, and spans a vast set of open source and commercial data technologies, tools, and products. At the same time, organizations are adopting multiple technologies to keep up with the scale, speed, and use cases that today\u2019s data environment demands. To remain competitive and maximize the value of their data \u2013 including sensitive data \u2013 organizations are developing DataOps functions and frameworks to varying degrees. DataOps tools and processes enable continuous and automated delivery of data to power BI, analytics, data science, and data-powered products.<\/p>\n\n\n\n<p>This comprehensive survey examines the changing landscape of data engineering and emerging challenges such as data privacy, security, and quality as companies shift data management from on-premises databases and BI tools to cloud-based data platforms.<\/p>\n\n\n\n<p>The survey probed data leaders, architects, and engineers about the trends they\u2019re seeing, challenges they face, and what they expect for the coming years, including:&nbsp;<\/p>\n\n\n\n<ul><li>The most and least challenging tasks that data teams face with data pipelines<\/li><li>Why the modern data landscape and data use are causing new challenges to emerge<\/li><li>The top cloud data platforms and tools that data teams are adopting to maximize their data\u2019s value<\/li><\/ul>\n\n\n\n<div class=\"wp-block-image is-style-default\"><figure class=\"aligncenter size-large\"><img decoding=\"async\" loading=\"lazy\" width=\"700\" height=\"293\" src=\"https:\/\/insidebigdata.com\/wp-content\/uploads\/2021\/11\/Immuta_whitepaper_fig.png\" alt=\"\" class=\"wp-image-27658\" srcset=\"https:\/\/insidebigdata.com\/wp-content\/uploads\/2021\/11\/Immuta_whitepaper_fig.png 700w, https:\/\/insidebigdata.com\/wp-content\/uploads\/2021\/11\/Immuta_whitepaper_fig-150x63.png 150w, https:\/\/insidebigdata.com\/wp-content\/uploads\/2021\/11\/Immuta_whitepaper_fig-300x126.png 300w\" sizes=\"(max-width: 700px) 100vw, 700px\" \/><\/figure><\/div>\n\n\n\n<p>This report aims to help data teams understand and learn from the choices others are making, and the challenges they face as the data landscape continues its rapid evolution.<\/p>\n\n\n\n<p>Download the full report <a href=\"https:\/\/insidebigdata.com\/white-paper\/2022-state-of-data-engineering-emerging-challenges-with-data-security-quality\/\" target=\"_blank\" rel=\"noreferrer noopener\">HERE<\/a>. <\/p>\n\n\n\n<p><em>Sign up for the free insideBIGDATA&nbsp;<a rel=\"noreferrer noopener\" href=\"http:\/\/insidebigdata.com\/newsletter\/\" target=\"_blank\">newsletter<\/a>.<\/em><\/p>\n\n\n\n<p><em>Join us on Twitter:&nbsp;@InsideBigData1 \u2013 <a href=\"https:\/\/twitter.com\/InsideBigData1\" target=\"_blank\" rel=\"noreferrer noopener\">https:\/\/twitter.com\/InsideBigData1<\/a><\/em><\/p>\n","protected":false},"excerpt":{"rendered":"<p>The 2022 Data Engineering Survey, from our friends over at Immuta, examined the changing landscape of data engineering and operations challenges, tools, and opportunities. The modern data engineering technology market is dynamic, driven by the tectonic shift from on-premise databases and BI tools to modern, cloud-based data platforms built on lakehouse architectures.<\/p>\n","protected":false},"author":10513,"featured_media":27653,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"footnotes":""},"categories":[115,87,180,67,56,84,1,58],"tags":[781,790,95],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v20.6 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>2022 State of Data Engineering: Emerging Challenges with Data Security &amp; Quality - insideBIGDATA<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/insidebigdata.com\/2021\/11\/18\/2022-state-of-data-engineering-emerging-challenges-with-data-security-quality\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"2022 State of Data Engineering: Emerging Challenges with Data Security &amp; Quality - insideBIGDATA\" \/>\n<meta property=\"og:description\" content=\"The 2022 Data Engineering Survey, from our friends over at Immuta, examined the changing landscape of data engineering and operations challenges, tools, and opportunities. The modern data engineering technology market is dynamic, driven by the tectonic shift from on-premise databases and BI tools to modern, cloud-based data platforms built on lakehouse architectures.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/insidebigdata.com\/2021\/11\/18\/2022-state-of-data-engineering-emerging-challenges-with-data-security-quality\/\" \/>\n<meta property=\"og:site_name\" content=\"insideBIGDATA\" \/>\n<meta property=\"article:publisher\" content=\"http:\/\/www.facebook.com\/insidebigdata\" \/>\n<meta property=\"article:published_time\" content=\"2021-11-18T14:00:00+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2021-11-19T17:20:48+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/insidebigdata.com\/wp-content\/uploads\/2021\/11\/Immuta_whitepaper_cover.png\" \/>\n\t<meta property=\"og:image:width\" content=\"300\" \/>\n\t<meta property=\"og:image:height\" content=\"391\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Editorial Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@insideBigData\" \/>\n<meta name=\"twitter:site\" content=\"@insideBigData\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Editorial Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"2 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/insidebigdata.com\/2021\/11\/18\/2022-state-of-data-engineering-emerging-challenges-with-data-security-quality\/\",\"url\":\"https:\/\/insidebigdata.com\/2021\/11\/18\/2022-state-of-data-engineering-emerging-challenges-with-data-security-quality\/\",\"name\":\"2022 State of Data Engineering: Emerging Challenges with Data Security & Quality - insideBIGDATA\",\"isPartOf\":{\"@id\":\"https:\/\/insidebigdata.com\/#website\"},\"datePublished\":\"2021-11-18T14:00:00+00:00\",\"dateModified\":\"2021-11-19T17:20:48+00:00\",\"author\":{\"@id\":\"https:\/\/insidebigdata.com\/#\/schema\/person\/2949e412c144601cdbcc803bd234e1b9\"},\"breadcrumb\":{\"@id\":\"https:\/\/insidebigdata.com\/2021\/11\/18\/2022-state-of-data-engineering-emerging-challenges-with-data-security-quality\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/insidebigdata.com\/2021\/11\/18\/2022-state-of-data-engineering-emerging-challenges-with-data-security-quality\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/insidebigdata.com\/2021\/11\/18\/2022-state-of-data-engineering-emerging-challenges-with-data-security-quality\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/insidebigdata.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"2022 State of Data Engineering: Emerging Challenges with Data Security &#038; Quality\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/insidebigdata.com\/#website\",\"url\":\"https:\/\/insidebigdata.com\/\",\"name\":\"insideBIGDATA\",\"description\":\"Your Source for AI, Data Science, Deep Learning &amp; Machine Learning Strategies\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/insidebigdata.com\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/insidebigdata.com\/#\/schema\/person\/2949e412c144601cdbcc803bd234e1b9\",\"name\":\"Editorial Team\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/insidebigdata.com\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/e137ce7ea40e38bd4d25bb7860cfe3e4?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/e137ce7ea40e38bd4d25bb7860cfe3e4?s=96&d=mm&r=g\",\"caption\":\"Editorial Team\"},\"sameAs\":[\"http:\/\/www.insidebigdata.com\"],\"url\":\"https:\/\/insidebigdata.com\/author\/editorial\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"2022 State of Data Engineering: Emerging Challenges with Data Security & Quality - insideBIGDATA","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/insidebigdata.com\/2021\/11\/18\/2022-state-of-data-engineering-emerging-challenges-with-data-security-quality\/","og_locale":"en_US","og_type":"article","og_title":"2022 State of Data Engineering: Emerging Challenges with Data Security & Quality - insideBIGDATA","og_description":"The 2022 Data Engineering Survey, from our friends over at Immuta, examined the changing landscape of data engineering and operations challenges, tools, and opportunities. The modern data engineering technology market is dynamic, driven by the tectonic shift from on-premise databases and BI tools to modern, cloud-based data platforms built on lakehouse architectures.","og_url":"https:\/\/insidebigdata.com\/2021\/11\/18\/2022-state-of-data-engineering-emerging-challenges-with-data-security-quality\/","og_site_name":"insideBIGDATA","article_publisher":"http:\/\/www.facebook.com\/insidebigdata","article_published_time":"2021-11-18T14:00:00+00:00","article_modified_time":"2021-11-19T17:20:48+00:00","og_image":[{"width":300,"height":391,"url":"https:\/\/insidebigdata.com\/wp-content\/uploads\/2021\/11\/Immuta_whitepaper_cover.png","type":"image\/png"}],"author":"Editorial Team","twitter_card":"summary_large_image","twitter_creator":"@insideBigData","twitter_site":"@insideBigData","twitter_misc":{"Written by":"Editorial Team","Est. reading time":"2 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/insidebigdata.com\/2021\/11\/18\/2022-state-of-data-engineering-emerging-challenges-with-data-security-quality\/","url":"https:\/\/insidebigdata.com\/2021\/11\/18\/2022-state-of-data-engineering-emerging-challenges-with-data-security-quality\/","name":"2022 State of Data Engineering: Emerging Challenges with Data Security & Quality - insideBIGDATA","isPartOf":{"@id":"https:\/\/insidebigdata.com\/#website"},"datePublished":"2021-11-18T14:00:00+00:00","dateModified":"2021-11-19T17:20:48+00:00","author":{"@id":"https:\/\/insidebigdata.com\/#\/schema\/person\/2949e412c144601cdbcc803bd234e1b9"},"breadcrumb":{"@id":"https:\/\/insidebigdata.com\/2021\/11\/18\/2022-state-of-data-engineering-emerging-challenges-with-data-security-quality\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/insidebigdata.com\/2021\/11\/18\/2022-state-of-data-engineering-emerging-challenges-with-data-security-quality\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/insidebigdata.com\/2021\/11\/18\/2022-state-of-data-engineering-emerging-challenges-with-data-security-quality\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/insidebigdata.com\/"},{"@type":"ListItem","position":2,"name":"2022 State of Data Engineering: Emerging Challenges with Data Security &#038; Quality"}]},{"@type":"WebSite","@id":"https:\/\/insidebigdata.com\/#website","url":"https:\/\/insidebigdata.com\/","name":"insideBIGDATA","description":"Your Source for AI, Data Science, Deep Learning &amp; Machine Learning Strategies","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/insidebigdata.com\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/insidebigdata.com\/#\/schema\/person\/2949e412c144601cdbcc803bd234e1b9","name":"Editorial Team","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/insidebigdata.com\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/e137ce7ea40e38bd4d25bb7860cfe3e4?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/e137ce7ea40e38bd4d25bb7860cfe3e4?s=96&d=mm&r=g","caption":"Editorial Team"},"sameAs":["http:\/\/www.insidebigdata.com"],"url":"https:\/\/insidebigdata.com\/author\/editorial\/"}]}},"jetpack_featured_media_url":"https:\/\/insidebigdata.com\/wp-content\/uploads\/2021\/11\/Immuta_whitepaper_cover.png","jetpack_shortlink":"https:\/\/wp.me\/p9eA3j-7c3","jetpack-related-posts":[{"id":25579,"url":"https:\/\/insidebigdata.com\/2021\/02\/03\/data-engineering-survey-2021-impact-report\/","url_meta":{"origin":27655,"position":0},"title":"Data Engineering Survey: 2021 Impact Report","date":"February 3, 2021","format":false,"excerpt":"This Data Engineering Survey: 2021 Impact Report summarizes key findings from the inaugural survey and provides a glimpse into the current and future state of data engineering and DataOps. The report highlights some of the major trends uncovered in this year\u2019s survey including the adoption of cloud data platforms, what\u2026","rel":"","context":"In &quot;AI Deep Learning&quot;","img":{"alt_text":"","src":"https:\/\/i0.wp.com\/insidebigdata.com\/wp-content\/uploads\/2021\/02\/Immuta_fig1.png?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":32641,"url":"https:\/\/insidebigdata.com\/2023\/06\/14\/largest-data-engineering-survey-reports-on-adoption-of-modern-data-stack-tools\/","url_meta":{"origin":27655,"position":1},"title":"Largest Data Engineering Survey Reports on Adoption of Modern Data Stack Tools","date":"June 14, 2023","format":false,"excerpt":"Airbyte, creators of a fast-growing open-source data integration platform, made available results of the biggest data engineering survey in the market which provides\u00a0insights\u00a0into the latest trends, tools, and practices in data engineering \u2013 especially adoption of tools in the modern data stack.\u00a0Its first worldwide\u00a0State of Data survey displays results in\u2026","rel":"","context":"In &quot;AI Deep Learning&quot;","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":20323,"url":"https:\/\/insidebigdata.com\/2018\/04\/30\/next-generation-devops-ml-ops\/","url_meta":{"origin":27655,"position":2},"title":"The Next Generation of DevOps: ML Ops","date":"April 30, 2018","format":false,"excerpt":"In this special guest feature, Debashis Saha, Vice President of Platform Engineering at Intuit, discusses how DevOps methodologies can be applied to machine learning, in what he calls \u201cML Ops.\u201d With ML Ops, he believes it can provide an end-to-end automation of the process, creating transparency and delivering efficiency and\u2026","rel":"","context":"In &quot;Big Data&quot;","img":{"alt_text":"","src":"https:\/\/i0.wp.com\/insidebigdata.com\/wp-content\/uploads\/2018\/04\/Debashis-Saha_headshot.jpg?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":28748,"url":"https:\/\/insidebigdata.com\/2022\/03\/21\/the-powerful-combination-of-cloud-data-engineering-and-analytics-automation\/","url_meta":{"origin":27655,"position":3},"title":"The Powerful Combination of Cloud Data Engineering and Analytics Automation","date":"March 21, 2022","format":false,"excerpt":"In this sponsored post, our friends over at Trifacta discuss how unlocking the value from data \u2013 whether it is in the cloud or on premises \u2013 remains out of reach to many. According to an Alteryx-commissioned survey by YouGov, only 12% of workers reported having the benefit of driving\u2026","rel":"","context":"In &quot;Analytics&quot;","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":21692,"url":"https:\/\/insidebigdata.com\/2018\/12\/20\/unified-analytics-big-data-ai-success\/","url_meta":{"origin":27655,"position":4},"title":"Using Unified Analytics &#038; Big Data as Path to AI Success","date":"December 20, 2018","format":false,"excerpt":"How can modern enterprises unlock the potential of AI to change their business? Today\u2019s businesses and enterprises are increasingly focused on big data that can help drive innovation and transformation through the potential of artificial intelligence. According to a survey and research report commissioned with IDG\u2019s CIO, nearly 90 percent\u2026","rel":"","context":"In &quot;Analytics&quot;","img":{"alt_text":"","src":"https:\/\/i0.wp.com\/insidebigdata.com\/wp-content\/uploads\/2018\/12\/shutterstock_529299211-e1544820052641.jpg?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":25890,"url":"https:\/\/insidebigdata.com\/2021\/04\/06\/tecton-announces-line-up-for-first-annual-machine-learning-data-engineering-conference-apply\/","url_meta":{"origin":27655,"position":5},"title":"Tecton Announces Line-Up for First Annual Machine Learning Data Engineering Conference &#8211; apply()","date":"April 6, 2021","format":false,"excerpt":"Tecton, the enterprise feature store company, announced the line-up for apply(), a virtual conference that it is hosting on data engineering for applied machine learning (ML) April 21 - 22. apply() is a practitioner-focused community event for data and ML teams to discuss the practical data engineering challenges faced when\u2026","rel":"","context":"In &quot;Big Data&quot;","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"_links":{"self":[{"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/posts\/27655"}],"collection":[{"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/users\/10513"}],"replies":[{"embeddable":true,"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/comments?post=27655"}],"version-history":[{"count":0,"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/posts\/27655\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/media\/27653"}],"wp:attachment":[{"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/media?parent=27655"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/categories?post=27655"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/tags?post=27655"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}