{"id":29332,"date":"2022-05-13T06:00:00","date_gmt":"2022-05-13T13:00:00","guid":{"rendered":"https:\/\/insidebigdata.com\/?p=29332"},"modified":"2022-05-14T10:21:27","modified_gmt":"2022-05-14T17:21:27","slug":"looking-ahead-observability-data-management-modernization","status":"publish","type":"post","link":"https:\/\/insidebigdata.com\/2022\/05\/13\/looking-ahead-observability-data-management-modernization\/","title":{"rendered":"Looking Ahead | Observability Data Management Modernization"},"content":{"rendered":"\n

In a perfect world, it should be easy to get your data and analyze it. We are not in a perfect world. In the real world, data is generated from many applications, IoT sensors, server logs, containers, cloud services, and more. Organizations today use real-time data streams to keep up with evolving business requirements. Setting up data pipelines is easy. Handling the errors at each stage of the pipeline and not losing data is hard.<\/p>\n\n\n\n

With all of these data streams generated from various sources and in various formats, it\u2019s difficult to analyze the data to get valuable insights when you\u2019re dealing with system failures, data quality, and data intake variation. Add to this that organizations are entering the multi-petabyte and eventually exabyte range, and all of these issues are compounded. <\/p>\n\n\n\n

Data provenance presents even more challenges. Many organizations could have several copies of data with the similar provenance in different places with slightly tweaked content. How do you tell how each of the copies is different or which version you want? <\/p>\n\n\n\n

You have to be able to follow the data \u2018through the crevices\u2019 to see what changed, when it changed, and the cause of the problem. This can take months for the most talented data scientist \u2013 many small and medium organizations don\u2019t have data scientists to solve these issues. <\/p>\n\n\n\n

Current solutions on the market are expensive. All sizes of companies need help with tool sprawl and keeping the budget in control and observability standards will help ease the pressure. <\/p>\n\n\n\n

Real-time observability is critical, and using tiered storage and the cloud is beneficial. Operational complexity can be managed from the number of engineers to the total cost to manage the systems.<\/p>\n\n\n\n

The ideal world, with observability standards, will give companies observability data management strategies that offer the ability to handle storage and the management of data at rest and in motion with a cohesive infrastructure where problems are easy to troubleshoot and diagnose. <\/p>\n\n\n\n

In this scenario, it\u2019s easy to see if the system is working as intended and everything is managed in a single place, like a data fabric. It\u2019s not practical for companies to learn to speak three different languages in order to monitor data or manage their database. You should be able to say what you want of the data and when and not spend a huge amount of time dealing with data structure or finding the needle in the haystack when business processes are interrupted.   <\/p>\n\n\n\n

A data pipeline that incorporates an effective approach to back-pressure management, visualization, and data provenance translates into less troubleshooting, faster recovery, and cost reduction for your business.<\/p>\n\n\n\n

About the Author<\/strong><\/p>\n\n\n\n

\"\"<\/figure><\/div>\n\n\n\n

Karen Pieper is currently VP of engineering at Era Software<\/a>. She holds a Ph.D. from Stanford in Computer Science and has focused her career on solving hard tech problems. She was in chip design for 20 years, working on simulation and synthesis algorithms. She then moved to AWS, Facebook, and ERA Software focusing on terabyte and petabyte scale databases and data pipelines.  <\/em><\/p>\n\n\n\n

Sign up for the free insideBIGDATA newsletter<\/a>.<\/em><\/p>\n\n\n\n

Join us on Twitter: @InsideBigData1 \u2013 https:\/\/twitter.com\/InsideBigData1<\/a><\/em><\/p>\n","protected":false},"excerpt":{"rendered":"

In this contributed article, Karen Pieper, VP of engineering at Era Software, discusses how organizations today use real-time data streams to keep up with evolving business requirements. Setting up data pipelines is easy. Handling the errors at each stage of the pipeline and not losing data is hard.<\/p>\n","protected":false},"author":10513,"featured_media":24971,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"footnotes":""},"categories":[65,115,87,180,56,1],"tags":[594,1067,96],"acf":[],"yoast_head":"\nLooking Ahead | Observability Data Management Modernization - insideBIGDATA<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/insidebigdata.com\/2022\/05\/13\/looking-ahead-observability-data-management-modernization\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Looking Ahead | Observability Data Management Modernization - insideBIGDATA\" \/>\n<meta property=\"og:description\" content=\"In this contributed article, Karen Pieper, VP of engineering at Era Software, discusses how organizations today use real-time data streams to keep up with evolving business requirements. Setting up data pipelines is easy. Handling the errors at each stage of the pipeline and not losing data is hard.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/insidebigdata.com\/2022\/05\/13\/looking-ahead-observability-data-management-modernization\/\" \/>\n<meta property=\"og:site_name\" content=\"insideBIGDATA\" \/>\n<meta property=\"article:publisher\" content=\"http:\/\/www.facebook.com\/insidebigdata\" \/>\n<meta property=\"article:published_time\" content=\"2022-05-13T13:00:00+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2022-05-14T17:21:27+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/insidebigdata.com\/wp-content\/uploads\/2020\/09\/data_engineering_shutterstock_669226057.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"300\" \/>\n\t<meta property=\"og:image:height\" content=\"168\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Editorial Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@insideBigData\" \/>\n<meta name=\"twitter:site\" content=\"@insideBigData\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Editorial Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/insidebigdata.com\/2022\/05\/13\/looking-ahead-observability-data-management-modernization\/\",\"url\":\"https:\/\/insidebigdata.com\/2022\/05\/13\/looking-ahead-observability-data-management-modernization\/\",\"name\":\"Looking Ahead | Observability Data Management Modernization - insideBIGDATA\",\"isPartOf\":{\"@id\":\"https:\/\/insidebigdata.com\/#website\"},\"datePublished\":\"2022-05-13T13:00:00+00:00\",\"dateModified\":\"2022-05-14T17:21:27+00:00\",\"author\":{\"@id\":\"https:\/\/insidebigdata.com\/#\/schema\/person\/2949e412c144601cdbcc803bd234e1b9\"},\"breadcrumb\":{\"@id\":\"https:\/\/insidebigdata.com\/2022\/05\/13\/looking-ahead-observability-data-management-modernization\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/insidebigdata.com\/2022\/05\/13\/looking-ahead-observability-data-management-modernization\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/insidebigdata.com\/2022\/05\/13\/looking-ahead-observability-data-management-modernization\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/insidebigdata.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Looking Ahead | Observability Data Management Modernization\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/insidebigdata.com\/#website\",\"url\":\"https:\/\/insidebigdata.com\/\",\"name\":\"insideBIGDATA\",\"description\":\"Your Source for AI, Data Science, Deep Learning & Machine Learning Strategies\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/insidebigdata.com\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/insidebigdata.com\/#\/schema\/person\/2949e412c144601cdbcc803bd234e1b9\",\"name\":\"Editorial Team\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/insidebigdata.com\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/e137ce7ea40e38bd4d25bb7860cfe3e4?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/e137ce7ea40e38bd4d25bb7860cfe3e4?s=96&d=mm&r=g\",\"caption\":\"Editorial Team\"},\"sameAs\":[\"http:\/\/www.insidebigdata.com\"],\"url\":\"https:\/\/insidebigdata.com\/author\/editorial\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Looking Ahead | Observability Data Management Modernization - insideBIGDATA","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/insidebigdata.com\/2022\/05\/13\/looking-ahead-observability-data-management-modernization\/","og_locale":"en_US","og_type":"article","og_title":"Looking Ahead | Observability Data Management Modernization - insideBIGDATA","og_description":"In this contributed article, Karen Pieper, VP of engineering at Era Software, discusses how organizations today use real-time data streams to keep up with evolving business requirements. Setting up data pipelines is easy. Handling the errors at each stage of the pipeline and not losing data is hard.","og_url":"https:\/\/insidebigdata.com\/2022\/05\/13\/looking-ahead-observability-data-management-modernization\/","og_site_name":"insideBIGDATA","article_publisher":"http:\/\/www.facebook.com\/insidebigdata","article_published_time":"2022-05-13T13:00:00+00:00","article_modified_time":"2022-05-14T17:21:27+00:00","og_image":[{"width":300,"height":168,"url":"https:\/\/insidebigdata.com\/wp-content\/uploads\/2020\/09\/data_engineering_shutterstock_669226057.jpg","type":"image\/jpeg"}],"author":"Editorial Team","twitter_card":"summary_large_image","twitter_creator":"@insideBigData","twitter_site":"@insideBigData","twitter_misc":{"Written by":"Editorial Team","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/insidebigdata.com\/2022\/05\/13\/looking-ahead-observability-data-management-modernization\/","url":"https:\/\/insidebigdata.com\/2022\/05\/13\/looking-ahead-observability-data-management-modernization\/","name":"Looking Ahead | Observability Data Management Modernization - insideBIGDATA","isPartOf":{"@id":"https:\/\/insidebigdata.com\/#website"},"datePublished":"2022-05-13T13:00:00+00:00","dateModified":"2022-05-14T17:21:27+00:00","author":{"@id":"https:\/\/insidebigdata.com\/#\/schema\/person\/2949e412c144601cdbcc803bd234e1b9"},"breadcrumb":{"@id":"https:\/\/insidebigdata.com\/2022\/05\/13\/looking-ahead-observability-data-management-modernization\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/insidebigdata.com\/2022\/05\/13\/looking-ahead-observability-data-management-modernization\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/insidebigdata.com\/2022\/05\/13\/looking-ahead-observability-data-management-modernization\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/insidebigdata.com\/"},{"@type":"ListItem","position":2,"name":"Looking Ahead | Observability Data Management Modernization"}]},{"@type":"WebSite","@id":"https:\/\/insidebigdata.com\/#website","url":"https:\/\/insidebigdata.com\/","name":"insideBIGDATA","description":"Your Source for AI, Data Science, Deep Learning & Machine Learning Strategies","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/insidebigdata.com\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/insidebigdata.com\/#\/schema\/person\/2949e412c144601cdbcc803bd234e1b9","name":"Editorial Team","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/insidebigdata.com\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/e137ce7ea40e38bd4d25bb7860cfe3e4?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/e137ce7ea40e38bd4d25bb7860cfe3e4?s=96&d=mm&r=g","caption":"Editorial Team"},"sameAs":["http:\/\/www.insidebigdata.com"],"url":"https:\/\/insidebigdata.com\/author\/editorial\/"}]}},"jetpack_featured_media_url":"https:\/\/insidebigdata.com\/wp-content\/uploads\/2020\/09\/data_engineering_shutterstock_669226057.jpg","jetpack_shortlink":"https:\/\/wp.me\/p9eA3j-7D6","jetpack-related-posts":[{"id":32502,"url":"https:\/\/insidebigdata.com\/2023\/05\/30\/modern-observability-when-small-data-beats-big-data\/","url_meta":{"origin":29332,"position":0},"title":"Modern Observability:\u00a0 When \u201cSmall\u201d Data Beats Big Data","date":"May 30, 2023","format":false,"excerpt":"In this contributed article, Ozan\u00a0Unlu, CEO and Founder of\u00a0Edge Delta, explores how a cloud-first world demands that observability be approached in a different way, one that favors \u201csmall data\u201d over \u201cBig Data.\u201d In some cases, Ozan believes, a central repository is no longer even needed.","rel":"","context":"In "Big Data"","img":{"alt_text":"","src":"https:\/\/i0.wp.com\/insidebigdata.com\/wp-content\/uploads\/2022\/09\/Observability_shutterstock_152448146.jpg?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":33187,"url":"https:\/\/insidebigdata.com\/2023\/08\/22\/data-observability-essential-for-your-modern-data-stack\/","url_meta":{"origin":29332,"position":1},"title":"Data Observability, Essential for your Modern Data Stack","date":"August 22, 2023","format":false,"excerpt":"In this contributed article, Mayank Mehra, head of product management at Modak, shares the importance of incorporating effective data observability practices to equip data and analytics leaders with essential insights into the health of their data stacks. Mayank also explains why this is becoming increasingly paramount, given the current trend\u2026","rel":"","context":"In "Big Data"","img":{"alt_text":"","src":"https:\/\/i0.wp.com\/insidebigdata.com\/wp-content\/uploads\/2023\/06\/Industry_Perspectives_shutterstock_1127578655_special.jpg?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":31910,"url":"https:\/\/insidebigdata.com\/2023\/03\/28\/acceldata-and-its-data-observability-platform-solving-big-data-management-challenges\/","url_meta":{"origin":29332,"position":2},"title":"Acceldata and its Data Observability Platform – Solving Big Data Management Challenges","date":"March 28, 2023","format":false,"excerpt":"In this video interview with Ashwin Rajeeva, co-founder and CTO of Acceldata, we talk about the company\u2019s data observability platform \u2013 what \"data observability\" is all about and why it\u2019s critically important in big data analytics and machine learning development environments.","rel":"","context":"In "Analytics"","img":{"alt_text":"","src":"https:\/\/i0.wp.com\/insidebigdata.com\/wp-content\/uploads\/2023\/03\/logo-acceldata-1100x825-1.png?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":30303,"url":"https:\/\/insidebigdata.com\/2022\/09\/13\/how-to-optimize-the-modern-data-stack-with-enterprise-data-observability\/","url_meta":{"origin":29332,"position":3},"title":"How to Optimize the Modern Data Stack with Enterprise Data Observability","date":"September 13, 2022","format":false,"excerpt":"In this sponsored post, our friends over at Acceldata examine how in their attempt to overcome various challenges and optimize for data success, organizations across all stages of the data journey are turning to data observability where they can get a continuous, comprehensive, and multidimensional view into all enterprise data\u2026","rel":"","context":"In "Big Data"","img":{"alt_text":"","src":"https:\/\/i0.wp.com\/insidebigdata.com\/wp-content\/uploads\/2022\/07\/Acceldata_logo.png?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":31508,"url":"https:\/\/insidebigdata.com\/2023\/02\/01\/the-real-value-of-data-observability\/","url_meta":{"origin":29332,"position":4},"title":"The Real Value of Data Observability","date":"February 1, 2023","format":false,"excerpt":"In this special guest feature, Andy Petrella, CPO and founder of Kensu, points out that as application observability became a central element for DevOps teams, data observability is set to follow the same path and help data teams to lower maintenance costs, scale up value creation from data, and maintain\u2026","rel":"","context":"In "Analytics"","img":{"alt_text":"","src":"https:\/\/i0.wp.com\/insidebigdata.com\/wp-content\/uploads\/2023\/01\/Andy-Petrella.png?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":32568,"url":"https:\/\/insidebigdata.com\/2023\/06\/07\/busting-data-observability-myths\/","url_meta":{"origin":29332,"position":5},"title":"Busting Data Observability Myths","date":"June 7, 2023","format":false,"excerpt":"In this sponsored article, Rohit Choudhary, co-founder and CEO of Acceldata, breaks down four common myths and misconceptions around observability. In today\u2019s economic climate, many companies are tightening their belts. They need solutions that help them run their business efficiently, smoothly, and reliably in order to maximize impact and keep\u2026","rel":"","context":"In "Big Data"","img":{"alt_text":"","src":"https:\/\/i0.wp.com\/insidebigdata.com\/wp-content\/uploads\/2022\/09\/Observability_shutterstock_152448146.jpg?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]}],"_links":{"self":[{"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/posts\/29332"}],"collection":[{"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/users\/10513"}],"replies":[{"embeddable":true,"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/comments?post=29332"}],"version-history":[{"count":0,"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/posts\/29332\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/media\/24971"}],"wp:attachment":[{"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/media?parent=29332"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/categories?post=29332"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/tags?post=29332"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}