{"id":32502,"date":"2023-05-30T12:02:00","date_gmt":"2023-05-30T19:02:00","guid":{"rendered":"https:\/\/insidebigdata.com\/?p=32502"},"modified":"2023-05-30T12:14:36","modified_gmt":"2023-05-30T19:14:36","slug":"modern-observability-when-small-data-beats-big-data","status":"publish","type":"post","link":"https:\/\/insidebigdata.com\/2023\/05\/30\/modern-observability-when-small-data-beats-big-data\/","title":{"rendered":"Modern Observability:\u00a0 When \u201cSmall\u201d Data Beats Big Data"},"content":{"rendered":"\n

If you take a look at the history of big data, one thing becomes clear – our ability to collect data has always been greater than our capacity to make meaning of it.  We\u2019re seeing this happen with observability, or the task of assessing and measuring a system\u2019s current state of health based on the external data it generates. Today\u2019s IT teams with observability practices in place are literally inundated with so much data that they can no longer harness and leverage it effectively. What was intended to be a blessing, becomes something of a curse: observability is ultimately meant to help these teams find and fix the root cause of system performance (speed, availability) issues faster, but ironically, the length of downtime<\/a> associated with publicly reported outages is actually growing at a disturbing rate. <\/p>\n\n\n\n

With the rise of the cloud, hybrid infrastructures and microservices, apps and systems are only going to be generating even more data.  For many, big data is simply getting too big, and this is going to force some organizations to modify their observability approaches significantly – reverting from big data to small data.  What does this mean?<\/p>\n\n\n\n

No More \u201cCentralize and Analyze\u201d<\/em> – Observability architectures have traditionally been built using a \u2018centralize then analyze\u2019 approach, meaning data is centralized in a monitoring platform before users can query or analyze it. The thinking behind this approach is that data becomes contextually richer, the more you have and the more you can correlate, in one central location. Building your architecture in this manner may have worked well in a previous era when data volumes were comparatively smaller. But given the tsunami of data now being generated – the vast majority of which is never used – organizations can no longer afford to aggregate their data in these expensive, \u201chot\u201d storage tiers. Rather, data needs to continue to be analyzed simultaneously and correlated, but in smaller volumes in different places, where it originates.<\/p>\n\n\n\n

Analyze data at the source<\/em> – To keep the storage costs associated with a central repository down, many organizations have resorted to indiscriminately discarding data sets. While it\u2019s true that the vast majority of data is never used, the reality is anomalies and problems can crop up anytime, anywhere – so if you\u2019re randomly omitting data, you\u2019re leaving yourself with significant blind spots. By analyzing data in smaller chunks, at the source (versus a central repository), you can effectively survey and have an eye on all your data – giving tremendous peace of mind – while then relegating lower-priority data to a lower cost storage tier and saving significantly on expenses.<\/p>\n\n\n\n

Ease the pressure on the pipes and downstream systems<\/em> – Another challenge of the \u201ccentralize and analyze\u201d approach is it can lead to clogged data pipelines and overstuffed central repositories, which slow down significantly and can take much longer to render returns on queries. So another benefit of analyzing data in smaller increments, at the source is that organizations become much more nimble in conducting real-time data analytics – helping identify growing hotspots and their root causes faster, which is critical to reducing MTTR. Some organizations find they don\u2019t even need a central repository at all. But for those who wish to keep one, high-volume, noisy datasets can be converted into lightweight KPIs that are baselined over time, making it much easier to tell when something is abnormal or anomalous \u2013 a good sign that you want to index that data. In this way, organizations can \u201cslim down\u201d their central repositories and maintain some control over what\u2019s getting routed there.<\/p>\n\n\n\n

Make Your Data Accessible – <\/em> The reality is there are going to be times when access to all of this data is needed, and it should be accessible – whether in a streamlined central repository, or in cold storage. Developer team members should have fast, easy access to all their smaller-sized datasets regardless of the storage tier they\u2019re in, not having to ask operations team members who often serve as the gatekeepers in the central repository-based, big data-\u201desque\u201d model. <\/p>\n\n\n\n

Conclusion<\/strong><\/p>\n\n\n\n

When it comes to managing increasingly complex and extensive IT infrastructures, big data remains, justifiably, a major focus of research and interest. But small data is still with us and sometimes small data will beat big data, enabling more agility in reaching the right conclusions faster, more reliably and at lower cost. Modern observability initiatives are a prime example, as surging data volumes are driving many organizations close to an inflection point.<\/p>\n\n\n\n

About the Author<\/strong><\/p>\n\n\n

\n
\"\"<\/figure><\/div>\n\n\n

Ozan Unlu is the CEO and Founder of Edge Delta<\/a>, an edge observability platform. Previously he served as a Senior Solutions Architect at Sumo Logic; a Software Development Lead and Program Manager at Microsoft; and a Data Engineer at Boeing. Ozan holds a BS in nanotechnology from the University of Washington. <\/em><\/p>\n\n\n\n

Sign up for the free insideBIGDATA newsletter<\/a>.<\/em><\/p>\n\n\n\n

Join us on Twitter: https:\/\/twitter.com\/InsideBigData1<\/a><\/em><\/p>\n\n\n\n

Join us on LinkedIn: https:\/\/www.linkedin.com\/company\/insidebigdata\/<\/a><\/em><\/p>\n\n\n\n

Join us on Facebook: https:\/\/www.facebook.com\/insideBIGDATANOW<\/a><\/em><\/p>\n","protected":false},"excerpt":{"rendered":"

In this contributed article, Ozan\u00a0Unlu, CEO and Founder of\u00a0Edge Delta, explores how a cloud-first world demands that observability be approached in a different way, one that favors \u201csmall data\u201d over \u201cBig Data.\u201d In some cases, Ozan believes, a central repository is no longer even needed.<\/p>\n","protected":false},"author":10513,"featured_media":30424,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"footnotes":""},"categories":[115,63,64,180,61,268,56,97,1],"tags":[280,1067,95],"acf":[],"yoast_head":"\nModern Observability:\u00a0 When \u201cSmall\u201d Data Beats Big Data - insideBIGDATA<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/insidebigdata.com\/2023\/05\/30\/modern-observability-when-small-data-beats-big-data\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Modern Observability:\u00a0 When \u201cSmall\u201d Data Beats Big Data - insideBIGDATA\" \/>\n<meta property=\"og:description\" content=\"In this contributed article, Ozan\u00a0Unlu, CEO and Founder of\u00a0Edge Delta, explores how a cloud-first world demands that observability be approached in a different way, one that favors \u201csmall data\u201d over \u201cBig Data.\u201d In some cases, Ozan believes, a central repository is no longer even needed.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/insidebigdata.com\/2023\/05\/30\/modern-observability-when-small-data-beats-big-data\/\" \/>\n<meta property=\"og:site_name\" content=\"insideBIGDATA\" \/>\n<meta property=\"article:publisher\" content=\"http:\/\/www.facebook.com\/insidebigdata\" \/>\n<meta property=\"article:published_time\" content=\"2023-05-30T19:02:00+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2023-05-30T19:14:36+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/insidebigdata.com\/wp-content\/uploads\/2022\/09\/Observability_shutterstock_152448146.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"300\" \/>\n\t<meta property=\"og:image:height\" content=\"214\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Editorial Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@insideBigData\" \/>\n<meta name=\"twitter:site\" content=\"@insideBigData\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Editorial Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/insidebigdata.com\/2023\/05\/30\/modern-observability-when-small-data-beats-big-data\/\",\"url\":\"https:\/\/insidebigdata.com\/2023\/05\/30\/modern-observability-when-small-data-beats-big-data\/\",\"name\":\"Modern Observability:\u00a0 When \u201cSmall\u201d Data Beats Big Data - insideBIGDATA\",\"isPartOf\":{\"@id\":\"https:\/\/insidebigdata.com\/#website\"},\"datePublished\":\"2023-05-30T19:02:00+00:00\",\"dateModified\":\"2023-05-30T19:14:36+00:00\",\"author\":{\"@id\":\"https:\/\/insidebigdata.com\/#\/schema\/person\/2949e412c144601cdbcc803bd234e1b9\"},\"breadcrumb\":{\"@id\":\"https:\/\/insidebigdata.com\/2023\/05\/30\/modern-observability-when-small-data-beats-big-data\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/insidebigdata.com\/2023\/05\/30\/modern-observability-when-small-data-beats-big-data\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/insidebigdata.com\/2023\/05\/30\/modern-observability-when-small-data-beats-big-data\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/insidebigdata.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Modern Observability:\u00a0 When \u201cSmall\u201d Data Beats Big Data\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/insidebigdata.com\/#website\",\"url\":\"https:\/\/insidebigdata.com\/\",\"name\":\"insideBIGDATA\",\"description\":\"Your Source for AI, Data Science, Deep Learning & Machine Learning Strategies\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/insidebigdata.com\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/insidebigdata.com\/#\/schema\/person\/2949e412c144601cdbcc803bd234e1b9\",\"name\":\"Editorial Team\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/insidebigdata.com\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/e137ce7ea40e38bd4d25bb7860cfe3e4?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/e137ce7ea40e38bd4d25bb7860cfe3e4?s=96&d=mm&r=g\",\"caption\":\"Editorial Team\"},\"sameAs\":[\"http:\/\/www.insidebigdata.com\"],\"url\":\"https:\/\/insidebigdata.com\/author\/editorial\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Modern Observability:\u00a0 When \u201cSmall\u201d Data Beats Big Data - insideBIGDATA","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/insidebigdata.com\/2023\/05\/30\/modern-observability-when-small-data-beats-big-data\/","og_locale":"en_US","og_type":"article","og_title":"Modern Observability:\u00a0 When \u201cSmall\u201d Data Beats Big Data - insideBIGDATA","og_description":"In this contributed article, Ozan\u00a0Unlu, CEO and Founder of\u00a0Edge Delta, explores how a cloud-first world demands that observability be approached in a different way, one that favors \u201csmall data\u201d over \u201cBig Data.\u201d In some cases, Ozan believes, a central repository is no longer even needed.","og_url":"https:\/\/insidebigdata.com\/2023\/05\/30\/modern-observability-when-small-data-beats-big-data\/","og_site_name":"insideBIGDATA","article_publisher":"http:\/\/www.facebook.com\/insidebigdata","article_published_time":"2023-05-30T19:02:00+00:00","article_modified_time":"2023-05-30T19:14:36+00:00","og_image":[{"width":300,"height":214,"url":"https:\/\/insidebigdata.com\/wp-content\/uploads\/2022\/09\/Observability_shutterstock_152448146.jpg","type":"image\/jpeg"}],"author":"Editorial Team","twitter_card":"summary_large_image","twitter_creator":"@insideBigData","twitter_site":"@insideBigData","twitter_misc":{"Written by":"Editorial Team","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/insidebigdata.com\/2023\/05\/30\/modern-observability-when-small-data-beats-big-data\/","url":"https:\/\/insidebigdata.com\/2023\/05\/30\/modern-observability-when-small-data-beats-big-data\/","name":"Modern Observability:\u00a0 When \u201cSmall\u201d Data Beats Big Data - insideBIGDATA","isPartOf":{"@id":"https:\/\/insidebigdata.com\/#website"},"datePublished":"2023-05-30T19:02:00+00:00","dateModified":"2023-05-30T19:14:36+00:00","author":{"@id":"https:\/\/insidebigdata.com\/#\/schema\/person\/2949e412c144601cdbcc803bd234e1b9"},"breadcrumb":{"@id":"https:\/\/insidebigdata.com\/2023\/05\/30\/modern-observability-when-small-data-beats-big-data\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/insidebigdata.com\/2023\/05\/30\/modern-observability-when-small-data-beats-big-data\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/insidebigdata.com\/2023\/05\/30\/modern-observability-when-small-data-beats-big-data\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/insidebigdata.com\/"},{"@type":"ListItem","position":2,"name":"Modern Observability:\u00a0 When \u201cSmall\u201d Data Beats Big Data"}]},{"@type":"WebSite","@id":"https:\/\/insidebigdata.com\/#website","url":"https:\/\/insidebigdata.com\/","name":"insideBIGDATA","description":"Your Source for AI, Data Science, Deep Learning & Machine Learning Strategies","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/insidebigdata.com\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/insidebigdata.com\/#\/schema\/person\/2949e412c144601cdbcc803bd234e1b9","name":"Editorial Team","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/insidebigdata.com\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/e137ce7ea40e38bd4d25bb7860cfe3e4?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/e137ce7ea40e38bd4d25bb7860cfe3e4?s=96&d=mm&r=g","caption":"Editorial Team"},"sameAs":["http:\/\/www.insidebigdata.com"],"url":"https:\/\/insidebigdata.com\/author\/editorial\/"}]}},"jetpack_featured_media_url":"https:\/\/insidebigdata.com\/wp-content\/uploads\/2022\/09\/Observability_shutterstock_152448146.jpg","jetpack_shortlink":"https:\/\/wp.me\/p9eA3j-8se","jetpack-related-posts":[{"id":31508,"url":"https:\/\/insidebigdata.com\/2023\/02\/01\/the-real-value-of-data-observability\/","url_meta":{"origin":32502,"position":0},"title":"The Real Value of Data Observability","date":"February 1, 2023","format":false,"excerpt":"In this special guest feature, Andy Petrella, CPO and founder of Kensu, points out that as application observability became a central element for DevOps teams, data observability is set to follow the same path and help data teams to lower maintenance costs, scale up value creation from data, and maintain\u2026","rel":"","context":"In "Analytics"","img":{"alt_text":"","src":"https:\/\/i0.wp.com\/insidebigdata.com\/wp-content\/uploads\/2023\/01\/Andy-Petrella.png?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":31210,"url":"https:\/\/insidebigdata.com\/2022\/12\/23\/data-leaders-looking-to-data-observability-to-overcome-data-quality-cost-and-pipeline-concerns\/","url_meta":{"origin":32502,"position":1},"title":"Data Leaders Looking to Data Observability to Overcome Data Quality, Cost and Pipeline Concerns","date":"December 23, 2022","format":false,"excerpt":"Acceldata, a market leader in data observability, conducted a survey of 200 data executives and found that they have major concerns including the lack of data pipeline visibility. Below are additional highlights from the results.","rel":"","context":"In "Big Data"","img":{"alt_text":"","src":"https:\/\/i0.wp.com\/insidebigdata.com\/wp-content\/uploads\/2022\/09\/Observability_shutterstock_152448146.jpg?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":32010,"url":"https:\/\/insidebigdata.com\/2023\/04\/04\/slidecast-ashwin-rajeev-co-founder-cto-of-acceldata-discusses-data-observability\/","url_meta":{"origin":32502,"position":2},"title":"Slidecast: Ashwin Rajeeva, Co-founder & CTO of Acceldata Discusses Data Observability","date":"April 4, 2023","format":false,"excerpt":"In this slidecast presentation, Ashwin Rajeev from Acceldata\u00a0describes the company\u2019s data observability solutions. Acceldata solutions allow you to gain comprehensive insights into your data stack to improve data and pipeline reliability, platform performance, and spend efficiency.","rel":"","context":"In "Big Data"","img":{"alt_text":"","src":"https:\/\/i0.wp.com\/insidebigdata.com\/wp-content\/uploads\/2023\/03\/Ashwin_Rajeev.jpg?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":31910,"url":"https:\/\/insidebigdata.com\/2023\/03\/28\/acceldata-and-its-data-observability-platform-solving-big-data-management-challenges\/","url_meta":{"origin":32502,"position":3},"title":"Acceldata and its Data Observability Platform – Solving Big Data Management Challenges","date":"March 28, 2023","format":false,"excerpt":"In this video interview with Ashwin Rajeeva, co-founder and CTO of Acceldata, we talk about the company\u2019s data observability platform \u2013 what \"data observability\" is all about and why it\u2019s critically important in big data analytics and machine learning development environments.","rel":"","context":"In "Analytics"","img":{"alt_text":"","src":"https:\/\/i0.wp.com\/insidebigdata.com\/wp-content\/uploads\/2023\/03\/logo-acceldata-1100x825-1.png?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":30422,"url":"https:\/\/insidebigdata.com\/2022\/09\/24\/video-highlights-why-does-observability-matter\/","url_meta":{"origin":32502,"position":4},"title":"Video Highlights: Why Does Observability Matter?","date":"September 24, 2022","format":false,"excerpt":"Why does observability matter? Isn\u2019t observability just a fancier word for monitoring? Observability has become a buzz word in the big data space. It\u2019s thrown around so often, it can be easy to forget what it even really means. In this video presentation, our friends over at Pepperdata provide some\u2026","rel":"","context":"In "Big Data"","img":{"alt_text":"","src":"https:\/\/i0.wp.com\/insidebigdata.com\/wp-content\/uploads\/2022\/09\/Observability_shutterstock_152448146.jpg?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":32568,"url":"https:\/\/insidebigdata.com\/2023\/06\/07\/busting-data-observability-myths\/","url_meta":{"origin":32502,"position":5},"title":"Busting Data Observability Myths","date":"June 7, 2023","format":false,"excerpt":"In this sponsored article, Rohit Choudhary, co-founder and CEO of Acceldata, breaks down four common myths and misconceptions around observability. In today\u2019s economic climate, many companies are tightening their belts. They need solutions that help them run their business efficiently, smoothly, and reliably in order to maximize impact and keep\u2026","rel":"","context":"In "Big Data"","img":{"alt_text":"","src":"https:\/\/i0.wp.com\/insidebigdata.com\/wp-content\/uploads\/2022\/09\/Observability_shutterstock_152448146.jpg?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]}],"_links":{"self":[{"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/posts\/32502"}],"collection":[{"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/users\/10513"}],"replies":[{"embeddable":true,"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/comments?post=32502"}],"version-history":[{"count":0,"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/posts\/32502\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/media\/30424"}],"wp:attachment":[{"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/media?parent=32502"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/categories?post=32502"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/tags?post=32502"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}