{"id":21762,"date":"2018-12-30T08:00:43","date_gmt":"2018-12-30T16:00:43","guid":{"rendered":"https:\/\/insidebigdata.com\/?p=21762"},"modified":"2018-12-31T09:35:48","modified_gmt":"2018-12-31T17:35:48","slug":"detecting-anomalies-time-series-data-deciphering-noise-zoning-signals","status":"publish","type":"post","link":"https:\/\/insidebigdata.com\/2018\/12\/30\/detecting-anomalies-time-series-data-deciphering-noise-zoning-signals\/","title":{"rendered":"Detecting Anomalies in Time Series Data: Deciphering the Noise and Zoning in on the Signals"},"content":{"rendered":"

You have the data. How do you make sense of it?<\/strong><\/p>\n

The Internet of Things (IoT) is no longer a fancy marketing term thrown in to close a critical deal \u2013 it\u2019s all around us. In the digital age, everyone has smart, connected machines, which are happily and continuously reeling in their data \u2013 by the truckload. In fact, according to Forbes, the global IoT market is estimated to hit US$457 billion<\/a> by 2020.<\/p>\n

The question is, what do you do with all this data? Granted, every self-respecting, agile and competitive organization in today\u2019s day and age has already realized the importance of gathering their data. They also know that they need to analyze said data to identify abnormalities, or what we call anomalies. Which is great. But how do you go about selecting the anomaly detection technique that works best for you? Let\u2019s first take a step back and run through each of them to get a better understanding.<\/p>\n

It\u2019s not what you do, but how you do it<\/strong><\/p>\n

There are broadly three techniques adopted today for detecting anomalies \u2013 supervised, semi-supervised and unsupervised.<\/p>\n

Supervised anomaly detection:<\/u>This technique hinges on the prior labeling of data as \u201cnormal\u201d or \u201canomalous\u201d. The algorithm is trained using existing current or historical data, and is then deployed to predict outcomes on new data. Though this technique finds application in fraud detection in the banking\/ fintech space, it can only be applied to predict known anomalies such as previously identified fraud\/ misappropriations.<\/p>\n

Semi-supervised anomaly detection:<\/u>This technique is inherently tricky. The algorithm in this case only has a set of \u201cnormal\u201d data points for reference \u2013 any data points that are outside this reference range are classified as anomalous. The downside of this technique is that it has the tendency to flag \u201cfalse positives\u201d, or anomalies that aren\u2019t actually anomalies. Thus, it ends up being counter-productive \u2013 in that you could end up wasting considerable time, effort and resources.<\/p>\n

Unsupervised anomaly detection (also called automated anomaly detection):<\/u> In this technique, which is entirely automated, anomalies can be identified from unlabelled data by assuming a majority of the data points to be normal. Deviating instances that are statistically significant (classified as a \u201c2* standard deviation\u201d or \u201c3* standard deviation\u201d) on either side of the established normal are regarded as anomalies. The more powerful the algorithm, the higher the accuracy of the anomaly detection. This method may be used for detecting anomalies in time series data, and also to predict and flag future anomalies.<\/p>\n

What you need is an algorithm powerful enough to analyze raw data<\/strong><\/p>\n

While each of the above techniques obviously has advantages as well as disadvantages, it\u2019s only unsupervised anomaly detection that is feasible in the case of raw, unlabelled time series data \u2013 which is what you get from just about any online asset in a modern-day digitised company. Anomaly detection in time series data has a variety of applications across industries \u2013 from identifying abnormalities in ECG data to finding glitches in aircraft sensor data.<\/p>\n

What\u2019s more, you normally only know 20% of the anomalies that you can expect. The remaining 80% are new\/ unpredictable. Unsupervised anomaly detection is the only technique that\u2019s capable of identifying these hidden signals or anomalies \u2013 and flagging them early enough to fix them before they occur.<\/p>\n

\"\"<\/p>\n

Let\u2019s take a closer look at how this happens.<\/p>\n

Anomaly detection for time series data with deep learning \u2013 identifying the \u201cunknown unknowns\u201d<\/strong><\/p>\n

One of the most effective ways of detecting anomalies in time series data is via deep learning. This technique involves the following steps:<\/p>\n

    \n
  1. Apply deep learning architecture to time series data:<\/strong> First, recurrent neural networks are applied to a series of input and output sets to establish the normal and accordingly predict the time series. This process is repeated until the predictions achieve a high level of accuracy. However, the models need to be updated regularly to accommodate changing trends and ensure accuracy and relevance. Long short-term memory (LSTM) neural networks are great at remembering seasonal and other trends.<\/li>\n
  2. Predict the next values from the latest available explanatory variables: <\/strong>Once the model has been trained, it can predict the next series based on real-time explanatory variables.<\/li>\n
  3. Predict the upper and lower limits based on the standard deviation calculated for the latest predicted values:<\/strong> Once the values are predicted, the algorithm creates upper and lower limits at a specified confidence level. For instance, a 95% confidence level means that limits need to be at a \u201c1.96 * standard deviation with respect to the mean on both sides\u201d for a normal distribution.<\/li>\n
  4. Identify and score anomalies: <\/strong>Whenever an actual perceived value falls beyond the predicted normal range, anomalies are marked \u2013 and scored based on their magnitude of deviation. A simple scoring methodology could be:<\/li>\n<\/ol>\n

    \"\"<\/p>\n

    Anomaly scores help users filter out anomalies that are less than a set threshold value (say 40), and also to prioritise them so that they can focus on more serious anomalies first and then move on to less serious ones. In case of critical metrics that involve huge expenses, the threshold value can be set to zero so that the tiniest of anomalies with the lowest of scores can be scrutinised for relevant action.<\/p>\n

    Anomaly detection in industrial data is by no means a simple process given the scale at which it needs to happen, and also the highly dynamic nature of business in today\u2019s world. However, it\u2019s still imperative to get it right, as no digital business can hope to stay relevant and competitive in an increasingly tough economy without the power of meaningful data analytics to back its growth.<\/p>\n

    About the Author<\/strong><\/p>\n

    \"\"Pratap Dangeti is the Principal Data Scientist at Subex<\/a>. He has close to 9 years of experience in the field of analytics across the domains like banking, IT, credit & risk, manufacturing, hi-tech, utilities and telecom. His technical expertise includes Statistical Modelling, Machine Learning, Big Data, Deep Learning, NLP, and artificial intelligence. As a hobbyist, he has written 2 books in the field of Machine Learning & NLP.<\/em><\/p>\n

     <\/p>\n

    Sign up for the free insideBIGDATA\u00a0newsletter<\/a>.<\/em><\/p>\n

     <\/p>\n

     <\/p>\n","protected":false},"excerpt":{"rendered":"

    In this contributed article, Pratap Dangeti, Principal Data Scientist at Subex, discusses how anomaly detection in industrial data is by no means a simple process given the scale at which it needs to happen, and also the highly dynamic nature of business in today\u2019s world. However, it\u2019s still imperative to get it right, as no digital business can hope to stay relevant and competitive in an increasingly tough economy without the power of meaningful data analytics to back its growth.<\/p>\n","protected":false},"author":10513,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"footnotes":""},"categories":[115,182,87,180,67,56,97,1],"tags":[718,96],"acf":[],"yoast_head":"\nDetecting Anomalies in Time Series Data: Deciphering the Noise and Zoning in on the Signals - insideBIGDATA<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/insidebigdata.com\/2018\/12\/30\/detecting-anomalies-time-series-data-deciphering-noise-zoning-signals\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Detecting Anomalies in Time Series Data: Deciphering the Noise and Zoning in on the Signals - insideBIGDATA\" \/>\n<meta property=\"og:description\" content=\"In this contributed article, Pratap Dangeti, Principal Data Scientist at Subex, discusses how anomaly detection in industrial data is by no means a simple process given the scale at which it needs to happen, and also the highly dynamic nature of business in today\u2019s world. However, it\u2019s still imperative to get it right, as no digital business can hope to stay relevant and competitive in an increasingly tough economy without the power of meaningful data analytics to back its growth.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/insidebigdata.com\/2018\/12\/30\/detecting-anomalies-time-series-data-deciphering-noise-zoning-signals\/\" \/>\n<meta property=\"og:site_name\" content=\"insideBIGDATA\" \/>\n<meta property=\"article:publisher\" content=\"http:\/\/www.facebook.com\/insidebigdata\" \/>\n<meta property=\"article:published_time\" content=\"2018-12-30T16:00:43+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2018-12-31T17:35:48+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/insidebigdata.com\/wp-content\/uploads\/2018\/12\/Pratap-Dangeti-PIC1.png\" \/>\n<meta name=\"author\" content=\"Editorial Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@insideBigData\" \/>\n<meta name=\"twitter:site\" content=\"@insideBigData\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Editorial Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"5 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/insidebigdata.com\/2018\/12\/30\/detecting-anomalies-time-series-data-deciphering-noise-zoning-signals\/\",\"url\":\"https:\/\/insidebigdata.com\/2018\/12\/30\/detecting-anomalies-time-series-data-deciphering-noise-zoning-signals\/\",\"name\":\"Detecting Anomalies in Time Series Data: Deciphering the Noise and Zoning in on the Signals - insideBIGDATA\",\"isPartOf\":{\"@id\":\"https:\/\/insidebigdata.com\/#website\"},\"datePublished\":\"2018-12-30T16:00:43+00:00\",\"dateModified\":\"2018-12-31T17:35:48+00:00\",\"author\":{\"@id\":\"https:\/\/insidebigdata.com\/#\/schema\/person\/2949e412c144601cdbcc803bd234e1b9\"},\"breadcrumb\":{\"@id\":\"https:\/\/insidebigdata.com\/2018\/12\/30\/detecting-anomalies-time-series-data-deciphering-noise-zoning-signals\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/insidebigdata.com\/2018\/12\/30\/detecting-anomalies-time-series-data-deciphering-noise-zoning-signals\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/insidebigdata.com\/2018\/12\/30\/detecting-anomalies-time-series-data-deciphering-noise-zoning-signals\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/insidebigdata.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Detecting Anomalies in Time Series Data: Deciphering the Noise and Zoning in on the Signals\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/insidebigdata.com\/#website\",\"url\":\"https:\/\/insidebigdata.com\/\",\"name\":\"insideBIGDATA\",\"description\":\"Your Source for AI, Data Science, Deep Learning & Machine Learning Strategies\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/insidebigdata.com\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/insidebigdata.com\/#\/schema\/person\/2949e412c144601cdbcc803bd234e1b9\",\"name\":\"Editorial Team\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/insidebigdata.com\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/e137ce7ea40e38bd4d25bb7860cfe3e4?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/e137ce7ea40e38bd4d25bb7860cfe3e4?s=96&d=mm&r=g\",\"caption\":\"Editorial Team\"},\"sameAs\":[\"http:\/\/www.insidebigdata.com\"],\"url\":\"https:\/\/insidebigdata.com\/author\/editorial\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Detecting Anomalies in Time Series Data: Deciphering the Noise and Zoning in on the Signals - insideBIGDATA","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/insidebigdata.com\/2018\/12\/30\/detecting-anomalies-time-series-data-deciphering-noise-zoning-signals\/","og_locale":"en_US","og_type":"article","og_title":"Detecting Anomalies in Time Series Data: Deciphering the Noise and Zoning in on the Signals - insideBIGDATA","og_description":"In this contributed article, Pratap Dangeti, Principal Data Scientist at Subex, discusses how anomaly detection in industrial data is by no means a simple process given the scale at which it needs to happen, and also the highly dynamic nature of business in today\u2019s world. However, it\u2019s still imperative to get it right, as no digital business can hope to stay relevant and competitive in an increasingly tough economy without the power of meaningful data analytics to back its growth.","og_url":"https:\/\/insidebigdata.com\/2018\/12\/30\/detecting-anomalies-time-series-data-deciphering-noise-zoning-signals\/","og_site_name":"insideBIGDATA","article_publisher":"http:\/\/www.facebook.com\/insidebigdata","article_published_time":"2018-12-30T16:00:43+00:00","article_modified_time":"2018-12-31T17:35:48+00:00","og_image":[{"url":"https:\/\/insidebigdata.com\/wp-content\/uploads\/2018\/12\/Pratap-Dangeti-PIC1.png"}],"author":"Editorial Team","twitter_card":"summary_large_image","twitter_creator":"@insideBigData","twitter_site":"@insideBigData","twitter_misc":{"Written by":"Editorial Team","Est. reading time":"5 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/insidebigdata.com\/2018\/12\/30\/detecting-anomalies-time-series-data-deciphering-noise-zoning-signals\/","url":"https:\/\/insidebigdata.com\/2018\/12\/30\/detecting-anomalies-time-series-data-deciphering-noise-zoning-signals\/","name":"Detecting Anomalies in Time Series Data: Deciphering the Noise and Zoning in on the Signals - insideBIGDATA","isPartOf":{"@id":"https:\/\/insidebigdata.com\/#website"},"datePublished":"2018-12-30T16:00:43+00:00","dateModified":"2018-12-31T17:35:48+00:00","author":{"@id":"https:\/\/insidebigdata.com\/#\/schema\/person\/2949e412c144601cdbcc803bd234e1b9"},"breadcrumb":{"@id":"https:\/\/insidebigdata.com\/2018\/12\/30\/detecting-anomalies-time-series-data-deciphering-noise-zoning-signals\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/insidebigdata.com\/2018\/12\/30\/detecting-anomalies-time-series-data-deciphering-noise-zoning-signals\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/insidebigdata.com\/2018\/12\/30\/detecting-anomalies-time-series-data-deciphering-noise-zoning-signals\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/insidebigdata.com\/"},{"@type":"ListItem","position":2,"name":"Detecting Anomalies in Time Series Data: Deciphering the Noise and Zoning in on the Signals"}]},{"@type":"WebSite","@id":"https:\/\/insidebigdata.com\/#website","url":"https:\/\/insidebigdata.com\/","name":"insideBIGDATA","description":"Your Source for AI, Data Science, Deep Learning & Machine Learning Strategies","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/insidebigdata.com\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/insidebigdata.com\/#\/schema\/person\/2949e412c144601cdbcc803bd234e1b9","name":"Editorial Team","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/insidebigdata.com\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/e137ce7ea40e38bd4d25bb7860cfe3e4?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/e137ce7ea40e38bd4d25bb7860cfe3e4?s=96&d=mm&r=g","caption":"Editorial Team"},"sameAs":["http:\/\/www.insidebigdata.com"],"url":"https:\/\/insidebigdata.com\/author\/editorial\/"}]}},"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/p9eA3j-5F0","jetpack-related-posts":[{"id":17518,"url":"https:\/\/insidebigdata.com\/2017\/03\/31\/mastering-internet-things\/","url_meta":{"origin":21762,"position":0},"title":"Mastering the Internet of Things","date":"March 31, 2017","format":false,"excerpt":"In this special guest feature, Michael Hiskey, CMO of Semarchy, discusses how in the age of big data and IoT, the well-known, but often misunderstood, area of Master Data Management (MDM) is the essential tool for bringing meaning and insight to troves of raw data.","rel":"","context":"In "Big Data"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":32796,"url":"https:\/\/insidebigdata.com\/2023\/07\/07\/addressing-the-challenges-of-real-time-data-sharing-in-iot\/","url_meta":{"origin":21762,"position":1},"title":"Addressing the Challenges of Real-Time Data Sharing in IoT","date":"July 7, 2023","format":false,"excerpt":"In this contributed article, Jeff Tao, Founder, CEO, and Core Developer of TDengine, discusses how the Internet of Things (IoT) has revolutionized how we live, work and share information. While IoT has made accessing data easier, real-time data sharing - which requires seamless and secure data transfer from connected devices\u2026","rel":"","context":"In "Analytics"","img":{"alt_text":"","src":"https:\/\/i0.wp.com\/insidebigdata.com\/wp-content\/uploads\/2019\/12\/IoT_shutterstock_510828820.jpg?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":13442,"url":"https:\/\/insidebigdata.com\/2015\/07\/27\/building-for-the-internet-of-things-3-things-the-data-industry-must-do\/","url_meta":{"origin":21762,"position":2},"title":"Building for the Internet of Things: 3 Things the Data Industry Must Do","date":"July 27, 2015","format":false,"excerpt":"In this special guest feature, Yaniv Mor of Xplenty talks about the upward trajectory of the Internet of Things (IoT) industry and examines the distinct opportunities as well as pitfalls present in this new space.","rel":"","context":"In "Google News Feed"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":22053,"url":"https:\/\/insidebigdata.com\/2019\/01\/29\/data-monetization-the-further-away-your-data-the-more-distant-your-profits\/","url_meta":{"origin":21762,"position":3},"title":"Data Monetization: The Further Away Your Data, the More Distant Your Profits","date":"January 29, 2019","format":false,"excerpt":"In this contributed article, Steve Todd, a software engineer and inventor for Dell EMC, discusses the steps necessary to avoid crippling data taxation in order to further the journey to the Holy Grail of digital business. He also outlines how the use of open-source and vendor-neutral data management technologies reduce\u2026","rel":"","context":"In "Big Data"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":17596,"url":"https:\/\/insidebigdata.com\/2017\/04\/12\/enterprise-lacks-big-data-strategy-iot-transformation\/","url_meta":{"origin":21762,"position":4},"title":"The Enterprise Lacks a Big Data Strategy for IoT Transformation","date":"April 12, 2017","format":false,"excerpt":"A new research report is now available describing how organizations are approaching IoT transformation. The Enterprise Lacks a Big Data Strategy for IoT Transformation by Verizon and Harvard Business Review Analytic Services, explores how companies are currently using IoT to improve customer experience and increase efficiency, the challenges they\u2019re facing\u2026","rel":"","context":"In "Big Data"","img":{"alt_text":"","src":"https:\/\/i0.wp.com\/insidebigdata.com\/wp-content\/uploads\/2014\/08\/IoT.png?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":20983,"url":"https:\/\/insidebigdata.com\/2018\/08\/26\/terbine-launches-blockchain-enabled-iot-data-exchange-accelerate-information-sharing-monetization\/","url_meta":{"origin":21762,"position":5},"title":"Terbine Launches Blockchain-enabled IoT Data Exchange To Accelerate Information Sharing and Monetization","date":"August 26, 2018","format":false,"excerpt":"Terbine announced the IoT Data Exchange, designed to radically increase the rate of adoption for sharing of machine-generated data between companies, public agencies and academic institutions.","rel":"","context":"In "Google News Feed"","img":{"alt_text":"","src":"https:\/\/i0.wp.com\/insidebigdata.com\/wp-content\/uploads\/2018\/08\/Terbine_data_exchange.png?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]}],"_links":{"self":[{"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/posts\/21762"}],"collection":[{"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/users\/10513"}],"replies":[{"embeddable":true,"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/comments?post=21762"}],"version-history":[{"count":0,"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/posts\/21762\/revisions"}],"wp:attachment":[{"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/media?parent=21762"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/categories?post=21762"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/tags?post=21762"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}