{"id":28644,"date":"2022-03-06T06:00:00","date_gmt":"2022-03-06T14:00:00","guid":{"rendered":"https:\/\/insidebigdata.com\/?p=28644"},"modified":"2022-03-07T09:25:41","modified_gmt":"2022-03-07T17:25:41","slug":"how-to-use-the-mann-kendall-test-to-assess-cloud-costs","status":"publish","type":"post","link":"https:\/\/insidebigdata.com\/2022\/03\/06\/how-to-use-the-mann-kendall-test-to-assess-cloud-costs\/","title":{"rendered":"How to Use the Mann-Kendall Test to Assess Cloud Costs"},"content":{"rendered":"\n

The public cloud now supports a vast variety of workloads as it provides a low-effort option to provision compute resources at scale. But as companies have learned over the past decade, cloud costs can easily spiral out of control without proper governance and continuous monitoring.<\/p>\n\n\n\n

Gaining control over cloud costs remains an issue, in large part because gaining visibility of the cloud\u2019s internal operations is challenging, at best. This leaves organizations to make rough estimates of how much they are actually consuming, until the bill arrives. According to a recent survey<\/a> of IT, finance and operations professionals nearly half of respondents saw cloud costs jump at least 25% in 2020, with 20% reporting increases of 50% or more. At the same time, fewer than 20% said they had the ability to immediately detect spikes in cloud costs, while 25% said it can take days or even weeks to do so.<\/p>\n\n\n\n

Drilling into Data<\/strong><\/p>\n\n\n\n

One particularly thorny issue is the difficulty involved in breaking down cloud usage by service or individual components, which prevents users from determining exactly why costs are rising or developing an effective strategy to reverse the trend. Usually, a cloud bill will contain thousands of services being used every month. Even when viewing overlapping time series data, which in theory should break down costs for each service, it is next to impossible to ascertain which series are trending up or down for anything more than 50 services at once.<\/p>\n\n\n\n

This problem is at the heart of what companies need to know in order to manage their cloud budget, and is only magnified when organizations start spreading workloads over multiple clouds.<\/p>\n\n\n\n

While there are many commercial, off-the-shelf solutions to track cloud costs, a lot of organizations find it helpful to craft their own customized approach. One of the most effective tools to accomplish this is the Mann-Kendall test, which has proven highly effective at separating the signal from the noise in environments generating multiple time series data feeds.<\/p>\n\n\n\n

The Mann-Kendall (MK) test is an amalgam of the work of mathematician H.B Mann and statistician Maurice Kendall. It provides a non-parametric means of identifying trends in a series that, when applied to cloud management data, can be used to produce a trend analysis that is capable of accurately measuring costs and other metrics over time. This provides cloud consumers with deep insight into their usage patterns and how they are affected by various factors. This can then be analyzed more deeply to determine the most effective way to streamline costs without substantially hampering operations, and in some cases, even improving them.<\/p>\n\n\n\n

The Right Approach<\/strong><\/p>\n\n\n\n

Implementing MK testing in a cloud management platform must be done correctly. For instance, the test delivers results that are similar to parametric linear regression analyses, but only if the residuals from a fitted regression are normally distributed. This, in turn, can determine whether the slope of the regression is different from zero.<\/p>\n\n\n\n

In order for this to provide a meaningful analysis of time series data, the MK test must make a number of assumptions. First, without a measurable trend already identified, all measurements over time must be independent and identically distributed. As well, the measurements must represent the true states of the observables at the time they are measured and the methods used to collect samples must be unbiased.<\/p>\n\n\n\n

Note that MK is not designed to account for seasonal (periodic) effects, so it is best to remove this data before the series is fed into the test process. It helps to apply a seasonal and trend decomposition tool that uses Loess regression to break the time series into its constituent components. This provides the ability to analyze inputs according to a range of desired parameters, such as the time series itself, data periodicity, seasonal width and model type to determine if data variations are proportional or do not trend with the time series.<\/p>\n\n\n\n

Ultimately, this is used to determine if trends exist within the time series data and to calculate the slope of the regression line to determine how quickly the data is changing. All of this can be done in a highly granular fashion to give cloud administrators a fine-grained view of which workloads are spiking and how they can be reined in.<\/p>\n\n\n\n

It should be noted that not all spikes in cloud consumption are a net-negative for a company. If workloads are on the rise because sales are up, that\u2019s a good thing. If the rise is not producing a quantifiable benefit to the business model, however, it\u2019s probably time to take action.<\/p>\n\n\n\n

Luckily, by using Mann-Kendall, organizations have a quick, reliable way to get to the truth of the matter.<\/p>\n\n\n\n

About the Author<\/strong><\/p>\n\n\n\n

\"\"<\/figure><\/div>\n\n\n\n

Vadim Solovey is General Manager at DoiT International<\/a><\/em><\/p>\n\n\n\n

Sign up for the free insideBIGDATA newsletter<\/a>.<\/em><\/p>\n\n\n\n

Join us on Twitter: @InsideBigData1 \u2013 https:\/\/twitter.com\/InsideBigData1<\/a><\/em><\/p>\n","protected":false},"excerpt":{"rendered":"

In this contributed article, Vadim Solovey, Chief Technology Officer at DoiT International, indicates that while there are many commercial, off-the-shelf solutions to track cloud costs, a lot of organizations find it helpful to craft their own customized approach. One of the most effective tools to accomplish this is the Mann-Kendall test, which has proven highly effective at separating the signal from the noise in environments generating multiple time series data feeds.<\/p>\n","protected":false},"author":10513,"featured_media":22516,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"footnotes":""},"categories":[115,66,182,87,180,67,56,97,1],"tags":[117,133,134,96],"acf":[],"yoast_head":"\nHow to Use the Mann-Kendall Test to Assess Cloud Costs - insideBIGDATA<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/insidebigdata.com\/2022\/03\/06\/how-to-use-the-mann-kendall-test-to-assess-cloud-costs\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"How to Use the Mann-Kendall Test to Assess Cloud Costs - insideBIGDATA\" \/>\n<meta property=\"og:description\" content=\"In this contributed article, Vadim Solovey, Chief Technology Officer at DoiT International, indicates that while there are many commercial, off-the-shelf solutions to track cloud costs, a lot of organizations find it helpful to craft their own customized approach. One of the most effective tools to accomplish this is the Mann-Kendall test, which has proven highly effective at separating the signal from the noise in environments generating multiple time series data feeds.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/insidebigdata.com\/2022\/03\/06\/how-to-use-the-mann-kendall-test-to-assess-cloud-costs\/\" \/>\n<meta property=\"og:site_name\" content=\"insideBIGDATA\" \/>\n<meta property=\"article:publisher\" content=\"http:\/\/www.facebook.com\/insidebigdata\" \/>\n<meta property=\"article:published_time\" content=\"2022-03-06T14:00:00+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2022-03-07T17:25:41+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/insidebigdata.com\/wp-content\/uploads\/2019\/04\/Cloud_data_SHUTTERSTOCK.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"250\" \/>\n\t<meta property=\"og:image:height\" content=\"187\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Editorial Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@insideBigData\" \/>\n<meta name=\"twitter:site\" content=\"@insideBigData\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Editorial Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/insidebigdata.com\/2022\/03\/06\/how-to-use-the-mann-kendall-test-to-assess-cloud-costs\/\",\"url\":\"https:\/\/insidebigdata.com\/2022\/03\/06\/how-to-use-the-mann-kendall-test-to-assess-cloud-costs\/\",\"name\":\"How to Use the Mann-Kendall Test to Assess Cloud Costs - insideBIGDATA\",\"isPartOf\":{\"@id\":\"https:\/\/insidebigdata.com\/#website\"},\"datePublished\":\"2022-03-06T14:00:00+00:00\",\"dateModified\":\"2022-03-07T17:25:41+00:00\",\"author\":{\"@id\":\"https:\/\/insidebigdata.com\/#\/schema\/person\/2949e412c144601cdbcc803bd234e1b9\"},\"breadcrumb\":{\"@id\":\"https:\/\/insidebigdata.com\/2022\/03\/06\/how-to-use-the-mann-kendall-test-to-assess-cloud-costs\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/insidebigdata.com\/2022\/03\/06\/how-to-use-the-mann-kendall-test-to-assess-cloud-costs\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/insidebigdata.com\/2022\/03\/06\/how-to-use-the-mann-kendall-test-to-assess-cloud-costs\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/insidebigdata.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"How to Use the Mann-Kendall Test to Assess Cloud Costs\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/insidebigdata.com\/#website\",\"url\":\"https:\/\/insidebigdata.com\/\",\"name\":\"insideBIGDATA\",\"description\":\"Your Source for AI, Data Science, Deep Learning & Machine Learning Strategies\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/insidebigdata.com\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/insidebigdata.com\/#\/schema\/person\/2949e412c144601cdbcc803bd234e1b9\",\"name\":\"Editorial Team\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/insidebigdata.com\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/e137ce7ea40e38bd4d25bb7860cfe3e4?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/e137ce7ea40e38bd4d25bb7860cfe3e4?s=96&d=mm&r=g\",\"caption\":\"Editorial Team\"},\"sameAs\":[\"http:\/\/www.insidebigdata.com\"],\"url\":\"https:\/\/insidebigdata.com\/author\/editorial\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"How to Use the Mann-Kendall Test to Assess Cloud Costs - insideBIGDATA","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/insidebigdata.com\/2022\/03\/06\/how-to-use-the-mann-kendall-test-to-assess-cloud-costs\/","og_locale":"en_US","og_type":"article","og_title":"How to Use the Mann-Kendall Test to Assess Cloud Costs - insideBIGDATA","og_description":"In this contributed article, Vadim Solovey, Chief Technology Officer at DoiT International, indicates that while there are many commercial, off-the-shelf solutions to track cloud costs, a lot of organizations find it helpful to craft their own customized approach. One of the most effective tools to accomplish this is the Mann-Kendall test, which has proven highly effective at separating the signal from the noise in environments generating multiple time series data feeds.","og_url":"https:\/\/insidebigdata.com\/2022\/03\/06\/how-to-use-the-mann-kendall-test-to-assess-cloud-costs\/","og_site_name":"insideBIGDATA","article_publisher":"http:\/\/www.facebook.com\/insidebigdata","article_published_time":"2022-03-06T14:00:00+00:00","article_modified_time":"2022-03-07T17:25:41+00:00","og_image":[{"width":250,"height":187,"url":"https:\/\/insidebigdata.com\/wp-content\/uploads\/2019\/04\/Cloud_data_SHUTTERSTOCK.jpg","type":"image\/jpeg"}],"author":"Editorial Team","twitter_card":"summary_large_image","twitter_creator":"@insideBigData","twitter_site":"@insideBigData","twitter_misc":{"Written by":"Editorial Team","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/insidebigdata.com\/2022\/03\/06\/how-to-use-the-mann-kendall-test-to-assess-cloud-costs\/","url":"https:\/\/insidebigdata.com\/2022\/03\/06\/how-to-use-the-mann-kendall-test-to-assess-cloud-costs\/","name":"How to Use the Mann-Kendall Test to Assess Cloud Costs - insideBIGDATA","isPartOf":{"@id":"https:\/\/insidebigdata.com\/#website"},"datePublished":"2022-03-06T14:00:00+00:00","dateModified":"2022-03-07T17:25:41+00:00","author":{"@id":"https:\/\/insidebigdata.com\/#\/schema\/person\/2949e412c144601cdbcc803bd234e1b9"},"breadcrumb":{"@id":"https:\/\/insidebigdata.com\/2022\/03\/06\/how-to-use-the-mann-kendall-test-to-assess-cloud-costs\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/insidebigdata.com\/2022\/03\/06\/how-to-use-the-mann-kendall-test-to-assess-cloud-costs\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/insidebigdata.com\/2022\/03\/06\/how-to-use-the-mann-kendall-test-to-assess-cloud-costs\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/insidebigdata.com\/"},{"@type":"ListItem","position":2,"name":"How to Use the Mann-Kendall Test to Assess Cloud Costs"}]},{"@type":"WebSite","@id":"https:\/\/insidebigdata.com\/#website","url":"https:\/\/insidebigdata.com\/","name":"insideBIGDATA","description":"Your Source for AI, Data Science, Deep Learning & Machine Learning Strategies","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/insidebigdata.com\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/insidebigdata.com\/#\/schema\/person\/2949e412c144601cdbcc803bd234e1b9","name":"Editorial Team","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/insidebigdata.com\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/e137ce7ea40e38bd4d25bb7860cfe3e4?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/e137ce7ea40e38bd4d25bb7860cfe3e4?s=96&d=mm&r=g","caption":"Editorial Team"},"sameAs":["http:\/\/www.insidebigdata.com"],"url":"https:\/\/insidebigdata.com\/author\/editorial\/"}]}},"jetpack_featured_media_url":"https:\/\/insidebigdata.com\/wp-content\/uploads\/2019\/04\/Cloud_data_SHUTTERSTOCK.jpg","jetpack_shortlink":"https:\/\/wp.me\/p9eA3j-7s0","jetpack-related-posts":[{"id":28509,"url":"https:\/\/insidebigdata.com\/2022\/02\/19\/enterprise-strategy-group-esg-cloud-observability-survey-sponsored-by-yotascale-reveals-enterprises-are-challenged-to-keep-track-of-cloud-costs-and-need-better-visibility-into-cloud-spend\/","url_meta":{"origin":28644,"position":0},"title":"Enterprise Strategy Group (ESG) Cloud Observability Survey, Sponsored by Yotascale, Reveals Enterprises are Challenged to Keep Track of Cloud Costs and Need Better Visibility into Cloud Spend","date":"February 19, 2022","format":false,"excerpt":"Yotascale, a leader in dynamic cloud cost management, announced the results of an ESG Observability Survey it sponsored to survey IT, DevOps, and AppDev professionals responsible for evaluating, purchasing, managing, and building application infrastructure. Out of 357 professionals, 64% agree that the adoption of public cloud or multiple public cloud\u2026","rel":"","context":"In "Big Data"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":23479,"url":"https:\/\/insidebigdata.com\/2019\/10\/26\/kubernetes-driving-google-cloud\/","url_meta":{"origin":28644,"position":1},"title":"Kubernetes Driving Google Cloud","date":"October 26, 2019","format":false,"excerpt":"Kubernetes is increasingly being used with big data deployments. This trend is driving more big data apps to move to GCP, which offers homegrown support for Kubernetes. To make these workloads simpler and cheaper, there's a need for a new solution for managing data workloads on Google Cloud Dataproc. With\u2026","rel":"","context":"In "Big Data"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":19428,"url":"https:\/\/insidebigdata.com\/2017\/11\/20\/sentenai-delivers-automated-data-engineering-data-science-machine-learning-applications\/","url_meta":{"origin":28644,"position":2},"title":"Sentenai Delivers Automated Data Engineering for Data Science and Machine Learning Applications","date":"November 20, 2017","format":false,"excerpt":"Sentenai, an emerging sensor data technology company, announced the availability of its flagship product, the Sentenai Sensor Data Cloud. By going beyond the initial harnessing of machine-based data and understanding the information that data provides, organizations can streamline their operational processes and develop predictive maintenance solutions that decrease unplanned downtime.","rel":"","context":"In "Big Data"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":23544,"url":"https:\/\/insidebigdata.com\/2019\/11\/10\/companies-are-bringing-data-back-from-the-cloud-now-they-need-a-place-to-put-it\/","url_meta":{"origin":28644,"position":3},"title":"Companies Are Bringing Data Back from the Cloud. Now They Need a Place to Put It","date":"November 10, 2019","format":false,"excerpt":"In this special guest feature, Shridar Subramanian, Vice President of Global Product Management and Marketing at StorageCraft, discusses the trend called \u201ccloud data repatriation,\" and how it appears to be gaining steam. A rethinking is happening where companies are looking to return at least some of their core data and\u2026","rel":"","context":"In "Big Data"","img":{"alt_text":"","src":"https:\/\/i0.wp.com\/insidebigdata.com\/wp-content\/uploads\/2019\/11\/Shridar-Subramanian.jpg?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":16754,"url":"https:\/\/insidebigdata.com\/2016\/12\/21\/hortonworks-advances-cloud-strategy-with-availability-of-hortonworks-data-cloud-for-amazon-web-services\/","url_meta":{"origin":28644,"position":4},"title":"Hortonworks Advances Cloud Strategy with Availability of Hortonworks Data Cloud for Amazon Web Services","date":"December 21, 2016","format":false,"excerpt":"Hortonworks, Inc. \u00ae (NASDAQ: HDP), a leading innovator of open and connected data platforms, announced the availability of Hortonworks Data Cloud on the Amazon Web Services (AWS) Cloud. Hortonworks Data Cloud for AWS enables users to harness the agility and elasticity of Apache\u00ae Hadoop\u2122 and Apache\u00ae Spark\u2122 in the cloud\u2026","rel":"","context":"In "Big Data"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":33222,"url":"https:\/\/insidebigdata.com\/2023\/08\/29\/how-ml-is-used-to-reveal-hidden-savings-in-your-cloud-infra\/","url_meta":{"origin":28644,"position":5},"title":"How ML is Used to Reveal Hidden Savings in Your Cloud Infra\u00a0","date":"August 29, 2023","format":false,"excerpt":"In this contributed article, Maxim Melamedov, CEO and co-founder of\u00a0Zesty, explores the cost-savings potential behind leveraging AI\/ML in the cloud. By implementing tools capable of real-time decision making and analysis, companies can truly unlock the promise of the cloud.","rel":"","context":"In "Cloud"","img":{"alt_text":"","src":"https:\/\/i0.wp.com\/insidebigdata.com\/wp-content\/uploads\/2023\/08\/Machine_Learning_shutterstock_742653250_special.jpg?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]}],"_links":{"self":[{"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/posts\/28644"}],"collection":[{"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/users\/10513"}],"replies":[{"embeddable":true,"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/comments?post=28644"}],"version-history":[{"count":0,"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/posts\/28644\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/media\/22516"}],"wp:attachment":[{"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/media?parent=28644"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/categories?post=28644"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/tags?post=28644"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}