{"id":22567,"date":"2019-05-04T08:00:34","date_gmt":"2019-05-04T15:00:34","guid":{"rendered":"https:\/\/insidebigdata.com\/?p=22567"},"modified":"2019-05-05T11:33:50","modified_gmt":"2019-05-05T18:33:50","slug":"accelerating-training-for-ai-deep-learning-networks-with-chunking","status":"publish","type":"post","link":"https:\/\/insidebigdata.com\/2019\/05\/04\/accelerating-training-for-ai-deep-learning-networks-with-chunking\/","title":{"rendered":"Accelerating Training for AI Deep Learning Networks with \u201cChunking\u201d"},"content":{"rendered":"\n

At the International Conference on Learning Representations<\/a> on May 6, IBM Research will share a deeper look around how chunk-based accumulation can speed the training for deep learning networks used for artificial intelligence (AI). <\/p>\n\n\n\n

The company first shared the concept and its vast potential at last year\u2019s NeurIPS conference, when it demonstrated the ability to train deep learning models with 8-bit precision while fully preserving model accuracy across all major AI data set categories: image, speech and text. The result? This technique could accelerate training time for deep neural networks by two to four times over today\u2019s 16-bit systems. <\/p>\n\n\n\n

In IBM Research\u2019s new paper, titled \u201cAccumulation Bit-Width Scaling For Ultralow Precision Training of Deep Networks<\/a>,\u201d researchers explain in greater depth exactly how the concept of chunk-based accumulation works to lower the precision of accumulation from 32-bits down to 16-bits. \u201cChunking\u201d takes the product and divides it into smaller groups of accumulation and then adds the result of each of these smaller groups together, leading to a significantly more accurate result than that of normal accumulation. This allows researchers to study new networks and improve the overall efficiency of deep learning hardware. <\/p>\n\n\n\n

Although this approach was previously considered infeasible to further reduce precision for training, IBM expects this 8-bit training platform to become a widely adopted industry standard in the coming years. <\/p>\n\n\n\n

Sign up for the free insideBIGDATA newsletter<\/a>.<\/em><\/p>\n","protected":false},"excerpt":{"rendered":"

At the International Conference on Learning Representations on May 6, IBM Research will share a deeper look around how chunk-based accumulation can speed the training for deep learning networks used for artificial intelligence (AI). <\/p>\n","protected":false},"author":37,"featured_media":22568,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"footnotes":""},"categories":[526,87,180,173,67,56,84,1],"tags":[264,96],"acf":[],"yoast_head":"\nAccelerating Training for AI Deep Learning Networks with \u201cChunking\u201d - insideBIGDATA<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/insidebigdata.com\/2019\/05\/04\/accelerating-training-for-ai-deep-learning-networks-with-chunking\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Accelerating Training for AI Deep Learning Networks with \u201cChunking\u201d - insideBIGDATA\" \/>\n<meta property=\"og:description\" content=\"At the International Conference on Learning Representations on May 6, IBM Research will share a deeper look around how chunk-based accumulation can speed the training for deep learning networks used for artificial intelligence (AI).\" \/>\n<meta property=\"og:url\" content=\"https:\/\/insidebigdata.com\/2019\/05\/04\/accelerating-training-for-ai-deep-learning-networks-with-chunking\/\" \/>\n<meta property=\"og:site_name\" content=\"insideBIGDATA\" \/>\n<meta property=\"article:publisher\" content=\"http:\/\/www.facebook.com\/insidebigdata\" \/>\n<meta property=\"article:published_time\" content=\"2019-05-04T15:00:34+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2019-05-05T18:33:50+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/insidebigdata.com\/wp-content\/uploads\/2019\/05\/Deep_Learning_shutterstock_386816095.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"300\" \/>\n\t<meta property=\"og:image:height\" content=\"240\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Daniel Gutierrez\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@AMULETAnalytics\" \/>\n<meta name=\"twitter:site\" content=\"@insideBigData\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Daniel Gutierrez\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"1 minute\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/insidebigdata.com\/2019\/05\/04\/accelerating-training-for-ai-deep-learning-networks-with-chunking\/\",\"url\":\"https:\/\/insidebigdata.com\/2019\/05\/04\/accelerating-training-for-ai-deep-learning-networks-with-chunking\/\",\"name\":\"Accelerating Training for AI Deep Learning Networks with \u201cChunking\u201d - insideBIGDATA\",\"isPartOf\":{\"@id\":\"https:\/\/insidebigdata.com\/#website\"},\"datePublished\":\"2019-05-04T15:00:34+00:00\",\"dateModified\":\"2019-05-05T18:33:50+00:00\",\"author\":{\"@id\":\"https:\/\/insidebigdata.com\/#\/schema\/person\/2540da209c83a68f4f5922848f7376ed\"},\"breadcrumb\":{\"@id\":\"https:\/\/insidebigdata.com\/2019\/05\/04\/accelerating-training-for-ai-deep-learning-networks-with-chunking\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/insidebigdata.com\/2019\/05\/04\/accelerating-training-for-ai-deep-learning-networks-with-chunking\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/insidebigdata.com\/2019\/05\/04\/accelerating-training-for-ai-deep-learning-networks-with-chunking\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/insidebigdata.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Accelerating Training for AI Deep Learning Networks with \u201cChunking\u201d\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/insidebigdata.com\/#website\",\"url\":\"https:\/\/insidebigdata.com\/\",\"name\":\"insideBIGDATA\",\"description\":\"Your Source for AI, Data Science, Deep Learning & Machine Learning Strategies\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/insidebigdata.com\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/insidebigdata.com\/#\/schema\/person\/2540da209c83a68f4f5922848f7376ed\",\"name\":\"Daniel Gutierrez\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/insidebigdata.com\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/5780282e7e567e2a502233e948464542?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/5780282e7e567e2a502233e948464542?s=96&d=mm&r=g\",\"caption\":\"Daniel Gutierrez\"},\"description\":\"Daniel D. Gutierrez is a Data Scientist with Los Angeles-based AMULET Analytics, a service division of AMULET Development Corp. He's been involved with data science and Big Data long before it came in vogue, so imagine his delight when the Harvard Business Review recently deemed \\\"data scientist\\\" as the sexiest profession for the 21st century. Previously, he taught computer science and database classes at UCLA Extension for over 15 years, and authored three computer industry books on database technology. He also served as technical editor, columnist and writer at a major computer industry monthly publication for 7 years. Follow his data science musings at @AMULETAnalytics.\",\"sameAs\":[\"http:\/\/www.insidebigdata.com\",\"https:\/\/twitter.com\/@AMULETAnalytics\"],\"url\":\"https:\/\/insidebigdata.com\/author\/dangutierrez\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Accelerating Training for AI Deep Learning Networks with \u201cChunking\u201d - insideBIGDATA","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/insidebigdata.com\/2019\/05\/04\/accelerating-training-for-ai-deep-learning-networks-with-chunking\/","og_locale":"en_US","og_type":"article","og_title":"Accelerating Training for AI Deep Learning Networks with \u201cChunking\u201d - insideBIGDATA","og_description":"At the International Conference on Learning Representations on May 6, IBM Research will share a deeper look around how chunk-based accumulation can speed the training for deep learning networks used for artificial intelligence (AI).","og_url":"https:\/\/insidebigdata.com\/2019\/05\/04\/accelerating-training-for-ai-deep-learning-networks-with-chunking\/","og_site_name":"insideBIGDATA","article_publisher":"http:\/\/www.facebook.com\/insidebigdata","article_published_time":"2019-05-04T15:00:34+00:00","article_modified_time":"2019-05-05T18:33:50+00:00","og_image":[{"width":300,"height":240,"url":"https:\/\/insidebigdata.com\/wp-content\/uploads\/2019\/05\/Deep_Learning_shutterstock_386816095.jpg","type":"image\/jpeg"}],"author":"Daniel Gutierrez","twitter_card":"summary_large_image","twitter_creator":"@AMULETAnalytics","twitter_site":"@insideBigData","twitter_misc":{"Written by":"Daniel Gutierrez","Est. reading time":"1 minute"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/insidebigdata.com\/2019\/05\/04\/accelerating-training-for-ai-deep-learning-networks-with-chunking\/","url":"https:\/\/insidebigdata.com\/2019\/05\/04\/accelerating-training-for-ai-deep-learning-networks-with-chunking\/","name":"Accelerating Training for AI Deep Learning Networks with \u201cChunking\u201d - insideBIGDATA","isPartOf":{"@id":"https:\/\/insidebigdata.com\/#website"},"datePublished":"2019-05-04T15:00:34+00:00","dateModified":"2019-05-05T18:33:50+00:00","author":{"@id":"https:\/\/insidebigdata.com\/#\/schema\/person\/2540da209c83a68f4f5922848f7376ed"},"breadcrumb":{"@id":"https:\/\/insidebigdata.com\/2019\/05\/04\/accelerating-training-for-ai-deep-learning-networks-with-chunking\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/insidebigdata.com\/2019\/05\/04\/accelerating-training-for-ai-deep-learning-networks-with-chunking\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/insidebigdata.com\/2019\/05\/04\/accelerating-training-for-ai-deep-learning-networks-with-chunking\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/insidebigdata.com\/"},{"@type":"ListItem","position":2,"name":"Accelerating Training for AI Deep Learning Networks with \u201cChunking\u201d"}]},{"@type":"WebSite","@id":"https:\/\/insidebigdata.com\/#website","url":"https:\/\/insidebigdata.com\/","name":"insideBIGDATA","description":"Your Source for AI, Data Science, Deep Learning & Machine Learning Strategies","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/insidebigdata.com\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/insidebigdata.com\/#\/schema\/person\/2540da209c83a68f4f5922848f7376ed","name":"Daniel Gutierrez","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/insidebigdata.com\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/5780282e7e567e2a502233e948464542?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/5780282e7e567e2a502233e948464542?s=96&d=mm&r=g","caption":"Daniel Gutierrez"},"description":"Daniel D. Gutierrez is a Data Scientist with Los Angeles-based AMULET Analytics, a service division of AMULET Development Corp. He's been involved with data science and Big Data long before it came in vogue, so imagine his delight when the Harvard Business Review recently deemed \"data scientist\" as the sexiest profession for the 21st century. Previously, he taught computer science and database classes at UCLA Extension for over 15 years, and authored three computer industry books on database technology. He also served as technical editor, columnist and writer at a major computer industry monthly publication for 7 years. Follow his data science musings at @AMULETAnalytics.","sameAs":["http:\/\/www.insidebigdata.com","https:\/\/twitter.com\/@AMULETAnalytics"],"url":"https:\/\/insidebigdata.com\/author\/dangutierrez\/"}]}},"jetpack_featured_media_url":"https:\/\/insidebigdata.com\/wp-content\/uploads\/2019\/05\/Deep_Learning_shutterstock_386816095.jpg","jetpack_shortlink":"https:\/\/wp.me\/p9eA3j-5RZ","jetpack-related-posts":[{"id":17885,"url":"https:\/\/insidebigdata.com\/2017\/05\/14\/nvidia-train-100000-developers-deep-learning-2017\/","url_meta":{"origin":22567,"position":0},"title":"NVIDIA to Train 100,000 Developers on Deep Learning in 2017","date":"May 14, 2017","format":false,"excerpt":"[GPU Technology Conference Coverage] To meet surging demand for expertise in the field of AI, NVIDIA announced that it plans to train 100,000 developers this year \u2014 a tenfold increase over 2016 \u2014 through the NVIDIA Deep Learning Institute. Analyst firm IDC estimates that 80 percent of all applications will\u2026","rel":"","context":"In "AI Deep Learning"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":17100,"url":"https:\/\/insidebigdata.com\/2017\/02\/08\/insidebigdata-guide-deep-learning-artificial-intelligence\/","url_meta":{"origin":22567,"position":1},"title":"insideBIGDATA Guide to Deep Learning and Artificial Intelligence","date":"February 8, 2017","format":false,"excerpt":"The insideBIGDATA Guide to Deep Learning & Artificial Intelligence is a useful new resource directed toward enterprise thought leaders who wish to gain strategic insights into this exciting area of technology. In this guide, we take a high-level view of AI and deep learning in terms of how it\u2019s being\u2026","rel":"","context":"In "AI Deep Learning"","img":{"alt_text":"","src":"https:\/\/i0.wp.com\/insidebigdata.com\/wp-content\/uploads\/2017\/02\/insideBIGDATA_Guide_DL_AI.png?resize=350%2C200","width":350,"height":200},"classes":[]},{"id":20604,"url":"https:\/\/insidebigdata.com\/2018\/06\/22\/advancements-dynamic-efficient-deep-learning-systems\/","url_meta":{"origin":22567,"position":2},"title":"Advancements in Dynamic and Efficient Deep Learning Systems","date":"June 22, 2018","format":false,"excerpt":"We're seeing much hype in the marketplace about the potential of AI, especially with respect to computer vision systems and its ability accelerate the development of everything from self-driving cars to autonomous robots. To create more dynamic and efficient deep learning systems, that don\u2019t compromise accuracy, IBM Research is exploring\u2026","rel":"","context":"In "AI Deep Learning"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":14589,"url":"https:\/\/insidebigdata.com\/2016\/03\/04\/hpc-the-computational-foundation-of-deep-learning\/","url_meta":{"origin":22567,"position":3},"title":"Scaling Deep Learning","date":"March 4, 2016","format":false,"excerpt":"Deep Learning is a relatively new area of Machine Learning research which has been introduced with the objective of moving Machine Learning closer to one of its original goals: Artificial Intelligence. The video presentation below is from the 2016 Stanford HPC Conference, where Brian Catanzaro from Baidu presents: \"Scaling Deep\u2026","rel":"","context":"In "AI Deep Learning"","img":{"alt_text":"","src":"https:\/\/i0.wp.com\/img.youtube.com\/vi\/x_i0Wq8D3ds\/0.jpg?resize=350%2C200","width":350,"height":200},"classes":[]},{"id":17874,"url":"https:\/\/insidebigdata.com\/2017\/05\/29\/enterprise-software-tools-ai\/","url_meta":{"origin":22567,"position":4},"title":"Enterprise Software Tools for AI","date":"May 29, 2017","format":false,"excerpt":"This post delves into a variety of enterprise software options for AI. This article is part of a special insideHPC report that explores trends in machine learning and deep learning. Find out how businesses are using machine learning and deep learning, differentiating between AI, machine learning and deep learning, what\u2026","rel":"","context":"In "Featured"","img":{"alt_text":"","src":"https:\/\/i0.wp.com\/insidebigdata.com\/wp-content\/uploads\/2017\/04\/Dell_Nivida_ML_DLguide.jpg?resize=350%2C200","width":350,"height":200},"classes":[]},{"id":17865,"url":"https:\/\/insidebigdata.com\/2017\/05\/22\/differentiating-between-ai-machine-learning-and-deep-learning\/","url_meta":{"origin":22567,"position":5},"title":"Differentiating between AI, Machine Learning and Deep Learning","date":"May 22, 2017","format":false,"excerpt":"With all the quickly evolving jargon in the industry today, it\u2019s important to be able to differentiate between AI, machine learning and deep learning. This article is part of a special insideHPC report that explores trends in machine learning and deep learning.","rel":"","context":"In "Featured"","img":{"alt_text":"AI","src":"https:\/\/i0.wp.com\/insidebigdata.com\/wp-content\/uploads\/2017\/05\/Screen-Shot-2017-05-12-at-7.23.52-AM.png?resize=350%2C200","width":350,"height":200},"classes":[]}],"_links":{"self":[{"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/posts\/22567"}],"collection":[{"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/users\/37"}],"replies":[{"embeddable":true,"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/comments?post=22567"}],"version-history":[{"count":0,"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/posts\/22567\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/media\/22568"}],"wp:attachment":[{"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/media?parent=22567"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/categories?post=22567"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/tags?post=22567"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}