{"id":24922,"date":"2020-08-27T06:00:00","date_gmt":"2020-08-27T13:00:00","guid":{"rendered":"https:\/\/insidebigdata.com\/?p=24922"},"modified":"2023-06-23T12:43:55","modified_gmt":"2023-06-23T19:43:55","slug":"research-highlights-attention-condensers","status":"publish","type":"post","link":"https:\/\/insidebigdata.com\/2020\/08\/27\/research-highlights-attention-condensers\/","title":{"rendered":"Research Highlights: Attention Condensers"},"content":{"rendered":"\n

A group of AI researchers from DarwinAI <\/a>and out of the University of Waterloo, announced an important theoretical development in deep learning around “attention condensers.” The paper describing this important advancement is: “TinySpeech: Attention Condensers for Deep Speech Recognition Neural Networks on Edge Devices<\/a>,” by Alexander Wong, et al. Wong is DarwinAI’s CTO. <\/p>\n\n\n\n

By integrating deep attention condensers into their GenSynth platform, the Darwin team was able to construct two highly efficient automatic speech recognition (ASR) models for edge devices. The smallest of these models, termed TinySpeech-B, is over 200x smaller and 20x less complex than previous models with the same accuracy. Moreover, as the models uses 8-bit low-precision parameters, its storage requirements are over 800 times lower than equivalent networks.<\/p>\n\n\n

\n
\"\"
Source: https:\/\/arxiv.org\/abs\/2008.04245 <\/a><\/figcaption><\/figure><\/div>\n\n\n
\n

\u201cThis is an important theoretical advancement that allows a deep learning model to hone in on the “important” aspects of input the way a human might,” said Sheldon Fernandez, CEO of DarwinAI. “In the main, this facilitates more effective and trustworthy decision-making in AI, resulting in neural networks that are both more efficient and robust\u201d.<\/p>\n<\/blockquote>\n\n\n\n

Mcuh liek you can pasre this snetnece evne thuogh its phoentcially flwaed, attention condensers are unique, stand-alone architectures that allow a deep learning model to better focus on “what\u2019s important,” facilitating more effective and trustworthy decisions.<\/p>\n\n\n\n

In a broad sense, this breakthrough is related to Moravec’s paradox<\/em> (see figure below), an observation by AI researchers that high-level reasoning in humans requires little computation whereas low-level sensorimotor skills require a lot. By using human-like shortcuts, we\u2019ve demonstrated that attention condensers can dramatically accelerate AI and reduce its complexity. <\/p>\n\n\n

\n
\"\"<\/figure><\/div>\n\n\n

Sign up for the free insideBIGDATA newsletter<\/a>.<\/em><\/p>\n","protected":false},"excerpt":{"rendered":"

A group of AI researchers from DarwinAI and out of the University of Waterloo, announced an important theoretical development in deep learning around “attention condensers.” The paper describing this important advancement is: “TinySpeech: Attention Condensers for Deep Speech Recognition Neural Networks on Edge Devices,” by Alexander Wong, et al. Wong is DarwinAI’s CTO. <\/p>\n","protected":false},"author":10513,"featured_media":22568,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"footnotes":""},"categories":[526,115,87,180,67,56,84,1303,1],"tags":[437,264,593,652,933,95],"acf":[],"yoast_head":"\nResearch Highlights: Attention Condensers - insideBIGDATA<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/insidebigdata.com\/2020\/08\/27\/research-highlights-attention-condensers\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Research Highlights: Attention Condensers - insideBIGDATA\" \/>\n<meta property=\"og:description\" content=\"A group of AI researchers from DarwinAI and out of the University of Waterloo, announced an important theoretical development in deep learning around "attention condensers." The paper describing this important advancement is: "TinySpeech: Attention Condensers for Deep Speech Recognition Neural Networks on Edge Devices," by Alexander Wong, et al. Wong is DarwinAI's CTO.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/insidebigdata.com\/2020\/08\/27\/research-highlights-attention-condensers\/\" \/>\n<meta property=\"og:site_name\" content=\"insideBIGDATA\" \/>\n<meta property=\"article:publisher\" content=\"http:\/\/www.facebook.com\/insidebigdata\" \/>\n<meta property=\"article:published_time\" content=\"2020-08-27T13:00:00+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2023-06-23T19:43:55+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/insidebigdata.com\/wp-content\/uploads\/2019\/05\/Deep_Learning_shutterstock_386816095.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"300\" \/>\n\t<meta property=\"og:image:height\" content=\"240\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Editorial Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@insideBigData\" \/>\n<meta name=\"twitter:site\" content=\"@insideBigData\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Editorial Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"1 minute\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/insidebigdata.com\/2020\/08\/27\/research-highlights-attention-condensers\/\",\"url\":\"https:\/\/insidebigdata.com\/2020\/08\/27\/research-highlights-attention-condensers\/\",\"name\":\"Research Highlights: Attention Condensers - insideBIGDATA\",\"isPartOf\":{\"@id\":\"https:\/\/insidebigdata.com\/#website\"},\"datePublished\":\"2020-08-27T13:00:00+00:00\",\"dateModified\":\"2023-06-23T19:43:55+00:00\",\"author\":{\"@id\":\"https:\/\/insidebigdata.com\/#\/schema\/person\/2949e412c144601cdbcc803bd234e1b9\"},\"breadcrumb\":{\"@id\":\"https:\/\/insidebigdata.com\/2020\/08\/27\/research-highlights-attention-condensers\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/insidebigdata.com\/2020\/08\/27\/research-highlights-attention-condensers\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/insidebigdata.com\/2020\/08\/27\/research-highlights-attention-condensers\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/insidebigdata.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Research Highlights: Attention Condensers\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/insidebigdata.com\/#website\",\"url\":\"https:\/\/insidebigdata.com\/\",\"name\":\"insideBIGDATA\",\"description\":\"Your Source for AI, Data Science, Deep Learning & Machine Learning Strategies\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/insidebigdata.com\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/insidebigdata.com\/#\/schema\/person\/2949e412c144601cdbcc803bd234e1b9\",\"name\":\"Editorial Team\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/insidebigdata.com\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/e137ce7ea40e38bd4d25bb7860cfe3e4?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/e137ce7ea40e38bd4d25bb7860cfe3e4?s=96&d=mm&r=g\",\"caption\":\"Editorial Team\"},\"sameAs\":[\"http:\/\/www.insidebigdata.com\"],\"url\":\"https:\/\/insidebigdata.com\/author\/editorial\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Research Highlights: Attention Condensers - insideBIGDATA","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/insidebigdata.com\/2020\/08\/27\/research-highlights-attention-condensers\/","og_locale":"en_US","og_type":"article","og_title":"Research Highlights: Attention Condensers - insideBIGDATA","og_description":"A group of AI researchers from DarwinAI and out of the University of Waterloo, announced an important theoretical development in deep learning around \"attention condensers.\" The paper describing this important advancement is: \"TinySpeech: Attention Condensers for Deep Speech Recognition Neural Networks on Edge Devices,\" by Alexander Wong, et al. Wong is DarwinAI's CTO.","og_url":"https:\/\/insidebigdata.com\/2020\/08\/27\/research-highlights-attention-condensers\/","og_site_name":"insideBIGDATA","article_publisher":"http:\/\/www.facebook.com\/insidebigdata","article_published_time":"2020-08-27T13:00:00+00:00","article_modified_time":"2023-06-23T19:43:55+00:00","og_image":[{"width":300,"height":240,"url":"https:\/\/insidebigdata.com\/wp-content\/uploads\/2019\/05\/Deep_Learning_shutterstock_386816095.jpg","type":"image\/jpeg"}],"author":"Editorial Team","twitter_card":"summary_large_image","twitter_creator":"@insideBigData","twitter_site":"@insideBigData","twitter_misc":{"Written by":"Editorial Team","Est. reading time":"1 minute"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/insidebigdata.com\/2020\/08\/27\/research-highlights-attention-condensers\/","url":"https:\/\/insidebigdata.com\/2020\/08\/27\/research-highlights-attention-condensers\/","name":"Research Highlights: Attention Condensers - insideBIGDATA","isPartOf":{"@id":"https:\/\/insidebigdata.com\/#website"},"datePublished":"2020-08-27T13:00:00+00:00","dateModified":"2023-06-23T19:43:55+00:00","author":{"@id":"https:\/\/insidebigdata.com\/#\/schema\/person\/2949e412c144601cdbcc803bd234e1b9"},"breadcrumb":{"@id":"https:\/\/insidebigdata.com\/2020\/08\/27\/research-highlights-attention-condensers\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/insidebigdata.com\/2020\/08\/27\/research-highlights-attention-condensers\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/insidebigdata.com\/2020\/08\/27\/research-highlights-attention-condensers\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/insidebigdata.com\/"},{"@type":"ListItem","position":2,"name":"Research Highlights: Attention Condensers"}]},{"@type":"WebSite","@id":"https:\/\/insidebigdata.com\/#website","url":"https:\/\/insidebigdata.com\/","name":"insideBIGDATA","description":"Your Source for AI, Data Science, Deep Learning & Machine Learning Strategies","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/insidebigdata.com\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/insidebigdata.com\/#\/schema\/person\/2949e412c144601cdbcc803bd234e1b9","name":"Editorial Team","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/insidebigdata.com\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/e137ce7ea40e38bd4d25bb7860cfe3e4?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/e137ce7ea40e38bd4d25bb7860cfe3e4?s=96&d=mm&r=g","caption":"Editorial Team"},"sameAs":["http:\/\/www.insidebigdata.com"],"url":"https:\/\/insidebigdata.com\/author\/editorial\/"}]}},"jetpack_featured_media_url":"https:\/\/insidebigdata.com\/wp-content\/uploads\/2019\/05\/Deep_Learning_shutterstock_386816095.jpg","jetpack_shortlink":"https:\/\/wp.me\/p9eA3j-6tY","jetpack-related-posts":[{"id":21136,"url":"https:\/\/insidebigdata.com\/2018\/09\/19\/darwinai-emerges-stealth-powerful-design-optimization-explainability-platform-deep-learning\/","url_meta":{"origin":24922,"position":0},"title":"DarwinAI Emerges from Stealth with Powerful Design, Optimization and Explainability Platform for Deep Learning","date":"September 19, 2018","format":false,"excerpt":"DarwinAI, a Waterloo, Canada startup creating next generation technologies for Artificial Intelligence development, announced it is emerging out of stealth with $3M in seed funding. The team\u2019s Generative Synthesis platform leverages AI to reduce the complexity and guesswork in designing efficient, high performance deep neural networks for real world applications.","rel":"","context":"In "AI Deep Learning"","img":{"alt_text":"","src":"https:\/\/i0.wp.com\/insidebigdata.com\/wp-content\/uploads\/2018\/09\/DarwinAI_logo.jpg?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":28903,"url":"https:\/\/insidebigdata.com\/2022\/04\/09\/research-highlights-deep-neural-networks-and-tabular-data-a-survey\/","url_meta":{"origin":24922,"position":1},"title":"Research Highlights: Deep Neural Networks and Tabular Data: A Survey","date":"April 9, 2022","format":false,"excerpt":"In this regular column, we take a look at highlights for important research topics of the day for big data, data science, machine learning, AI and deep learning. It\u2019s important to keep connected with the research arm of the field in order to see where we\u2019re headed. In this edition,\u2026","rel":"","context":"In "AI Deep Learning"","img":{"alt_text":"","src":"https:\/\/i0.wp.com\/insidebigdata.com\/wp-content\/uploads\/2022\/04\/Research_highlights_2.png?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":21207,"url":"https:\/\/insidebigdata.com\/2018\/10\/07\/introduction-deep-learning-neural-networks\/","url_meta":{"origin":24922,"position":2},"title":"An Introduction to Deep Learning and Neural Networks","date":"October 7, 2018","format":false,"excerpt":"In this contributed article, Agile SEO technical writer and editor Limor Wainstein outlines how deep learning, neural networks, and machine learning are not interchangeable terms. This article helps to clarify the definitions for you with an introduction to deep learning and neural networks.","rel":"","context":"In "AI Deep Learning"","img":{"alt_text":"","src":"https:\/\/i0.wp.com\/insidebigdata.com\/wp-content\/uploads\/2018\/10\/Neural-Network-diagram.jpg?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":21994,"url":"https:\/\/insidebigdata.com\/2019\/01\/16\/best-of-arxiv-org-for-ai-machine-learning-and-deep-learning-december-2018\/","url_meta":{"origin":24922,"position":3},"title":"Best of arXiv.org for AI, Machine Learning, and Deep Learning \u2013 December 2018","date":"January 16, 2019","format":false,"excerpt":"In this recurring monthly feature, we will filter all the recent research papers appearing in the arXiv.org preprint server for subjects relating to AI, machine learning and deep learning \u2013 from disciplines including statistics, mathematics and computer science \u2013 and provide you with a useful \u201cbest of\u201d list for the\u2026","rel":"","context":"In "AI Deep Learning"","img":{"alt_text":"","src":"https:\/\/i0.wp.com\/insidebigdata.com\/wp-content\/uploads\/2013\/12\/arxiv.jpg?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":14937,"url":"https:\/\/insidebigdata.com\/2016\/04\/28\/movidius-announces-fathom-deep-learning-accelerator-compute-stick\/","url_meta":{"origin":24922,"position":4},"title":"Movidius Announces Fathom Deep Learning Accelerator Compute Stick","date":"April 28, 2016","format":false,"excerpt":"Movidius, a leader in low-power machine vision technology, today announced both the Fathom Neural Compute Stick \u2013 the world\u2019s first deep learning acceleration module, and Fathom deep learning software framework. Both tools hand-in-hand will allow powerful neural networks to be moved out of the cloud, and deployed natively in end-user\u2026","rel":"","context":"In "Big Data"","img":{"alt_text":"","src":"https:\/\/i0.wp.com\/insidebigdata.com\/wp-content\/uploads\/2016\/04\/Fathom1.jpg?resize=350%2C200","width":350,"height":200},"classes":[]},{"id":23221,"url":"https:\/\/insidebigdata.com\/2019\/09\/08\/research-highlight-compressing-neural-networks-with-minimal-sacrifice-in-accuracy\/","url_meta":{"origin":24922,"position":5},"title":"Research Highlight: Compressing Neural Networks with Minimal Sacrifice in Accuracy","date":"September 8, 2019","format":false,"excerpt":"Most data scientists soon realize that deep learning models can be unwieldy and often impractical to run on smaller devices without major modification. Our friends over at deeplearning.ai recently communicated about a group of researchers at Facebook AI Research determined a new technique to compress neural networks with minimal loss\u2026","rel":"","context":"In "AI Deep Learning"","img":{"alt_text":"","src":"https:\/\/i0.wp.com\/insidebigdata.com\/wp-content\/uploads\/2019\/06\/Data-Scientist-shutterstock_768047488.jpg?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]}],"_links":{"self":[{"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/posts\/24922"}],"collection":[{"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/users\/10513"}],"replies":[{"embeddable":true,"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/comments?post=24922"}],"version-history":[{"count":0,"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/posts\/24922\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/media\/22568"}],"wp:attachment":[{"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/media?parent=24922"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/categories?post=24922"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/tags?post=24922"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}