{"id":24721,"date":"2020-07-11T06:00:00","date_gmt":"2020-07-11T13:00:00","guid":{"rendered":"https:\/\/insidebigdata.com\/?p=24721"},"modified":"2020-07-12T11:14:36","modified_gmt":"2020-07-12T18:14:36","slug":"research-highlights-exbert","status":"publish","type":"post","link":"https:\/\/insidebigdata.com\/2020\/07\/11\/research-highlights-exbert\/","title":{"rendered":"Research Highlights: ExBERT"},"content":{"rendered":"\n

In the insideBIGDATA Research Highlights column we take a look at new and upcoming results from the research community for data science, machine learning, AI and deep learning. Our readers need to get a glimpse for technology coming down the pipeline that will make their efforts more strategic and competitive. In this installment we review a new paper: EXBERT: A Visual Analysis Tool to Explore Learned Representations in Transformers Models<\/a> by researchers from the MIT-IBM Watson AI Lab and Harvard. The group presents the latest from “ExBERT,” a tool that allows you to go under the hood of a language model and gather previously inaccessible details, like what information the model uses to autocomplete words and phrases.<\/p>\n\n\n\n

Large Transformer-based language models can route and reshape complex information via their multi-headed attention mechanism. Although the attention never receives explicit supervision, it can exhibit understandable patterns following linguistic or positional information. To further our understanding of the inner workings of these models, we need to analyze both the learned representations and the attentions.<\/p>\n\n\n\n

\"\"<\/figure><\/div>\n\n\n\n

To support analysis for a wide variety of Transformer models, the researchers introduce exBERT, a tool to help humans conduct flexible, interactive investigations and formulate hypotheses for the model-internal reasoning process. exBERT provides insights into the meaning of the contextual representations and attention by matching a human-specified input to similar contexts in large annotated datasets.<\/p>\n\n\n\n

The fully-featured demo video below shows select Transformer models with the Wizard of Oz and a subset of Wikipedia pre-annotated for the hidden representations for each model.<\/p>\n\n\n\n

\n