# Exploiting BERT For Multimodal Target Sentiment Classification Through Input Space Translation

**Authors:** Zaid Khan, Yun Fu  
**arXiv:** [2108.01682](https://arxiv.org/abs/2108.01682)  
**Date:** 2021-08-03  
**Summary page:** https://www.summarizepaper.com/en/paper/2108.01682v2/

## Key points

- Authors Zaid Khan and Yun Fu focus on multimodal target/aspect sentiment classification
- Combination of multimodal sentiment analysis with aspect/target sentiment classification merges vision and language
- Twitter is highlighted as an optimal platform due to its multimodal nature, emotional content, and real-world impact
- Challenge lies in brevity of tweets with potentially irrelevant images
- Two-stream model introduced to translate images in input space
- Single-pass non-autoregressive text generation approach leverages translation for enriching language model with multimodal information
- Methodology achieves state-of-the-art performance on two multimodal Twitter datasets without modifying internal structure of language model
- Failure mode observed in prevalent approach for aspect sentiment analysis when applied to tweets
- Research provides insights into enhancing sentiment classification accuracy in social media contexts
- Availability of code on GitHub promotes reproducibility and future research efforts

## Summary

In their paper titled "Exploiting BERT For Multimodal Target Sentiment Classification Through Input Space Translation," authors Zaid Khan and Yun Fu delve into the realm of multimodal target/aspect sentiment classification. This innovative approach combines multimodal sentiment analysis with aspect/target sentiment classification to merge vision and language in discerning the sentiment towards a specific entity within a sentence. The study highlights Twitter as an optimal platform for such tasks due to its inherently multimodal nature, high emotional content, and significant impact on real-world events. The challenge arises from the brevity of multimodal tweets often accompanied by intricate and potentially irrelevant images. To address this issue, Khan and Fu introduce a two-stream model that employs an object-aware transformer to translate images in input space. This is followed by a single-pass non-autoregressive text generation approach that leverages the translation to create an auxiliary sentence enriching the language model with multimodal information. By expanding the amount of textual data available to the language model and distilling object-level details from complex images, their methodology achieves state-of-the-art performance on two multimodal Twitter datasets. Notably, this success is attained without necessitating modifications to the internal structure of the language model for handling multimodal data, underscoring the efficacy of their translation technique. Furthermore, the authors shed light on a failure mode observed in a prevalent approach for aspect sentiment analysis when applied to tweets. Their findings provide valuable insights into enhancing sentiment classification accuracy in social media contexts. Overall, Khan and Fu's research offers a compelling exploration of leveraging BERT for multimodal target sentiment classification through input space translation. Their work not only advances understanding in this domain but also provides practical implications for improving sentiment analysis in diverse multimedia settings. The availability of their code on GitHub further facilitates reproducibility and future research endeavors in this evolving field.