# ITEm: Unsupervised Image-Text Embedding Learning for eCommerce

**Authors:** Baohao Liao, Michael Kozielski, Sanjika Hewavitharana, Jiangbo Yuan, Shahram Khadivi, Tomer Lancewicki  
**arXiv:** [2311.02084](https://arxiv.org/abs/2311.02084)  
**Date:** 2023-10-22  
**Summary page:** https://www.summarizepaper.com/en/paper/2311.02084v2/

## Key points

- Product embedding is crucial in eCommerce applications
- Integration of multiple modalities in learning product embeddings shows advancements over single-modality embeddings
- Challenges exist in effectively incorporating all modalities without neglecting valuable information
- Image-text embedding model (ITEm) developed at eBay enhances attention to image and text modalities
- ITEm model extends BERT by learning embeddings from both text and images without prior knowledge of regions of interest
- Evaluation of the pre-trained ITEm model demonstrates substantial improvements over baseline models in tasks like searching for similar products and predicting categories
- Importance of both product images and titles in distinguishing products highlighted, with a focus on avoiding biased embeddings dominated by a single modality
- ITEm model aims to learn fine-grained embeddings from both images and titles in an unsupervised manner to mitigate dominance issues caused by one modality containing dominant information
- Study proposes a method to generate fine-grained representations from product images and titles for eCommerce applications
- Evaluation involved collecting a large-scale image-text product dataset annotated for tasks like searching for similar products and classification against state-of-the-art unimodal and multimodal models
- Exploration of learning image-title embeddings without prior knowledge of regions of interest expands applicability beyond eCommerce datasets
- ITEm proves more efficient than traditional two-stream models by adopting a single-stream model approach with one encoder for vision and text inputs

## Summary

In the realm of eCommerce, product embedding plays a crucial role in various applications. The integration of multiple modalities in learning product embeddings has shown significant advancements compared to single-modality embeddings. offers complementary information from different modalities. However, some modalities may hold more informative power than others, posing a challenge in effectively incorporating all modalities without neglecting valuable information. During a research tenure at eBay, an image-text embedding model (ITEm) was developed as an unsupervised learning method to enhance the attention given to image and text modalities. The ITEm model extends BERT by learning embeddings from both text and images without prior knowledge of regions of interest. It trains a global representation to predict masked words and construct masked image patches without individual representations. Evaluation of the pre-trained ITEm model on tasks such as searching for extremely similar products and predicting product categories demonstrated substantial improvements over strong baseline models. The study introduces the , showcasing examples where both product images and titles contribute significantly to distinguishing products. While titles often contain dominant information for most products, neglecting the importance of images can lead to biased embeddings dominated by a single modality. To address this issue, the ITEm model aims to learn fine-grained embeddings from both images and titles in an unsupervised manner. Key contributions of the study include proposing a method to generate fine-grained representations from product images and titles for eCommerce applications, mitigating over-dominance issues caused by one modality containing dominant information. The evaluation involved collecting a large-scale image-text product dataset annotated for tasks like searching for extremely similar products and classification against state-of-the-art unimodal and multimodal models. Furthermore, the study explores learning image-title embeddings without prior knowledge of regions of interest, expanding the applicability of the proposed method beyond eCommerce datasets. By adopting a single-stream model approach with one encoder for both vision and text inputs, ITEm proves to be more efficient than traditional two-stream models that require sequential processing of different modalities' embeddings. Overall, this research contributes valuable insights into effectively leveraging multiple modalities in learning product embeddings for enhanced performance in eCommerce applications.