# MAKE: Product Retrieval with Vision-Language Pre-training in Taobao Search

**Authors:** Xiaoyang Zheng, Zilong Wang, Sen Li, Ke Xu, Tao Zhuang, Qingwen Liu, Xiaoyi Zeng  
**arXiv:** [2301.12646](https://arxiv.org/abs/2301.12646)  
**Date:** 2023-01-30  
**Summary page:** https://www.summarizepaper.com/en/paper/2301.12646v1/

## Key points

- The paper focuses on improving text-to-multimodal retrieval in Taobao Search.
- It consists of two phases: retrieval and ranking.
- A Modal Adaptation module is proposed for cross-modal fusion, assigning appropriate weights to texts and images.
- A separate text encoder and Keyword Enhancement mechanism are designed to address the semantic imbalance between user queries and product titles.
- The paper introduces a vision-language (V+L) pre-training method that leverages multimodal information from user queries, product titles, and product images.
- Extensive experiments show that their retrieval-specific pre-training model called MAKE outperforms existing V+L pre-training methods in text-to-multimodal retrieval tasks.
- Significant improvements have been observed in the retrieval system of Taobao Search after deploying MAKE online.

## Summary

The paper titled "MAKE: Product Retrieval with Vision-Language Pre-training in Taobao Search" focuses on improving the text-to-multimodal retrieval in Taobao Search. It consists of two phases: retrieval and ranking. The retrieval phase returns a subset of candidate products based on user queries, while the ranking phase determines the final order of these products. To address the problem of incorporating visual clues into retrieval tasks, the authors propose a novel Modal Adaptation module for cross-modal fusion. This module assigns appropriate weights to texts and images across different products, taking into account users' varying attention on titles or images. Additionally, user queries are often brief in e-commerce search which leads to semantic imbalance between user queries and product titles. To overcome this challenge, the authors design a separate text encoder and a Keyword Enhancement mechanism to enrich query representations and improve text-to-multimodal matching. The paper introduces a vision-language (V+L) pre-training method that leverages multimodal information from user queries, product titles, and product images. Extensive experiments demonstrate that their retrieval-specific pre-training model called MAKE outperforms existing V+L pre-training methods in text-to-multimodal retrieval tasks. As a result of deploying MAKE online, significant improvements have been observed in the retrieval system of Taobao Search. The paper is authored by Xiaoyang Zheng, Zilong Wang, Sen Li, Ke Xu, Tao Zhuang, Qingwen Liu and Xiaoyi Zeng and has been accepted for publication at The Industry Track of the Web Conference 2023 spanning five pages.