The paper titled "MAKE: Product Retrieval with Vision-Language Pre-training in Taobao Search" focuses on improving the text-to-multimodal retrieval in Taobao Search. It consists of two phases: retrieval and ranking. The retrieval phase returns a subset of candidate products based on user queries, while the ranking phase determines the final order of these products. To address the problem of incorporating visual clues into retrieval tasks, the authors propose a novel Modal Adaptation module for cross-modal fusion. This module assigns appropriate weights to texts and images across different products, taking into account users' varying attention on titles or images. Additionally, user queries are often brief in e-commerce search which leads to semantic imbalance between user queries and product titles. To overcome this challenge, the authors design a separate text encoder and a Keyword Enhancement mechanism to enrich query representations and improve text-to-multimodal matching. The paper introduces a vision-language (V+L) pre-training method that leverages multimodal information from user queries, product titles, and product images. Extensive experiments demonstrate that their retrieval-specific pre-training model called MAKE outperforms existing V+L pre-training methods in text-to-multimodal retrieval tasks. As a result of deploying MAKE online, significant improvements have been observed in the retrieval system of Taobao Search. The paper is authored by Xiaoyang Zheng, Zilong Wang, Sen Li, Ke Xu, Tao Zhuang, Qingwen Liu and Xiaoyi Zeng and has been accepted for publication at The Industry Track of the Web Conference 2023 spanning five pages.
The paper is about making it easier to find things on Taobao Search. There are two parts to this: finding the right items and putting them in order. They have a special way of combining words and pictures called Modal Adaptation. They also have a way of making sure the words match what people are looking for. They use a special method that looks at what people say, what the item is called, and what it looks like. They did lots of tests and found that their new method works better than other methods. Taobao Search got much better after they started using it."
Definitions
- Text-to-multimodal retrieval: Finding things based on both words and pictures.
- Phases: Different parts or steps of a process.
- Cross-modal fusion: Combining words and pictures together.
- Semantic imbalance: When the meaning of words doesn't match with something else.
- Vision-language (V+L) pre-training: Teaching computers to understand both images and words together.
Improving Text-to-Multimodal Retrieval in Taobao Search with MAKE
E-commerce search engines are becoming increasingly popular as more and more people shop online. To provide users with the best possible experience, it is essential for these search engines to accurately match user queries with product titles and images. The paper titled “MAKE: Product Retrieval with Vision-Language Pre-training in Taobao Search” by Xiaoyang Zheng, Zilong Wang, Sen Li, Ke Xu, Tao Zhuang, Qingwen Liu and Xiaoyi Zeng proposes a novel vision-language (V+L) pre-training method called MAKE that can improve text-to-multimodal retrieval tasks in Taobao Search. This paper has been accepted for publication at The Industry Track of the Web Conference 2023 spanning five pages.
Retrieval and Ranking Phases
The authors propose two phases for improving text-to-multimodal retrieval tasks: retrieval and ranking. In the retrieval phase, a subset of candidate products based on user queries is returned while the ranking phase determines the final order of these products.
Modal Adaptation Module
To address the problem of incorporating visual clues into retrieval tasks, the authors introduce a Modal Adaptation module for cross modal fusion which assigns appropriate weights to texts and images across different products taking into account users' varying attention on titles or images.
Keyword Enhancement Mechanism
User queries are often brief in ecommerce search which leads to semantic imbalance between user queries and product titles. To overcome this challenge, the authors design a separate text encoder as well as a Keyword Enhancement mechanism to enrich query representations and improve text to multimodal matching.
Vision Language Pre Training Method - MAKE
The proposed V+L pre training method leverages multimodal information from user queries including product titles and product images . Extensive experiments demonstrate that their retrieval specific pre training model called MAKE outperforms existing V+L pre training methods in text to multimodal retrieval tasks . As a result of deploying MAKE online , significant improvements have been observed in the retrieval system of Taobao Search .
Conclusion This research paper presents an effective way to improve text -to -multimodal retrieval tasks using V + L pre -training method called MAKE . By leveraging multimodal information from user queries , product titles ,and product images , it enables better matching between user query representation snd relevant products . Experiments show that this approach yields improved results compared to existing methods when deployed online on Taobao Search .
Taobao Search consists of two phases: the retrieval phase and the ranking phase. Given a user query, the retrieval phase returns a subset of candidate products for the following ranking phase. Recently, the paradigm of pre-training and fine-tuning has shown its potential in incorporating visual clues into retrieval tasks. In this paper, we focus on solving the problem of text-to-multimodal retrieval in Taobao Search. We consider that users' attention on titles or images varies on products. Hence, we propose a novel Modal Adaptation module for cross-modal fusion, which helps assigns appropriate weights on texts and images across products. Furthermore, in e-commerce search, user queries tend to be brief and thus lead to significant semantic imbalance between user queries and product titles. Therefore, we design a separate text encoder and a Keyword Enhancement mechanism to enrich the query representations and improve text-to-multimodal matching. To this end, we present a novel vision-language (V+L) pre-training methods to exploit the multimodal information of (user query, product title, product image). Extensive experiments demonstrate that our retrieval-specific pre-training model (referred to as MAKE) outperforms existing V+L pre-training methods on the text-to-multimodal retrieval task. MAKE has been deployed online and brings major improvements on the retrieval system of Taobao Search.
