# Unifying Visual and Vision-Language Tracking via Contrastive Learning

**Authors:** Yinchao Ma, Yuyang Tang, Wenfei Yang, Tianzhu Zhang, Jinpeng Zhang, Mengxue Kang  
**arXiv:** [2401.11228](https://arxiv.org/abs/2401.11228)  
**Date:** 2024-01-20  
**Summary page:** https://www.summarizepaper.com/en/paper/2401.11228v1/

## Key points

- Single object tracking challenge: accurately locating a target object in video sequences based on modal references like BBOX, NL, or NL+BBOX
- Existing trackers specialize in limited reference settings, hindering effective handling of diverse modalities
- UVLTrack: unified tracker accommodating all three reference settings using the same parameters
- Modality-unified feature extractor for joint visual and language feature learning
- Multi-modal contrastive loss mechanism aligning visual and language features into a cohesive semantic space
- Modality-adaptive box head component leveraging target references to dynamically extract scenario features from video contexts
- UVLTrack enhances performance across different reference settings through contrastive target distinction
- Impressive capabilities demonstrated on various datasets including visual tracking, vision-language tracking, and visual grounding datasets
- Developed by authors Yinchao Ma, Yuyang Tang, Wenfei Yang, Tianzhu Zhang, Jinpeng Zhang, and Mengxue Kang as a versatile solution for unifying visual and vision-language tracking through contrastive learning
- Research findings and associated codes/models to be open-sourced at https://github.com/OpenSpaceAI/UVLTrack

## Summary

In the realm of single object tracking, the challenge lies in accurately locating a target object within a video sequence based on various modal references such as the initial bounding box (BBOX), natural language (NL), or a combination of both (NL+BBOX). Existing trackers often specialize in one or a subset of these reference settings, leading to limitations in handling diverse modalities effectively. However, this issue has been addressed by a groundbreaking unified tracker known as UVLTrack. UVLTrack sets itself apart by seamlessly accommodating all three reference settings (BBOX, NL, NL+BBOX) using the same parameters. This innovative tracker offers several key advantages. Firstly, it incorporates a modality-unified feature extractor that facilitates joint visual and language feature learning. Additionally, it introduces a multi-modal contrastive loss mechanism to align visual and language features into a cohesive semantic space. Moreover, UVLTrack introduces a modality-adaptive box head component that leverages target references to dynamically extract scenario features from video contexts. By distinguishing the target in a contrastive manner, this adaptive box head enhances performance across different reference settings. Extensive experiments have demonstrated the impressive capabilities of UVLTrack on seven visual tracking datasets, three vision-language tracking datasets, and three visual grounding datasets. The collaborative efforts of authors Yinchao Ma, Yuyang Tang, Wenfei Yang, Tianzhu Zhang, Jinpeng Zhang and Mengxue Kang have culminated in the development of UVLTrack - a versatile and robust solution for unifying visual and vision-language tracking through contrastive learning. The research findings and associated codes/models are set to be open-sourced at https://github.com/OpenSpaceAI/UVLTrack for further exploration and application in the field of computer vision.