# An Inverse Scaling Law for CLIP Training

**Authors:** Xianhang Li, Zeyu Wang, Cihang Xie  
**arXiv:** [2305.07017](https://arxiv.org/abs/2305.07017)  
**Date:** 2023-05-11  
**Summary page:** https://www.summarizepaper.com/en/paper/2305.07017v1/

## Key points

- Xianhang Li, Zeyu Wang, and Cihang Xie present a groundbreaking finding that could significantly reduce the training cost of CLIP
- CLIP has enabled many recent breakthroughs in computer vision but its associated training cost is prohibitively high
- The authors discovered an inverse scaling law for CLIP training, whereby using larger image/text encoders leads to shorter sequence lengths of image/text tokens that can be applied in training
- The strategy for reducing image/text token length plays a crucial role in determining the quality of this scaling law
- The authors were able to train CLIP successfully even using academic resources
- On an A100 eight-GPU server, their CLIP models achieved zero-shot top-1 ImageNet accuracies of 63.2% in approximately two days, 67.8% in about three days, and 69.3% in roughly four days
- This research could have significant implications for computer vision researchers who may now be able to explore the full potential of CLIP without being hindered by high training costs
- The authors' code is available at https://github.com/UCSC-VLAA/CLIPA for anyone interested in exploring this further

## Summary

In their paper titled "An Inverse Scaling Law for CLIP Training," Xianhang Li, Zeyu Wang, and Cihang Xie present a groundbreaking finding that could significantly reduce the training cost of CLIP, the first foundation model that connects images and text. While CLIP has enabled many recent breakthroughs in computer vision, its associated training cost is prohibitively high, making it challenging to explore its full potential. The authors discovered an inverse scaling law for CLIP training, whereby using larger image/text encoders leads to shorter sequence lengths of image/text tokens that can be applied in training. They also found that the strategy for reducing image/text token length plays a crucial role in determining the quality of this scaling law. As a result of this discovery, the authors were able to train CLIP successfully even using academic resources. For instance, on an A100 eight-GPU server, their CLIP models achieved zero-shot top-1 ImageNet accuracies of 63.2% in approximately two days, 67.8% in about three days, and 69.3% in roughly four days. This research could have significant implications for computer vision researchers who may now be able to explore the full potential of CLIP without being hindered by high training costs. By reducing the computation barrier associated with CLIP through this finding, they hope to inspire more research in this field from academics. The authors' code is available at https://github.com/UCSC-VLAA/CLIPA for anyone interested in exploring this further.