# Sigmoid Loss for Language Image Pre-Training

**Authors:** Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, Lucas Beyer  
**arXiv:** [2303.15343](https://arxiv.org/abs/2303.15343)  
**Date:** 2023-03-27  
**Summary page:** https://www.summarizepaper.com/en/paper/2303.15343v1/

## Key points

- Proposal of a simpler alternative to standard contrastive learning with softmax normalization in the form of a sigmoid loss
- Decoupling batch size from task definition for scaling up batch sizes and better performance at smaller batch sizes
- Comparison of sigmoid loss with standard softmax loss in CLIP and LiT setups, showing superiority of sigmoid loss for batch sizes smaller than 16k
- Simplicity and efficiency of distributed implementation of sigmoid loss without needing operations across the full batch
- Successful training of SigLiT model at a large batch size of one million due to symmetric nature of sigmoid loss
- Importance of disentangling batch size from loss function for training efficiency and quality in language-image pre-training tasks

## Summary

, , , , 

In their paper titled "Sigmoid Loss for Language Image Pre-Training," Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer propose a simpler alternative to standard contrastive learning with softmax normalization in the form of a sigmoid loss. This loss operates solely on image-text pairs and eliminates the need for a global view of pairwise similarities for normalization. By decoupling the batch size from the task definition, the sigmoid loss allows for further scaling up of batch sizes while also performing better at smaller batch sizes. The authors compare the proposed sigmoid loss with the standard softmax loss across various setups, particularly focusing on two prominent approaches for image-text learning: CLIP and LiT. They introduce sigmoid language image pre-training (SigLIP) and sigmoid LiT (SigLiT), finding that the sigmoid loss outperforms the softmax loss significantly when the batch size is smaller than 16k. As the train batch size increases, however, the performance gap between the two losses diminishes. One key advantage of the sigmoid loss is its simplicity and efficiency in distributed loss implementation. It requires no operation across the full batch, simplifying implementation and boosting efficiency. The symmetric nature of the sigmoid loss allows for successful training of a SigLiT model at a large batch size of one million. Overall, this research sheds light on how disentangling batch size from loss function can impact training efficiency and quality in language-image pre-training tasks. The findings suggest that while growing batch sizes can offer benefits up to a certain point, more reasonable batch sizes may be sufficient to achieve similar quality results without sacrificing efficiency. This study paves the way for further exploration into improving both quality and efficiency in language-image pre-training methodologies.