# DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Data

**Authors:** Stephanie Fu, Netanel Tamir, Shobhita Sundaram, Lucy Chai, Richard Zhang, Tali Dekel, Phillip Isola  
**arXiv:** [2306.09344](https://arxiv.org/abs/2306.09344)  
**Date:** 2023-06-15  
**Summary page:** https://www.summarizepaper.com/en/paper/2306.09344v1/

## Key points

- Introduction of a new holistic perceptual metric called DreamSim to address limitations of current perceptual similarity metrics
- Motivation for perceiving similarities between images in various ways, including higher-level concepts like object pose and semantic content
- Development of the NIGHTS dataset containing human similarity judgments over image triplets focusing on mid-level similarities
- Use of features from large pre-trained vision models to outperform standard perceptual metrics on the NIGHTS dataset
- Creation of DreamSim by tuning these models on their data to align better with human perception
- Consideration of foreground objects, color, and layout in DreamSim compared to previous metrics or modern image embeddings
- Expansion of measuring perceptual image similarity to encompass factors beyond low-level similarities with DreamSim
- Contribution of the NIGHTS dataset and DreamSim to advancing understanding of human visual perception for tasks like image retrieval and synthesis

## Summary

In this paper, the authors introduce a new holistic perceptual metric called DreamSim to address the limitations of current perceptual similarity metrics. Their motivation stems from the importance of perceiving similarities between images in various ways, including higher-level concepts such as object pose and semantic content. Existing metrics like PSNR and SSIM are limited in capturing these higher-level similarities, leading researchers to explore image-level embeddings from large vision models like DINO and CLIP. To bridge this gap, the authors collect a new dataset named NIGHTS (Novel Image Generations with Human-Tested Similarity) containing human similarity judgments over image triplets. This dataset focuses on mid-level similarities rather than just low-level perturbations or high-level category differences. The authors find that features from recent large pre-trained vision models outperform standard perceptual metrics on their dataset and develop DreamSim by tuning these models on their data. DreamSim aligns better with human perception and demonstrates high agreement in both quantitative assessments and qualitative comparisons using real images. It heavily considers foreground objects while also taking color and layout into account compared to previous metrics or modern image embeddings. In summary, this work expands the task of measuring perceptual image similarity to encompass factors beyond low-level similarities and provides a new metric that effectively captures mid-level similarities. The NIGHTS dataset and DreamSim contribute significantly to advancing our understanding of human visual perception and can be integrated into existing pipelines for tasks such as image retrieval and synthesis.