In this paper, the authors introduce a new holistic perceptual metric called DreamSim to address the limitations of current perceptual similarity metrics. Their motivation stems from the importance of perceiving similarities between images in various ways, including higher-level concepts such as object pose and semantic content. Existing metrics like PSNR and SSIM are limited in capturing these higher-level similarities, leading researchers to explore image-level embeddings from large vision models like DINO and CLIP. To bridge this gap, the authors collect a new dataset named NIGHTS (Novel Image Generations with Human-Tested Similarity) containing human similarity judgments over image triplets. This dataset focuses on mid-level similarities rather than just low-level perturbations or high-level category differences. The authors find that features from recent large pre-trained vision models outperform standard perceptual metrics on their dataset and develop DreamSim by tuning these models on their data. DreamSim aligns better with human perception and demonstrates high agreement in both quantitative assessments and qualitative comparisons using real images. It heavily considers foreground objects while also taking color and layout into account compared to previous metrics or modern image embeddings. In summary, this work expands the task of measuring perceptual image similarity to encompass factors beyond low-level similarities and provides a new metric that effectively captures mid-level similarities. The NIGHTS dataset and DreamSim contribute significantly to advancing our understanding of human visual perception and can be integrated into existing pipelines for tasks such as image retrieval and synthesis.
- Introduction of a new holistic perceptual metric called DreamSim to address limitations of current perceptual similarity metrics
- Motivation for perceiving similarities between images in various ways, including higher-level concepts like object pose and semantic content
- Development of the NIGHTS dataset containing human similarity judgments over image triplets focusing on mid-level similarities
- Use of features from large pre-trained vision models to outperform standard perceptual metrics on the NIGHTS dataset
- Creation of DreamSim by tuning these models on their data to align better with human perception
- Consideration of foreground objects, color, and layout in DreamSim compared to previous metrics or modern image embeddings
- Expansion of measuring perceptual image similarity to encompass factors beyond low-level similarities with DreamSim
- Contribution of the NIGHTS dataset and DreamSim to advancing understanding of human visual perception for tasks like image retrieval and synthesis
Summary1. A new way to measure how similar images are called DreamSim was created to improve existing methods.
2. People want to compare images in different ways, like looking at objects and what they mean.
3. A special dataset called NIGHTS was made where people said how similar groups of images were.
4. Big vision models were used to make DreamSim better than other methods on the NIGHTS dataset.
5. DreamSim was made by adjusting these models to match how people see things.
Definitions- Perceptual: How we see and understand things with our senses.
- Similarity: How much two things are alike or resemble each other.
- Dataset: A collection of data or information for research or analysis.
- Features: Specific characteristics or qualities of something.
- Perception: How we interpret and understand the world around us.
Introduction
Perceiving similarities between images is a crucial task in computer vision, with applications ranging from image retrieval to image synthesis. However, current perceptual similarity metrics such as PSNR and SSIM have limitations in capturing higher-level similarities between images. This has led researchers to explore image-level embeddings from large vision models like DINO and CLIP. In this paper, the authors introduce a new holistic perceptual metric called DreamSim to address these limitations and expand the task of measuring perceptual image similarity.
Motivation
The motivation for this research stems from the importance of perceiving similarities between images in various ways, including higher-level concepts such as object pose and semantic content. Existing metrics like PSNR and SSIM are limited in capturing these higher-level similarities, leading researchers to explore alternative methods for measuring perceptual similarity. The authors argue that there is a need for a more comprehensive metric that considers mid-level similarities rather than just low-level perturbations or high-level category differences.
Dataset Collection
To address this gap, the authors collect a new dataset named NIGHTS (Novel Image Generations with Human-Tested Similarity) containing human similarity judgments over image triplets. This dataset focuses on mid-level similarities rather than just low-level perturbations or high-level category differences. The authors carefully select 1,000 unique images from diverse categories such as animals, landscapes, objects, people, etc., resulting in 10 million possible triplets for comparison.
Development of DreamSim
The authors find that features from recent large pre-trained vision models outperform standard perceptual metrics on their dataset. They then develop DreamSim by tuning these models on their data to align better with human perception. DreamSim takes into account factors beyond low-level similarities and heavily considers foreground objects while also taking color and layout into account compared to previous metrics or modern image embeddings.
Evaluation
To evaluate the effectiveness of DreamSim, the authors conduct both quantitative assessments and qualitative comparisons using real images. They compare DreamSim with existing metrics such as PSNR, SSIM, and modern image embeddings like DINO and CLIP. The results show that DreamSim outperforms all other metrics in capturing mid-level similarities between images.
Significance
This work expands the task of measuring perceptual image similarity to encompass factors beyond low-level similarities. By considering mid-level similarities, DreamSim provides a more comprehensive measure of image similarity that aligns better with human perception. This is a significant contribution to advancing our understanding of human visual perception and can be integrated into existing pipelines for tasks such as image retrieval and synthesis.
Conclusion
In conclusion, the authors introduce a new holistic perceptual metric called DreamSim to address the limitations of current perceptual similarity metrics. Their motivation stems from the importance of perceiving similarities between images in various ways, including higher-level concepts such as object pose and semantic content. The NIGHTS dataset and DreamSim contribute significantly to advancing our understanding of human visual perception and provide a new metric that effectively captures mid-level similarities between images. This research has implications for various applications in computer vision and sets the stage for further exploration into measuring perceptual image similarity beyond traditional methods.
