# Gaussian Grouping: Segment and Edit Anything in 3D Scenes

**Authors:** Mingqiao Ye, Martin Danelljan, Fisher Yu, Lei Ke  
**arXiv:** [2312.00732](https://arxiv.org/abs/2312.00732)  
**Date:** 2023-12-01  
**Summary page:** https://www.summarizepaper.com/en/paper/2312.00732v1/

## Key points

- Gaussian Splatting allows for high-quality and real-time synthesis of novel views in 3D scenes
- Gaussian Splatting lacks fine-grained object-level scene understanding
- Gaussian Grouping is a new approach that enables joint reconstruction and segmentation of objects in open-world 3D scenes
- Each Gaussian in Gaussian Grouping is augmented with a compact Identity Encoding for grouping based on object instance or stuff membership
- Identity Encodings are supervised during differentiable rendering using 2D mask predictions from SAM and incorporating 3D spatial consistency regularization
- Discrete and grouped 3D Gaussians offer advantages over implicit NeRF representations, including high visual quality, fine granularity, and efficiency

## Summary

The recent development of Gaussian Splatting has allowed for high-quality and real-time synthesis of novel views in 3D scenes. However, this method primarily focuses on appearance and geometry modeling, lacking in fine-grained object-level scene understanding. To address this limitation, we propose a new approach called Gaussian Grouping. This extension of Gaussian Splatting enables the joint reconstruction and segmentation of objects in open-world 3D scenes. In our method, each Gaussian is augmented with a compact Identity Encoding, which allows for grouping based on object instance or stuff membership within the scene. Unlike other methods that rely on expensive 3D labels, we supervise the Identity Encodings during differentiable rendering by leveraging 2D mask predictions from SAM (Segment Anything Model) and incorporating 3D spatial consistency regularization. Compared to implicit NeRF representations, our discrete and grouped 3D Gaussians offer several advantages. They can reconstruct, segment, and edit objects in 3D scenes with high visual quality, fine granularity, and efficiency.