# Search3D: Hierarchical Open-Vocabulary 3D Segmentation

**Authors:** Ayca Takmaz, Alexandros Delitzas, Robert W. Sumner, Francis Engelmann, Johanna Wald, Federico Tombari  
**arXiv:** [2409.18431](https://arxiv.org/abs/2409.18431)  
**Date:** 2024-09-27  
**Summary page:** https://www.summarizepaper.com/en/paper/2409.18431v1/

## Key points

- Search3D is a novel approach introduced by Ayca Takmaz, Alexandros Delitzas, Robert W. Sumner, Francis Engelmann, Johanna Wald, and Federico Tombari focusing on hierarchical open-vocabulary 3D segmentation.
- The method enables the search for entities at varying levels of granularity within a scene through free-form text descriptions.
- Search3D goes beyond traditional open-vocabulary 3D instance segmentation by addressing fine-grained scene entities like object parts and regions described by generic attributes such as materials.
- By building a hierarchical open-vocabulary 3D scene representation, Search3D offers flexible searching capabilities less anchored to explicit object-centric queries.
- The authors introduce a scene-scale open-vocabulary 3D part segmentation benchmark based on MultiScan to systematically evaluate their method.
- They provide open-vocabulary fine-grained part annotations on ScanNet++ to validate the effectiveness of Search3D across various tasks.
- Through rigorous testing and comparison with baselines, the authors demonstrate that Search3D outperforms existing methods in scene-scale open-vocabulary 3D part segmentation while maintaining strong performance in segmenting 3D objects and materials.

## Summary

Search3D is a novel approach introduced by Ayca Takmaz, Alexandros Delitzas, Robert W. Sumner, Francis Engelmann, Johanna Wald, and Federico Tombari that focuses on hierarchical open-vocabulary 3D segmentation. The method aims to enhance the exploration of 3D spaces through free-form text descriptions by enabling the search for entities at varying levels of granularity within a scene. Existing methods in open-vocabulary 3D instance segmentation primarily concentrate on identifying object-level instances; however, Search3D goes beyond this limitation by addressing more fine-grained scene entities such as object parts and regions described by generic attributes like materials. By building a hierarchical open-vocabulary 3D scene representation, Search3D allows for flexible searching capabilities that are less anchored to explicit object-centric queries compared to previous approaches. This shift towards a more adaptable open-vocabulary 3D search setting expands the capabilities of instance-level 3D segmentation and offers a more comprehensive understanding of complex scenes. To ensure a systematic evaluation of their method, the authors also introduce a scene-scale open-vocabulary 3D part segmentation benchmark based on MultiScan. Additionally, they provide a set of open-vocabulary fine-grained part annotations on ScanNet++, further validating the effectiveness of Search3D across various tasks. Through rigorous testing and comparison with baselines, the authors demonstrate that Search3D outperforms existing methods in scene-scale open-vocabulary 3D part segmentation while maintaining strong performance in segmenting 3D objects and materials. This work has been submitted to IEEE for possible publication, showcasing its potential impact on advancing research in the field of computer vision and .