# Open3DIS: Open-Vocabulary 3D Instance Segmentation with 2D Mask Guidance

**Authors:** Phuc D. A. Nguyen, Tuan Duc Ngo, Evangelos Kalogerakis, Chuang Gan, Anh Tran, Cuong Pham, Khoi Nguyen  
**arXiv:** [2312.10671](https://arxiv.org/abs/2312.10671)  
**Date:** 2023-12-17  
**Summary page:** https://www.summarizepaper.com/en/paper/2312.10671v3/

## Key points

- Open3DIS is a cutting-edge solution for object identification in diverse 3D environments
- The challenge involves accurately identifying objects with varying shapes, sizes, and colors at the instance level
- Introduces a new module that aggregates 2D instance masks across frames and maps them to geometrically coherent point cloud regions as high-quality object proposals
- Combining refined proposals with class-agnostic 3D proposals from ISBNet leads to significant performance improvements on datasets like ScanNet200, S3DIS, and Replica
- Outperforms previous methods like OVIR-3D and OpenMask3D on the ScanNet200 dataset in terms of Average Precision (AP) and APtail metrics
- Achieves notable enhancement in AP compared to existing approaches by incorporating both 2D and class-agnostic 3D proposals
- Competes closely with fully supervised techniques on various metrics, demonstrating effectiveness in segmenting rare objects
- Showcased superior performance on different datasets such as ScanNet20 and Replica compared to other state-of-the-art methods across novel and base classes
- Even under zero-shot scenarios on the Replica dataset without using class-agnostic 3D proposals, outperformed competing methods like OpenMask3D and OVIR-3D

## Summary

In this study, we present Open3DIS, a cutting-edge solution for  in diverse 3D environments. The challenge lies in accurately identifying objects with varying shapes, sizes, and colors at the instance level. Previous advancements in  have utilized class-agnostic 3D instance proposal networks to localize objects and learn queryable features for each 3D mask. While these methods have shown promise in generating high-quality instance proposals, they struggle with identifying small-scale and geometrically ambiguous objects. Our novel approach introduces a new module that aggregates 2D instance masks across frames and maps them to geometrically coherent point cloud regions as high-quality object proposals. This innovative method overcomes the limitations of existing techniques by providing precise 3D instance masks independently of any pre-existing 3D models. By combining these refined proposals with class-agnostic 3D proposals from ISBNet, our model achieves significant performance improvements on prominent datasets such as ScanNet200, S3DIS, and Replica. Specifically, on the ScanNet200 dataset, our Open3DIS outperforms previous methods like OVIR-3D and OpenMask3D by substantial margins in terms of Average Precision (AP) and APtail metrics. By incorporating both  and class-agnostic 3D proposals, we achieve a notable enhancement in AP compared to existing approaches. Furthermore, our method competes closely with fully supervised techniques on various metrics, demonstrating its effectiveness in segmenting rare objects. To assess the generalizability of our approach, on different datasets such as ScanNet20 and Replica. In both cases, Open3DIS showcased superior performance compared to other state-of-the-art methods across novel and base classes. Even under zero-shot scenarios on the Replica dataset without using class-agnostic 3D proposals, our approach still outperformed competing methods like OpenMask3D and OVIR-3D. Overall, our study highlights the effectiveness of merging 2D and 3D proposals for improved object segmentation in diverse real-world environments. The results demonstrate the robustness and versatility of Open3DIS in accurately identifying objects with varying characteristics across different datasets.