# GraphDreamer: Compositional 3D Scene Synthesis from Scene Graphs

**Authors:** Gege Gao, Weiyang Liu, Anpei Chen, Andreas Geiger, Bernhard Schölkopf  
**arXiv:** [2312.00093](https://arxiv.org/abs/2312.00093)  
**Date:** 2023-11-30  
**Summary page:** https://www.summarizepaper.com/en/paper/2312.00093v1/

## Key points

- Utilization of pretrained text-to-image diffusion models in text-guided 3D modeling has gained traction
- Limitation of existing methods in handling complex textual descriptions with multiple objects due to inability of vectorized text embeddings to capture intricate relationships
- Introduction of GraphDreamer framework focusing on generating compositional 3D scenes from structured scene graphs
- Use of signed distance fields by GraphDreamer for representing objects and preventing object inter-penetration
- Novel text prompt designed for ChatGPT to automatically generate structured representations based on textual inputs
- Extensive experiments validating the efficacy of GraphDreamer in producing high-fidelity compositional 3D scenes with disentangled object entities
- Advancement in text-guided 3D modeling research through GraphDreamer's ability to accurately translate complex textual descriptions into detailed and coherent 3D visualizations

## Summary

In the realm of <kw>text-guided 3D modeling,</kw> the utilization of <kw>pretrained text-to-image diffusion models</kw> has gained significant traction due to their enhanced capabilities. However, existing methods often fall short when faced with complex textual descriptions depicting scenes with multiple objects. This limitation stems from the inherent inability of vectorized text embeddings to capture intricate relationships between entities within a scene. To address these challenges, a groundbreaking framework known as <kw>GraphDreamer</kw> has been introduced. This innovative approach focuses on generating compositional 3D scenes from structured scene graphs, where objects are represented as nodes and their interactions as edges. By leveraging the rich node and edge information encoded in scene graphs, GraphDreamer optimally harnesses pretrained text-to-image diffusion models to disentangle different objects without relying on image-level supervision. One key aspect of GraphDreamer is its use of <kw>signed distance fields</kw> for representing objects and enforcing constraints to prevent object inter-penetration. This enables the model to effectively capture object-wise relationships within the scene while maintaining spatial coherence. Additionally, to streamline the process of creating scene graphs, a novel text prompt has been designed for ChatGPT to automatically generate structured representations based on textual inputs. Extensive qualitative and quantitative experiments have been conducted to validate the efficacy of GraphDreamer in producing high-fidelity compositional 3D scenes with disentangled object entities. The results showcase the model's ability to accurately translate complex textual descriptions into detailed and coherent 3D visualizations, marking a significant advancement in <kw>text-guided 3D modeling</kw> research. Overall, GraphDreamer represents a pioneering step towards enhancing the interpretability and fidelity of 3D scene synthesis by incorporating <kw>structured scene graphs</kw> and leveraging advanced deep learning techniques for optimal knowledge distillation from pretrained models.