# FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects

**Authors:** Bowen Wen, Wei Yang, Jan Kautz, Stan Birchfield  
**arXiv:** [2312.08344](https://arxiv.org/abs/2312.08344)  
**Date:** 2023-12-13  
**Summary page:** https://www.summarizepaper.com/en/paper/2312.08344v2/

## Key points

- FoundationPose is a unified model for 6D object pose estimation and tracking, supporting both model-based and model-free setups.
- The model uses a neural implicit representation to bridge the gap between these setups, enabling effective novel view synthesis and ensuring downstream modules remain invariant.
- A novel synthetic data generation pipeline was developed using 3D model databases, large language models (LLMs), and diffusion models to facilitate large-scale training without manual effort.
- Object-centric neural field for RGBD rendering enables render-and-compare processes in both model-free and model-based scenarios.
- LLM-aided texture augmentation enhances object textures in a realistic and automatic manner by generating textured models with text prompts, object shapes, and noisy textures.

## Summary

, , , , 

The researchers introduce FoundationPose, a unified foundation model for 6D object pose estimation and tracking. This model supports both model-based and model-free setups, allowing for instant application to novel objects without the need for fine-tuning. The approach bridges the gap between these two setups through a neural implicit representation that enables effective novel view synthesis, ensuring that downstream pose estimation modules remain invariant under a unified framework. To facilitate large-scale training without extensive manual effort, the researchers developed a novel synthetic data generation pipeline leveraging techniques such as 3D model databases, large language models (LLMs), and diffusion models. Additionally, they implemented an object-centric neural field for RGBD rendering to enable render-and-compare processes in both model-free and model-based scenarios. The pose estimation process involves initializing global poses uniformly around the object, refining them using a network, and selecting the best pose based on predicted scores. Furthermore, LLM-aided texture augmentation was employed to enhance object textures in a more realistic and automatic manner compared to previous methods. By utilizing recent advancements in large language models and diffusion models, textured models were generated by providing text prompts along with object shapes and noisy textures. A hierarchical prompt strategy was introduced to streamline this process for augmenting diverse objects with different styles under various prompt guidance. Extensive evaluations on multiple public datasets were conducted to demonstrate the superior performance of FoundationPose compared to existing specialized methods in challenging scenarios involving various objects. Despite reduced assumptions, FoundationPose achieved comparable results to instance-level methods while showcasing strong generalizability through large-scale synthetic training. In summary, this research presents a versatile and high-performing foundation model for 6D pose estimation and tracking of novel objects, with potential future applications in state estimation beyond single rigid objects.