AG3D: Learning to Generate 3D Avatars from 2D Image Collections

AI-generated keywords: Generative Model 3D Avatars 2D Image Collections Adversarial Generative Model Discriminators

AI-generated Key Points

  • Zijian Dong and Xu Chen propose a novel method for generating realistic 3D avatars from unstructured 2D image collections.
  • The authors' approach is to learn generative models of 3D avatars from abundant unstructured 2D image collections, which can capture shape and deformation of the body and loose clothing.
  • The proposed method outperforms previous methods in terms of geometry and appearance, as demonstrated through systematic ablation studies.
  • The authors' generator design uses a monolithic approach that models humans holistically in a canonical space using an efficient tri-plane representation.
  • The proposed method also incorporates multiple discriminators specialized for improving geometric detail as well as perceptually important regions like the face.
  • Normal information is used for guiding geometry in the generative setting by discriminating normal maps rendered from their generative model against off-the-shelf monocular estimators applied to images of human subjects.
  • The authors contribute a generative model of articulated 3D humans with state-of-the-art appearance and geometry, a new generator that is efficient and can generate and deform loose clothing, and several specialized discriminators that significantly improve visual and geometric fidelity.
Also access our AI generated: Comprehensive summary, Lay summary, Blog-like article; or ask questions about this paper to our AI assistant.

Authors: Zijian Dong, Xu Chen, Jinlong Yang, Michael J. Black, Otmar Hilliges, Andreas Geiger

Project Page: https://zj-dong.github.io/AG3D/
License: CC BY 4.0

Abstract: While progress in 2D generative models of human appearance has been rapid, many applications require 3D avatars that can be animated and rendered. Unfortunately, most existing methods for learning generative models of 3D humans with diverse shape and appearance require 3D training data, which is limited and expensive to acquire. The key to progress is hence to learn generative models of 3D avatars from abundant unstructured 2D image collections. However, learning realistic and complete 3D appearance and geometry in this under-constrained setting remains challenging, especially in the presence of loose clothing such as dresses. In this paper, we propose a new adversarial generative model of realistic 3D people from 2D images. Our method captures shape and deformation of the body and loose clothing by adopting a holistic 3D generator and integrating an efficient and flexible articulation module. To improve realism, we train our model using multiple discriminators while also integrating geometric cues in the form of predicted 2D normal maps. We experimentally find that our method outperforms previous 3D- and articulation-aware methods in terms of geometry and appearance. We validate the effectiveness of our model and the importance of each component via systematic ablation studies.

Submitted to arXiv on 03 May. 2023

Ask questions about this paper to our AI assistant

You can also chat with multiple papers at once here.

AI assistant instructions?

Results of the summarizing process for the arXiv paper: 2305.02312v1

In their paper titled "AG3D: Learning to Generate 3D Avatars from 2D Image Collections," Zijian Dong and Xu Chen propose a novel method for generating realistic 3D avatars from unstructured 2D image collections. While there has been significant progress in generative models of human appearance in 2D, creating animated and rendered 3D avatars remains a challenge due to the limited availability and high cost of acquiring 3D training data. The authors' approach is to learn generative models of 3D avatars from abundant unstructured 2D image collections, which can capture shape and deformation of the body and loose clothing. To achieve this goal, the authors propose a new adversarial generative model that captures shape and deformation using a holistic 3D generator and an efficient articulation module. They also integrate multiple discriminators to improve realism by focusing on perceptually important regions such as the face while incorporating geometric cues in the form of predicted 2D normal maps. The proposed method outperforms previous methods in terms of geometry and appearance, as demonstrated through systematic ablation studies. The authors' generator design uses a monolithic approach that models humans holistically in a canonical space using an efficient tri-plane representation. To enable fast volume rendering, they adapt the efficient articulation module Fast-SNARF to their generative setting while further accelerating rendering via empty-space skipping informed by a coarse human body prior. This approach is more flexible than prior methods that base deformations on SMPL, enabling it to model deformations for points far away from the body. The proposed method also incorporates multiple discriminators specialized for improving geometric detail as well as perceptually important regions like the face. Normal information is used for guiding geometry in the generative setting by discriminating normal maps rendered from their generative model against off-the-shelf monocular estimators applied to images of human subjects. The authors demonstrate that this additional normal supervision significantly improves the quality of generated 3D shapes. In summary, the authors contribute a generative model of articulated 3D humans with state-of-the-art appearance and geometry, a new generator that is efficient and can generate and deform loose clothing, and several specialized discriminators that significantly improve visual and geometric fidelity. They plan to release code and models for their method.
Created on 04 May. 2023

Assess the quality of the AI-generated content by voting

Score: 0

Why do we need votes?

Votes are used to determine whether we need to re-run our summarizing tools. If the count reaches -10, our tools can be restarted.

Similar papers summarized with our AI tools

Navigate through even more similar papers through a

tree representation

Look for similar papers (in beta version)

By clicking on the button above, our algorithm will scan all papers in our database to find the closest based on the contents of the full papers and not just on metadata. Please note that it only works for papers that we have generated summaries for and you can rerun it from time to time to get a more accurate result while our database grows.

Disclaimer: The AI-based summarization tool and virtual assistant provided on this website may not always provide accurate and complete summaries or responses. We encourage you to carefully review and evaluate the generated content to ensure its quality and relevance to your needs.