# SCULPT: Shape-Conditioned Unpaired Learning of Pose-dependent Clothed and Textured Human Meshes

**Authors:** Soubhik Sanyal, Partha Ghosh, Jinlong Yang, Michael J. Black, Justus Thies, Timo Bolkart  
**arXiv:** [2308.10638](https://arxiv.org/abs/2308.10638)  
**Date:** 2023-08-21  
**Summary page:** https://www.summarizepaper.com/en/paper/2308.10638v1/

## Key points

- Authors introduce SCULPT model for creating pose-dependent clothed and textured human meshes
- Model uses deep neural network to capture geometry and appearance distribution of clothed human bodies
- Challenges in training due to limited availability of datasets addressed by leveraging medium-sized 3D scan datasets like CAPE and large-scale 2D image datasets
- Proposed approach involves developing a geometry conditioned texture generator using both 3D scan data and 2D image data
- Attribute labels such as clothing types for geometry and clothing colors for texture generation used to disentangle pose from clothing type and appearance
- Conditioning labels automatically generated for 2D images based on visual question answering models BLIP and CLIP
- Method validated on SCULPT dataset and compared against state-of-the-art 3D generative models for clothed human bodies
- Codebase to be released for research purposes to facilitate further exploration in the field

## Summary

In their paper titled "SCULPT: Shape-Conditioned Unpaired Learning of Pose-dependent Clothed and Textured Human Meshes," authors Soubhik Sanyal, Partha Ghosh, Jinlong Yang, Michael J. Black, Justus Thies, and Timo Bolkart introduce a groundbreaking  for creating . The model utilizes a deep neural network to capture the geometry and appearance distribution of clothed human bodies. One of the main challenges faced in training such a model is the limited availability of datasets containing textured 3D meshes for humans. To address this issue, the authors leverage medium-sized 3D scan datasets like CAPE and large-scale 2D image datasets of clothed humans, demonstrating that multiple appearances can be mapped to a single geometry. To effectively learn from these two data modalities, the authors propose an  approach for . They first develop a  using 3D scan data by representing it as per vertex displacements with respect to the SMPL model. Subsequently, they train a geometry conditioned texture generator in an unsupervised manner using 2D image data. Intermediate activations from the learned geometry model are used to condition the texture generator. To disentangle pose from clothing type and appearance, attribute labels such as clothing types for geometry and clothing colors for texture generation are employed as conditioning factors. Furthermore, the authors automatically generate these conditioning labels for 2D images based on the visual question answering models BLIP and CLIP. The proposed method is validated on the SCULPT dataset and compared against state-of-the-art 3D generative models for clothed human bodies. The authors plan to release their codebase for research purposes to facilitate further exploration in this field. Overall, this innovative approach presents a significant advancement in generating realistic clothed and textured human meshes through , showcasing potential applications in various fields such as virtual try-on systems, gaming industry, virtual reality environments, and more.