# From Audio to Photoreal Embodiment: Synthesizing Humans in Conversations

**Authors:** Evonne Ng, Javier Romero, Timur Bagautdinov, Shaojie Bai, Trevor Darrell, Angjoo Kanazawa, Alexander Richard  
**arXiv:** [2401.01885](https://arxiv.org/abs/2401.01885)  
**Date:** 2024-01-03  
**Summary page:** https://www.summarizepaper.com/en/paper/2401.01885v1/

## Key points

- Framework for generating full-bodied photorealistic avatars that can gesture according to conversational dynamics
- Combination of vector quantization and diffusion for more dynamic and expressive motion
- Generated motion includes gestures of the face, body, and hands
- Introduction of a multi-view conversational dataset for photorealistic reconstruction
- Experiments show model outperforms diffusion-only and vector quantization-only methods
- Importance of photorealism in accurately assessing subtle motion details in conversational gestures
- Code and dataset available online for further exploration by other researchers

## Summary

In their paper titled "From Audio to Photoreal Embodiment: Synthesizing Humans in Conversations," authors Evonne Ng, Javier Romero, Timur Bagautdinov, Shaojie Bai, Trevor Darrell, Angjoo Kanazawa, and Alexander Richard present a framework for generating full-bodied photorealistic avatars that can gesture according to the conversational dynamics of a dyadic interaction. The goal is to create avatars that can express crucial nuances in gestures such as sneers and smirks. The proposed method combines the benefits of sample diversity from vector quantization with the high-frequency details obtained through diffusion. By doing so, they are able to generate more dynamic and expressive motion for the avatars. The generated motion includes gestures of the face, body, and hands. , , , , and  are all key components of this research. To support their work, the authors introduce a first-of-its-kind multi-view conversational dataset that allows for photorealistic reconstruction. This dataset enables the visualization of the generated motion using highly photorealistic avatars. Experiments conducted by the authors demonstrate that their model generates appropriate and diverse gestures, outperforming both diffusion-only and vector quantization-only methods. Additionally, a perceptual evaluation highlights the importance of photorealism in accurately assessing subtle motion details in conversational gestures compared to traditional mesh-based approaches. The code and dataset associated with this research are made available online for further exploration and use by other researchers in this field. Overall, that can effectively communicate through realistic gestural motions.