# To React or not to React: End-to-End Visual Pose Forecasting for Personalized Avatar during Dyadic Conversations

**Authors:** Chaitanya Ahuja, Shugao Ma, Louis-Philippe Morency, Yaser Sheikh  
**arXiv:** [1910.02181](https://arxiv.org/abs/1910.02181)  
**Date:** 2019-10-05  
**Summary page:** https://www.summarizepaper.com/en/paper/1910.02181v1/

## Key points

- Nonverbal behaviors play a crucial role in improving telepresence through personalized avatars
- Gestures, facial expressions, body posture, and para-linguistic cues complement verbal messages
- The Dyadic Residual-Attention Model (DRAM) integrates intrapersonal and interpersonal dynamics using selective attention mechanisms
- DRAM generates sequences of body poses conditioned on audio inputs and the interlocutor's pose and audio
- Adaptive attention between monadic and dyadic dynamics improves avatar pose prediction
- Evaluation results confirm the significance of adaptive attention in accurately predicting avatar pose
- User study shows that DRAM produces more natural body poses capturing both individual and interactive aspects of communication better than non-adaptive models.

## Summary

The paper titled "To React or not to React: End-to-End Visual Pose Forecasting for Personalized Avatar during Dyadic Conversations" explores the importance of nonverbal behaviors in improving telepresence through personalized avatars. Nonverbal cues such as gestures, facial expressions, body posture, and para-linguistic cues have been shown to complement and clarify verbal messages. To create more realistic avatars, it is crucial to model these behaviors, especially in dyadic interactions. The authors propose a neural architecture called the Dyadic Residual-Attention Model (DRAM) that integrates both intrapersonal (monadic) and interpersonal (dyadic) dynamics using selective attention mechanisms. The model generates sequences of body poses conditioned on audio inputs and the body pose of the interlocutor, as well as the audio of the human operating the avatar. By incorporating adaptive attention between monadic and dyadic dynamics, DRAM aims to improve the prediction of avatar pose. To evaluate their proposed model, the authors use dyadic conversational data consisting of pose and audio recordings from both participants. The results confirm the significance of adaptive attention in predicting avatar pose accurately. Additionally, a user study is conducted to analyze judgments made by human observers. The findings demonstrate that the generated body poses are more natural and better capture both intrapersonal and interpersonal dynamics compared to non-adaptive monadic/dyadic models. In conclusion, this paper highlights the importance of modeling nonverbal behaviors in personalized avatars for enhancing telepresence during dyadic conversations. The Dyadic Residual-Attention Model (DRAM) presented in this study effectively integrates intrapersonal and interpersonal dynamics using selective attention mechanisms. The evaluation results and user study confirm that DRAM produces more realistic body poses while capturing both individual and interactive aspects of communication better than existing models without adaptive attention.