# Automated Deception Detection from Videos: Using End-to-End Learning Based High-Level Features and Classification Approaches

**Authors:** Laslo Dinges, Marc-André Fiedler, Ayoub Al-Hamadi, Thorsten Hempel, Ahmed Abdelrahman, Joachim Weimann, Dmitri Bershadskyy  
**arXiv:** [2307.06625](https://arxiv.org/abs/2307.06625)  
**Date:** 2023-07-13  
**Summary page:** https://www.summarizepaper.com/en/paper/2307.06625v1/

## Key points

- Deception detection is a multidisciplinary field attracting researchers from various disciplines
- The study proposes a multimodal approach combining deep learning and discriminative models for automated deception detection from videos
- Video modalities are used to analyze gaze, head pose, and facial expressions using convolutional end-to-end learning
- The approach yields promising results compared to state-of-the-art methods
- Discriminative models are employed due to limited training data, outperforming sequence-to-class approaches
- Experiments are conducted on five datasets, including a new Rolling-Dice Experiment motivated by economic factors
- Facial expressions are found to be more effective than gaze and head pose in detecting deception
- Combining multiple modalities with feature selection enhances overall detection performance
- Scenario-specific training data and contextual factors influence deceptive behavior
- Differences in expressed features across datasets support the findings
- Cross-dataset experiments reinforce the effectiveness of the proposed approach
- Deception detection performance exceeds chance levels even with low-stake datasets like the Rolling-Dice Experiment
- Automating deception detection from video modalities holds significant potential

## Summary

Deception detection is a multidisciplinary field that attracts researchers from various disciplines such as psychology, criminology, computer science, and economics. In this study, the authors propose a multimodal approach that combines deep learning and discriminative models for automated deception detection from videos. The researchers utilize video modalities to analyze gaze, head pose, and facial expressions using convolutional end-to-end learning. This approach yields promising results compared to state-of-the-art methods. However, due to limited training data, they also employ discriminative models for deception detection. Although sequence-to-class approaches are explored, the discriminative models outperform them due to the scarcity of data. To evaluate their approach, the authors conduct experiments on five datasets including a new Rolling-Dice Experiment motivated by economic factors. The results indicate that facial expressions are more effective than gaze and head pose in detecting deception. Additionally, combining multiple modalities with feature selection enhances the overall detection performance. The study highlights the importance of scenario-specific training data and emphasizes how contextual factors influence deceptive behavior. The differences in expressed features across datasets further support these findings. Cross-dataset experiments reinforce the effectiveness of the proposed approach. Despite the challenges posed by low-stake datasets like the Rolling-Dice Experiment, the performance of deception detection exceeds chance levels. This suggests that automating deception detection from video modalities holds significant potential. Overall, this study presents a comprehensive evaluation of a multimodal approach for automated deception detection which sheds light on its effectiveness and open avenues for future research in this field.