# LoRA-like Calibration for Multimodal Deception Detection using ATSFace Data

**Authors:** Shun-Wen Hsiao, Cheng-Yuan Sun  
**arXiv:** [2309.01383](https://arxiv.org/abs/2309.01383)  
**Date:** 2023-09-04  
**Summary page:** https://www.summarizepaper.com/en/paper/2309.01383v1/

## Key points

- Deception detection on human videos has gained attention
- AI models in this domain lack interpretability
- A new attention-aware neural network model has been introduced
- The model assesses visual, audio, and text features to identify deceptive cues
- The model achieves 92% accuracy on a real-life trial dataset
- The model provides insights into the underlying process by indicating attention focus in videos
- An experiment involving university students was conducted to enrich the study
- A calibration method inspired by Low-Rank Adaptation (LoRA) was introduced to refine deception detection accuracy
- A new dataset called ATSFace was created, consisting of 309 video clips
- The dataset involved posing questions and recording participants' responses using an iPhone 14 Pro in Chinese language
- Transcripts of sample deceptive and truthful statements were obtained using an automatic speech recognition (ASR) system called CapCut
- The research presents a comprehensive approach to deception detection on human videos.

## Summary

Recently, deception detection on human videos has gained significant attention due to its potential applications. While AI models in this domain have shown high accuracy, they often lack interpretability. To address this issue, a new attention-aware neural network model has been introduced. This model tackles the challenges associated with video data and deception dynamics by continuously assessing visual, audio, and text features to identify deceptive cues. By employing a multimodal fusion strategy, the model achieves an impressive 92% accuracy rate on a real-life trial dataset. One of the key contributions of this research is that the model not only detects deceit but also provides valuable insights into the underlying process by indicating the attention focus in the videos. To enrich the study further, an experiment involving university students answering questions either truthfully or deceitfully was conducted. This resulted in a new dataset called ATSFace, which consists of 309 video clips. To refine individual-based deception detection accuracy, a calibration method inspired by Low-Rank Adaptation (LoRA) was introduced. The evaluation of this approach involved creating a new dataset called ATSFace1 through a multimodal approach to deception detection. The dataset was collected by posing general questions about school life and finance to university students who were instructed to respond truthfully initially. They were then asked to create fictitious narratives about selected topics and provide honest narratives for other topics. The experiment involved recording participants using an iPhone 14 Pro in a 1080p HD/30fps format while they responded to questions in Chinese. The resulting dataset consisted of 309 videos, with 147 being deceptive and 162 being truthful clips. The average duration of these videos was 23.32 seconds. Transcripts of sample deceptive and truthful statements from the dataset were obtained using an automatic speech recognition (ASR) system called CapCut. These transcripts comprised 35,069 words in total, averaging 113 words per transcript. Overall, this research presents a comprehensive approach to deception detection on human videos. The attention-aware neural network model, combined with the multimodal fusion strategy and calibration method, achieves high accuracy in detecting deceit while providing insights into the underlying process.