# Automated reproducibility assessments in the social and behavioral sciences using large language models

**Authors:** Tobias Holtdirk, Pietro Marcolongo, Anna Steinberg Schulten, Felix Henninger, Stefan Rose, Sarah Ball, Bolei Ma, Frauke Kreuter, Markus Weinmann, Stefan Feuerriegel  
**arXiv:** [2606.13670](https://arxiv.org/abs/2606.13670)  
**Date:** 2026-06-11  
**Summary page:** https://www.summarizepaper.com/en/paper/2606.13670v1/

## Key points

- Reproducibility of research findings is crucial for building a solid foundation of knowledge in social and behavioral sciences.
- Traditional reproducibility assessments are labor-intensive and challenging to scale across a large number of studies.
- A recent study led by Tobias Holtdirk et al. introduced an automated approach using large language models (LLMs) for reproducibility assessments in social and behavioral sciences.
- The study evaluated 76 published studies with predefined claims and found that LLMs can effectively automate reproducibility assessments.
- LLMs successfully recovered original effect sizes in 41% of studies and reached the same qualitative conclusion as the original study in 96% of cases.
- LLMs outperformed human analysts in replicating original effect sizes and reaching consistent qualitative conclusions, suggesting their potential as a scalable tool for automating reproducibility assessments.
- This study highlights the promising role of LLMs in streamlining reproducibility evaluations and improving efficiency and accuracy in assessing scientific findings.

## Summary

In the field of social and behavioral sciences, ensuring reproducibility of research findings is crucial for building a solid foundation of knowledge. Traditionally, reproducibility assessments have been conducted by independent researchers who reanalyze original data to determine if published results can be replicated. However, this process is often labor-intensive and challenging to scale across a large number of studies. A recent study led by Tobias Holtdirk, Pietro Marcolongo, Anna Steinberg Schulten, Felix Henninger, Stefan Rose, Sarah Ball, Bolei Ma, Frauke Kreuter, Markus Weinmann, and Stefan Feuerriegel has introduced a novel approach to automating reproducibility assessments using large language models (LLMs). The researchers focused on evaluating 76 published studies in the social and behavioral sciences that had predefined claims. The results of their study demonstrated that LLMs can effectively automate reproducibility assessments. In cases where the LLM could generate a viable effect size estimate with a tolerance of +/-0.05 in Cohen's d., it successfully recovered the original effect sizes in 41% of studies. Additionally,<Organization>the LLM pipeline reached the same qualitative conclusion as the original study in an impressive 96% of cases. Comparing these results to those obtained through human reanalysis revealed that LLMs outperformed human analysts in terms of replicating original effect sizes and reaching consistent qualitative conclusions. This suggests that LLMs have the potential to serve as a scalable tool for automating reproducibility assessments in the social and behavioral sciences. Overall,<Organization>this study highlights the promising role that LLMs can play in streamlining reproducibility evaluations and laying the groundwork for systematic auditing of empirical results in social and behavioral research fields. By leveraging advanced technology like LLMs, researchers can enhance efficiency and accuracy in assessing the reliability and validity of scientific findings.