# The Diversity-Innovation Paradox in Science

**Authors:** Bas Hofstra, Vivek V. Kulkarni, Sebastian Munoz-Najar Galvez, Bryan He, Dan Jurafsky, Daniel A. McFarland  
**arXiv:** [1909.02063](https://arxiv.org/abs/1909.02063)  
**Date:** 2019-09-04  
**Summary page:** https://www.summarizepaper.com/en/paper/1909.02063v2/

## Key points

- Diversity paradox in scientific innovation and career success
- Previous research shows diversity promotes innovation but underrepresented groups have less successful careers
- Study analyzes dataset of 1.2 million US doctoral recipients from 1977 to 2015
- Structural Topic Models (STMs) used to detect scientific innovations
- Fit metrics plateau at K = 400, 500, and 600 for topic models
- Three weighing schemes for extracting top terms: equal balance, more weight to frequency, more weight to exclusivity
- Sensitivity analyses conducted on nine hyperparameter scenarios
- Dataset includes doctoral abstracts from 1982-2010 as input documents for assessing students' scholarship
- Year fixed-effects account for left and right-censoring for comparisons within years
- Co-occurring concepts in abstracts aggregated each year to identify students introducing novel links
- Spurious links removed using significance score based on log-odds ratio of link occurrence probability compared to component concept term occurrence probabilities independently over the corpus
- Outcome variable measured as novelty and impactful novelty in text
- Underrepresented groups produce higher rates of scientific novelty but contributions undervalued and less likely to be taken up by other scholars compared to majority groups.
- Equally impactful contributions from underrepresented groups less likely to lead to successful scientific careers compared to majority groups.

## Summary

This study examines the diversity paradox in the context of scientific innovation and career success. Previous research has shown that while diversity promotes innovation, underrepresented groups within organizations often have less successful careers. The researchers aim to determine if this paradox also applies to scientists by analyzing a dataset of approximately 1.2 million US doctoral recipients from 1977 to 2015. To detect scientific innovations, the researchers utilize Structural Topic Models (STMs), which are commonly used to identify latent thematic dimensions in large text corpora. They fit topic models at various levels (K = [50-1000]) and find that fit metrics plateau at K = 400, 500, and 600. They extract concepts by identifying terms of relative importance within each latent theme using the STM output. The researchers explore three weighing schemes for extracting top terms: equally balancing frequency and exclusivity (50/50), attaching more weight to frequency and less to exclusivity (75/25), and attaching more weight to exclusivity and less to frequency (25/75). They conduct sensitivity analyses on nine hyperparameter scenarios and report results where frequency and exclusivity are equally balanced at K = 500. The dataset includes doctoral abstracts from 1982-2010 as input documents for assessing students' scholarship at the beginning of their careers. Year fixed-effects account for left- and right-censoring, enabling comparisons within years. Using extracted scientific concepts, co-occurring concepts in abstracts are aggregated each year to identify students who introduce novel links. Spurious links are removed by computing a significance score based on the log-odds ratio of link occurrence probability compared to component concept term occurrence probabilities independently over the corpus. The outcome variable is measured as novelty and impactful novelty in text. The researchers filter for meaningful links using top FREX terms from structural topic models and remove spurious links through a link significance score. If a link is introduced by two students, it indicates that the concept is novel and impactful. The results show that underrepresented groups produce higher rates of scientific novelty. However, their novel contributions are undervalued and less likely to be taken up by other scholars compared to contributions from gender and racial majorities. Additionally, equally impactful contributions from underrepresented groups are less likely to lead to successful scientific careers compared to majority groups.