AI summary · arXiv: 2303.16779v1 · cs.CL
Language Models Trained on Media Diets Can Predict Public Opinion
The essentials
- The study proposes a new approach to measuring public opinion by using language models adapted to online news, TV broadcast, or radio show content that can emulate the opinions of subpopulations that have consumed a set of media.
- The researchers used U.S. nationally representative surveys on COVID-19 and consumer confidence as ground truth for opinions expressed to validate this method.
- Results indicate that this approach is predictive of human judgments found in survey response distributions and robust to phrasing and channels of media exposure.
- This method is more accurate at modeling people who follow media more closely and aligned with literature on which types of opinions are affected by media consumption.
- Probing language models provides a powerful new method for investigating media effects, supplementing polls, and forecasting public opinion.
- Neural language models can predict human responses with surprising fidelity, suggesting a need for further research in this area.
- The authors of the study are Eric Chu, Jacob Andreas, Stephen Ansolabehere, and Deb Roy.
Key points written by AI: they may contain errors, check them against the paper.
Ask questions about this paper to our AI assistant
How to chat with this paper? Illustrated guide
You can also chat with multiple papers at once here.
Keep this conversation — a free account saves your questions and answers, your collections of papers and your alerts.
Continue with Google Create a free accountTools for this paper
Advanced tools Premium
For researchers who follow a field closely: citation alerts, journal-club slides, version comparison and Zotero export. Discover Premium
Among the 31 papers around this one, the name that comes back most: Esin Durmus (2 papers)
Papers around this one
See them on a mapRanked by how close their title and abstract are to this paper’s. Select some to read them together.
Closest papers
-
Public Opinion Mining Using Large Language Models on COVID-19 Related TweetsVu Tran, Tomoko Matsui · 2023 · International Conference on Knowledge and Systems Engineering · cited 1 time · Not summarized yet
Abstract
Social media platforms have emerged as a significant source of public opinion, offering a massive user-generated data in which user-opinions are valuable if obtainable. Large language models (LLMs) have been in the spotlight recently with suggestions on the emergent abilities to solve tasks that are not explicitly trained for. Thus, this study explores the potential of utilizing LLMs for opinion mining on social media data by asking LLMs difficult questions, instead of simply asking whether the text's sentiment polarity is either positive, negative, or neutral. This study compares the LLM response statistics and the corresponding public surveys related to COVID-19, including the intention to…
-
MindVote: When AI Meets the Wild West of Social Media OpinionXu-Tao Mao, Ezra Xuanru Tao, Le-Yao Wang · 2025 · AAAI Conference on Artificial Intelligence · Not summarized yet
Abstract
Large Language Models (LLMs) are increasingly used as scalable tools for pilot testing, predicting public opinion distributions before deploying costly surveys. However, the prevailing paradigm for evaluating these models relies on traditional structured surveys—a methodology misaligned with the more realistic scenarios like social media where opinions are rich in digital contexts. By design, surveys strip away the social and cultural context that shapes public opinion, and LLM benchmarks built on this paradigm inherit these critical limitations. To bridge this gap, we introduce MindVote, the first benchmark for public opinion prediction grounded in authentic social media discourse. MindVote…
-
Whose Opinions Do Language Models Reflect?Shibani Santurkar, Esin Durmus, Faisal Ladhak et al. · 2023 · Computation and Language · cited 1,170 times · AI summary available
Abstract
Language models (LMs) are increasingly being used in open-ended contexts, where the opinions reflected by LMs in response to subjective queries can have a profound impact, both on user satisfaction, as well as shaping the views of society at large. In this work, we put forth a quantitative framework to investigate the opinions reflected by LMs -- by leveraging high-quality public opinion polls and their associated human responses. Using this framework, we create OpinionsQA, a new dataset for evaluating the alignment of LM opinions with those of 60 US demographic groups over topics ranging from abortion to automation. Across topics, we find substantial misalignment between the views reflected…
-
Comparing the algorithmic fidelity of large language models in predicting human decision making: a case study of vaccination choiceShao-Chong Xu, Li-Yue Zhang, Amelia M. Jamison et al. · 2026 · npj Digital Public Health · cited 1 time · Not summarized yet
Abstract
This study explores the algorithmic fidelity of five well-established LLM architectures when attempting to replicate complex health decisions like vaccination. The analysis leverages individual survey responses and real-world online news content within prescribed model designs that vary in their relative inputs—namely, basic demographic information, revealed attitudes based on survey data, and personalized media diets. We evaluate the predictive accuracy and sensitivity for each model across all five LLMs, and reveal differing biases inherent to each LLM design. We further explore the role of media exposure on predicted vaccination choice within a counterfactual scenario analysis, by varying…
-
Simulating Public Opinion: Comparing Distributional and Individual-Level Predictions from LLMs and Random ForestsFernando Miranda, P. Balbi · 2025 · Entropy · cited 6 times · Not summarized yet
Abstract
Understanding and modeling the flow of information in human societies is essential for capturing phenomena such as polarization, opinion formation, and misinformation diffusion. Traditional agent-based models often rely on simplified behavioral rules that fail to capture the nuanced and context-sensitive nature of human decision-making. In this study, we explore the potential of Large Language Models (LLMs) as data-driven, high-fidelity agents capable of simulating individual opinions under varying informational conditions. Conditioning LLMs on real survey data from the 2020 American National Election Studies (ANES), we investigate their ability to predict individual-level responses across a…
-
AI-Augmented Surveys: Leveraging Large Language Models and Surveys for Opinion PredictionJunsol Kim, Byungkyu Lee · 2023 · Computation and Language · cited 85 times · AI summary available
Abstract
Large language models (LLMs) that produce human-like responses have begun to revolutionize research practices in the social sciences. This paper shows how we can integrate LLMs and social surveys to accurately predict individual responses to survey questions that were not asked before. We develop a novel methodological framework to personalize LLMs by considering the meaning of survey questions derived from their text, the latent beliefs of individuals inferred from their response patterns, and the temporal contexts across different survey periods through fine-tuning LLMs with survey data. Using the General Social Survey from 1972 to 2021, we show that the fine-tuned model based on Alpaca-7b…
-
Simulating Human Opinions with Large Language Models: Opportunities and Challenges for Personalized Survey Data ModelingCarolin Kaiser, Jakob Kaiser, Vladimir Manewitsch et al. · 2025 · User Modeling, Adaptation, and Personalization · cited 34 times · Not summarized yet
Abstract
Public and private organizations rely on opinion surveys to inform business and policy decisions. Yet, empirical surveys are costly and time-consuming. Recent advances in large language models (LLMs) have sparked interest in generating synthetic survey data, i.e., simulated answers based on target demographics, as an alternative to real human data. But how well can LLMs replicate human opinions? In this ongoing project, we develop and critically evaluate methods for synthetic survey sampling. As an empirical benchmark, we collected responses from a representative U.S. sample (n = 461) on preferences for a common consumer good (soft drinks). Then, we developed ASPIRE (Automated Synthetic Pers…
-
Designing Domain-Specific Large Language Models: The Critical Role of Fine-Tuning in Public Opinion SimulationHao Lin · 2024 · arXiv.org · preprint, not peer reviewed · cited 7 times · Not summarized yet
Abstract
Large language models (LLMs) have transformed natural language processing, yet face challenges in specialized tasks such as simulating opinions on environmental policies. This paper introduces a novel fine-tuning approach that integrates socio-demographic data from the UK Household Longitudinal Study, uniquely using profiling factors, such as age, gender, income, education, and region. This method enhances the accuracy and representation of generated views. By emulating diverse synthetic profiles, the fine-tuned models significantly outperform pre-trained counterparts, achieving measurable improvements in capturing demographic nuances. Evaluation metrics, including Chi-Squared, Cosine Simila…
Prior work it builds on
-
From Tweets to Polls: Linking Text Sentiment to Public Opinion Time SeriesBrendan T. O'Connor, R. Balasubramanyan, Bryan R. Routledge et al. · 2010 · International Conference on Web and Social Media · cited 2,184 times · Not summarized yet
Abstract
We connect measures of public opinion measured from polls with sentiment measured from text. We analyze several surveys on consumer confidence and political opinion over the 2008 to 2009 period, and find they correlate to sentiment word frequencies in contempora- neous Twitter messages. While our results vary across datasets, in several cases the correlations are as high as 80%, and capture important large-scale trends. The re- sults highlight the potential of text streams as a substi- tute and supplement for traditional polling. consumer confidence and political opinion, and can also pre- dict future movements in the polls. We find that temporal smoothing is a critically important issue to supp…
-
Media Effects Across Time and Subject: How News Coverage Affects Two Out of Four Attributes of Consumer ConfidenceM. Boukes, A. Damstra, R. Vliegenthart · 2019 · Communication Research · cited 46 times · Not summarized yet
Abstract
Previous studies using aggregate-level designs demonstrated that the tone of economic news affects consumer confidence. However, the individual-level mechanisms underlying this effect remain to be investigated: It is not clear which consumer confidence attributes are most susceptible to media effects. Theoretically, we integrate the economic voting literature and extend media system dependency by differentiating between the media effects on sociotropic versus egocentric evaluations and the effects on prospective versus retrospective economic evaluations. Methodologically, data from a manual content analysis are linked to data from a three-wave panel survey, containing repeated measurements o…
-
How the news media activate public expression and influence national agendasGary King, Benjamin Schneer, Ariel White · 2017 · Science · cited 303 times · Not summarized yet
Abstract
Measuring the impact of the media The active participation of the people is one of the central components of a functioning democracy. King et al. performed a real-world randomized experiment in the United States to understand the causal effect of news stories on increasing public discussion of a specific topic (see the Policy Forum by Gentzkow). Social media posts increased by almost 20% the first day after the publication of news stories on a wide range of topics. Furthermore, the posts were relatively evenly distributed across political affiliation, gender, and region of the United States. Science, this issue p. 776; see also p. 726 Quantifying the effects of the press on public discourse …
-
Measuring Public Opinion with SurveysAdam J. Berinsky · 2017 · Annual Review of Political Science · cited 124 times · Not summarized yet
-
Ethical and social risks of harm from Language ModelsLaura Weidinger, John F. J. Mellor, Maribeth Rauh et al. · 2021 · arXiv.org · preprint, not peer reviewed · cited 1,735 times · AI summary available
Abstract
This paper aims to help structure the risk landscape associated with large-scale Language Models (LMs). In order to foster advances in responsible innovation, an in-depth understanding of the potential risks posed by these models is needed. A wide range of established and anticipated risks are analysed in detail, drawing on multidisciplinary expertise and literature from computer science, linguistics, and social sciences. We outline six specific risk areas: I. Discrimination, Exclusion and Toxicity, II. Information Hazards, III. Misinformation Harms, V. Malicious Uses, V. Human-Computer Interaction Harms, VI. Automation, Access, and Environmental Harms. The first area concerns the perpetuati…
Papers that cite it
-
Random Silicon Sampling: Simulating Human Sub-Population Opinion Using a Large Language Model Based on Group-Level Demographic InformationSeungjong Sun, Eungu Lee, Dongyan Nan et al. · 2024 · arXiv.org · preprint, not peer reviewed · cited 71 times · Not summarized yet
Abstract
Large language models exhibit societal biases associated with demographic information, including race, gender, and others. Endowing such language models with personalities based on demographic data can enable generating opinions that align with those of humans. Building on this idea, we propose"random silicon sampling,"a method to emulate the opinions of the human population sub-group. Our study analyzed 1) a language model that generates the survey responses that correspond with a human group based solely on its demographic distribution and 2) the applicability of our methodology across various demographic subgroups and thematic questions. Through random silicon sampling and using only grou…
-
Vox Populi, Vox AI? Using Large Language Models to Estimate German Vote ChoiceLeah von der Heyde, Anna Haensch, Alexander Wenz · 2024 · Social science computer review · cited 28 times · Not summarized yet
Abstract
“Synthetic samples” generated by large language models (LLMs) have been argued to complement or replace traditional surveys, assuming their training data is grounded in human-generated data that potentially reflects attitudes and behaviors prevalent in the population. Initial US-based studies that have prompted LLMs to mimic survey respondents found that the responses match survey data. However, the relationship between the respective target population and LLM training data might affect the generalizability of such findings. In this paper, we critically evaluate the use of LLMs for public opinion research in a different context, by investigating whether LLMs can estimate vote choice in Germa…
-
Aligning Language Models to User OpinionsEunjeong Hwang, Bodhisattwa Prasad Majumder, Niket Tandon · 2023 · Conference on Empirical Methods in Natural Language Processing · cited 151 times · Not summarized yet
Abstract
An important aspect of developing LLMs that interact with humans is to align models' behavior to their users. It is possible to prompt an LLM into behaving as a certain persona, especially a user group or ideological persona the model captured during its pertaining stage. But, how to best align an LLM with a specific user and not a demographic or ideological group remains an open question. Mining public opinion surveys (by Pew Research), we find that the opinions of a user and their demographics and ideologies are not mutual predictors. We use this insight to align LLMs by modeling both user opinions as well as user demographics and ideology, achieving up to 7 points accuracy gains in predic…
-
Can Large Language Models Transform Computational Social Science?Caleb Ziems, William Held, Omar Shaikh et al. · 2023 · Computation and Language · cited 692 times · Not summarized yet
Abstract
Large Language Models (LLMs) like ChatGPT are capable of successfully performing many language processing tasks zero-shot (without the need for training data). If this capacity also applies to the coding of social phenomena like persuasiveness and political ideology, then LLMs could effectively transform Computational Social Science (CSS). This work provides a road map for using LLMs as CSS tools. Towards this end, we contribute a set of prompting best practices and an extensive evaluation pipeline to measure the zero-shot performance of 13 language models on 24 representative CSS benchmarks. On taxonomic labeling tasks (classification), LLMs fail to outperform the best fine-tuned models but…
-
Community-Cross-Instruct: Unsupervised Instruction Generation for Aligning Large Language Models to Online CommunitiesZihao He, Rebecca Dorn, Si-Yi Guo et al. · 2024 · Conference on Empirical Methods in Natural Language Processing · cited 17 times · Not summarized yet
Abstract
Social scientists use surveys to probe the opinions and beliefs of populations, but these methods are slow, costly, and prone to biases. Recent advances in large language models (LLMs) enable the creating of computational representations or “digital twins” of populations that generate human-like responses mimicking the population’s language, styles, and attitudes. We introduce Community-Cross-Instruct, an unsupervised framework for aligning LLMs to online communities to elicit their beliefs. Given a corpus of a community’s online discussions, Community-Cross-Instruct automatically generates instruction-output pairs by an advanced LLM to (1) finetune a foundational LLM to faithfully represent…
Go further
How are these papers found?
We compare the title and abstract of this paper with those of every paper summarized on this site, and with the papers Semantic Scholar recommends, those it cites and those citing it (for a much-cited paper, its most recent citations and, from OpenAlex, the most cited ones). Among the prior work and the follow-ups, the papers that use each other in depth, a method it uses and the much-cited follow-ups come a little higher. The levels (very close, close, related) come from the similarity of the texts, not from a judgment of quality; * marks a paper without an abstract, compared by its title. The sentence under a paper is written by an AI from the two abstracts: check it in the papers. A paper marked not relevant by three readers leaves the list.
Updated on 6 November 2026.
The previous summary was created more than a year ago and can be re-run (if necessary) by clicking on the Run button below.
Disclaimer: The AI-based summarization tool and virtual assistant provided on this website may not always provide accurate and complete summaries or responses. We encourage you to carefully review and evaluate the generated content to ensure its quality and relevance to your needs.