Running summarizing tools on a new article

Abstract: Concerns about bias in large language models (LLMs) often focus on technical metrics, yet limited research examines how marginalized communities perceive and interpret model behavior. Through 25 interviews with participants across gender identities (non-binary, men, and women), we investigate how ChatGPT responds to gendered versus neutral prompts and how users evaluate bias in these responses. Our findings reveal striking differences: non-binary participants identified condescension, stereotyping, and identity erasure in outputs that cisgender participants often found acceptable, demonstrating that bias perception is not universal, but rather shaped by lived experience and social positioning. This differential perception reveals a fundamental limitation in current fairness evaluation: technical metrics and aggregate user feedback systematically miss harms that are visible primarily to marginalized users. Our work demonstrates that evaluating LLM fairness requires centering the perspectives of those most affected by algorithmic bias, rather than relying solely on technical detection or treating all user feedback as equivalent, and makes recommendations for how to go about this.