# Data Bias According to Bipol: Men are Naturally Right and It is the Role of Women to Follow Their Lead

**Authors:** Irene Pagliai, Goya van Boven, Tosin Adewumi, Lama Alkhaled, Namrata Gurung, Isabella Södergren, Elisa Barney  
**arXiv:** [2404.04838](https://arxiv.org/abs/2404.04838)  
**Date:** 2024-04-07  
**Summary page:** https://www.summarizepaper.com/en/paper/2404.04838v2/

## Key points

- Introduction of new large labeled datasets on bias in Italian, Dutch, and German languages
- Bias detected in all evaluated datasets across five languages, including English GLUE/SuperGLUE leaderboards
- Utilization of state-of-the-art multilingual pretrained models like mT5 and mBERT for benchmarking
- Motivation from recent events highlighting social bias in AI and large language models (LLMs)
- Comparison of various bias evaluation methods, including the bipol metric for explainability
- Confirmation of bias in toxic comments through annotation with a confidence level of 95% and error margin of 7%
- Identification of male bias along with other forms of bias in many datasets
- Emphasis on gender biases as well as biases related to origin and age in previous research efforts
- Focus on binary gender bias in Dutch language studies
- Conducting experiments using SotA pre-trained multilingual models mT5-small and mBERT-base to compare macro F1 performance for toxic comments containing bias
- Methodology involving rigorous annotation techniques with multiple annotators and gold samples for high-quality results
- Contribution of new labeled datasets, lexica of sensitive terms, models, and codes for detecting bias in multiple languages

## Summary

In this study, we introduce new large labeled datasets on bias in three languages - Italian, Dutch, and German. These datasets consist of almost 2 million samples each and are accompanied by lexica of sensitive terms for bias detection in the respective languages. Our experiments reveal that bias exists in all ten datasets of five languages evaluated, including benchmark datasets on the English GLUE/SuperGLUE leaderboards. We utilize state-of-the-art multilingual pretrained models such as mT5 and mBERT to benchmark on these datasets. The motivation behind our research stems from recent events highlighting the prevalence of social bias in AI and large language models (LLMs). We aim to estimate bias in multiple datasets by comparing various bias evaluation methods, including the bipol metric which offers explainability. Additionally, we confirm the assumption that toxic comments contain bias by annotating 200 samples from a toxic dataset population with a confidence level of 95% and an error margin of 7%. To ensure annotation quality, we include 30 gold samples with unanimous agreement from the original data. Our findings indicate that many datasets exhibit male bias (prejudice against women) along with other forms of bias. The literature review highlights previous efforts to measure and mitigate bias in languages other than English, emphasizing gender biases as well as biases related to origin and age. For Dutch language studies specifically, binary gender bias is a common focus. To verify our assumption regarding toxic comments containing bias, we conduct experiments using SotA pre-trained multilingual models mT5-small and mBERT-base to compare their macro F1 performance. Our methodology involves rigorous annotation techniques with multiple annotators and gold samples to ensure high-quality results. Overall, this study contributes new labeled datasets, lexica of sensitive terms, models, and codes for detecting bias in multiple languages. By shedding light on the presence of biases across various datasets and languages, we aim to address the challenge of social bias in AI systems effectively.