# Measuring Taiwanese Mandarin Language Understanding

**Authors:** Po-Heng Chen, Sijia Cheng, Wei-Lin Chen, Yen-Ting Lin, Yun-Nung Chen  
**arXiv:** [2403.20180](https://arxiv.org/abs/2403.20180)  
**Date:** 2024-03-29  
**Summary page:** https://www.summarizepaper.com/en/paper/2403.20180v1/

## Key points

- Authors Po-Heng Chen, Sijia Cheng, Wei-Lin Chen, Yen-Ting Lin, and Yun-Nung Chen focus on evaluating large language models (LLMs) with a specific emphasis on Traditional Chinese in a Chinese context.
- They introduce TMLU, a comprehensive evaluation suite designed to assess the advanced knowledge and reasoning capabilities of LLMs within the context of Taiwanese Mandarin.
- TMLU encompasses 37 subjects spanning various fields and educational levels from middle school to professional expertise.
- The authors provide chain-of-thought-like few-shot explanations for each subject to facilitate the assessment of complex reasoning skills.
- Results show that Chinese open-weight models perform worse than multilingual proprietary models, with open-weight models tailored for Taiwanese Mandarin lagging behind Simplified-Chinese counterparts.
- The study highlights significant room for improvement and emphasizes TMLU's goal of developing localized Taiwanese-Mandarin LLMs.
- The authors have made their benchmark data and evaluation scripts publicly available to support further research in enhancing Taiwanese Mandarin language understanding through advanced LLM technologies.

## Summary

In their preprint titled "Measuring Taiwanese Mandarin Language Understanding," authors Po-Heng Chen, Sijia Cheng, Wei-Lin Chen, Yen-Ting Lin, and Yun-Nung Chen delve into the evaluation of large language models (LLMs) with a specific focus on Traditional Chinese in a Chinese context. They highlight the lack of representation of Traditional Chinese in existing benchmarks and introduce TMLU, a comprehensive evaluation suite designed to assess the advanced knowledge and reasoning capabilities of LLMs within the context of Taiwanese Mandarin. TMLU encompasses 37 subjects spanning social science, STEM, humanities, Taiwan-specific content, and more. It caters to educational levels ranging from middle school to professional expertise. The authors also provide chain-of-thought-like few-shot explanations for each subject to facilitate the assessment of complex reasoning skills. To establish a robust baseline for comparison  conduct extensive experiments and analysis involving 24 advanced LLMs. The results of their study reveal that Chinese open-weight models exhibit inferior performance compared to multilingual proprietary models. Furthermore  open-weight models tailored for Taiwanese Mandarin lag behind their Simplified-Chinese counterparts. These findings underscore significant room for improvement  highlight TMLU's objective of nurturing the development of localized Taiwanese-Mandarin LLMs. In an effort to support further research in this area  the authors have made their benchmark data and evaluation scripts publicly available. This initiative aims to encourage collaboration within the research community towards enhancing Taiwanese Mandarin language understanding through advanced LLM technologies.