# KLUE: Korean Language Understanding Evaluation

**Authors:** Sungjoon Park, Jihyung Moon, Sungdong Kim, Won Ik Cho, Jiyoon Han, Jangwon Park, Chisung Song, Junseong Kim, Yongsook Song, Taehwan Oh, Joohong Lee, Juhyun Oh, Sungwon Lyu, Younghoon Jeong, Inkwon Lee, Sangwoo Seo, Dongjun Lee, Hyunwoo Kim, Myeonghwa Lee, Seongbo Jang  
**arXiv:** [2105.09680](https://arxiv.org/abs/2105.09680)  
**Date:** 2021-05-20  
**Summary page:** https://www.summarizepaper.com/en/paper/2105.09680v1/

## Key points

- The paper introduces the Korean Language Understanding Evaluation (KLUE) benchmark, consisting of 8 NLU tasks in Korean.
- The tasks include Topic Classification, Semantic Textual Similarity, Natural Language Inference, Named Entity Recognition (NER), Relation Extraction, Dependency Parsing, Machine Reading Comprehension, and Dialogue State Tracking.
- The authors built these tasks from scratch using diverse source corpora while respecting copyrights.
- Annotation protocols were designed with ethical considerations in mind.
- Suitable evaluation metrics and fine-tuning recipes for pretrained language models are provided for each task.
- Two pretrained language models (KLUE-BERT and KLUE-RoBERTa) are released to reproduce baseline models on KLUE and facilitate future research.
- Preliminary experiments show that KLUE-RoBERTa-large outperforms other baselines and existing open-source Korean PLMs.
- Performance is minimally affected when personally identifiable information is replaced from the pretraining corpus, suggesting privacy and NLU capability are not at odds with each other.
- BPE tokenization combined with morpheme-level pre-tokenization is effective in tasks involving morpheme-level tagging detection and generation.
- Comprehensive documentation on creating KLUE is provided to accelerate Korean NLP research and facilitate similar resources for other languages in the future.
- Section 2 discusses source corpora selection criteria; Section 3 presents detailed information about each task; Section 4 focuses on the Named Entity Recognition (NER) task.

## Summary

The paper introduces the Korean Language Understanding Evaluation (KLUE) benchmark, which consists of 8 natural language understanding (NLU) tasks in Korean. These tasks include Topic Classification, Semantic Textual Similarity, Natural Language Inference, Named Entity Recognition (NER), Relation Extraction, Dependency Parsing, Machine Reading Comprehension and Dialogue State Tracking. The authors built these tasks from scratch using diverse source corpora while respecting copyrights to ensure accessibility for anyone without restrictions. They also designed annotation protocols with ethical considerations in mind. In addition to the benchmark tasks and data, the authors provide suitable evaluation metrics and fine-tuning recipes for pretrained language models for each task. They also release two pretrained language models (PLMs), KLUE-BERT and KLUE-RoBERTa, to help reproduce baseline models on KLUE and facilitate future research. Preliminary experiments using the proposed KLUE benchmark suite have yielded interesting observations. First, KLUE-RoBERTa-large outperforms other baselines including multilingual PLMs and existing open-source Korean PLMs. Second, there is minimal degradation in performance even when personally identifiable information is replaced from the pretraining corpus suggesting that privacy and NLU capability are not at odds with each other. Lastly, using BPE tokenization in combination with morpheme-level pre-tokenization is effective in tasks involving morpheme-level tagging detection and generation. The paper also provides comprehensive documentation on creating KLUE to accelerate Korean NLP research and facilitate the creation of similar resources for other languages in the future. Section 2 discusses source corpora selection criteria and provides details about selected corpora; Section 3 presents detailed information about each task in the KLUE benchmark suite; Section 4 focuses on the Named Entity Recognition (NER) task including dataset construction evaluation metrics related work and conclusions. The authors use two corpora WIKITREE and NSMC to incorporate both formal and informal writing styles in the NER task. WIKITREE is a news article corpus suitable for NER due to its formal sentences with many entity types while NSMC includes colloquial reviews of movies or TV shows providing a noisy dataset that broadens the application field of NER models.