Evaluating Language Understanding and World Knowledge of Large Language Models in Turkic Languages
Date Issued
2024-04
Author(s)
Isbarov, Jafar
Abstract
MMLU is one of the most widely used and referred-to benchmarks for LLM performance [19]. It
consists of multiple-choice questions on elementary, high school, college, and professional topics.
MMLU benchmark was designed with English in mind, but we have seen a rise of multilingual
benchmarks alongside multilingual models in recent years. One example is the MMMLU, which
has multiple alternatives now. Such benchmarks allow researchers to compare the performance of
LLMs across languages, speeding up the development in this area.
While multilingual benchmarks exist, such works tend to have a global scope. As a result,
low-resource languages are left out of the spotlight yet again. This applies to Turkic languages as
well. With the exception of Turkish, Turkic languages have been left of most such benchmarks.
In this work, we have concentrated on evaluation of Turkic languages, including extremely lowresource languages such as Tatar and Crimean Tatar. We have prepared a benchmark consisting of
high-school-level multiple-choice questions. This benchmark has allowed us to evaluate the NLU
capabilities of SOTA LLMs in Turkic languages.
Turkish is the only Turkic language with a native MMLU benchmark. TurkishMMLU was released a few months ago [65], and we have collaborated with their team on this project. There
have been attempts to create MMLU benchmarks for Turkic (and other) languages by synthetic
generation or machine translation. However, these approaches are known to produce potentially
erroneous text and do not consider the linguistic and cultural nuances of the target language. Since
MMLU was designed for the English language, its contents are biased towards an Anglocentric
worldview. Therefore, any successful benchmark should be created from native sources instead
of direct or indirect translations. Since TurkishMMLU already exists, our project expanded this
to seven more Turkic languages – Azerbaijani, Kazakh, Uzbek, Tatar, Crimean Tatar, Uyghur, and
Karakalpak.
In this project, we have created a unified and native language understanding benchmark for
the aforementioned languages. We have later used this benchmark to evaluate state-of-the-art
LLMs with multilingual capabilities. We have also evaluated the effect of the writing system on the
performance of the model. Finally, we have analysed the output of LLMs in low-resource languages
and identified that LLMs can respond to questions in low-resource languages using a high-resource
language without being prompted to do so.
consists of multiple-choice questions on elementary, high school, college, and professional topics.
MMLU benchmark was designed with English in mind, but we have seen a rise of multilingual
benchmarks alongside multilingual models in recent years. One example is the MMMLU, which
has multiple alternatives now. Such benchmarks allow researchers to compare the performance of
LLMs across languages, speeding up the development in this area.
While multilingual benchmarks exist, such works tend to have a global scope. As a result,
low-resource languages are left out of the spotlight yet again. This applies to Turkic languages as
well. With the exception of Turkish, Turkic languages have been left of most such benchmarks.
In this work, we have concentrated on evaluation of Turkic languages, including extremely lowresource languages such as Tatar and Crimean Tatar. We have prepared a benchmark consisting of
high-school-level multiple-choice questions. This benchmark has allowed us to evaluate the NLU
capabilities of SOTA LLMs in Turkic languages.
Turkish is the only Turkic language with a native MMLU benchmark. TurkishMMLU was released a few months ago [65], and we have collaborated with their team on this project. There
have been attempts to create MMLU benchmarks for Turkic (and other) languages by synthetic
generation or machine translation. However, these approaches are known to produce potentially
erroneous text and do not consider the linguistic and cultural nuances of the target language. Since
MMLU was designed for the English language, its contents are biased towards an Anglocentric
worldview. Therefore, any successful benchmark should be created from native sources instead
of direct or indirect translations. Since TurkishMMLU already exists, our project expanded this
to seven more Turkic languages – Azerbaijani, Kazakh, Uzbek, Tatar, Crimean Tatar, Uyghur, and
Karakalpak.
In this project, we have created a unified and native language understanding benchmark for
the aforementioned languages. We have later used this benchmark to evaluate state-of-the-art
LLMs with multilingual capabilities. We have also evaluated the effect of the writing system on the
performance of the model. Finally, we have analysed the output of LLMs in low-resource languages
and identified that LLMs can respond to questions in low-resource languages using a high-resource
language without being prompted to do so.