Pith. sign in

REVIEW 6 major objections 6 minor 46 references

This paper builds the first large-scale, natively authored Kyrgyz evaluation suite and argues that English-to-Kyrgyz model rankings transfer on reasoning and reading tasks, while a large HellaSwag gap reflects translation artifacts rather t

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 18:47 UTC pith:YUPGXVZM

load-bearing objection A real benchmark resource, but the WinoGrande transfer claim is contradicted by their own numbers — all Kyrgyz WinoGrande results sit at chance. the 6 major comments →

arxiv 2607.17173 v1 pith:YUPGXVZM submitted 2026-07-19 cs.CL

KyrgyzLLM-Bench: Benchmarking Kyrgyz Language Understanding

Timur Turatali , Aida Turdubaeva , Rustem Izmailov , Anton M. Alekseev , Sergey I. Nikolenko This is my paper
classification cs.CL
keywords Kyrgyz NLPLLM evaluationmultilingual benchmarkslow-resource languagescross-lingual transfertranslation artifactsreading comprehensionTurkic languages
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper aims to make Kyrgyz, a low-resource Turkic language, a place where LLM ability can be measured rather than guessed. It assembles KyrgyzLLM-Bench — 7,977 natively written multiple-choice exam questions, 400 native reading-comprehension items, and carefully post-edited translations of four standard English tasks — and scores 26 open and proprietary models on it. Its central claim is that relative model quality transfers from English to Kyrgyz on reading and pronoun-resolution tasks (WinoGrande, BoolQ) and partly on knowledge questions (MMLU), but collapses on the event-continuation task HellaSwag; the authors attribute that collapse to translation-induced plausibility shifts, not to a genuine commonsense deficit. If the measurements hold, they give researchers the first reliable way to compare models in Kyrgyz and a caution about trusting translated benchmarks for low-resource languages. A careful reader would care because almost all multilingual evaluation currently relies on translated data that can silently mislead.

Core claim

On the paper's own terms, the discovery is a measurable pattern: model rankings across families and scales are broadly preserved between English and Kyrgyz on BoolQ and WinoGrande, and to a lesser degree on MMLU, which the authors read as evidence that core reasoning and comprehension capabilities partially transfer across the language boundary. The same measurements show a large and irregular English–Kyrgyz gap on HellaSwag, and the paper explains this gap through plausibility shifts: translation disrupts the morphological cohesion, discourse flow, and idiomatic naturalness that event-continuation judgments depend on, so the Kyrgyz items no longer measure the same competence. The corollary

What carries the argument

The carrying object is the benchmark suite itself, built from three parts: KyrgyzMMLU, 7,977 multiple-choice items sourced from the official national school test and reviewed by domain experts; KyrgyzRC, 400 reading-comprehension questions over encyclopedic, news, literary, and math passages authored by native-speaking students and checked by supervisors and a professional linguist; and Kyrgyz translations of WinoGrande, HellaSwag, BoolQ, and TruthfulQA produced by two independent machine translations, ensemble-checked, then post-edited by native speakers. The explanatory mechanism doing the argumentative work is the plausibility-shift hypothesis — translated continuations lose the naturalne

Load-bearing premise

Every score and ranking in the paper rests on the assumption that the released KyrgyzMMLU and KyrgyzRC items are correct, unambiguous, and culturally sound — the review was sequential rather than parallel and no inter-annotator agreement was computed (the paper states Cohen's κ was not measured), so systematic item errors would quietly invalidate all conclusions built on these datasets.

What would settle it

Re-score a random sample of KyrgyzMMLU and KyrgyzRC with an independent panel of native-speaker annotators unconnected to the authors, and build a natively authored Kyrgyz HellaSwag-style continuation task; then rerun the same 26 models on it. Widespread item disagreements, or near-English accuracy on the native continuation task, would overturn the paper's data foundation or its translation-artifact explanation respectively.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the benchmark is sound, it becomes the reference yardstick for Kyrgyz LLM capability: all datasets, code, and per-model results are released, so future model releases can be checked against it directly.
  • The cross-lingual transfer finding implies that English performance on BoolQ- and WinoGrande-style tasks is a useful predictor of Kyrgyz performance, which can guide model choice for Kyrgyz applications before expensive native evaluation.
  • The HellaSwag result implies that translated event-continuation benchmarks should be abandoned or rewritten natively for agglutinative languages, since their scores conflate translation fidelity with reasoning.
  • The few-shot results imply that in-context learning is not a guaranteed win for low-resource languages: gains on native reading comprehension are real for open models, but translated-task scores can drop, so few-shot comparisons should always be read against zero-shot baselines.
  • The overall English–Kyrgyz accuracy gap on native tasks implies that pretraining corpus imbalance, not task difficulty, is the binding constraint, pointing to native Kyrgyz data collection as the highest-leverage next step.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: The plausibility-shift account makes a testable prediction the paper does not run — a natively authored Kyrgyz event-continuation task should largely erase the English–Kyrgyz gap; building one would confirm or refute the central explanation.
  • Editorial extension: Because item quality rests on a sequential review with no inter-annotator agreement, an independent audit of a few hundred released KyrgyzMMLU and KyrgyzRC items would directly test the data foundation; the public release makes this the cheapest possible check.
  • Editorial extension: The pattern plausibly generalizes to other under-resourced agglutinative languages, where translated event-continuation benchmarks may systematically understate model ability for the same morphological reasons.
  • Editorial extension: If ranking transfer is as broad as reported, the benchmark could also serve as a lightweight multilingual probe — a model's Kyrgyz score on BoolQ and WinoGrande could be used as a low-cost monitor of language-coverage drift during fine-tuning.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

6 major / 6 minor

Summary. The paper introduces KyrgyzLLM-Bench, a benchmark suite for Kyrgyz consisting of two natively authored datasets (KyrgyzMMLU with 7,977 items from the national school exam and KyrgyzRC with 400 reading-comprehension items) and four translated benchmarks (WinoGrande, HellaSwag, BoolQ, TruthfulQA) that were machine-translated, post-edited by native speakers, and reviewed. The authors evaluate 26 open and proprietary LLMs under zero-shot and few-shot settings using Lighteval, report English baselines for 14 open models, and analyze cross-lingual transfer, few-shot effects, and translation artifacts. The central claims are that model rankings transfer broadly from English to Kyrgyz on WinoGrande and BoolQ, to a lesser extent on MMLU, and that the large English–Kyrgyz gap on HellaSwag is consistent with translation-induced plausibility shifts. The paper also releases datasets, code, and per-model results and integrates tasks into Lighteval.

Significance. If the benchmark and conclusions are validated, the contribution is useful: KyrgyzMMLU and KyrgyzRC are scarce native-resource additions for a low-resource Turkic language, and the transparent documentation of construction and quality-control steps, plus public release, would support reproducibility. The paper also ships machine-checkable artifacts (datasets, code, per-model results) and an explicit falsifiable prediction about HellaSwag being translation-sensitive. However, the headline cross-lingual ranking claim is only partially supported by the reported tables, and the absence of uncertainty quantification or repeated-seed evaluation weakens some numerical comparisons. The benchmark resource itself is valuable even if the cross-lingual transfer claim requires revision.

major comments (6)
  1. [Abstract and Section 6 (Table 7)] The claim that 'model rankings transfer broadly from English to Kyrgyz on WinoGrande and BoolQ' is not supported for WinoGrande. In Table 7, all 14 open models score between 48.3% and 51.5% zero-shot on Kyrgyz WinoGrande, a two-choice task: this spread is within binomial sampling noise (approximately ±1.6 points for ~1000 items at p=0.5). Few-shot scores are similarly flat (48–52%). English WinoGrande scores in Table 6 range from 49.6% to 71.3%, so the English signal does not transfer. The few-shot English ranking is also not mirrored: Llama-3.1-8B-Instruct, top on English WinoGrande (71.3% zero-shot, 74.0% few-shot), is mid-pack on Kyrgyz (50.6%/51.6%). The conclusion as stated should be revised: the data indicate that Kyrgyz WinoGrande is non-discriminative at chance level, not that rankings transfer. A Spearman correlation computed from the tables would likely be near zero. The paper
  2. [Section 5 (decoding) and Tables 6–10] The evaluation uses temperature=0.6 and top-p=0.9, a stochastic decoding regime, but each number is reported as a single point estimate with no repeated seeds, no variance, and no confidence intervals. This is particularly problematic for the cross-lingual comparison, since many differences in Tables 6–7 are a few percentage points; with stochastic decoding and ~1000-item tasks, the standard error alone is ~1.6 points, and seed-to-seed variation at nonzero temperature can be several points. The paper should either report mean and standard deviation over multiple runs, or use greedy decoding with a justification. This is needed to support claims such as 'few-shot gains' and 'proprietary models exhibit less consistent few-shot behavior' (Section 6).
  3. [Section 4.2 and Limitations] The validity of Myanmar/English aside, the KyrgyzRC and KyrgyzMMLU items are load-bearing for every model score, but the annotation process is sequential (one author, one domain supervisor, one linguist) with no inter-annotator agreement. The paper itself notes Cohen's kappa was not computed and that items were not independently dual-annotated. This is an acknowledged limitation, but its consequences for the central MMLU/RC claims are not quantified. For a benchmark paper, a small held-out parallel-annotation study or a post-hoc item-error audit would substantially increase confidence that systematic item errors do not drive the results. As it stands, the reader cannot rule out that some aggregate differences—particularly the low KyrgyzMMLU scores—reflect item ambiguity or formatting issues rather than model capability.
  4. [Section 4.3 / Table 5] The WinoGrande example in Table 5 is a red flag for the translated benchmark's validity. The original English item uses 'home' vs 'house'; the Kyrgyz translation renders both as 'үй' (home) and 'турак жай' (dwelling), which are near-synonyms in this context. The sentence 'Ал менин үйүмө эч качан келбейт, бирок мен ар дайым анын турак жайына барам' does not preserve the pragmatic distinction that makes the original Winograd schema solvable. If this is representative, Kyrgyz WinoGrande may be unsolvable by design, which would explain the chance-level performance in Table 7. The authors should provide a per-item translation-quality audit or at least report the proportion of items where the two options are not semantically distinguished in Kyrgyz. This is directly relevant to the paper's own recommendation to audit translated HellaSwag, but the same scrutiny should apply to WinoGrande.
  5. [Section 6 and Table 8] The proprietary-model results are based on the condensed KyrgyzLLM Tiny Bench (100 randomly selected questions per subject for KyrgyzMMLU sub-parts, and tasks like BoolQ/HellaSwag/WinoGrande on a 100-item subset). With 100 binary items, the binomial standard error is ~5 points, so many differences in Table 8 (e.g., 48% vs 53% on WinoGrande) are within noise. The paper presents these numbers as task-level comparisons without acknowledging the reduced statistical power. Moreover, Gemini 2.5 Flash's scores are flagged as affected by safety refusals, but the table still includes those numbers; the paper should state how many refusals occurred and whether those answers were scored as incorrect, since this materially changes interpretation.
  6. [Section 6 cross-lingual consistency paragraph] The claim that rankings are 'broadly preserved on WinoGrande/BoolQ' is made without reporting any correlation coefficient or rank-order statistic. Given the chance-level WinoGrande data and the narrow BoolQ range in Table 7 (zero-shot: 37.9–58.3; few-shot: 57.4–76.9), a quantitative measure (e.g., Spearman rho with a confidence interval) is needed. Without it, the phrase 'broadly preserved' is not verifiable. The paper already computes English baselines; the additional analysis is straightforward.
minor comments (6)
  1. [Table 3] The sample KyrgyzMMLU question is a simple arithmetic word problem; it would help to show a more linguistically or culturally specific item to support the claim of cultural grounding.
  2. [Section 4.2 / Appendix C] Appendix C shows the same Wikipedia passage as Table 4; consider using a different example to illustrate the metadata schema and avoid redundancy.
  3. [Tables 6 and 7] The zero-shot and few-shot blocks in Tables 6–7 would be easier to read if the few-shot delta columns were aligned with the corresponding zero-shot columns; cell colors help but a numeric delta would be clearer.
  4. [Section 3] The claim that tokenizer fragmentation 'increases effective context length' should be phrased as 'increases the number of tokens (and thus context consumption) for the same content', to avoid ambiguity about what 'effective' means.
  5. [References] Reference [14] on the Kyrgyz Seed dataset is cited for a quality analysis of NLLB OPUS; the citation is plausible but the sentence could be more specific about which proportion and which language pairs were analyzed.
  6. [Appendix B] The GSMT8K exclusion rationale is clear, but the paper should mention the size of the translated GSM8K set and whether the translation will be released separately, since the current statement leaves ambiguity.

Circularity Check

0 steps flagged

No significant circularity: all load-bearing results are direct measurements against released Kyrgyz and English benchmarks, not derivations from fitted inputs or self-citation.

full rationale

The paper's central claims are empirical measurements: model accuracies on KyrgyzLLM-Bench, cross-lingual comparisons against English baselines, and a qualitative hypothesis about translation-induced plausibility shifts on HellaSwag. None of these quantities is defined in terms of another predicted quantity, and none is obtained by fitting a parameter to the outcome it is later said to predict. The natively authored KyrgyzMMLU and KyrgyzRC items are described as externally sourced from national educational materials and authentic texts, with documented multi-stage curation; the translated benchmarks are post-edited versions of standard English datasets. The HellaSwag 'plausibility-shift' explanation is explicitly presented as a hypothesis ('consistent with', 'we hypothesize', 'could reflect'), not as a quantity derived from the data. Self-citations to the authors' earlier KyrgyzNLP and TurkLang-2025 work appear, but they are used to situate the benchmark lineage and release provenance, not to justify a derivation or to forbid alternative explanations; the present paper reports direct runs over publicly released data and integrates into the external Lighteval framework. The Limitations section openly notes that Cohen's kappa was not computed for KyrgyzRC and that proprietary-model results may be affected by service refusals; these are validity and interpretability concerns, not circularity. The reported near-chance WinoGrande results noted by skeptics would challenge the empirical claim of rank transfer on that task, but a wrong or unsupported empirical conclusion is not a circularity: the numbers are measurements, not outputs forced by the inputs. The paper is therefore self-contained for circularity purposes, with no step reducing to its own inputs by construction.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The benchmark does not introduce new theoretical entities. Its load-bearing assumptions are about data quality, translation validity, parsing fidelity, and contamination absence; the paper handles the first three partly through disclosure but provides no quantitative support. The evaluation hyperparameters are manual choices rather than fitted values.

free parameters (4)
  • Decoding temperature / top-p = 0.6 / 0.9
    Chosen decoder settings for all model evaluations; results may shift under different settings, and no seed averaging or confidence intervals are reported.
  • Few-shot example counts = 5-shot (10-shot for HellaSwag)
    Manual choice following common practice; since ICL effects vary by task and model, this directly conditions the observed few-shot gains.
  • Tiny Bench subset size = 100 per subject
    Proprietary-model comparisons use only 100 randomly selected questions per subject with a fixed seed; sampling variance is not reported.
  • English MMLU subject subset = 17 subjects
    A hand-selected subset of English MMLU subjects chosen as 'analogous to' KyrgyzMMLU; the cross-lingual MMLU transfer claim depends on this selection.
axioms (4)
  • domain assumption KyrgyzMMLU and KyrgyzRC items are correct and unambiguous as produced by the sequential student-supervisor-linguist review.
    Load-bearing for all accuracy scores; no inter-annotator agreement was computed, as the paper acknowledges in Limitations.
  • domain assumption Translated benchmarks preserve the construct measured in English after post-editing.
    The HellaSwag analysis interprets the English-Kyrgyz gap as a plausibility shift rather than a translation failure (Sections 4.3 and 6).
  • domain assumption Regex-based text parsing captures the model's intended choice despite formatting variation.
    Text-based answer extraction is the scoring protocol; its sensitivity is acknowledged in Section 8 and evidenced by implausible table cells.
  • domain assumption GRT exam items are not substantially memorized by models during pretraining.
    No contamination analysis is reported; if public official exam questions appear in pretraining data, KyrgyzMMLU scores would overstate true Kyrgyz knowledge.

pith-pipeline@v1.3.0-alltime-deepseek · 23054 in / 10992 out tokens · 105891 ms · 2026-08-01T18:47:45.336089+00:00 · methodology

0 comments
read the original abstract

Evaluating large language models (LLMs) across languages remains challenging, as most multilingual benchmarks rely on translated English datasets, often obscuring linguistic and cultural specificity in the target language. This issue is particularly pronounced for less-resourced languages such as Kyrgyz, where reliable natively authored evaluation data are scarce. Building on previously introduced Kyrgyz-language evaluation datasets, this work reports the first systematic and large-scale evaluation of LLMs in Kyrgyz using the KyrgyzLLM-Bench benchmark suite. KyrgyzLLM-Bench comprises two natively authored datasets$-$KyrgyzMMLU and KyrgyzRC$-$together with carefully translated and manually post-edited versions of WinoGrande, HellaSwag, BoolQ, and TruthfulQA. We evaluate 26 open- and closed-source LLMs under zero-shot and few-shot settings, analyzing model performance, cross-lingual transfer, and the impact of translation artifacts on evaluation reliability. Across families and tasks, model rankings transfer broadly from English to Kyrgyz on WinoGrande and BoolQ, and to a lesser extent on MMLU, while HellaSwag exhibits a substantial English-Kyrgyz performance gap consistent with translation-induced plausibility shifts. Few-shot prompting improves several open-source models on reading comprehension but behaves inconsistently for proprietary models on translated tasks. We publicly release all datasets, evaluation code, and per-model results, and integrate the Kyrgyz tasks into a widely used multilingual evaluation framework to support future research on Kyrgyz NLP.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

46 extracted references · 1 linked inside Pith

  1. [1]

    Alekseev and T

    A. Alekseev and T. Turatali. KyrgyzNLP: challenges, progress, and future. InInternational Conference on Analysis of Images, Social Networks and Texts, pages 3–39. Springer, 2024

    2024

  2. [2]

    J. Bai, S. Bai, Y. Chu, Z. Cui, K. Dang, X. Deng, Y. Fan, W. Ge, Y. Han, F. Huang, B. Hui, L. Ji, M. Li, J. Lin, R. Lin, D. Liu, G. Liu, C. Lu, K. Lu, J. Ma, R. Men, X. Ren, X. Ren, C. Tan, S. Tan, J. Tu, P. Wang, S. Wang, W. Wang, S. Wu, B. Xu, J. Xu, A. Yang, H. Yang, J. Yang, S. Yang, Y. Yao, B. Yu, H. Yuan, Z. Yuan, J. Zhang, X. Zhang, Y. Zhang, Z. Zh...

    2023

  3. [3]

    Clark, K

    C. Clark, K. Lee, M.-W. Chang, T. Kwiatkowski, M. Collins, and K. Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. InProc. Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), Minneapolis, MN, USA, Jun. 2019

    2019

  4. [4]

    Cobbe, V

    K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, et al. Training verifiers to solve math word problems. InProc. International Conference on Learning Representations (ICLR), Virtual, May 2021

    2021

  5. [5]

    Dar¸ gis, G

    R. Dar¸ gis, G. B¯arzdi¸ nˇs, I. Skadi¸ na, N. Gr¯uz¯ıtis, and B. Saul¯ıte. Evaluating open-source llms in low-resource languages: Insights from latvian high school exams. InProceedings of the 4th International Conference on Natural Language Processing for Digital Humanities, pages 289–293, 2024. 15

    2024

  6. [6]

    L. Gao, J. Tow, S. Biderman, S. Black, A. DiPofi, C. Foster, L. Golding, J. Hsu, K. Mc- Donell, N. Muennighoff, et al. Language model evaluation harness.https://github.com/ EleutherAI/lm-evaluation-harness, 2023

    2023

  7. [7]

    Mesnard, C

    GemmaTeam, T. Mesnard, C. Hardin, R. Dadashi, S. Bhupatiraju, S. Pathak, L. Sifre, M. Rivi`ere, M. S. Kale, J. Love, P. Tafti, L. Hussenot, P. G. Sessa, A. Chowdhery, A. Roberts, A. Barua, A. Botev, A. Castro-Ros, A. Slone, A. H´eliou, A. Tacchetti, A. Bulanova, A. Paterson, B. Tsai, B. Shahriari, C. Le Lan, C. A. Choquette-Choo, C. Crepy, D. Cer, D. Ippo...

    2024

  8. [8]

    Goyal, C

    N. Goyal, C. Gao, V. Chaudhary, P.-J. Chen, G. Wenzek, D. Ju, et al. The flores-101 evaluation benchmark for low-resource and multilingual machine translation. InProceedings of ACL, 2022

    2022

  9. [9]

    Grattafiori, A

    A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, A. Yang, A. Fan, A. Goyal, A. Hartshorn, A. Yang, A. Mitra, A. Sravankumar, A. Korenev, A. Hinsvark, A. Rao, A. Zhang, A. Rodriguez, A. Gregerson, A. Spataru, B. Roziere, B. Biron, B. Tang, B. Chern, C. Caucheteux, C. Nayak, C. Bi, C. Mar...

    2024

  10. [10]

    Habib, C

    N. Habib, C. Fourrier, H. Kydl ´ıˇcek, T. Wolf, and L. Tunstall. Lighteval: A lightweight framework for llm evaluation, 2023

    2023

  11. [11]

    Hendrycks, C

    D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding.Proceedings of the International Conference on Learning Representations (ICLR), 2021. 17

    2021

  12. [12]

    J. Hu, S. Ruder, A. Siddhant, G. Neubig, O. Firat, and M. Johnson. Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalization. InProceed- ings of the 37th International Conference on Machine Learning, volume 119, pages 4411–4421, 2020

    2020

  13. [13]

    Open llm leaderboard (archived evaluation protocol).https://huggingface

    Hugging Face. Open llm leaderboard (archived evaluation protocol).https://huggingface. co/docs/leaderboards/en/open_llm_leaderboard/archive, 2024

    2024

  14. [14]

    Jumashev, A

    M. Jumashev, A. Tillabaeva, A. Kasieva, T. Omurkanov, A. Musaeva, M. E. Kyzy, G. Cha- gataeva, and J. Washington. The kyrgyz seed dataset submission to the wmt25 open language data initiative shared task. InProceedings of the Tenth Conference on Machine Translation, pages 1088–1102, 2025

    2025

  15. [15]

    A. Kan. Akylai smart speaker: Artificial intelligence speaking kyrgyz language (june 18th, 2024).https://web.archive.org/web/20240619010036/https://24.kg/english/296874_ AkylAI_smart_speaker_Artificial_intelligence_speaking_Kyrgyz_language/, 2024. Accessed: 2024-09-14

    arXiv 2024

  16. [16]

    V. D. Lai, C. Van Nguyen, N. T. Ngo, T. Nguyen, F. Dernoncourt, R. A. Rossi, et al. Okapi: Instruction-tuned large language models in multiple languages with reinforcement learning from human feedback. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 318–327, 2023

    2023

  17. [17]

    Z. Lin, J. Hilton, and O. Evans. Truthfulqa: Measuring how models mimic human falsehoods. InProc. Annual Meeting of the Association for Computational Linguistics (ACL), Dublin, Ireland, May 2022

    2022

  18. [18]

    Mirzakhalov, A

    J. Mirzakhalov, A. Babu, A. Kunafin, A. Wahab, B. Moydinboyev, S. Ivanova, et al. Evaluating multiway multilingual nmt in the turkic languages. InProceedings of the Sixth Conference on Machine Translation, pages 518–530, 2021

    2021

  19. [19]

    Nvidia nemo microservices: Model parameter tuning guidelines.https://docs

    NVIDIA. Nvidia nemo microservices: Model parameter tuning guidelines.https://docs. nvidia.com/nemo/microservices/latest/design-synthetic-data-from-scratch-or-seeds/ configure-models.html, 2025

    2025

  20. [20]

    Sakaguchi, R

    K. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y. Choi. Winogrande: An adversarial winograd schema challenge at scale. InProc. Thirty-Fourth AAAI Conf. on Artificial Intelligence (AAAI), New York, NY, USA, Feb. 2020

    2020

  21. [21]

    Salmorbekova, A

    R. Salmorbekova, A. Alymbaev, and A. Tukhtamatov. Kyrgyz and russian languages in the eurasian space.Bulletin of Science and Practice, 9(6):722–733, 2023. In Russian

    2023

  22. [22]

    Singh, A

    S. Singh, A. Romanou, C. Fourrier, D. I. Adelani, J. G. Ngui, D. Vila-Suero, P. Limkonchotiwat, K. Marchisio, W. Q. Leong, Y. Susanto, R. Ng, S. Longpre, S. Ruder, W.-Y. Ko, A. Bosselut, A. Oh, A. Martins, L. Choshen, D. Ippolito, E. Ferrante, M. Fadaee, B. Ermis, and S. Hooker. Global MMLU: Understanding and addressing cultural and linguistic biases in m...

    2025

  23. [23]

    Skadi¸ na, B

    I. Skadi¸ na, B. Bakanovs, and R. Dar¸ gis. First steps in benchmarking Latvian in large language models. InProceedings of the Third Workshop on Resources and Representations for Under- Resourced Languages and Domains (RESOURCEFUL-2025), pages 86–95. University of Tartu Library, 2025. 18

    2025

  24. [24]

    Turatali, A

    T. Turatali, A. Turdubaeva, I. Zhenishbekov, Z. Suranbaev, A. Alekseev, and R. Izmailov. Bridging the gap in less-resourced languages: Building a benchmark for kyrgyz language models. In2025 10th International Conference on Computer Science and Engineering (UBMK), pages 1673–1677, 2025

    2025

  25. [25]

    A chatbot for teenagers about puberty, relationships, and health launched in kyrgyzstan (may 24th, 2022).http://web.archive.org/web/20240525072322/https://iite

    UNESCO-IITE. A chatbot for teenagers about puberty, relationships, and health launched in kyrgyzstan (may 24th, 2022).http://web.archive.org/web/20240525072322/https://iite. unesco.org/highlights/oilo-chatbot-sex-ed-kyrgyzstan-en/, 2022. Accessed: 2024-09-14

    arXiv 2022

  26. [26]

    Vanmassenhove, D

    E. Vanmassenhove, D. Shterionov, and M. Gwilliam. Machine translationese: Effects of algorithmic bias on linguistic complexity in machine translation. InProceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2203–2213, 2021

    2021

  27. [27]

    Veitsman and M

    Y. Veitsman and M. Hartmann. Recent advancements and challenges of turkic central asian language processing. InProceedings of the First Workshop on Language Models for Low-Resource Languages, pages 309–324, 2025

    2025

  28. [28]

    A. Wang, Y. Pruksachatkun, N. Nangia, A. Singh, J. Michael, and F. Hill. Superglue: A stickier benchmark for general-purpose language understanding systems. InAdvances in Neural Information Processing Systems, 2019

    2019

  29. [29]

    A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. InProceedings of the 2018 EMNLP Workshop BlackboxNLP, pages 353–355, 2018

    2018

  30. [30]

    M. Wu, W. Wang, S. Liu, H. Yin, X. Wang, Y. Zhao, C. Lyu, L. Wang, W. Luo, and K. Zhang. The bitter lesson learned from 2,000+ multilingual benchmarks.arXiv preprint arXiv:2504.15521, 2025

    Pith/arXiv arXiv 2025

  31. [31]

    Zellers, A

    R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi. Hellaswag: Can a machine really finish your sentence? InProc. Conf. Empirical Methods in Natural Language Processing (EMNLP), Hong Kong, China, Nov. 2019. 19 A Prompting strategies For benchmarks other thanKyrgyzMMLUorKyrgyzRC, the prompts we have used are direct translations of original English q...

    2019

  32. [34]

    {example_01_choices[2]} Туура жоопту тандаңыз: {example_01_answer} Текст: {example_02_text} Суроо: {example_02_question} Сунушталган жооптор:

  33. [38]

    {example_02_choices[3]} Туура жоопту тандаңыз: {example_02_answer} Текст: {example_03_text} Суроо: {example_03_question} Сунушталган жооптор:

  34. [39]

    {example_03_choices[0]}

  35. [40]

    {example_03_choices[1]} Туура жоопту тандаңыз: {example_03_answer} Текст: {text} Суроо: {question} Сунушталган жооптор:

  36. [41]

    Сиз билимиңизге жана жөндөмүңүзгө жараша суроолорго жооп берген AIсыз

    {choices[3]} Туура жоопту тандаңыз: 20 Listing A2: Prompt for few-shot solution forKyrgyzMMLU(actual prompt is built dynamically in Python code, some details have been removed). Сиз билимиңизге жана жөндөмүңүзгө жараша суроолорго жооп берген AIсыз. Сизге суроо жана 2-5 жооп варианты берилет, туура жооптун НОМЕРИН (индексин) гана кайтарышыңыз керек.,→ Суро...

  37. [42]

    {example_01_choices[0]}

  38. [43]

    {example_01_choices[1]}

  39. [44]

    {example_01_choices[2]}

  40. [45]

    {example_01_choices[3]}

  41. [46]

    {example_01_choices[4]} Туура жоопту тандаңыз: {example_01_answer} Суроо: {example_02_question} Сунушталган жооптор:

  42. [47]

    {example_02_choices[0]}

  43. [48]

    {example_02_choices[1]}

  44. [49]

    {example_02_choices[2]}

  45. [50]

    {example_02_choices[3]} Туура жоопту тандаңыз: {example_02_answer} Суроо: {question} Сунушталган жооптор:

  46. [51]

    source_type

    {choices[4]} Туура жоопту тандаңыз: BGSM8KTranslation Although theGSM8Kdataset was translated into Kyrgyz, we do not include its results in the main reported version of the benchmark. This decision was made for several reasons. First, the evaluation protocol ofGSM8Kdiffers substantially from that of the other benchmarks considered in this work, as it reli...