MM-Eval: A Hierarchical Benchmark for Modern Mongolian Evaluation in LLMs

Bo Xia; Mengyuan Zhang; Ruihui Wang; Xiaobing Zhao; Yuan Sun

REVIEW 4 major objections 4 minor 19 references

MM-Eval: A Hierarchical Benchmark for Modern Mongolian Evaluation in LLMs

T0 review · 4 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper introduces MM-Eval, a four-level benchmark for modern Mongolian in Cyrillic script, and reports that every LLM it tests handles syntax better than semantics, with reasoning scores far lower than either.

desk verdict A useful new benchmark for Mongolian LLM evaluation whose main finding rests on the least-verified section of the dataset; deserves review but needs a label audit. read the letter →

arxiv 2411.09492 v1 pith:3TVZLTPS submitted 2024-11-14 cs.CL cs.AI

Mengyuan Zhang , Ruihui Wang , Bo Xia , Yuan Sun , Xiaobing Zhao This is my paper · ORCID

classification cs.CLcs.AI

keywords MM-EvalModernMongolianLLMevaluationlow-resourcelanguagesyntaxsemanticsknowledgereasoning

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MM-Eval is a new benchmark for evaluating large language models on modern Mongolian, a low-resource language written in Cyrillic. The dataset is organized into four levels: 569 syntax questions, 677 semantics questions, 344 knowledge questions, and 250 reasoning problems. The paper tests five open- and closed-source models and finds a consistent pattern: all models score higher on syntax than on semantics, and all struggle with reasoning, where the best accuracy is 29.6%. The authors argue this reveals a gap between surface grammatical ability and deeper meaning understanding, while knowledge performance suggests that general knowledge transfers from high-resource languages to Mongolian.

What carries the argument

The Dual Capability Framework organizes the benchmark into language abilities (syntax, semantics) and cognitive abilities (knowledge, reasoning). Syntax items are built by shuffling word order in textbook sentences, semantics items are fill-in-the-blank questions with same-part-of-speech distractors, knowledge items come from filtered WebQSP facts plus ChatGPT-generated and manually verified common-knowledge pairs, and reasoning items are MGSM math word problems translated into Mongolian. This framework lets the paper attribute low scores to either a missing language-specific ability or a missing general cognitive capacity.

What would settle it

Have professional Mongolian teachers independently re-annotate all 677 semantic items and a random sample of the syntax, knowledge, and reasoning sets. If agreement on the gold answers is low, or if many shuffled-order syntax options are judged acceptable in colloquial Mongolian, then the reported syntax-over-semantics gap and the reasoning ceiling of 29.6% would not survive.

Watch

Extended reading notes

Core claim

The central claim is that current LLMs display a clear capability hierarchy in modern Mongolian: syntactic competence exceeds semantic competence, and reasoning is the weakest area. GPT-4-Turbo reaches 90.69% on syntax and 80.52% on knowledge, Qwen2-7B-Instruct leads semantics at 54.8%, and DeepseekV2.5 achieves the highest reasoning score at 29.6%. The paper interprets this as evidence that models partially master Mongolian grammar but lack deeper language understanding and complex reasoning in this low-resource language.

Load-bearing premise

The benchmark's gold labels and distractor options are accurate: every multiple-choice and numeric answer was manually verified, but no inter-annotator agreement or independent quality metric is reported, so a substantial share of mislabeled items would change every reported accuracy.

Editorial extensions

If this is right

Model rankings change across the four levels, so a single aggregate score for Mongolian would hide that GPT-4-Turbo leads syntax and knowledge while Qwen2-7B-Instruct leads semantics.
The near-ordering syntax > knowledge > semantics > reasoning across most tested models suggests that language ability and cognitive ability should be evaluated separately for low-resource languages.
Knowledge accuracy of 59–80% alongside weaker semantics implies that general knowledge transfers across languages better than language-specific semantic skill.
The reasoning floor of 5–29.6% identifies Mongolian math word problems as a clear target for future training data and model improvement.
The dataset provides a reusable test set for tracking whether future models improve on Mongolian without relying on machine-translation benchmarks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the syntax–semantics gap is genuine, a plausible follow-up is to test whether it narrows when models receive longer context or prompts in traditional Mongolian script, since the current design uses isolated sentences and may underestimate semantic ability.
The reasoning ceiling may partly reflect translation quality rather than pure reasoning capacity; running the same MGSM problems in English with the same models would separate translation failure from reasoning failure.
Because part of the knowledge section was generated with ChatGPT and then translated, models trained on similar outputs may score artificially high; a contamination-controlled version would be needed for stable conclusions.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

The useful thing here is the resource, not the results. MM-Eval gives the first hierarchical benchmark for Modern Mongolian — 1,840 MCQ/math items across syntax, semantics, knowledge, and reasoning, with baselines for five LLMs. That is a real contribution for a language that barely appears in LLM evaluation. The Dual Capability Framework is a simple taxonomy, not deep theory, but it is a reasonable way to organize the evaluation. The choice to source syntax and semantics from a textbook and to translate WebQSP and MGSM is sensible, and the manual checks described for knowledge and reasoning are a point in the paper's favor.

The soft spots are mostly around the Semantics section, and one is load-bearing. Section 3.5 says distractor words were chosen to be 'plausible yet definitively incorrect,' but unlike sections 3.6 and 3.7 it reports no manual verification or quality check for those distractors. Same-part-of-speech distractors drawn from a vocabulary list can easily be grammatically and pragmatically compatible with the cloze sentence. If a non-trivial share of the 677 semantics items have more than one defensible answer, the paper's main empirical finding — every model scores higher on syntax than on semantics — becomes a labeling artifact rather than a measurement of deeper understanding. The paper needs an audit: at least a human agreement study or a clearly reported manual pass with numbers.

There is also a plain internal inconsistency that undermines confidence in the tables. Table 1 lists ChatGPT-4-Turbo as best on Semantics with 72.53% and Qwen2-7B at 54.8%, but Section 4.3 says 'Qwen2-7B-Instruct performs well in semantics (72.53%)'. That is not a subtle error; it attributes the table's best to the wrong model. The authors should fix it and double-check the rest of the reported numbers.

Missing baselines and statistics are minor compared with the label problem, but still worth listing: no random baseline (25% for four-option items), no confidence intervals, no contamination analysis. The circularity (GPT-4 generated part of the data it is scored on) is real but partial, since manual verification is claimed for knowledge and reasoning. The transfer claim about knowledge relies on a comparison with high-resource languages that is not in the paper.

Bottom line: this paper is for anyone working on low-resource language evaluation, especially Mongolian NLP. The benchmark is a genuine resource, and the paper deserves peer review. I would condition acceptance on fixing the attribution error, validating the semantics labels with an independent manual pass, and adding simple baselines and confidence intervals.

Referee Report

4 major / 4 minor

Summary. The paper introduces MM-Eval, a benchmark for evaluating LLMs on Modern Mongolian (Cyrillic script), organized into a Dual Capability Framework with four hierarchical levels: syntax, semantics, knowledge, and reasoning. The dataset comprises 569 syntax MCQs drawn from a Mongolian textbook with shuffled-word distractors, 677 semantics MCQs constructed by cloze deletion with same-part-of-speech distractors from textbook vocabulary, 344 knowledge MCQs derived from WebQSP and ChatGPT-generated content, and 250 reasoning math problems translated from MGSM. The authors evaluate five models (Qwen2-7B-Instruct, GLM4-9b-chat, Llama3.1-8B-Instruct, GPT-4-Turbo, and DeepseekV2.5) and report that all models score higher on syntax than on semantics, that knowledge performance shows a moderate decline, and that all models perform poorly on reasoning (best 29.6%). The dataset is publicly released.

Significance. If the benchmark's labels and claims are sound, MM-Eval would be a useful resource for a genuinely under-served language: it provides a structured, multi-level evaluation covering both language proficiency and cognitive transfer, and the finding that semantic understanding lags syntactic competence in low-resource settings is potentially important. The paper also makes a plausible conceptual contribution in separating language abilities from cognitive abilities. However, the empirical value of the contribution currently rests on unverified assumptions about label quality and on an experimental design that involves the tested model in the construction of part of the data. The release of the dataset and the explicit descriptions of the construction pipeline are positive features, but they do not by themselves establish the reliability of the reported accuracies.

major comments (4)

[§3.5 (Semantics Eval)] The semantics labels are the load-bearing component of the paper's central claim that syntax accuracy exceeds semantics accuracy for all models. The construction procedure selects same-part-of-speech distractors from the vocabulary list, but the manuscript reports no manual verification or correction step for these distractors, in contrast with §3.6 and §3.7 where manual proofreading is explicitly described. A cloze item with same-part-of-speech distractors can admit multiple plausible completions: a noun or adjective that is grammatical may still be semantically compatible with the sentence context, and whether it is 'definitively incorrect' is precisely the judgment that is never independently validated. If a substantial fraction of the 677 Semantics items have more than one correct answer, the reported semantics accuracies and the uniform syntax-over-semantics ordering are not a reliable measure of deeper language understanding. The authors must add a manual verification and correction step, report inter-annotator agreement, or otherwise demonstrate that each item has exactly one correct answer and three unambiguously incorrect distractors, and then re-release the dataset and recompute Table 1 if any items change.
[§4.3 and Table 1] There is a direct inconsistency between Table 1 and §4.3 regarding the best-performing semantic model. Table 1 lists chatgpt4-turbo as achieving 72.53% on Semantics and qwen2-7b-instruct as 54.8%, while §4.3 states that 'Qwen2-7B-Instruct performs well in semantics (72.53%)', attributing the table's best value to the wrong model. Figure 2 is also described in a duplicated paragraph that swaps the roles of Table 1 and Figure 2. These inconsistencies make it impossible to know which numbers the authors actually stand behind and undermine confidence in the reported experimental results. The authors must correct the text, reconcile Table 1, Figure 2, and §4.3, and re-audit all reported numbers.
[§4.1 and Table 1] The experimental comparison lacks a chance baseline and any measure of statistical uncertainty. All multiple-choice items have four options, so a random model would be expected to score 25% on Syntax, Semantics, and Knowledge by chance; several reported scores (e.g., Llama-3.1-8B on Semantics at 28.06%, and Qwen2 on Reasoning at 6%) are only slightly above or even below this floor, and without confidence intervals or significance tests the claim that 'all models performed better on syntactic tasks than semantic tasks' is not statistically supported. The authors should report binomial confidence intervals or standard errors for each accuracy, provide a random baseline, and ideally run multiple inference seeds (the current temperature=0 setting gives one deterministic outcome).
[§3.6, §3.7, and §4.2] The evaluation is partially circular for the closed-source models. The knowledge distractors in §3.6 are generated with the ChatGPT API, and the Mongolian translations of reasoning problems in §3.7 are produced and verified by the ChatGPT API; GPT-4-Turbo is then one of the evaluated models in §4.2. This means the tested model directly contributed content to the benchmark on which its own knowledge and reasoning scores are computed, so the reported 80.52% knowledge accuracy and 26.8% reasoning accuracy for GPT-4-Turbo are not independent measurements. The authors should either remove GPT-4-Turbo from the evaluated models, construct the knowledge and reasoning sections without using the evaluated model family, or demonstrate through a contamination analysis that the model's scores are not inflated by its own generated content. They should also report whether any contamination check was performed against model training corpora for the textbook- and translated-based items.

minor comments (4)

[§5 (Discussion)] The Discussion states that MM-Eval is 'limited by its single content source', but the benchmark actually uses three sources: the textbook, WebQSP, and MGSM, not to mention ChatGPT-generated knowledge items. This wording should be clarified to indicate that the language-ability section relies on a single textbook, while the cognitive-ability section uses multiple sources.
[Table 1 and §4.2] Model names are inconsistent: Table 1 uses lowercase 'deepseekv2.5' and 'chatgpt4-turbo', while §4.2 refers to 'GPT-4-Turbo-04-09' and 'DeepseekV2.5'. Please standardize model names, including for GLM4-9b-chat and Llama3.1-8B-Instruct.
[§4.3] The two consecutive paragraphs beginning 'Figure 2 presents the corresponding results...' and 'Table 1 presents...' contain redundant and conflicting descriptions; one of the paragraphs should be removed and the remaining text should accurately describe which figure shows what.
[§4.2] The paper cites a reference for GLM-130B but evaluates GLM4-9b-chat; a reference or model card for the actual evaluated GLM4 checkpoint should be provided.

Circularity Check

0 steps flagged · score 0.0 of 10

Benchmark construction with fixed external labels; no derivation reduces to its inputs.

full rationale

MM-Eval is a benchmark-construction paper rather than a predictive or first-principles derivation: there are no fitted parameters, no equations, and no claim that a model's score is derived from the data-generation procedure. The central empirical claim (all models score higher on syntax than semantics, and all struggle on reasoning) is computed directly from model outputs against fixed, pre-existing labels drawn from a textbook, WebQSP, and MGSM, so it cannot be circular in the definitional or statistical-forcing sense. The use of ChatGPT API in Sections 3.6 and 3.7 to generate knowledge distractors and to check Mongolian translations is a legitimate benchmark-contamination and validity concern, since GPT-4-Turbo is also an evaluated model, but it does not make any reported score equal to an input by construction: correct answers originate from external sources or rule-based facts and were manually verified, and model accuracies are not fitted parameters. The unvalidated 'definitively incorrect' semantics distractors in Section 3.5 threaten construct validity, and the Section 4.3 misattribution of the 72.53% semantics score to Qwen2 rather than ChatGPT-4-Turbo is an internal inconsistency, but neither is a circular step. No self-citation chain, imported uniqueness theorem, or ansatz-via-citation is load-bearing. Thus the paper shows no significant circularity.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The benchmark rests on the assumed correctness of textbook-derived sentences, ChatGPT-generated and translated items, and manually verified labels. No inter-annotator agreement, quality statistics, or contamination checks are reported. The inference hyperparameters are arbitrary choices that could affect absolute accuracies.

free parameters (1)

inference_hyperparameters = temperature=0, top_p=0.1, frequency_penalty=1
These settings are chosen by hand and are not justified; the reported accuracies depend on them. The unusual frequency_penalty=1 may suppress numeric output in the reasoning tasks and could lower all reasoning scores.

assumptions (5)

domain assumption Sentences from Modern Mongolian Language Textbook I are representative of modern Mongolian and were correctly OCR'd and cleaned.
Section 3.3 states OCR, cleaning, and manual correction, but no quality metrics are provided.
domain assumption ChatGPT-generated knowledge items and distractors, after manual proofreading, are correct and natural Mongolian.
Section 3.6 describes generation and manual verification; this relies on the annotators' Mongolian proficiency and is not validated externally.
domain assumption Translated MGSM problems preserve the original mathematical content and answers.
Section 3.7 relies on ChatGPT comparison and manual checks; no evaluation of translation quality or answer preservation is reported.
ad hoc to paper The Dual Capability Framework's split into syntax, semantics, knowledge, and reasoning is a meaningful and non-overlapping taxonomy.
Section 3.1 introduces the framework; its validity as a measurement model is not justified beyond assertion.
domain assumption The tested models have not memorized the evaluation items (no contamination), despite GPT-4 being used to generate part of the data.
No contamination check is reported; Section 3.6 and 3.7 show GPT-4 involvement in data creation.

how reviews work

0 comments

Cite this review

Pith. "Pith review of MM-Eval: A Hierarchical Benchmark for Modern Mongolian Evaluation in LLMs." pith.science (2026). https://pith.science/paper/3TVZLTPS

@misc{pith2026241109492,
  author       = {Pith},
  title        = {Pith review of: MM-Eval: A Hierarchical Benchmark for Modern Mongolian Evaluation in LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3TVZLTPS}},
  note         = {Machine review of arXiv:2411.09492}
}

read the original abstract

Large language models (LLMs) excel in high-resource languages but face notable challenges in low-resource languages like Mongolian. This paper addresses these challenges by categorizing capabilities into language abilities (syntax and semantics) and cognitive abilities (knowledge and reasoning). To systematically evaluate these areas, we developed MM-Eval, a specialized dataset based on Modern Mongolian Language Textbook I and enriched with WebQSP and MGSM datasets. Preliminary experiments on models including Qwen2-7B-Instruct, GLM4-9b-chat, Llama3.1-8B-Instruct, GPT-4, and DeepseekV2.5 revealed that: 1) all models performed better on syntactic tasks than semantic tasks, highlighting a gap in deeper language understanding; and 2) knowledge tasks showed a moderate decline, suggesting that models can transfer general knowledge from high-resource to low-resource contexts. The release of MM-Eval, comprising 569 syntax, 677 semantics, 344 knowledge, and 250 reasoning tasks, offers valuable insights for advancing NLP and LLMs in low-resource languages like Mongolian. The dataset is available at https://github.com/joenahm/MM-Eval.

Figures

Figures reproduced from arXiv: 2411.09492 by the authors.

**Figure 3.** Syntax Eval Examples A.2 Semantics Eval [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

**Figure 6.** Reasoning Eval Examples 7 [PITH_FULL_IMAGE:figures/full_fig_p007_6.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

19 extracted references · 11 canonical work pages

[3]

CoRR, abs/2110.14168

Training verifiers to solve math word problems. CoRR, abs/2110.14168. Damai Dai, Chengqi Deng, Chenggang Zhao, R. X. Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, et al

arXiv
[5]

In 9th Inter- national Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7,

Measuring mas- sive multitask language understanding. In 9th Inter- national Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7,

work page 2021
[6]

In Forty- first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27,

LLM maybe longlm: Selfex- tend LLM context window without tuning. In Forty- first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27,

work page 2024
[7]

In Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024 , pages 15670–15693

From tarzan to tolkien: Con- trolling the language proficiency level of llms for content generation. In Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024 , pages 15670–15693. Association for Computational Linguistics. OpenAI

work page 2024
[9]

CoRR, abs/2311.12022

GPQA: A graduate-level google-proof q&a bench- mark. CoRR, abs/2311.12022. Tobias Schimanski, Jingwei Ni, Mathias Kraus, Elliott Ash, and Markus Leippold

arXiv
[10]

In Proceedings of the 62nd An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pages 1913–

Towards faith- ful and robust LLM specialists for evidence-based question-answering. In Proceedings of the 62nd An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pages 1913–

work page 2024
[11]

In The Eleventh International Conference on Learning Representa- tions, ICLR 2023, Kigali, Rwanda, May 1-5,

Language models are mul- tilingual chain-of-thought reasoners. In The Eleventh International Conference on Learning Representa- tions, ICLR 2023, Kigali, Rwanda, May 1-5,

work page 2023
[12]

Finesure: Fine-grained summarization evaluation using llms. In Proceed- ings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pages 906–922. Association for Computational Linguistics. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi...

work page 2024

Show all 19 references

[13]

CoRR, abs/2307.09288

Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307.09288. Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Benjamin Feuer, et al

arXiv
[14]

CoRR, abs/2406.19314

Livebench: A chal- lenging, contamination-free LLM benchmark. CoRR, abs/2406.19314. Guohai Xu, Jiayi Liu, Ming Yan, Haotian Xu, Jinghui Si, et al

arXiv
[15]

CoRR, abs/2307.09705

Cvalues: Measuring the values of chinese large language models from safety to responsibility. CoRR, abs/2307.09705. Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Mur- ray, and Young Jin Kim

arXiv
[16]

In Forty-first In- ternational Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27,

Contrastive prefer- ence optimization: Pushing the boundaries of LLM performance in machine translation. In Forty-first In- ternational Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27,

2024
[17]

CoRR, abs/2407.10671

Qwen2 technical report. CoRR, abs/2407.10671. Ziqing Yang, Zihang Xu, Yiming Cui, Baoxin Wang, Min Lin, et al

arXiv
[19]

In The Eleventh Inter- national Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5,

GLM-130B: an open bilin- gual pre-trained model. In The Eleventh Inter- national Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5,

2023
[2016]

In Proceedings of the 54th Annual Meet- ing of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Vol- ume 2: Short Papers

The value of semantic parse labeling for knowledge base question answering. In Proceedings of the 54th Annual Meet- ing of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Vol- ume 2: Short Papers. The Association for Computer Lingui...

2016
[2020]

MTNER: A corpus for mongolian tourism named entity recognition. In Machine Trans- lation - 16th China Conference, CCMT 2020, Hohhot, China, October 10-12, 2020, Revised Selected Pa- pers, volume 1328 of Communications in Computer and Information Science, pages 11–23. Springer....

2020
[2021]

CoRR, abs/2107.03374

Evalu- ating large language models trained on code. CoRR, abs/2107.03374. Xiao Cheng, Weihua Wang, Feilong Bao, and Guanglai Gao

arXiv
[2023]

CoRR, abs/2303.08774

GPT-4 technical report. CoRR, abs/2303.08774. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, et al

arXiv
[2024]

Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, Au- gust 11-16, 2024, pages 1280–1297. As...

2024

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [3]

CoRR, abs/2110.14168

Training verifiers to solve math word problems. CoRR, abs/2110.14168. Damai Dai, Chengqi Deng, Chenggang Zhao, R. X. Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, et al

arXiv

[2] [5]

In 9th Inter- national Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7,

Measuring mas- sive multitask language understanding. In 9th Inter- national Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7,

work page 2021

[3] [6]

In Forty- first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27,

LLM maybe longlm: Selfex- tend LLM context window without tuning. In Forty- first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27,

work page 2024

[4] [7]

In Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024 , pages 15670–15693

From tarzan to tolkien: Con- trolling the language proficiency level of llms for content generation. In Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024 , pages 15670–15693. Association for Computational Linguistics. OpenAI

work page 2024

[5] [9]

CoRR, abs/2311.12022

GPQA: A graduate-level google-proof q&a bench- mark. CoRR, abs/2311.12022. Tobias Schimanski, Jingwei Ni, Mathias Kraus, Elliott Ash, and Markus Leippold

arXiv

[6] [10]

In Proceedings of the 62nd An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pages 1913–

Towards faith- ful and robust LLM specialists for evidence-based question-answering. In Proceedings of the 62nd An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pages 1913–

work page 2024

[7] [11]

In The Eleventh International Conference on Learning Representa- tions, ICLR 2023, Kigali, Rwanda, May 1-5,

Language models are mul- tilingual chain-of-thought reasoners. In The Eleventh International Conference on Learning Representa- tions, ICLR 2023, Kigali, Rwanda, May 1-5,

work page 2023

[8] [12]

Finesure: Fine-grained summarization evaluation using llms. In Proceed- ings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pages 906–922. Association for Computational Linguistics. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi...

work page 2024