Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

LingBench++: A Linguistically-Informed Benchmark and Reasoning Framework for Multi-Step and Cross-Cultural Inference with LLMs

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LingBench++ claims that language models with access to reference grammars and iterative multi-agent reasoning outperform single-pass models on linguistics-olympiad-style puzzles, and that stepwise evaluation reveals where reasoning fails.

desk verdict Benchmark contributions are real and worth refereeing; the headline accuracy/interpretability claim rests on single-run noise that the authors themselves hedge against. read the letter →

arxiv 2507.16809 v2 pith:HUWW3WI4 submitted 2025-07-22 cs.CL

Da-Chen Lian , Ri-Sheng Huang , Pin-Er Chen , Chunki Lim , You-Kuan Lin , Guan-Yu Tseng , Zi-Cheng Yang , Zhen-Yu Lin
show 2 more authors
Pin-Cheng Chen Shu-Kai Hsieh
This is my paper · ORCID
classification cs.CL
keywords linguisticsolympiadLLMreasoningevaluationmulti-agentretrieval-augmentedgenerationtypologicalmetadatalow-resourcelanguagesstepwisecross-culturalinference
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

LingBench++ is a bid to change how language models are tested on linguistics-olympiad-style puzzles: instead of only checking final answers, it scores the reasoning that produced them. The benchmark contains 96 curated IOL problems (225 sub-problems) across more than 90 low-resource and cross-cultural languages, each with typological metadata and an expert-verified gold reasoning trace. On top of this, the paper develops a multi-agent architecture with a Solver, an Aggregator, and a Grammar Agent that retrieves reference grammars, and argues that such models, equipped with external knowledge and iterative hypothesis testing, outperform single-pass approaches in both accuracy and interpretability. The experiments are preliminary, and the authors note that gains from guided prompts and the Grammar Agent are less clear than gains from more aggregation rounds, but the framework is presented as a diagnostic foundation for reasoning evaluation.

What carries the argument

The load-bearing object is the gold-standard reasoning trace (GRR): for each problem, a step-by-step solution showing the deductions and linguistic rules that lead to the official answer, generated by an LLM and then manually verified and refined by seven linguistic experts and three IOL contestants. Every GRR anchors two evaluation instruments: the final-solution score, which combines exact-match (or fuzzy/select-tagged) answer grading with an LLM-graded ‘rule checklist’ extracted from the official solution, and the Check-of-Thought protocol, in which an LLM judge scores reasoning along five dimensions, three aligned to the GRR and two assessing internal consistency. The second mechanism is the Grammar Agent’s retrieval pipeline: a knowledge base of reference grammars for over 1,100 languages is chunked, embedded, and searched by full-text, vector, hybrid, or metadata queries, and the retrieved grammatical information is summarized for the Solver.

What would settle it

Have independent linguists re-derive the reasoning traces for a random sample of the 96 problems without seeing the gold trace, and measure agreement on each Check-of-Thought dimension; if agreement is low, or if the official IOL solutions admit multiple valid derivation paths that the traces omit, the stepwise scores and any interpretability comparison are not trustworthy. A second decisive check is to rerun the multi-agent and grammar-agent experiments across many seeds to see whether the reported average-score gains exceed run-to-run stochasticity.

Watch

Extended reading notes

Core claim

The central claim is that, for IOL-style linguistic puzzles, a model with external grammatical knowledge and iterative, multi-round reasoning outperforms a single-pass model on both correctness and explainability. The evidence reported is consistent with the iterative-reasoning half: in a multi-round aggregation setup, average scores rose with each additional round for both tested models, with Gemini-2.5-pro rising from 0.381 in the single-pass baseline to 0.459 after six rounds. The paper also argues that final-answer grading is misleading without an explanation check: it scores answers and rule explanations separately, and proposes the Check-of-Thought protocol, which grades reasoning steps against the gold trace along dimensions such as stepwise logical validity, hypothesis adequacy, rule-induction coverage, and contradiction detection. A companion multilingual translation study shows that output quality and refusal rates track resource availability and language family, motivating the use of external linguistic knowledge. The paper presents these as preliminary results, with the grammar-agent and guided-prompt effects explicitly flagged as less clear.

Load-bearing premise

The load-bearing premise is that the gold-standard reasoning traces are correct and complete references for judging reasoning quality; they were generated by an LLM and manually verified, but no inter-annotator agreement or alternative-solution audit is reported, so any model that reasons correctly via a different path could be systematically scored down.

Editorial extensions

If this is right

  • Agentic retrieval over reference grammars becomes a practical route to improving LLM performance on low-resource linguistic reasoning, where pre-training data are sparse.
  • Stepwise evaluation can distinguish a model that guesses the right answer from one that reasons soundly, and can localize failures to specific reasoning dimensions.
  • Typological metadata linking problems to language family, script, and speaker-count data would let researchers diagnose why models fail on certain language types.
  • Adding aggregation rounds to a multi-agent setup is expected to keep raising average scores at additional compute cost, so ablating parallel generation from iterative refinement is the next necessary experiment.
  • Multilingual evaluations should report resource class and translation direction, since English-to-target generation is both weaker and more sensitive to data scarcity.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The trace-based evaluation presumes one correct way to reason; a model that reaches the official answer through a different but valid derivation could be scored down, and no inter-annotator agreement is reported for the trace verification.
  • The claimed interpretability advantage rests on a single demonstrated Check-of-Thought application; scaling it would require validating that automated LLM judges agree with human linguists.
  • A clean test of the Grammar Agent’s value would compare its score on problems whose languages have reference grammars versus those that do not, holding problem difficulty fixed.
  • The multi-round gains may come from the aggregator selecting the best draft rather than synthesizing new reasoning; feeding the aggregator one strong draft versus several weaker drafts would separate those mechanisms.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces LingBench++, a benchmark of 96 IOL problems (225 subproblems) with transcribed problem texts, official solutions, expert-verified reasoning traces generated with Gemini-2.5-Pro, typological annotations, and a two-component final-answer grading protocol (answer exact-match plus a rule-checklist explanation score). It also proposes a Check-of-Thought protocol for stepwise reasoning evaluation, reports a FLORES-200 translation study of gemini-2.5-flash as a function of resource class, language family, and script, and evaluates a multi-agent framework (Solver, Aggregator, Grammar Agent) with Mixture-of-Agents iterative rounds. The central claim, stated in the abstract, is that models with external knowledge and iterative reasoning outperform single-pass approaches in both accuracy and interpretability.

Significance. The benchmark itself is a useful contribution: final-answer scoring is grounded in official IOL solutions rather than in the proposing model, the typological metadata enables structured diagnostics, and the grammar-agent knowledge base covers over 1,100 languages. The paper also honestly discloses that its agentic experiments are single runs and that statistical tests are needed, which is an appropriate caution. The FLORES-200 analysis is more carefully controlled and provides credible evidence that resource class and language family affect translation quality. The interpretability protocol is innovative in principle. However, the paper's headline comparative claim is not supported by the current evidence: the accuracy gains in Table 7 are on the scale of run-to-run noise, and interpretability is demonstrated only on a single example with no comparative data. The strengths are real, but the claims outrun the experiments.

major comments (4)
  1. [Section 5.2, Table 7] The abstract's claim that models with external knowledge and iterative reasoning "outperform single-pass approaches in both accuracy and interpretability" is not supported by the reported evidence. Every agentic row is a single run, and the dagger rows show that conceptually identical settings differ by 0.028 for Gemini-2.5-pro (baseline 0.381 vs. single-agent first round 0.353) and by 0.007 for OpenAI-o4-mini, placing run-to-run noise on the same scale as several reported effects. On the matched grammar-agent subset (Table 10), the Grammar Agent gain over the Gemini baseline is 0.014 (0.373 to 0.387), while Gemini MoA-2 on the same subset scores 0.409. The paper itself states that "statistical tests on multiple runs would be required" (Section 5.2). Without confidence intervals, paired tests, or repeated runs, the superiority claim cannot be asserted; the abstract should be revised to present these results as preliminary trends rather than as demonstrations.
  2. [Section 3.5.3, Table 3] The Check-of-Thought protocol is demonstrated on exactly one problem and only for the baseline model's reasoning; the paper contains no comparison of interpretability scores between single-pass and agentic models. Consequently, the abstract's claim about improved interpretability is not supported by any comparative experiment. The protocol is a promising proposal, but the paper should either add a comparative interpretability evaluation or explicitly restrict the claim to final-answer accuracy.
  3. [Section 3.2] The gold-standard reasoning traces (GRR) are generated by Gemini-2.5-Pro and then "manually verified" by seven experts and three IOL contestants, but no inter-annotator agreement, resolution criteria, or quality metrics are reported, and the paper does not state how alternative valid solution paths are handled. Since the Check-of-Thought alignment metrics (SLVS, ISC, HGA, RIC, IJC, CCS, SCR) all compare against this GRR, incomplete or biased traces would systematically penalize models that reason correctly but differently. At minimum, the authors should report verification statistics and a policy for handling multiple valid solution paths before the reasoning scores are used as evidence.
  4. [Section 5.1.1, Table 10] The Grammar Agent comparison is confounded by selection: it is evaluated only on the 77 problems whose languages have reference grammars, and the paper notes that the relationship between grammar-book coverage and problem properties remains unexplored. Even after matching to the same subset in Table 10, the Grammar Agent gain over the Gemini baseline is 0.014, which is within the run-to-run noise quantified by the dagger rows in Table 7. This portion of the analysis should be reframed as a proof-of-concept for the retrieval mechanism rather than as evidence of a robust performance advantage.
minor comments (5)
  1. [Section 3.5.1] For subproblems tagged <fuzzy>, the text says "another metric of the user's choice should be applied," but the paper never states which metric was actually used in the 96-problem evaluation; this should be specified for reproducibility.
  2. [Section 5.2] The number of graded problems varies across settings (89 to 95), and Table 7 reports averages without clarifying whether the scores are paired per problem; the paper should state how missing format-following problems are treated.
  3. [Section 3.5.1] The default 50/50 weighting between the answer and explanation components is a free parameter, and the paper does not report how sensitive the main comparisons are to this choice; a short sensitivity discussion would strengthen the protocol.
  4. [Section 4.2] The sentence "We use the 10196 unique languages" appears to contain a typo or truncation; the surrounding text suggests the intended meaning is a count of language-script pairs from FLORES-200.
  5. [Section 2.2] There are several typographical errors, including "at lease" and "T asks" in Section 1, and the reference to Neacs,u (2024) uses a nonstandard author formatting; a copyedit pass is needed.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: benchmark accuracy is anchored to external official IOL solutions; the only self-referential element is an explicitly illustrative single-problem reasoning check, which is a support gap rather than a circular reduction.

full rationale

The central benchmark results are not circular. Final-answer and rule-checklist scores (Section 3.5.1) are graded against official IOL solutions converted into structured references, and the grammar-agent and Mixture-of-Agents comparisons (Section 5.2 and Appendix H) are measured against those external references. No parameter is fitted to the reported scores, and no equation defines a prediction in terms of the benchmark input, so there is no reduction of the claimed outcome to its own inputs. The only self-referential element is the Check-of-Thought demonstration in Section 3.5.3: the gold reasoning reference was generated by Gemini-2.5-Pro and manually verified, the baseline model whose reasoning is scored is Gemini-2.5-pro, and the judge is also an LLM. This creates a similarity-to-source bias and is not validated against human ratings, but the paper explicitly frames it as a single illustrative example ('we introduce the possible prompt and results of evaluating a single problem rather than performing a large-scale application'), and it is not used to compute the benchmark's accuracy numbers. The abstract's claim that agentic models outperform in 'interpretability' is not actually demonstrated by a comparison of interpretability scores between single-pass and agentic models, so that part of the central claim is under-supported; however, that is an evidence and correctness gap, not a circular derivation. Self-citations (e.g., Shih et al. 2025 for rare-script reasoning) are background statements and are not load-bearing for the benchmark's conclusions. Overall, the paper is self-contained against external official solutions, and the mild self-referentiality in the illustrative reasoning evaluation warrants at most a low score.

Assumptions & free parameters 4 free parameters · 6 assumptions · 2 invented entities

The central accuracy claim rests on the external IOL solutions, so the ledger is relatively light. The main unverified inputs are the LLM-generated reasoning traces, the LLM-as-judge grading, and the unsupported assumption that the grammar database contains accurate references; each of these is a domain assumption rather than a fitted constant.

free parameters (4)
  • answer_explanation_weight = 0.5 / 0.5
    Default equal weighting for answer and explanation components in the final score (Section 3.5.1). Chosen by hand; affects all reported average scores.
  • MoA_configuration = N=2, M=2, R=0..4
    Number of agents per layer and number of aggregator layers in the multi-agent framework (Section 5.2). Chosen by the authors, not tuned on a validation set.
  • sampling_temperature = 0.75 (IOL experiments), 0.1 (FLORES translation)
    Temperature settings for LLM generation (Sections 4.2 and 5.2); both influence output stochasticity and therefore scores.
  • chunk_size = 256 tokens
    Chunk length for vector search in the Grammar Agent (Section 5.1.1); arbitrary choice affecting retrieval quality.
assumptions (6)
  • domain assumption IOL problems are self-contained and internally consistent; all information needed to solve them is inside the problem.
    Invoked in Section 1 to justify the benchmark; a property of IOL but still an assumption about the collected data.
  • domain assumption Official IOL solutions are correct and complete.
    Used as ground truth for answer grading in Sections 3.2 and 3.5.
  • domain assumption The manual decomposition of official solutions into rule checklists is exhaustive and unambiguous.
    Section 3.5.1; no inter-annotator agreement or consistency check is reported.
  • domain assumption LLM judges can reliably map free-text explanations to rule checklist items.
    Section 3.5.1; the LLM-as-judge approach is not validated against human grading.
  • domain assumption The reference grammar books in the knowledge base are accurate and relevant for the target languages.
    Section 5.1.1; no quality control or coverage analysis of the 1100-language grammar database is provided.
  • standard math chrF is a valid language-independent metric for translation quality.
    Section 4.2; a standard NLP metric, though it does not capture semantic equivalence.
invented entities (2)
  • Check-of-Thought evaluation protocol
    purpose: Quantify LLM reasoning quality across five dimensions with ten metrics.
    Introduced in Section 3.5.3; demonstrated on one example and not validated against human ratings or replicated.
  • Gold-standard reasoning traces (GRR)
    purpose: Reference for judging model reasoning steps in the Check-of-Thought protocol.
    Generated by Gemini-2.5-Pro and human-verified (Section 3.2); not publicly released and no reliability metrics reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LingBench++: A Linguistically-Informed Benchmark and Reasoning Framework for Multi-Step and Cross-Cultural Inference with LLMs." pith.science (2026). https://pith.science/paper/HUWW3WI4

@misc{pith2026250716809,
  author       = {Pith},
  title        = {Pith review of: LingBench++: A Linguistically-Informed Benchmark and Reasoning Framework for Multi-Step and Cross-Cultural Inference with LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HUWW3WI4}},
  note         = {Machine review of arXiv:2507.16809}
}
read the original abstract

We propose LingBench++, a linguistically-informed benchmark and reasoning framework designed to evaluate large language models (LLMs) on complex linguistic tasks inspired by the International Linguistics Olympiad (IOL). Unlike prior benchmarks that focus solely on final answer accuracy, LingBench++ provides structured reasoning traces, stepwise evaluation protocols, and rich typological metadata across over 90 low-resource and cross-cultural languages. We further develop a multi-agent architecture integrating grammatical knowledge retrieval, tool-augmented reasoning, and deliberate hypothesis testing. Through systematic comparisons of baseline and our proposed agentic models, we demonstrate that models equipped with external knowledge sources and iterative reasoning outperform single-pass approaches in both accuracy and interpretability. LingBench++ offers a comprehensive foundation for advancing linguistically grounded, culturally informed, and cognitively plausible reasoning in LLMs.

Figures

Figures reproduced from arXiv: 2507.16809 by the authors.

Figure 1
Figure 1. Example of an IOL Problem rules are discovered, they will apply to all examples in the problem (though some rules may have exceptions), and all information needed to solve the problem is found within the problem itself. Every linguistics problem is structured into four parts: an introduction, a corpus, tasks, and notes. An example is shown in [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Three Problems in IOL 2017 One distinctive aspect of IOL problems lies in their cross-cultural and semantic depth. Beyond the structural reasoning over phonology, morphology, and syntax, many tasks require solvers to engage with low-resource or endangered languages under conditions of micro-data. More significantly, some problems explicitly involve semantic inference, cultural conceptualization, or sociolinguistic r… view at source ↗
Figure 3
Figure 3. IOL Problem Distribution by Language Family [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (14 more)
Figure 4
Figure 4. Figure 4: Subject vs Type Distribution 3.4 Preliminary Data Analysis The distribution charts of each typological category in our benchmark are shown in Appendix F. Key findings include: Subject and Type Distribution: As shown in [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Subject vs Top 10 Language Family Distribution [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Type vs Top 10 Language Family Distribution [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: Distribution of Final Grades by Language Family [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 8
Figure 8. Figure 8: Distribution of Final Grades by Subject [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]
Figure 9
Figure 9. Figure 9: Distribution of Final Grades by Type on Rosetta problems and better, though inconsistently, on Pattern and Match-up types. Overall, the model shows strong performance in certain areas but inconsistent reasoning across languages, subjects, and problem types. 3.5.3 Evalu…
Figure 10
Figure 10. Figure 10: Comparison of chrF score distributions for English-to-Target (E → T) and Target-to-English (T → E) translations, grouped by resource class. The plot shows a clear positive trend where quality increases with resource availability, with the T → E direction consistently …
Figure 11
Figure 11. Figure 11: Multi-Agent Framework for Solving Linguistics Olympiad Problems [PITH_FULL_IMAGE:figures/full_fig_p019_11.png]
Figure 12
Figure 12. Figure 12: Original Problem 3 in 2024. 28 [PITH_FULL_IMAGE:figures/full_fig_p028_12.png]
Figure 13
Figure 13. Figure 13: Statistical distributions of various features in the IOL problems dataset. [PITH_FULL_IMAGE:figures/full_fig_p033_13.png]
Figure 14
Figure 14. Figure 14: Translation Score Distribution by Language Family. This plot compares the distribution of chrF scores for English-to-Target (E → T) and Target-to-English (T → E) directions across language families. A consistent performance gap is evident, with T → E scores being almo…
Figure 15
Figure 15. Figure 15: Translation Score Distribution by Script. This plot compares chrF score distributions across different writing systems. As with the family-based plot, the T → E direction consistently outperforms the E → T direction. Performance for languages using Latin and Cyrillic …
Figure 16
Figure 16. Figure 16: Score vs. Class Distribution within each Language Family. This faceted plot details the relationship between resource class and chrF score for each language family individually. A positive trend, where higher scores are associated with higher resource classes, is visi…
Figure 17
Figure 17. Figure 17: Score vs. Class Distribution within each Script. This faceted plot shows the relationship between resource class and chrF score for each writing system. The Latin script subplot contains the most data across all resource classes and most clearly demonstrates the posit…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. The Gold Medals in an Empty Room: Diagnosing Metalinguistic Reasoning in LLMs with Camlang

    cs.CL 2025-08 conditional novelty 7.0 of 10

    A novel constructed language with explicit grammar and dictionary reveals a large gap between human metalinguistic learning (87%) and the best LLM (47%) on translated CommonsenseQA.

  2. Think Through a Bottleneck: Hourglass Reasoning for Rigorous Induction

    cs.AI 2026-07 conditional novelty 6.0 of 10

    Strict stage isolation that passes only a compressed symbolic schema and rule between LLM calls improves few-shot inductive reasoning more than self-refinement or explicit verbalization alone.

Reference graph

Works this paper leans on

47 extracted references · 43 canonical work pages · cited by 2 Pith papers

  1. [1]

    Aycock, S., Stap, D., Wu, D., Monz, C., and Sima'an, K. (2025). Can LLM s really learn to translate a low-resource language from one grammar book? In The Thirteenth International Conference on Learning Representations

    work page 2025

  2. [2]

    A., and Kirk, H

    Bean, A., Hellsten, S., Mayne, H., Magomere, J., A., E., Chi, R., Hale, S. A., and Kirk, H. R. (2024). Lingoly: A benchmark of olympiad-level linguistic reasoning puzzles in low resource and extinct languages. In Globerson, A., Mackey, L., Belgrave, D., Fan, A., Paquet, U., Tomczak, J., and Zhang, C., editors, Advances in Neural Information Processing Sys...

    work page 2024

  3. [3]

    and Derzhanski, I

    Bozhanov, B. and Derzhanski, I. (2013). Rosetta stone linguistic problems. In Derzhanski, I. and Radev, D., editors, Proceedings of the Fourth Workshop on Teaching NLP and CL , pages 1--8, Sofia, Bulgaria. Association for Computational Linguistics

    work page 2013

  4. [4]

    Chi, N., Malchev, T., Kong, R., Chi, R., Huang, L., Chi, E., McCoy, R., and Radev, D. (2024). M ode L ing: A novel dataset for testing linguistic reasoning in language models. In Hahn, M., Sorokin, A., Kumar, R., Shcherbakov, A., Otmakhova, Y., Yang, J., Serikov, O., Rani, P., Ponti, E. M., Murado g lu, S., Gao, R., Cotterell, R., and Vylomova, E., editor...

    work page 2024

  5. [5]

    Docling technical report

    Deep Search Team (2024). Docling technical report. Technical report, AI4K Group, IBM Research

    work page 2024

  6. [6]

    Durham, S. P. and Rogers, D. E. (1969). An application of computer programming to the reconstruction of a proto-language. In I nternational C onference on C omputational L inguistics COLING 1969: Preprint No. 5 , S a nga S \"a by, Sweden

    work page 1969

  7. [7]

    Gao, K., Cai, H., Shuai, Q., Gong, D., and Li, Z. (2025). Embedding self-correction as an inherent ability in large language models for enhanced mathematical reasoning

    work page 2025

  8. [8]

    and Dan, S

    Goyal, S. and Dan, S. (2025). Iolbench: Benchmarking llms on linguistic reasoning

    work page 2025

Show all 47 references
  1. [9]

    He, J., Lin, H., Wang, Q., Fung, Y., and Ji, H. (2025). Self-correction is more than refinement: A learning framework for visual and language reasoning tasks

    2025

  2. [10]

    Joshi, P., Santy, S., Budhiraja, A., Bali, K., and Choudhury, M. (2020). The state and fate of linguistic diversity and inclusion in the NLP world. In Jurafsky, D., Chai, J., Schluter, N., and Tetreault, J., editors, Proceedings of the 58th Annual Meeting of the Association fo...

    2020

  3. [11]

    S., Reid, M., Matsuo, Y., and Iwasawa, Y

    Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. (2023). Large language models are zero-shot reasoners

    2023

  4. [12]

    u ttler, H., Lewis, M., Yih, W.-t., Rockt\

    Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., K\" u ttler, H., Lewis, M., Yih, W.-t., Rockt\" a schel, T., Riedel, S., and Kiela, D. (2020). Retrieval-augmented generation for knowledge-intensive nlp tasks. In Proceedings of the 34th International Co...

    2020

  5. [13]

    Li, W., Lin, Y., Xia, M., and Jin, C. (2025). Rethinking mixture-of-agents: Is mixing different large language models beneficial?

    2025

  6. [14]

    Long, J. (2023). Large language model guided tree-of-thought

    2023

  7. [15]

    Lu, L., Xie, P., and Mortensen, D. (2024). Semisupervised neural proto-language reconstruction. In Ku, L.-W., Martins, A., and Srikumar, V., editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 14715--...

    2024

  8. [16]

    Naik, A., Zhang, K., Robinson, N., Mysore, A., Marr, C., Byrnes, H. S. R., Cai, A., Chang, K., and Mortensen, D. (2024). Can large language models code like a linguist?: A case study in low resource sound law induction

    2024

  9. [17]

    Neacșu, V. A. (2024). Linguistics Olympiad . Number 13 in Textbooks in Language Sciences. Language Science Press, Berlin

    2024

  10. [18]

    NLLB Team , Costa-jussà, M. R., Cross, J., Çelebi, O., Elbayad, M., Heafield, K., Heffernan, K., Kalbassi, E., Lam, J., Licht, D., Maillard, J., Sun, A., Wang, S., Wenzek, G., Youngblood, A., Akula, B., Barrault, L., Gonzalez, G. M., Hansanti, P., Hoffman, J., Jarrett, S., Sad...

    2022

  11. [19]

    Paranjape, B., Lundberg, S., Singh, S., Hajishirzi, H., Zettlemoyer, L., and Ribeiro, M. T. (2023). Art: Automatic multi-step reasoning and tool-use for large language models

    2023

  12. [20]

    Pires, T., Schlinger, E., and Garrette, D. (2019). How multilingual is multilingual BERT ? In Korhonen, A., Traum, D., and M \`a rquez, L., editors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 4996--5001, Florence, Italy. Ass...

    2019

  13. [21]

    Popovi \'c , M. (2015). chrf: character n-gram f-score for automatic mt evaluation. In Proceedings of the tenth workshop on statistical machine translation , pages 392--395

    2015

  14. [22]

    Popovi \'c , M. (2017). chrf++: words helping character n-grams. In Proceedings of the second conference on machine translation , pages 612--618

    2017

  15. [23]

    Post, M. (2018). A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers , pages 186--191, Belgium, Brussels. Association for Computational Linguistics

    2018

  16. [24]

    Qi, Z., Fang, Y., Zhang, M., Sun, Z., Wu, T., Liu, Z., Lin, D., Wang, J., and Zhao, H. (2023). Gemini vs gpt-4v: A preliminary comparison and combination of vision-language models through qualitative cases

    2023

  17. [25]

    S., and Zanzotto, F

    Ranaldi, L., Pucci, G., Ranaldi, F., Ruzzetti, E. S., and Zanzotto, F. M. (2024). A tree-of-thoughts to broaden multi-step reasoning across languages. In Duh, K., Gomez, H., and Bethard, S., editors, Findings of the Association for Computational Linguistics: NAACL 2024 , pages...

    2024

  18. [26]

    G., Kementchedjhieva, Y., Rust, P., and Gurevych, I

    S ahin, G. G., Kementchedjhieva, Y., Rust, P., and Gurevych, I. (2020). P uzz L ing M achines: A C hallenge on L earning F rom S mall D ata. In Jurafsky, D., Chai, J., Schluter, N., and Tetreault, J., editors, Proceedings of the 58th Annual Meeting of the Association for Compu...

    2020

  19. [27]

    Schick, T., Dwivedi-Yu, J., Dessì, R., Raileanu, R., Lomeli, M., Zettlemoyer, L., Cancedda, N., and Scialom, T. (2023). Toolformer: Language models can teach themselves to use tools

    2023

  20. [28]

    Shih, Y.-F., Lin, Z.-L., and Hsieh, S.-K. (2025). Reasoning over the glyphs: Evaluation of llm's decipherment of rare scripts

    2025

  21. [29]

    Sánchez, E., Alastruey, B., Ropers, C., Stenetorp, P., Artetxe, M., and Costa-jussà, M. R. (2024). Linguini: A benchmark for language-agnostic linguistic reasoning

    2024

  22. [30]

    Tanzer, G., Suzgun, M., Visser, E., Jurafsky, D., and Melas-Kyriazi, L. (2024). A benchmark for learning to translate a new language from one grammar book. In Kim, B., Yue, Y., Chaudhuri, S., Fragkiadaki, K., Khan, M., and Sun, Y., editors, International Conference on Represen...

    2024

  23. [31]

    Vaduguru, S., Sathe, A., Choudhury, M., and Sharma, D. (2021). Sample-efficient linguistic generalizations through program synthesis: Experiments with phonology problems. In Nicolai, G., Gorman, K., and Cotterell, R., editors, Proceedings of the 18th SIGMORPHON Workshop on Com...

    2021

  24. [32]

    Wang, B., Min, S., Deng, X., Shen, J., Wu, Y., Zettlemoyer, L., and Sun, H. (2023). Towards understanding chain-of-thought prompting: An empirical study of what matters

    2023

  25. [33]

    Wang, J., Wang, J., Athiwaratkun, B., Zhang, C., and Zou, J. Y. (2025). Mixture-of-agents enhances large language model capabilities. In Yue, Y., Garg, A., Peng, N., Sha, F., and Yu, R., editors, International Conference on Representation Learning , volume 2025, pages 33944--33963

    2025

  26. [34]

    Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., and Zhou, D. (2023). Chain-of-thought prompting elicits reasoning in large language models

    2023

  27. [35]

    Wu, M., Zhu, T., Han, H., Zhang, X., Shao, W., and Chen, W. (2025). Chain-of-tools: Utilizing massive unseen tools in the cot reasoning of frozen language models

    2025

  28. [36]

    L., Cao, Y., and Narasimhan, K

    Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T. L., Cao, Y., and Narasimhan, K. (2023). Tree of thoughts: Deliberate problem solving with large language models

    2023

  29. [37]

    Yu, Z., He, L., Wu, Z., Dai, X., and Chen, J. (2023). Towards better chain-of-thought prompting strategies: A survey

    2023

  30. [38]

    Y., and Li, L

    Zhang, K., Choi, Y., Song, Z., He, T., Wang, W. Y., and Li, L. (2024a). Hire a linguist!: Learning endangered languages in LLM s with in-context linguistic descriptions. In Ku, L.-W., Martins, A., and Srikumar, V., editors, Findings of the Association for Computational Linguis...

    2024

  31. [39]

    Zhang, Z., Zhang, A., Li, M., and Smola, A. (2022). Automatic chain of thought prompting in large language models

    2022

  32. [40]

    Zhang, Z., Zhang, A., Li, M., Zhao, H., Karypis, G., and Smola, A. (2024b). Multimodal chain-of-thought reasoning in language models

    2024

  33. [41]

    Zhou, D., Schärli, N., Hou, L., Wei, J., Scales, N., Wang, X., Schuurmans, D., Cui, C., Bousquet, O., Le, Q., and Chi, E. (2023). Least-to-most prompting enables complex reasoning in large language models

    2023

  34. [42]

    Neacsu, Vlad A. 2024. Linguistics Olympiad: Training guide. Language Science Press

    2024

  35. [43]

    LingOly (Linguistic Olympiad Dataset): https://github.com/am-bean/lingOly

  36. [44]

    IOLing Competition on LLMs: https://arxiv.org/abs/2409.12126

    arXiv

  37. [45]

    International Linguistics Olympiad Past Problems: https://ioling.org/problems/by_year

  38. [46]

    [Other Linguistic Textbook 1]

  39. [47]

    [Other Grammar Book 1]

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.