Pith. sign in

REVIEW 4 major objections 4 minor 39 references

Easy to Complete, Hard to Choose: Investigating LLM Performance on the ProverbIT Benchmark

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read AI models ace proverb completion, yet fail when the right ending is absent.

desk verdict Useful new Italian proverb benchmark and a plausible completion/selection gap, but the missing human baseline keeps the headline conclusion unproven. read the letter →

arxiv 2608.04670 v1 pith:2QIXR3KP submitted 2026-08-05 cs.CL cs.AI

Enrico Mensa , Lorenzo Zane , Calogero Jerik Scozzaro , Matteo Delsanto , Tommaso Milani , Daniele Paolo Radicioni This is my paper · ORCID
classification cs.CLcs.AI
keywords Italianproverbsproverbcompletionmultiple-choicebenchmarklargelanguagemodelschain-of-thoughtanalysisfigurativenegativereasoningculturalknowledge
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces ProverbIT, a set of 100 Italian multiple-choice questions built so that none of the four listed endings completes the proverb correctly; the only sound answer is "None of the others." Across 13 language models, the authors find that nearly every model can freely complete the same proverbs, yet accuracy collapses when the correct ending is removed from the choices, with some models falling to 4-6%. Chain-of-thought traces from two reasoning models show the correct proverb ending appearing inside their reasoning even when they go on to choose a wrong option, usually a literal synonym. The authors' conclusion is that current models lean on memorized surface patterns rather than a semantic grasp of culturally grounded expressions.

What carries the argument

The load-bearing object is the ProverbIT item format, in which the true ending is never one of the options: four manually constructed distractors (A assonant, B literal synonym, C inverse, D trivial) appear alongside E, "None of the other answers." This shifts the task from pattern completion to negative discrimination, and it is what makes the performance gap interpretable. The second piece of machinery is the chain-of-thought mention-tracing applied to DeepSeek R1 and Qwen 3, which counts how often the true ending and each option are named during reasoning; this is what exposes the disconnect between knowing the correct ending and choosing it.

What would settle it

Run the same 100 ProverbIT items with a panel of native Italian speakers under the same prompt; if they choose a listed option, especially the synonym, on a substantial share of items, the benchmark's assumption that E is the only correct answer fails, and the model performance gap would then reflect exact-wording recall rather than lack of understanding.

Watch

Extended reading notes

Core claim

The central finding is a knowledge-versus-discrimination gap: proverb completion succeeds, but selecting the absence of the correct continuation fails. The benchmark removes the true ending from every question, replacing it with four constructed distractors (assonant, literal-synonym, inverse, and trivial) plus option E, "None of the other answers." On this base task, accuracy ranges from GPT o3's 86% down to Gemma 3's 4%, while the same models score much higher on direct completion and on a version where the true ending is added back as an option; some models lose tens of percentage points and one drops from roughly 75-88% to 6%. The paper interprets this as evidence that models rely on memorized linguistic patterns rather than deeper semantic understanding, and that even reasoning models only partially compensate. Error analysis shows a strong preference for the literal-synonym option, and CoT analysis of DeepSeek R1 and Qwen 3 reveals that both models frequently state the correct proverb ending while still failing to recognize that it is absent from the choices.

Load-bearing premise

The load-bearing premise is that every invented ending is clearly wrong for a knowledgeable native speaker, so that "None of the others" is the only reasonable choice; because no human baseline is reported, a reader cannot verify that premise, and a few synonym options are close enough to blur it.

Editorial extensions

If this is right

  • Completion scores should not be read as evidence that a model understands a proverb; a model can complete it fluently and still fail to reject wrong alternatives.
  • Multiple-choice evaluations should include a "none of the others" condition, otherwise a model can score well by recognizing surface patterns without verifying the answer's semantic correctness.
  • Reasoning traces are not reliable evidence of a model's conclusion: models can state the correct ending and still output a different letter.
  • Literal-synonym distractors are the main failure attractor, so model behavior on cultural expressions is better described as lexical matching than as meaning comparison.
  • Language-specific benchmarks like ProverbIT are needed because English-centric tests can hide these gaps.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If native Italian speakers were given the same 100 questions and did not choose E on every item, the benchmark would be measuring exact-wording recall rather than proverb understanding; that human calibration is the paper's untested premise.
  • A direct follow-up could ask models to explain the meaning of a proverb after answering E, or to select the best paraphrase; this would separate recognition of absence from genuine semantic representation.
  • The same construction, removing the correct answer and adding "None of the others," could be applied to idioms, catchphrases, and formulaic expressions in other low-resource languages; the paper's pattern suggests a general test for memorized routine versus understanding.
  • The bias toward the synonym option may be amplified by the letter E being the "None" answer; randomizing option positions, which the paper cites as future work, would show whether the effect is semantic or positional.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces ProverbIT, a new benchmark of 100 Italian proverb-completion multiple-choice questions in which the correct ending is deliberately absent and option E ('None of the other answers') is the intended answer. Thirteen LLMs and LRMs are evaluated on three tasks: open completion, multiple-choice with the true ending added, and the base ProverbIT task with no correct ending. The central reported finding is that models complete proverbs accurately (67-96%) but drop sharply on the base task (4-86%), with error analysis showing a bias toward literal-synonym distractors. Chain-of-thought analysis of DeepSeek R1 and Qwen 3 indicates that models often mention the correct proverb ending during reasoning yet still select a wrong option, and sometimes produce answers inconsistent with their own reasoning.

Significance. If the finding survives validation, it is a useful contribution: ProverbIT addresses an under-resourced language and a culturally embedded phenomenon, the dataset is publicly released, and the 'easy to complete, hard to choose' result offers a concrete test bed for the memorization-versus-understanding debate in LLMs. The paper's strengths include the manual construction of distractors with a clear taxonomy, the inclusion of both LRMs and smaller local models, and the detailed CoT analysis with quantified phenomena such as answer/reasoning mismatch. However, the central interpretation depends on an untested premise about item correctness, and the completion metric is described ambiguously; these issues must be resolved before the benchmark's conclusions can be accepted.

major comments (4)
  1. [§3.2 and §4.3] The claim that 'Since all provided endings are completely invented and thus incorrect, we expect models to always answer E' is not validated by any human baseline. The supplementary examples show that some distractors are semantically plausible variants of the canonical ending (e.g., 'ciò che brilla' for 'quel che luccica', 'è breve' for 'dura poco'), so a native speaker might accept them as legitimate completions. Without item-level human judgment, the 'dramatic drop' in base-task accuracy could reflect disagreement about canonical wording rather than failure to detect absence. I request a human baseline on all 100 items (or at least a subset with inter-annotator agreement) to establish that E is uniquely correct per item.
  2. [§4.1.1] The completion metric is ambiguous as stated: 'If the edit distance exceeds a threshold of 0.8 in at least two out of three runs, we consider the answer correct.' Since the implementation is from difflib, the value is likely a similarity ratio (higher = more similar), not an edit distance; as written, the sentence implies a larger edit distance indicates a correct answer, which would inflate completion accuracy. The threshold must be clarified (whether it is SequenceMatcher.ratio() > 0.8) and justified, because the completion results are the 'easy to complete' side of the paper's central contrast.
  3. [§4.1 and Table 2] No confidence intervals or significance tests are reported. With 100 items and three runs per condition, the differences between models (e.g., GPT o3 at 86% vs. Gemini 2.5 Pro at 77%) and between conditions may not be reliable. I recommend reporting bootstrap confidence intervals or applying McNemar's test per model for the base versus base+true-ending comparison, at minimum.
  4. [§4.1, Base + true ending Task] It is unclear what happens to option E when the true ending is added. Does the true ending replace E, or is it added as a sixth option while E remains? The interpretation of the base+true-ending results changes substantially depending on whether models can still choose 'None of the others' in that condition. Please specify the option structure.
minor comments (4)
  1. [Table 2] The two Claude models are both labeled 'Claude Sonnet 4' in Table 2; one should be labeled 'Claude Sonnet 4 Thinking' to match Table 1 and the text.
  2. [§4.3.1] The sentence 'we set a reasonable thinking budget of 2000 tokens for o3, Sonnet 4, and Gemini 2.5 Pro' is followed by 'only 22 out of 600 CoTs from these two models exceeded the 2000-token limit'; the connection is confusing because the 600 CoTs come from DeepSeek R1 and Qwen 3, which were given unlimited budgets.
  3. [References] Reference [39] contains a formatting typo in the title: '" my answer is c": First-token probabilities...' has mismatched quotes.
  4. [Figure captions] The figure captions for Figures 1 and 2 would benefit from stating explicitly that the 'correct proverb ending' is never present among the options A-D, so the dotted-line mentions demonstrate recall without recognition of absence; this is stated in the text but not in the captions.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the results are direct empirical measurements of model behavior, not consequences derived from the benchmark's construction.

full rationale

ProverbIT is an empirical evaluation paper rather than a derivation. The central quantities (base-task accuracy, base+true-ending accuracy, completion accuracy, error-type distributions, and CoT mention counts) are all directly measured from model responses. No parameter is fitted to a subset of the data and then renamed as a prediction; the 'None of the others' design is explicitly imported from the external work of Salido et al. [37], whose authors do not overlap with the present paper. The sentence 'Since all provided endings are completely invented and thus incorrect, we expect models to always answer E' is an evaluation expectation, not an equation that forces the observed accuracies. The finding that models predominantly choose option B (the synonym distractor) is an empirical result about model behavior and would not follow automatically from the distractor-construction rationale. The paper's interpretation that low base-task accuracy indicates reliance on memorized patterns is a substantive claim, but it is not circular: it depends on the untested assumption that every E-only item is uniquely correct for knowledgeable humans. That missing human baseline is a validity and interpretation risk, not a self-referential derivation step. Overall, the central results are self-contained measurements against an externally proposed evaluation technique, so no circular step meeting the quoted-evidence standard is present.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The paper introduces no fitted numeric parameters; the only hand-chosen number is the completion similarity threshold. The central claim rests on benchmark-validity assumptions (distractors are incorrect, E is the only correct answer, near-match completion implies knowledge) that are stated but not independently verified with human baselines.

free parameters (1)
  • Completion similarity threshold = 0.8 (described as edit distance threshold)
    Section 4.1.1 marks a completion as correct if the difflib similarity ratio between the model's completion and the true ending is above 0.8 in at least two of three runs. The threshold is chosen by hand, and the direction of the comparison is described ambiguously (it says 'exceeds 0.8' but difflib ratio is a similarity where 1.0 is identical).
assumptions (3)
  • domain assumption The 100 proverbs have canonical endings and the four distractors are always incorrect; E is unambiguously the only correct answer.
    Assumed in Section 3.1 and tested in the base task; no human validation is reported, so the benchmark's ground truth is an author judgment.
  • domain assumption Successful completion (edit-distance similarity above 0.8) indicates the model 'knows' the proverb.
    Used in Section 4.3 to contrast completion performance with multiple-choice performance; the threshold is arbitrary and the equivalence between near-match completion and knowledge is not tested.
  • domain assumption The 'None of the others' multiple-choice format measures discriminative or negative reasoning independently of the specific phrasing.
    The prompt instructs models to complete the proverb 'exactly' and warns that options have no typos; this framing could bias models toward selecting a listed option rather than E, which is an alternative explanation for the performance drop.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Easy to Complete, Hard to Choose: Investigating LLM Performance on the ProverbIT Benchmark." pith.science (2026). https://pith.science/paper/2QIXR3KP

@misc{pith2026260804670,
  author       = {Pith},
  title        = {Pith review of: Easy to Complete, Hard to Choose: Investigating LLM Performance on the ProverbIT Benchmark},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2QIXR3KP}},
  note         = {Machine review of arXiv:2608.04670}
}
read the original abstract

Large Language Models (LLMs) have transformed computational linguistics and achieved remarkable performance across numerous natural language processing tasks, yet significant gaps persist in understanding how these systems process culturally embedded linguistic expressions. This paper introduces ProverbIT, a novel Italian benchmark comprising 100 multiple-choice questions designed to evaluate LLMs' ability to complete Italian proverbs. We assess 13 frontier models, including Large Reasoning Models (LRMs) and traditional LLMs, across three tasks: proverb completion, multiple-choice selection with correct answers, and multiple-choice selection without correct answers. Our evaluation reveals surprising results: while nearly all models demonstrate knowledge of the proverbs through successful completion tasks, performance drops dramatically when transitioning to multiple-choice formats without correct answers, with even state-of-the-art reasoning models showing substantial degradation. Through detailed Chain-of-Thought analysis of two LRMs, we uncover that models exhibit a strong bias toward selecting literal synonyms and frequently mention correct proverb endings during reasoning without successfully identifying their absence from the given options. These findings suggest that current LLMs rely heavily on memorized patterns rather than deeper semantic understanding of culturally grounded expressions, highlighting important limitations in their reasoning capabilities for figurative language comprehension.

Figures

Figures reproduced from arXiv: 2608.04670 by the authors.

Figure 1
Figure 1. Analyses of the DeepSeek R1 CoTs. Left (the model answers correctly): tracing mentions of the correct answer (E) and any incorrect option (A/B/C/D). Right (the model answers incorrectly): tracing mentions of the correct (E) option and the exact provided incorrect answer (either A or B or C or D). The dotted line shows the mentions of the true ending of the proverb (which was not given as option). 0 0.2 0.4 0.6 0.8 1… view at source ↗
Figure 2
Figure 2. Analyses of the Qwen 3 CoTs. Left (the model answers correctly): tracing mentions of the correct answer (E) and any incorrect option (A/B/C/D). Right (the model answers incorrectly): tracing mentions of the correct (E) option and the exact provided incorrect answer (either A or B or C or D). The dotted line shows the mentions of the true ending of the proverb (which was not given as option). The dotted lines clearly… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

39 extracted references · 19 canonical work pages

  1. [1]

    Lewkowycz, A

    A. Lewkowycz, A. Andreassen, D. Dohan, E. Dyer, H. Michalewski, V. Ramasesh, A. Slone, C. Anil, I. Schlag, T. Gutman-Solo, et al., Solving quanti- tative reasoning problems with language models, Advances in Neural Information Processing Sys- tems 35 (2022) 3843–3857

    work page 2022

  2. [2]

    Chang, X

    Y. Chang, X. Wang, J. Wang, Y. Wu, L. Yang, K. Zhu, H. Chen, X. Yi, C. Wang, Y. Wang, et al., A sur- vey on evaluation of large language models, ACM transactions on intelligent systems and technology 15 (2024) 1–45

    work page 2024

  3. [3]

    F. D. L. Fornaciari, B. Altuna, I. Gonzalez-Dios, M. Melero, A hard nut to crack: Idiom detection with conversational large language models, in: Pro- ceedings of the 4th Workshop on Figurative Lan- guage Processing (FigLang 2024), 2024, pp. 35–44

    work page 2024

  4. [4]

    F. Xu, Q. Hao, Z. Zong, J. Wang, Y. Zhang, J. Wang, X. Lan, J. Gong, T. Ouyang, F. Meng, et al., Towards large reasoning models: A survey of reinforced rea- soning with large language models, arXiv preprint arXiv:2501.09686 (2025)

    arXiv 2025

  5. [5]

    M. Wu, W. Wang, S. Liu, H. Yin, X. Wang, Y. Zhao, C. Lyu, L. Wang, W. Luo, K. Zhang, The bitter les- son learned from 2,000+ multilingual benchmarks, arXiv preprint arXiv:2504.15521 (2025)

    arXiv 2025

  6. [6]

    A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, S. R. Bowman, Glue: A multi-task benchmark and anal- ysis platform for natural language understanding, arXiv preprint arXiv:1804.07461 (2018)

    arXiv 2018

  7. [7]

    A. Wang, Y. Pruksachatkun, N. Nangia, A. Singh, J. Michael, F. Hill, O. Levy, S. Bowman, Superglue: A stickier benchmark for general-purpose language understanding systems, Advances in neural infor- mation processing systems 32 (2019)

    work page 2019

  8. [8]

    Cobbe, V

    K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al., Training verifiers to solve math word problems, arXiv preprint arXiv:2110.14168 (2021)

    arXiv 2021

Show all 39 references
  1. [9]

    Glazer, E

    E. Glazer, E. Erdil, T. Besiroglu, D. Chicharro, E. Chen, A. Gunning, C. F. Olsson, J.-S. Denain, A. Ho, E. d. O. Santos, et al., Frontiermath: A bench- mark for evaluating advanced mathematical reason- ing in ai, arXiv preprint arXiv:2411.04872 (2024)

    2024 arXiv

  2. [10]

    C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, K. R. Narasimhan, Swe-bench: Can lan- guage models resolve real-world github issues?, in: ICLR, 2024

    2024

  3. [11]

    N. Jain, K. Han, A. Gu, W.-D. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, I. Stoica, Live- codebench: Holistic and contamination free eval- uation of large language models for code, arXiv preprint arXiv:2403.07974 (2024)

    2024 arXiv

  4. [12]

    Chollet, On the measure of intelligence, arXiv preprint arXiv:1911.01547 (2019)

    F. Chollet, On the measure of intelligence, arXiv preprint arXiv:1911.01547 (2019)

    2019 arXiv

  5. [13]

    Chollet, M

    F. Chollet, M. Knoop, G. Kamradt, B. Landers, H. Pinkard, Arc-agi-2: A new challenge for frontier ai reasoning systems, arXiv preprint arXiv:2505.11831 (2025)

    2025 arXiv

  6. [14]

    Attanasio, P

    G. Attanasio, P. Basile, F. Borazio, D. Croce, M. Fran- cis, J. Gili, E. Musacchio, M. Nissim, V. Patti, M. Ri- naldi, et al., Calamita: Challenge the abilities of language models in italian, in: Proceedings of the 10th Italian Conference on Computational Linguis- tics (CLiC-i...

    2024

  7. [15]

    Magnini, R

    B. Magnini, R. Zanoli, M. Resta, M. Cimmino, P. Al- bano, M. Madeddu, V. Patti, Evalita-llm: Bench- marking large language models on italian, arXiv preprint arXiv:2502.02289 (2025)

    2025 arXiv

  8. [16]

    I. A. Sag, T. Baldwin, F. Bond, A. Copestake, D. Flickinger, Multiword Expressions: A Pain in the Neck for NLP, Springer Berlin Heidelberg, 2002, p. 1–15. URL: http://dx.doi.org/10.1007/3-540-45715- 1_1. doi:10.1007/3-540-45715-1_1

    2002 doi

  9. [17]

    Tedeschi, F

    S. Tedeschi, F. Martelli, R. Navigli, Id10m: Idiom identification in 10 languages, in: Findings of the Association for Computational linguistics: NAACL 2022, 2022, pp. 2715–2726

    2022

  10. [18]

    Khoshtab, D

    P. Khoshtab, D. Namazifard, M. Masoudi, A. Akhgary, S. M. Sani, Y. Yaghoobzadeh, Com- parative study of multilingual idioms and similes in large language models, in: Proceedings of the 31st International Conference on Computational Linguistics, 2025, pp. 8680–8698

    2025

  11. [19]

    J. Kim, Y. Shin, U. Hwang, J. Choi, R. Xuan, T. Kim, Memorization or reasoning? exploring the idiom understanding of llms, arXiv preprint arXiv:2505.16216 (2025)

    2025

  12. [20]

    J. Lee, A. Liu, O. Ahia, H. Gonen, N. A. Smith, That was the last straw, we need more: Are translation systems sensitive to disambiguating context?, in: Findings of the Association for Computational Lin- guistics: EMNLP 2023, 2023, pp. 4555–4569

    2023

  13. [21]

    S. Li, J. Chen, S. Yuan, X. Wu, H. Yang, S. Tao, Y. Xiao, Translate meanings, not just words: Id- iomkb’s role in optimizing idiomatic translation with language models, in: Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, 2024, pp. 18554–18563

    2024

  14. [22]

    Donthi, M

    S. Donthi, M. Spencer, O. B. Patel, J. Y. Doh, E. Ro- dan, K. Zhu, S. O’Brien, Improving llm abilities in idiomatic translation, in: Proceedings of the First Workshop on Language Models for Low-Resource Languages, 2025, pp. 175–181

    2025

  15. [23]

    C. Liu, F. Koto, T. Baldwin, I. Gurevych, Are multi- lingual llms culturally-diverse reasoners? an inves- tigation into multicultural proverbs and sayings, in: Proceedings of the 2024 Conference of the North American Chapter of the Association for Computa- tional Linguistics: ...

    2024

  16. [24]

    Wang, V.-T

    M. Wang, V.-T. Pham, F. Moghimifar, T.-T. Vu, Proverbs run in pairs: Evaluating proverb trans- lation capability of large language model, arXiv preprint arXiv:2501.11953 (2025)

    2025 arXiv

  17. [25]

    Caramagna, I 200 proverbi italiani più belli e famosi (con significato), 2025

    F. Caramagna, I 200 proverbi italiani più belli e famosi (con significato), 2025. URL: https://aforisticamente.com/i-200-proverbi- italiani-piu-belli-e-famosi-con-significato/

    2025

  18. [26]

    URL: https://openrouter .ai/, accessed: 2025-06-15

    OpenRouter, Openrouter: A unified interface for llms, 2024. URL: https://openrouter .ai/, accessed: 2025-06-15

    2024

  19. [27]

    Hurst, A

    A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford, et al., Gpt-4o system card, arXiv preprint arXiv:2410.21276 (2024)

    2024 arXiv

  20. [28]

    A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan, et al., Deepseek-v3 technical report, arXiv preprint arXiv:2412.19437 (2024)

    2024 arXiv

  21. [29]

    D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, et al., Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, arXiv preprint arXiv:2501.12948 (2025)

    2025 arXiv

  22. [30]

    Team, Qwen3 technical report, 2025

    Q. Team, Qwen3 technical report, 2025. URL: https: //arxiv.org/abs/2505.09388.arXiv:2505.09388

    2025 arXiv

  23. [31]

    G. Team, A. Kamath, J. Ferret, S. Pathak, N. Vieil- lard, R. Merhej, S. Perrin, T. Matejovicova, A. Ramé, M. Rivière, et al., Gemma 3 technical report, arXiv preprint arXiv:2503.19786 (2025)

    2025 arXiv

  24. [32]

    Orlando, L

    R. Orlando, L. Moroni, P.-L. H. Cabot, S. Conia, E. Barba, S. Orlandini, G. Fiameni, R. Navigli, Min- erva llms: The first family of large language models trained from scratch on italian data, in: Proceedings of the 10th Italian Conference on Computational Linguistics (CLiC-it...

    2024

  25. [33]

    URL: https://huggingface .co/ mistralai/Mistral-Small-3.1-24B-Instruct-2503

    mistralai, Model card for mistral-small-3.1-24b- instruct-2503, 2025. URL: https://huggingface .co/ mistralai/Mistral-Small-3.1-24B-Instruct-2503

    2025

  26. [34]

    Sui, Y.-N

    Y. Sui, Y.-N. Chuang, G. Wang, J. Zhang, T. Zhang, J. Yuan, H. Liu, A. Wen, S. Zhong, H. Chen, et al., Stop overthinking: A survey on efficient reason- ing for large language models, arXiv preprint arXiv:2503.16419 (2025)

    2025 arXiv

  27. [35]

    Etxaniz, G

    J. Etxaniz, G. Azkune, A. Soroa, O. Lopez de Lacalle, M. Artetxe, Do multilingual language models think better in English?, in: K. Duh, H. Gomez, S. Bethard (Eds.), Proceedings of the 2024 Conference of the North American Chap- ter of the Association for Computational Linguis-...

    2024 doi

  28. [36]

    Ranaldi, G

    L. Ranaldi, G. Pucci, F. Ranaldi, E. S. Ruzzetti, F. M. Zanzotto, The limits of Italian in reasoning tasks, in: F. Dell’Orletta, A. Lenci, S. Montemagni, R. Sprug- noli (Eds.), Proceedings of the 10th Italian Confer- ence on Computational Linguistics (CLiC-it 2024), CEUR Works...

    2024

  29. [37]

    E. S. Salido, J. Gonzalo, G. Marco, None of the oth- ers: a general technique to distinguish reasoning from memorization in multiple-choice llm evalua- tion benchmarks, arXiv preprint arXiv:2502.12896 (2025)

    2025

  30. [38]

    Y. Chen, J. Benton, A. Radhakrishnan, J. Uesato, C. Denison, J. Schulman, A. Somani, P. Hase, M. Wagner, F. Roger, et al., Reasoning models don’t always say what they think, arXiv preprint arXiv:2505.05410 (2025)

    2025 arXiv

  31. [39]

    my answer is c

    X. Wang, B. Ma, C. Hu, L. Weber-Genzel, P. Röttger, F. Kreuter, D. Hovy, B. Plank, " my answer is c": First-token probabilities do not match text an- swers in instruction-tuned language models, arXiv preprint arXiv:2402.14499 (2024). Supplementary Materials Proverb Examples In...

    2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.