Pith. sign in

REVIEW 4 major objections 6 minor 57 references

QUENCH: Measuring the gap between Indic and Non-Indic Contextual General Reasoning in LLMs

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper introduces QUENCH, a 400-question English quiz benchmark, and reports that all seven LLMs it tests score 12 to 32 points lower on Indic-context questions than on non-Indic ones, as judged by an LLM jury.

desk verdict A genuinely new dataset and a plausible but under-validated headline gap; the Indic/non-Indic result rests almost entirely on an LLM jury that is never checked against humans. read the letter →

arxiv 2412.11763 v1 pith:MNO6J2EX submitted 2024-12-16 cs.CL

Mohammad Aflah Khan , Neemesh Yadav , Sarah Masud , Md. Shad Akhtar This is my paper · ORCID
classification cs.CL
keywords QUENCHbenchmarkLLMevaluationIndicknowledgegapquiz-basedreasoningentitypredictionrationalegenerationchain-of-thoughtpromptingLLM-as-judge
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

QUENCH is a benchmark of 400 English quiz questions, transcribed from public quizzing videos, in which answer entities are masked as X, Y, and Z and each question carries a gold rationale. The paper's central claim is that all seven LLMs it evaluates do consistently worse on the 70 Indic-context questions than on the 330 non-Indic questions, for both predicting the masked entity and generating the justifying rationale. On the paper's LLM-jury metric the gap runs from about 12 points for GPT-4-Turbo to about 32 points for Gemini 1.5 Flash. Because the questions are all in English and differ only in cultural context, the paper reads the gap as evidence of a real shortfall in non-Western knowledge and deduction, and presents QUENCH as a reusable instrument for measuring that gap.

What carries the argument

The central object is the QUENCH dataset: 400 English quiz questions with entities masked as X, Y, or Z and manually written free-text rationales, each tagged Indic or non-Indic based on whether the answer hinges on Indian geography, history, or culture. The evaluation machinery is a two-stage zero-shot protocol: the model first predicts the masked entities under a strict answer format, then generates a rationale from either its own predicted entities or the gold entities, and the whole pipeline is run both with and without chain-of-thought prompting. Scores come from BLEU, ROUGE-L, BERTScore, and an LLM-as-judge jury made of three models that each score every other model's outputs; the jury's binary entity verdict and 5-point rationale score are what produce the paper's headline Indic versus non-Indic gap.

What would settle it

Have a small panel of human quiz experts, blind to the Indic label, score a random sample of roughly 30 Indic and 30 matched non-Indic QUENCH answers for entity correctness and rationale quality; if the human-measured gap is close to zero while the LLM jury still reports 12 to 32 points, the paper's central claim of a large knowledge gap would not survive.

Watch

Extended reading notes

Core claim

The paper establishes QUENCH as a valid open-domain, zero-shot quiz benchmark and reports the stable finding that every benchmarked LLM performs worse on Indic-context questions than non-Indic ones: an average gap of about 21 points for entity prediction and 14.7 points for rationale generation as judged by a three-model LLM jury. GPT-4-Turbo shows the smallest gap, around 12 points, while Gemini 1.5 Flash shows the largest at 32 points. The authors attribute the gap to pretraining corpora with a predominantly North American context, and they verify that the benchmark questions themselves are not present in major pretraining corpora, arguing that the gap is not a memorization artifact.

Load-bearing premise

The headline gap assumes the LLM jury grades Indic-context answers fairly, yet no human validation of the jury is reported and one of the jury members is itself an evaluated model, so a systematic jury bias against Indian-context answers would shrink the measured gap toward the near-zero BERTScore differences of 0.6 to 2.4 points.

Editorial extensions

If this is right

  • Model rankings shift when cultural context is part of the test: GPT-4-Turbo leads on every metric, but the open-weight Meta-Llama-3-70B matches GPT-3.5-Turbo overall and shows a smaller performance swing across subsets.
  • English-only benchmarks that omit non-Western context overstate the general world-knowledge and deduction abilities of LLMs.
  • Chain-of-thought prompting does not reliably improve QUENCH scores, indicating the bottleneck is entity recall and cultural knowledge rather than the reasoning format.
  • Supplying gold entities instead of predicted ones raises rationale quality sharply (up to roughly 32 points for one model), showing LLMs can justify a known answer much better than they can retrieve it on their own.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the gap is computed from jury scores while BERTScore differences are only 0.6 to 2.4 points, a human-scored validation study on the Indic subset would reveal whether the true knowledge gap is as large as the jury reports or partly an artifact of judge preferences.
  • The same masked-entity-plus-rationale format could be applied to quiz content from other non-Western regions to test whether the gap is India-specific or a general property of LLM pretraining distributions.
  • A natural next experiment the paper does not run is to give the models retrieval access or web search and measure whether the Indic gap narrows, which would separate memory failures from reasoning failures.
  • The finding that chain-of-thought does not help on QUENCH suggests the benchmark could serve as a stress test for prompt-engineering claims that are otherwise validated on Western-centric tasks.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces QUENCH, a manually curated English quiz benchmark of 400 open-domain questions with masked entities and human-written rationales, tagged as Indic (70 questions) or Non-Indic (330 questions). Seven LLMs (GPT-4-Turbo, GPT-3.5-Turbo, Gemini-1.5-Flash, Gemma-1.1-7B, Mixtral-8x7B, Llama-3-8B, Llama-3-70B) are evaluated zero-shot with and without chain-of-thought prompting using BLEU, ROUGE-L, BERTScore, and an LLM-as-judge jury (GPT-4-Turbo, Mixtral-8x7B, Llama-3-70B). The central empirical claim is that all benchmarked models perform substantially worse on Indic-context questions than on Non-Indic ones, with GEval gaps of 12 to 32 points for entity prediction and 8 to 20 points for rationale prediction. The authors also report that CoT has little effect, that larger models tend to outperform smaller ones, that gold-labeled rationales improve rationale generation, and that even the best model commits characteristic entity-recognition errors. A data-contamination check using WIMBD and Infinigram finds no significant leakage of the source material into common pretraining corpora.

Significance. QUENCH is a potentially valuable contribution: it is an open-domain, non-MCQ quiz benchmark with multi-entity masking and gold rationales, it is released with code and data, and the contamination check is careful and reassuring. If the reported Indic/non-Indic gap is taken at face value, the benchmark provides a reusable instrument for quantifying the Western-centric bias of LLM world knowledge. The strengths of the paper are the dataset construction, the multi-metric evaluation protocol, and the explicit leakage analysis. However, the headline gap is carried almost entirely by LLM-jury scores that are not validated against human judgments, and the standard lexical/semantic metrics show only small or mixed differences; the paper's central claim therefore needs additional evidence before it can be considered established.

major comments (4)
  1. [§4, Evaluation Metrics; Table 4; Appendix C] The central Indic/non-Indic gap is computed with an LLM jury that is never validated against human judgments. The gap magnitudes differ dramatically across metrics: BERTScore deltas are only 0.6–2.4 points and BLEU/ROUGE deltas are often negative or mixed, while the GEval deltas are 12–32 points. Because the judges are themselves LLMs and one of them (GPT-4-Turbo) is also a benchmarked model, the possibility that the jury is biased against acceptable Indic paraphrases or transliterations is a real threat to the main claim. The paper should report per-judge scores, inter-judge agreement, and, crucially, a human-annotated sample scored with the same binary/Likert rubric. Without such validation, the headline gap may reflect judge bias rather than model knowledge.
  2. [§6, Human Benchmarking; §5.3] The human benchmarking section is internally inconsistent and too small to serve as a jury validation. It states that 20 questions were sampled with equal numbers from both subsets, but then reports 10 Non-Indic and 13 Indic questions (totaling 23). The authors explicitly refrain from broad conclusions, and the human scores are not compared to GEval scores on the same questions. As written, this section cannot substantiate the claim in §5.3 that "all the benchmarking LLMs perform poorly at questions with an Indic context for both entity and rationale generation tasks." The authors should either provide a proper human-evaluation study on LLM outputs or temper the claim to acknowledge that the jury metric is not externally validated.
  3. [§5.3 and Tables 7–8] The claim that the Indic gap is observable "across all metrics" is not supported by the full tables. For rationale generation with predicted entities, BLEU deltas are negative for several models (e.g., Gemini 1.5 Flash without CoT: −11.8; GPT-4-Turbo: −1.7), BERTScore deltas are near zero (0.0 to 0.4), and only the GEval deltas are large. The manuscript does not provide confidence intervals, significance tests, or effect sizes for any of the reported differences. Given the small Indic subset (70 questions), the authors should report uncertainty estimates and base the headline conclusion on metrics that are robust to the judge-bias concern, or at least clearly separate the jury-based result from the standard-metric result.
  4. [§4, Evaluation Metrics; footnote on jury composition] The jury description is ambiguous about self-scoring. The text says each judge scores "every other benchmarked LLM," which would give 3×6=18 judge-model combinations, yet the paper reports 21. If the judges in fact scored themselves (3×7=21), then the self-preference bias documented by Panickssery et al. (2024), which the authors cite, directly affects the reported scores and the model ranking. The authors should clarify whether self-evaluation was included, and if so, analyze its effect on the results.
minor comments (6)
  1. [Abstract and throughout] There are several typos and word-choice errors: "B enchmark" in the abstract, "access" should be "assess" in multiple places (e.g., Section 1 and Related Work), and "explainations" in Section 6.
  2. [§6, Human Benchmarking] The wording "we randomly sample 20 questions ... We sample equal numbers from both subsets and across all themes" is contradicted by the reported 10 Non-Indic and 13 Indic questions; this arithmetic inconsistency should be fixed.
  3. [Appendix B, gold-label rationale prompt] The prompt listed for "generating rationale using gold labels" appears to be identical to the entity-prediction prompt and does not actually provide the gold entity or ask for a rationale. This appears to be a copy-paste error, but as written it makes the gold-label rationale experiment non-reproducible from the paper alone.
  4. [Figure 4 and Table 4 captions] Figure 4(c)–(d) refer to "Indic and Non-Indic languages," but the dataset is entirely in English; these should be labeled as "contexts" or "subsets" to avoid confusion.
  5. [§7, Error Analysis, Table 5] The error taxonomy lists "Correct Answer" as an error type, which is conceptually odd; the authors should rename this category (e.g., "Correct answer despite failed reasoning") or clarify why a correct answer is counted as an error.
  6. [§8, Conclusion] The conclusion says "by accessing CoT," which should read "by assessing CoT" or "by comparing CoT." Also, the statement that the Indic gap is "significant" should be qualified in light of the lack of statistical significance testing noted in the major comments.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: QUENCH is an externally curated dataset and the LLM-jury evaluation is a measurement protocol, not a fitted derivation; the judge overlap is a validity risk but not a constructional reduction.

full rationale

The paper's central contribution is a manually curated benchmark (QUENCH) and an empirical evaluation of seven LLMs. There is no derivation chain in which an output quantity is defined in terms of the input quantity: the masked entities and rationales are human-annotated from YouTube quiz sources (Section 3), and the Indic/Non-Indic split is a human tagging decision, not a fitted parameter. The evaluation uses standard metrics (BLEU, ROUGE-L, BERTScore) plus an LLM-jury GEval score; the jury is not fitted to the benchmark, and the jury protocol is explicitly motivated by the self-favoritism literature (Panickssery et al., 2024; Verga et al., 2024) rather than smuggled in as a premise. The stated 21 judge-model combinations are ambiguous (if self-scoring occurred, GPT-4-Turbo would be judging its own outputs), and the unvalidated jury is a genuine threat to the reliability of the headline Indic/non-Indic gap, especially since BERTScore gaps are only 0.6-2.4 points while GEval gaps are 12-32 points. However, this is a measurement-validity concern, not a circularity: the GEval score is not constructed to equal the input labels, no parameter is fitted to make the gap appear, and the gap is not an identity. No self-citation is load-bearing; the only author-overlapping citation (Biderman et al. 2023, Pythia) is used to explain model exclusion. Accordingly, no specific circular step can be exhibited, and the appropriate finding is no significant circularity.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No free parameters are fitted in this empirical benchmark paper. The assessment depends on the stated axioms about ground truth, evaluation validity, and leakage. The only notable new artifact is the QUENCH dataset itself, which is a measurement instrument rather than a postulated entity, so it is not listed as an invented entity.

assumptions (4)
  • domain assumption LLM-jury GEval scores are a valid and unbiased measure of entity and rationale correctness.
    The headline Indic gap is primarily derived from GEval scores (Section 4, Appendix C). The jury includes GPT-4-Turbo, itself an evaluated model, and no human agreement study validates the jury scores. If the judges are biased, the gap estimate is biased.
  • domain assumption The manually curated answers and rationales are correct ground truth for all 400 questions.
    Two annotators compiled answers and rationales; inter-annotator agreement was measured on only 10 samples (Section 3). Entity answers are objective in principle, but rationale quality is subjective and sparsely checked.
  • domain assumption The contamination check covers the actual pretraining corpora of the evaluated models.
    Appendix A checks public corpora (C4, Dolma, RedPajama, Pile, etc.) and finds no exact matches, but closed models like GPT-4 and Gemini may have trained on undisclosed data, so absence of public-corpus leakage does not prove absence of memorization.
  • domain assumption Questions transcribed from YouTube videos are faithful to the original quiz text.
    Transcription used OCR and annotator correction, but there is no second independent verification for all items (Section 3, Annotation Process). Errors in transcription would propagate to benchmark quality.

how reviews work

0 comments
Cite this review

Pith. "Pith review of QUENCH: Measuring the gap between Indic and Non-Indic Contextual General Reasoning in LLMs." pith.science (2026). https://pith.science/paper/MNO6J2EX

@misc{pith2026241211763,
  author       = {Pith},
  title        = {Pith review of: QUENCH: Measuring the gap between Indic and Non-Indic Contextual General Reasoning in LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MNO6J2EX}},
  note         = {Machine review of arXiv:2412.11763}
}
read the original abstract

The rise of large language models (LLMs) has created a need for advanced benchmarking systems beyond traditional setups. To this end, we introduce QUENCH, a novel text-based English Quizzing Benchmark manually curated and transcribed from YouTube quiz videos. QUENCH possesses masked entities and rationales for the LLMs to predict via generation. At the intersection of geographical context and common sense reasoning, QUENCH helps assess world knowledge and deduction capabilities of LLMs via a zero-shot, open-domain quizzing setup. We perform an extensive evaluation on 7 LLMs and 4 metrics, investigating the influence of model size, prompting style, geographical context, and gold-labeled rationale generation. The benchmarking concludes with an error analysis to which the LLMs are prone.

Figures

Figures reproduced from arXiv: 2412.11763 by the authors.

Figure 1
Figure 1. Themes and their distribution in QUENCH. one of the themes. In each question, some entities are masked with ‘X.’ The aim is to connect the concepts in the question to predict ‘X.’ The ques￾tions contain adequate cues to deduce the entities. The questions in the quiz already have the entities masked, and we have not modified these. However, we manually annotate the explanations/rationale to arrive at the answer. Some… view at source ↗
Figure 2
Figure 2. Screenshots showing the upper (a) and lower (b) half of the custom annotation tool’s landing page has [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The number of masked entities in QUENCH. variables inserted by the annotator. We also store the question’s source so that anyone can proof￾check these annotations if needed. Curated Dataset. The annotation process span￾ning 4 months is completed via a custom annotation tool as described below. We observe that a subset of questions exclusively pertains to Indian entities. Thus, we further tag each question as Indic o… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Figures (a) and (b) display our aggregated results comparing scenarios with and without Chain-of-Thought [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Analysis of human benchmarking split across the indic and non-indic subsets capturing the distribution. [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Screenshots showing the instructions (a) and one of the questions with multiple rationales (b) for the [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

57 extracted references · 10 canonical work pages

  1. [1]

    Julien Abadji, Pedro Ortiz Suarez, Laurent Romary, and Beno \^ t Sagot. 2022. Towards a cleaner document-oriented multilingual crawled corpus. arXiv preprint arXiv:2201.06642

    arXiv 2022

  2. [2]

    Daman Arora, Himanshu Singh, and Mausam . 2023. https://doi.org/10.18653/v1/2023.emnlp-main.468 Have LLM s advanced enough? a challenging problem solving benchmark for large language models . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7527--7543, Singapore. Association for Computational Linguistics

    doi:10.18653/v1/2023.emnlp-main.468 2023

  3. [3]

    Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. 2023. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pages 2397--2430. PMLR

    2023

  4. [4]

    Yu, Qiang Yang, and Xing Xie

    Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, Wei Ye, Yue Zhang, Yi Chang, Philip S. Yu, Qiang Yang, and Xing Xie. 2024. https://doi.org/10.1145/3641289 A survey on evaluation of large language models . ACM Trans. Intell. Syst. Technol., 15(3)

    doi:10.1145/3641289 2024

  5. [5]

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradb...

    2023

  6. [6]

    Hyung Won Chung, Noah Constant, Xavier Garcia, Adam Roberts, Yi Tay, Sharan Narang, and Orhan Firat. 2023. Unimax: Fairer and more effective language sampling for large-scale multilingual pretraining. arXiv preprint arXiv:2304.09151

    arXiv 2023

  7. [7]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

    arXiv 2021

  8. [8]

    Together Computer. 2023. https://github.com/togethercomputer/RedPajama-Data Redpajama: an open dataset for training large language models

    2023

Show all 57 references
  1. [9]

    Ernest Davis. 2023. https://doi.org/10.1145/3615355 Benchmarks for automated commonsense reasoning: A survey . ACM Comput. Surv., 56(4)

    2023 doi

  2. [10]

    Esin Durmus, Karina Nguyen, Thomas I. Liao, Nicholas Schiefer, Amanda Askell, Anton Bakhtin, Carol Chen, Zac Hatfield-Dodds, Danny Hernandez, Nicholas Joseph, Liane Lovitt, Sam McCandlish, Orowa Sikder, Alex Tamkin, Janel Thamkul, Jared Kaplan, Jack Clark, and Deep Ganguli. 20...

    2024 arXiv

  3. [11]

    Smith, and Jesse Dodge

    Yanai Elazar, Akshita Bhagia, Ian Magnusson, Abhilasha Ravichander, Dustin Schwenk, Alane Suhr, Pete Walsh, Dirk Groeneveld, Luca Soldaini, Sameer Singh, Hanna Hajishirzi, Noah A. Smith, and Jesse Dodge. 2024. https://arxiv.org/abs/2310.20707 What's in my big data? Preprint, a...

    2024 arXiv

  4. [12]

    Kanishk Gandhi, Jan-Philipp Fränken, Tobias Gerstenberg, and Noah D. Goodman. 2023. https://arxiv.org/abs/2306.15448 Understanding social reasoning in language models with language models . Preprint, arXiv:2306.15448

    2023 arXiv

  5. [13]

    Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020. https://arxiv.org/abs/2101.00027 The pile: An 800gb dataset of diverse text for language modeling . P...

    2020 arXiv

  6. [14]

    Panagiotis Giadikiaroglou, Maria Lymperaiou, Giorgos Filandrianos, and Giorgos Stamou. 2024. https://arxiv.org/abs/2402.11291 Puzzle solving using reasoning of large language models: A survey . Preprint, arXiv:2402.11291

    2024 arXiv

  7. [15]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. https://openreview.net/forum?id=d7KBjmI3GmQ Measuring massive multitask language understanding . In International Conference on Learning Representations

    2021

  8. [16]

    Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. 2024. https://openreview.net/forum?id=IkmD3fKBPQ Large language models cannot self-correct reasoning yet . In The Twelfth International Conference on Learning Representations

    2024

  9. [17]

    Nayeon Lee, Chani Jung, Junho Myung, Jiho Jin, Jose Camacho-Collados, Juho Kim, and Alice Oh. 2024. https://arxiv.org/abs/2308.16705 Exploring cross-cultural differences in english hate speech annotations: From dataset construction to analysis . Preprint, arXiv:2308.16705

    2024 arXiv

  10. [18]

    Tony Lee, Michihiro Yasunaga, Chenlin Meng, Yifan Mai, Joon Sung Park, Agrim Gupta, Yunzhi Zhang, Deepak Narayanan, Hannah Teufel, Marco Bellagente, Minguk Kang, Taesung Park, Jure Leskovec, Jun-Yan Zhu, Fei-Fei Li, Jiajun Wu, Stefano Ermon, and Percy S Liang. 2023. https://pr...

    2023

  11. [19]

    Chin-Yew Lin. 2004. https://www.aclweb.org/anthology/W04-1013 ROUGE : A package for automatic evaluation of summaries . In Text Summarization Branches Out, pages 74--81, Barcelona, Spain. Association for Computational Linguistics

    2004

  12. [20]

    Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. https://doi.org/10.18653/v1/2022.acl-long.229 T ruthful QA : Measuring how models mimic human falsehoods . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pa...

    2022 doi

  13. [21]

    Jiacheng Liu, Sewon Min, Luke Zettlemoyer, Yejin Choi, and Hannaneh Hajishirzi. 2024. Infini-gram: Scaling unbounded n-gram language models to a trillion tokens. arXiv preprint arXiv:2401.17377

    2024 arXiv

  14. [22]

    Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. 2020. https://doi.org/10.24963/ijcai.2020/501 Logiqa: A challenge dataset for machine reading comprehension with logical reasoning . In Proceedings of the Twenty-Ninth International Joint Conference on ...

    2020 doi

  15. [23]

    Xiao Liu, Da Yin, Yansong Feng, and Dongyan Zhao. 2022. https://doi.org/10.18653/v1/2022.acl-long.168 Things not written in text: Exploring spatial commonsense from visual signals . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Vol...

    2022 doi

  16. [24]

    Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.153 G -eval: NLG evaluation using gpt-4 with better human alignment . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language ...

    2023 doi

  17. [25]

    Nicholas Lourie, Ronan Le Bras , Chandra Bhagavatula, and Yejin Choi. 2021. Unicorn on rainbow: A universal commonsense reasoning model on a new multitask benchmark. AAAI

    2021

  18. [26]

    Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Processing Sy...

    2022

  19. [27]

    Pan Lu, Liang Qiu, Wenhao Yu, Sean Welleck, and Kai-Wei Chang. 2023. https://doi.org/10.18653/v1/2023.acl-long.817 A survey of deep learning for mathematical reasoning . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long ...

    2023 doi

  20. [28]

    Theo Olausson, Alex Gu, Ben Lipkin, Cedegao Zhang, Armando Solar-Lezama, Joshua Tenenbaum, and Roger Levy. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.313 LINC : A neurosymbolic approach for logical reasoning by combining language models with first-order logic provers . ...

    2023 doi

  21. [29]

    Liangming Pan, Alon Albalak, Xinyi Wang, and William Wang. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.248 Logic- LM : Empowering large language models with symbolic solvers for faithful logical reasoning . In Findings of the Association for Computational Linguistics...

    2023 doi

  22. [30]

    Bowman, and Shi Feng

    Arjun Panickssery, Samuel R. Bowman, and Shi Feng. 2024. https://arxiv.org/abs/2404.13076 Llm evaluators recognize and favor their own generations . Preprint, arXiv:2404.13076

    2024 arXiv

  23. [31]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. pages 311--318

    2002

  24. [32]

    Ji-Lun Peng, Sijia Cheng, Egil Diau, Yung-Yu Shih, Po-Heng Chen, Yen-Ting Lin, and Yun-Nung Chen. 2024. https://arxiv.org/abs/2406.00936 A survey of useful llm evaluation . Preprint, arXiv:2406.00936

    2024 arXiv

  25. [33]

    Shuofei Qiao, Yixin Ou, Ningyu Zhang, Xiang Chen, Yunzhi Yao, Shumin Deng, Chuanqi Tan, Fei Huang, and Huajun Chen. 2023. https://doi.org/10.18653/v1/2023.acl-long.294 Reasoning with language model prompting: A survey . In Proceedings of the 61st Annual Meeting of the Associat...

    2023 doi

  26. [34]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. https://arxiv.org/abs/1910.10683 Exploring the limits of transfer learning with a unified text-to-text transformer . arXiv e-prints

    2019 arXiv

  27. [35]

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. https://doi.org/10.1145/3474381 Winogrande: an adversarial winograd schema challenge at scale . Commun. ACM, 64(9):99–106

    2021 doi

  28. [36]

    Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. 2023. Whose opinions do language models reflect? In Proceedings of the 40th International Conference on Machine Learning, ICML'23. JMLR.org

    2023

  29. [37]

    Soumya Sanyal, Harman Singh, and Xiang Ren. 2022. https://doi.org/10.18653/v1/2022.acl-long.77 F ai RR : Faithful and robust deductive reasoning over natural language . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long P...

    2022 doi

  30. [38]

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. 2022. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural I...

    2022

  31. [39]

    Agrima Seth, Sanchit Ahuja, Kalika Bali, and Sunayana Sitaram. 2024. https://aclanthology.org/2024.lrec-main.474 DOSA : A dataset of social artifacts from different I ndian geographical subcultures . In Proceedings of the 2024 Joint International Conference on Computational Li...

    2024

  32. [40]

    Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennigh...

    2024

  33. [41]

    Settaluri Lakshmi Sravanthi, Meet Doshi, Tankala Pavan Kalyan, Rudra Murthy, Pushpak Bhattacharyya, and Raj Dabre. 2024. https://arxiv.org/abs/2401.07078 Pub: A pragmatics understanding benchmark for assessing llms' pragmatics capabilities . Preprint, arXiv:2401.07078

    2024 arXiv

  34. [42]

    Pat Verga, Sebastian Hofstatter, Sophia Althammer, Yixuan Su, Aleksandra Piktus, Arkady Arkhangorodsky, Minjie Xu, Naomi White, and Patrick Lewis. 2024. https://arxiv.org/abs/2404.18796 Replacing judges with juries: Evaluating llm generations with a panel of diverse models . P...

    2024 arXiv

  35. [43]

    Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. https://proceedings.neurips.cc/paper_files/paper/2019/file/4496bf24afe7fab6f046bf4923da8de6-Paper.pdf Superglue: A stickier benchmark for general-purp...

    2019

  36. [44]

    Ishaan Watts, Varun Gumma, Aditya Yadavalli, Vivek Seshadri, Manohar Swaminathan, and Sunayana Sitaram. 2024. https://arxiv.org/abs/2406.15053 Pariksha : A large-scale investigation of human-llm evaluator agreement on multilingual and multi-cultural data . Preprint, arXiv:2406.15053

    2024 arXiv

  37. [45]

    Chi, Quoc V Le, and Denny Zhou

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. 2022. https://openreview.net/forum?id=_VjQlMeSB_J Chain of thought prompting elicits reasoning in large language models . In Advances in Neural Information Proc...

    2022

  38. [46]

    Yongkang Wu, Meng Han, Yutao Zhu, Lei Li, Xinyu Zhang, Ruofei Lai, Xiaoguang Li, Yuanhang Ren, Zhicheng Dou, and Zhao Cao. 2023. https://doi.org/10.18653/v1/2023.findings-acl.148 Hence, socrates is mortal: A benchmark for natural language syllogistic reasoning . In Findings of...

    2023 doi

  39. [47]

    Ruijie Xu, Zengzhi Wang, Run-Ze Fan, and Pengfei Liu. 2024. https://arxiv.org/abs/2404.18824 Benchmarking benchmark leakage in large language models . arXiv preprint arXiv:2404.18824

    2024 arXiv

  40. [48]

    Mohd Zaki, Jayadeva, Mausam, and N. M. Anoop Krishnan. 2024. https://doi.org/10.1039/d3dd00188a Mascqa: investigating materials science knowledge of large language models . Digital Discovery, 3(2):313–327

    2024 doi

  41. [49]

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. https://doi.org/10.18653/v1/P19-1472 H ella S wag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791--4...

    2019 doi

  42. [50]

    Weinberger, and Yoav Artzi

    Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020. https://openreview.net/forum?id=SkeHuCVFDr Bertscore: Evaluating text generation with bert . In International Conference on Learning Representations

    2020

  43. [51]

    Yadong Zhang, Shaoguang Mao, Tao Ge, Xun Wang, Adrian de Wynter, Yan Xia, Wenshan Wu, Ting Song, Man Lan, and Furu Wei. 2024 a . https://arxiv.org/abs/2404.01230 Llm as a mastermind: A survey of strategic reasoning with large language models . Preprint, arXiv:2404.01230

    2024 arXiv

  44. [52]

    Zhuosheng Zhang, Aston Zhang, Mu Li, hai zhao, George Karypis, and Alex Smola. 2024 b . https://openreview.net/forum?id=y1pPWFVfvR Multimodal chain-of-thought reasoning in language models . Transactions on Machine Learning Research

    2024

  45. [53]

    Gonzalez, Ion Stoica, and Hao Zhang

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric Xing, Joseph E. Gonzalez, Ion Stoica, and Hao Zhang. 2024. https://openreview.net/forum?id=BOfDKxfwt0 LMSYS -chat-1m: A large-scale real-world LLM convers...

    2024

  46. [54]

    Kaitlyn Zhou, Kawin Ethayarajh, and Dan Jurafsky. 2022. https://doi.org/10.18653/v1/2022.findings-acl.164 Richer countries and richer representations . In Findings of the Association for Computational Linguistics: ACL 2022, pages 2074--2085, Dublin, Ireland. Association for Co...

    2022 doi

  47. [55]

    Kaijie Zhu, Jindong Wang, Jiaheng Zhou, Zichen Wang, Hao Chen, Yidong Wang, Linyi Yang, Wei Ye, Neil Zhenqiang Gong, Yue Zhang, et al. 2023. Promptbench: Towards evaluating the robustness of large language models on adversarial prompts. arXiv preprint arXiv:2306.04528

    2023 arXiv

  48. [56]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  49. [57]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.