Pith. sign in

REVIEW 4 major objections 4 minor 36 references

Understanding LLM Scientific Reasoning through Promptings and Model's Explanation on the Answers

T0 review · 4 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read GPT-4o's answer accuracy and explanation quality come apart depending on the prompting technique used, with the most accurate prompt producing the least ground-truth-aligned explanations.

desk verdict A useful prompt-accuracy baseline on GPQA, but the explanation-quality metric is too confounded to support the headline decoupling claim. read the letter →

arxiv 2505.01482 v2 pith:OYMRVK4M submitted 2025-05-02 cs.AI

Alice Rueda , Mohammed S. Hassan , Argyrios Perivolaris , Bazen G. Teferra , Reza Samavi , Sirisha Rambhatla , Yuqi Wu , Yanbo Zhang
show 4 more authors
Bo Cao Divya Sharma Sridhar Krishnan Venkat Bhat
This is my paper · ORCID
classification cs.AI
keywords largelanguagemodelsscientificreasoningpromptengineeringGPQAGPT-4ochain-of-thoughtself-consistencyexplanationsimilarity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish how GPT-4o's scientific reasoning behaves on the 448-question GPQA-main benchmark under seven prompting techniques, measuring not only how often the model answers correctly but also how closely its explanations resemble expert-written ground-truth explanations. The central finding is a decoupling: self-consistency produces the highest accuracy (52.9%), while the simplest prompts—direct answer, chain-of-thought, and zero-shot chain-of-thought—produce the explanations most similar to the ground truth (median cosine similarity around 0.73). The paper takes this split as evidence that LLMs can appear to reason well on a task while the process behind correct answers is not the same as the process that produces a good scientific explanation. The result matters because it offers a repeatable way to treat accuracy and reasoning quality as separate dimensions when building LLM systems for science, medicine, and law.

What carries the argument

The machinery is a paired measurement: each answer is scored for correctness and, separately, for the similarity of its explanation to the GPQA ground-truth explanation. Similarity is computed as cosine similarity between MPNet sentence embeddings of GPT-4o's prompted explanation and the dataset's expert explanation, grouped into weak, moderate, and strong bands. The paper also fits linear regressions to the cumulative correctness curves and uses R-squared values above 0.99 to argue that the slope of each curve is a stable per-prompt summary. The load-bearing identity is the comparison of the two resulting rankings: the accuracy ranking places self-consistency first, while the explanation-similarity ranking places the three simplest prompts first, and this inversion is the evidence for the paper's claim that reasoning quality and accuracy can part ways.

What would settle it

Regress the Table 2 cosine similarities on explanation length or on verbatim term overlap with the ground truth: if either predictor alone reproduces the technique rankings, the explanation-quality measure tracks style or coverage rather than reasoning. Alternatively, rephrase GPQA questions without changing content and rerun the prompts: if accuracy on rephrased questions drops well below 52.9%, the 'pattern recognition' interpretation is confounded by memorized benchmark answers.

Watch

Extended reading notes

Core claim

The paper's central claim is that on GPQA-main, GPT-4o's answer accuracy and the quality of its scientific reasoning, as measured by the cosine similarity between the model's elicited explanations and GPQA's ground-truth explanations, do not correlate across prompting techniques. In its accuracy ranking, self-consistency leads with 237 out of 448 correct (52.9%), followed by direct answer (52.2%), zero-shot CoT (50%), multipath (48.4%), decomposition (47.8%), self-ask (46.9%), and chain-of-thought (43.8%). In the explanation-similarity ranking, the order is different at the top: direct answer, CoT, and zero-shot CoT reach median similarity values around 0.73 and are classified as strong, while self-consistency has the lowest median similarity (0.645) and is classified as moderate. The paper interprets this separation as evidence that the model often relies on pattern recognition and prompt-induced response habits rather than consistently applying logical inference, and it proposes a research agenda for structured reasoning frameworks, hybrid approaches, and human-in-the-loop verification.

Load-bearing premise

The load-bearing premise is that the cosine similarity between MPNet embeddings of GPT-4o's explanation and GPQA's ground-truth explanation actually measures scientific reasoning quality, together with the assumption that the model's December 2023 knowledge cutoff prevents it from having memorized the GPQA questions and answers.

Editorial extensions

If this is right

  • Self-consistency achieves the highest accuracy (237/448, 52.9%) yet ranks last in explanation similarity, so maximizing accuracy alone can select the prompt whose explanations least resemble expert reasoning.
  • Direct answer, CoT, and zero-shot CoT explanations all reach median cosine similarity around 0.73, placing simple single-path prompts closest to the GPQA ground-truth explanations.
  • Each prompt's cumulative correctness curve is nearly perfectly linear (R-squared above 0.99), so the slope of the curve can serve as a stable summary of a prompt's per-question accuracy.
  • GPT-4o refuses to answer certain questions under most prompts, notably question 257 where its computed answer is not among the choices, which the paper reads as evidence of internal computation that can override the required choice format.
  • Accuracy and explanation quality should be reported as separate dimensions when evaluating scientific reasoning, since the two rankings differ.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A length-controlled re-run of the same seven prompts would test whether the cosine-similarity ranking reflects logical content or simply the verbosity of each prompt's explanations.
  • The decoupling implies a practical trade-off not explicitly recommended by the paper: deployments that prioritize auditable reasoning might choose direct-answer or CoT prompts, while deployments that prioritize raw accuracy might choose self-consistency.
  • Question 257, where most prompts refuse to answer because no listed choice matches the model's computed value, offers a natural probe for whether the model's 'pattern recognition' behavior masks a consistent latent computation.
  • Running the same accuracy-plus-similarity protocol on other frontier models would show whether the decoupling is a GPT-4o-specific artifact or a general property of LLM prompting.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper evaluates seven prompting techniques (direct answer, chain-of-thought, zero-shot chain-of-thought, self-ask, self-consistency, decomposition, and multipath) with GPT-4o on the 448-question GPQA-main dataset. It measures answer accuracy and, for each prompt, the cosine similarity between the model's elicited explanation and the GPQA ground-truth explanation using an MPNet sentence transformer. The paper reports that self-consistency achieves the highest accuracy (237/448, 52.90% in Table 1) but the lowest median explanation similarity, while direct answer, chain-of-thought, and zero-shot chain-of-thought produce explanations most similar to the ground truth. On this basis the paper concludes that reasoning quality does not always correlate with accuracy, and proposes a research agenda for improving interpretability and robustness.

Significance. If the central claim held, the paper would provide a useful empirical baseline: on a challenging scientific benchmark, prompt choice can decouple answer accuracy from explanation quality, with self-consistency improving accuracy while lowering explanation similarity. The paper has clear strengths: it uses all 448 GPQA-main questions, reports domain-level breakdowns, gives transparent accuracy tables, and checks distributional assumptions with Shapiro-Wilk tests. However, the headline conclusion currently rests on an unvalidated embedding-similarity measure and an unreported rule for handling missing explanations, so the significance of the empirical finding is conditional on those issues being resolved. The paper does not provide code or human expert validation, but the tabulated measurements are detailed enough to reanalyze.

major comments (4)
  1. [Section V, Table 2] Section V states that 'For some questions, self-consistency only provided the answers with no explanation,' but Table 2 reports cosine-similarity counts that sum to 448 for every prompt, including self-consistency. The manuscript never states how many explanations were missing or how missing entries entered the cosine similarity computation (excluded, empty string, zero vector, or imputed). Because the central conclusion that reasoning quality does not correlate with accuracy depends on self-consistency's median of 0.645 being lower than the other prompts, the missing-explanation rule is load-bearing. Please report the number of missing explanations per prompt and provide a sensitivity analysis (for example, excluding those questions, or using worst-case and best-case imputation) to show that the ranking is stable.
  2. [Section III.C, Table 2] The paper equates MPNet cosine similarity between the prompt-induced explanation and the GPQA ground-truth explanation with 'scientific reasoning quality,' but provides no validation that this metric tracks reasoning quality. The authors state that they are non-experts and did not grade explanations, and the metric is plausibly confounded by prompt-induced style: multipath prompting explicitly forces justifications for all four choices, and self-consistency aggregates multiple reasoning paths, so lower cosine similarities may reflect stylistic divergence rather than worse scientific reasoning. Please add human expert ratings on a subset of explanations or a control analysis that compares cosine similarity with surface-form variables (for example, explanation length, number of equations, or template phrasing) to support the construct validity of the measure. Without such evidence, the accuracy-explanation decoupling claim is not established.
  3. [Section III, leakage control] Section III claims that setting the knowledge cutoff to December 2023 and the model date to April 1, 2024 'ensure[s] no data leakage,' but GPQA was published in November 2023 and the evaluated snapshot is GPT-4o-2024-08-06. These API parameters do not alter the training data, so they cannot guarantee that GPQA was not seen during training. The interpretation that accuracy reflects reasoning rather than memorization is therefore unsupported. Please add a discussion of contamination risk and, if possible, a quantitative bound (for example, evaluation on a post-cutoff held-out set or a citation to a contamination analysis of the model snapshot).
  4. [Section V, Fig. 1, Table V] The paper interprets R-squared values above 0.99 for the linear fits to cumulative correctness as evidence that 'the model's accuracy does not fluctuate wildly.' For a cumulative sum of per-question Bernoulli outcomes regressed on question index, a high R-squared is expected from the accumulation process itself and is not informative about consistency; the slope being close to the mean accuracy is also definitional. This claim should be removed or reframed, or replaced with a direct measure of variability such as per-subdomain accuracy with confidence intervals or a runs test.
minor comments (4)
  1. [Abstract, Section IV.A, Table 1] The abstract reports self-consistency accuracy as 52.99%, but Table 1 gives 237/448 = 52.90%; the same inconsistency appears in Section IV.A. Please correct the abstract and body to match the table.
  2. [Abstract, Section IV.B, Table 2] The abstract says self-consistency performed 'second worst' in explaining answers, while Section IV.B ranks by median and places self-consistency last, and Table 2 shows multipath as lower only by mean. Please state the ranking convention explicitly and reconcile the abstract, the text, and Table 2.
  3. [Section IV.B, Table 2] The text says the descending order by median is 'direct answer, CoT, zero-shot CoT,' but the table values are CoT 0.736, direct answer 0.733, and zero-shot CoT 0.730. Reorder the text or note the rounding that makes the order differ.
  4. [Section IV.A, Section V] Decomposition 'Option 1' is said to have been removed because of hallucination, but the paper does not report how many questions were affected or what the Option 1 results were beyond that statement. A brief quantitative description would help readers assess the removal.

Circularity Check

1 steps flagged · score 2.0 of 10

Cumulative-correctness regression is a definitional sub-claim; the central accuracy–explanation comparison is anchored to external GPQA ground truth.

  1. self definitional [Section IV-A (Accuracy of Answers), paragraph describing Fig. 1; corroborated by Table V in Appendix C]
    "The slope represents the mean, and the distance from the slope represents the variation. A higher R-squared value demonstrates that the mean can be an accurate representation of accuracy. All prompts have a linear characteristic on the number of correct answers, as supported by the linear models presented in the Supplementary Table C1 in Appendix C. The coefficients (slopes) of the linear models are very close to the accuracy values, as all R-squared scored over 0.99."

    Cumulative correctness is, by definition, the running sum of the per-question binary outcomes whose average is the reported accuracy. Regressing that cumulative sum against question index therefore yields a slope close to the mean accuracy and an R-squared near 1 for any fixed-rate Bernoulli process; the near-perfect linear fit is a mathematical consequence of the cumulative-sum construction, not an independent empirical check that the model's accuracy is stable. The paper presents this as evidence that 'the slope of the accuracy curve serves as a reliable indicator of performance,' but it carries no information beyond the Table 1 accuracy numbers already used to define the cumulative curve.

full rationale

The paper's central claim—that prompt choice can decouple answer accuracy from explanation quality—is not circular. Accuracy is measured directly from GPT-4o's multiple-choice responses, and explanation quality is measured as cosine similarity between MPNet embeddings of the model's elicited explanations and the GPQA ground-truth explanations. Neither quantity is fitted from the other, and the external GPQA benchmark anchors the comparison. The prompt techniques are standard external methods with original citations, and no load-bearing self-citation chain is present. The one construction-reduction is the cumulative-correctness regression: cumulative correctness is the running sum of the same binary outcomes whose mean defines accuracy, so a slope near the mean and R-squared above 0.99 are expected by construction. That sub-claim is a definitional consistency check rather than independent evidence, and it does not support the decoupling result. Other issues (e.g., the 52.99% versus 52.90% inconsistency and the unvalidated similarity metric) are correctness and validity concerns, not circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new entities, so the invented-entities list is empty. The central claims rest on GPQA ground-truth explanations being correct, on MPNet cosine similarity being a valid proxy for reasoning quality, on the knowledge-cutoff setting preventing leakage, and on psychology correlation thresholds transferring to embedding similarities. Hand-chosen settings (temperature 0.5, three self-consistency paths, 4096-token limit) are free parameters that the reported rankings depend on. The post-hoc removal of decomposition option 1 is a selection decision and is recorded under red flags rather than in this ledger.

free parameters (3)
  • Sampling temperature = 0.5
    Chosen by hand in Section III to 'reduce randomness' while permitting variability; all accuracy and similarity numbers depend on it, and rankings may not be robust to it.
  • Self-consistency path count = 3
    Set as 'the minimum setting of a small number of three' (Section III.B.5); self-consistency's headline 52.90 percent accuracy depends directly on this count.
  • Completion token limit = 4096
    Set to capture full explanations (Section III); if explanations are truncated, the cosine similarity scores change.
assumptions (4)
  • domain assumption MPNet embedding cosine similarity between GPT-4o's explanation and GPQA's ground-truth explanation is a valid proxy for scientific reasoning quality.
    Assumed in Section III.C and used to rank techniques in Section V; not validated against expert grading, and the authors state in Section V that their studies are conducted by non-experts with no doctoral degrees in the relevant sciences.
  • ad hoc to paper Setting the knowledge cutoff to December 2023 and the model date to April 1, 2024 prevents GPQA data leakage.
    Asserted in Sections II and III; API knowledge-cutoff settings do not remove content already in the training corpus, and no leakage test is performed, so the no-leakage premise is specific to this paper's setup rather than established.
  • domain assumption GPQA ground-truth explanations are correct and complete reference explanations.
    Used as the reference for all similarity scores (Section III.C); any errors or stylistic quirks in the reference explanations would bias the ranking of techniques, especially the claim that simpler prompts explain better.
  • domain assumption Correlation-strength thresholds (weak below 0.3, moderate 0.3 to 0.7, strong at least 0.7) from psychology apply directly to embedding cosine similarities.
    Applied in Table 2 to label 'Overall Similarity Strength'; the scale of MPNet cosine values is not the same as a Pearson correlation coefficient, so the labels are only heuristic.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Understanding LLM Scientific Reasoning through Promptings and Model's Explanation on the Answers." pith.science (2026). https://pith.science/paper/OYMRVK4M

@misc{pith2026250501482,
  author       = {Pith},
  title        = {Pith review of: Understanding LLM Scientific Reasoning through Promptings and Model's Explanation on the Answers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OYMRVK4M}},
  note         = {Machine review of arXiv:2505.01482}
}
read the original abstract

Large language models (LLMs) have demonstrated remarkable capabilities in natural language understanding, reasoning, and problem-solving across various domains. However, their ability to perform complex, multi-step reasoning task-essential for applications in science, medicine, and law-remains an area of active investigation. This paper examines the reasoning capabilities of contemporary LLMs, analyzing their strengths, limitations, and potential for improvement. The study uses prompt engineering techniques on the Graduate-Level GoogleProof Q&A (GPQA) dataset to assess the scientific reasoning of GPT-4o. Five popular prompt engineering techniques and two tailored promptings were tested: baseline direct answer (zero-shot), chain-of-thought (CoT), zero-shot CoT, self-ask, self-consistency, decomposition, and multipath promptings. Our findings indicate that while LLMs exhibit emergent reasoning abilities, they often rely on pattern recognition rather than true logical inference, leading to inconsistencies in complex problem-solving. The results indicated that self-consistency outperformed the other prompt engineering technique with an accuracy of 52.99%, followed by direct answer (52.23%). Zero-shot CoT (50%) outperformed multipath (48.44%), decomposition (47.77%), self-ask (46.88%), and CoT (43.75%). Self-consistency performed the second worst in explaining the answers. Simple techniques such as direct answer, CoT, and zero-shot CoT have the best scientific reasoning. We propose a research agenda aimed at bridging these gaps by integrating structured reasoning frameworks, hybrid AI approaches, and human-in-the-loop methodologies. By critically evaluating the reasoning mechanisms of LLMs, this paper contributes to the ongoing discourse on the future of artificial general intelligence and the development of more robust, trustworthy AI systems.

Figures

Figures reproduced from arXiv: 2505.01482 by the authors.

Figure 1
Figure 1. Cumulated correctness. (a) Each correct answer for the prompts is stored as a cumulative sum. (b) Model the accumulated correctness with linear [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Cumulated correctness sorted by subdomain. The answers were first sorted by subdomain before calculating the accumulated sum. (a) Cumulated [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Distribution of the cosine similarity that GPT-4o’s explanation for [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Cosine similarity for the first 30 questions that GPT-4o’s explanation [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

36 extracted references · 10 canonical work pages

  1. [1]

    Large language models and cognitive science: A comprehensive review of similarities, differences, and challenges,

    Q. Niu, J. Liu, Z. Bi et al. , “Large language models and cognitive science: A comprehensive review of similarities, differences, and challenges,” 2024, published online Sept 3. [Online]. Available: http://arxiv.org/abs/2409.02387

    arXiv 2024

  2. [2]

    Mulcogbench: A multi-modal cognitive benchmark dataset for evaluating chinese and english computational language models,

    Y . Zhang, X. Zhang, C. Li, S. Wang, and C. Zong, “Mulcogbench: A multi-modal cognitive benchmark dataset for evaluating chinese and english computational language models,” 2024, published online March

    work page 2024

  3. [3]

    Available: http://arxiv.org/abs/2403.01116

    [Online]. Available: http://arxiv.org/abs/2403.01116

    arXiv

  4. [4]

    Large language models for mathematical reasoning: Progresses and challenges,

    J. Ahn, R. Verma, R. Lou, D. Liu, R. Zhang, and W. Yin, “Large language models for mathematical reasoning: Progresses and challenges,” 2024, published online Jan 31. [Online]. Available: http://arxiv.org/abs/2402.00157 8 TABLE III BEST-PERFORMED SETTING FOR EACH OF THE WELL -KNOWN REASONING MODEL FAMILY ON GPQA D IAMOND DATASET OpenAI-o1-0912 DeepSeekR1-Z...

    arXiv 2024

  5. [5]

    Explaining and Improving Contrastive Decoding by Extrapolating the Probabilities of a Huge and Hypothetical LM

    H.-S. Chang, N. Peng, M. Bansal, A. Ramakrishna, and T. Chung, “Explaining and improving contrastive decoding by extrapolating the probabilities of a huge and hypothetical lm,” 2024, published online Nov 3. [Online]. Available: http://arxiv.org/abs/2411.01610

    work page Pith review arXiv 2024

  6. [6]

    Gpqa: A graduate-level google-proof q&a benchmark,

    D. Rein, B. L. Hou, A. C. Stickland et al. , “Gpqa: A graduate-level google-proof q&a benchmark,” 2023, published online Nov 20. [Online]. Available: http://arxiv.org/abs/2311.12022

    arXiv 2023

  7. [8]

    Iteration of thought: Leveraging inner dialogue for autonomous large language model reasoning,

    S. K. Radha, Y . N. Jelyani, A. Ghukasyan, and O. Goktas, “Iteration of thought: Leveraging inner dialogue for autonomous large language model reasoning,” 2024, published online Sept 19. [Online]. Available: http://arxiv.org/abs/2409.12618

    arXiv 2024

  8. [9]

    Can Stories Help LLMs Reason? Curating Information Space Through Narrative

    V . S. Javadi, J. R. Trippas, Y . K. Lal, and L. Flek, “Can stories help llms reason? curating information space through narrative,” 2024, published online Oct 24. [Online]. Available: http://arxiv.org/abs/2410.19221

    work page Pith review arXiv 2024

Show all 36 references
  1. [10]

    Mtmt: Consolidating multiple thinking modes to form a thought tree for strengthening llm,

    C. Li, X. Wang, Q. Chen, X. Zhou, and H. Chen, “Mtmt: Consolidating multiple thinking modes to form a thought tree for strengthening llm,” 2024, published online Dec 5. [Online]. Available: http://arxiv.org/abs/2412.03987

    2024 arXiv

  2. [11]

    Boosting scientific concepts understanding: Can analogy from teacher models empower student models?

    S. Yuan, C. Jiayang, L. Qiu, and D. Yang, “Boosting scientific concepts understanding: Can analogy from teacher models empower student models?” 2024, published online June 17. [Online]. Available: http://arxiv.org/abs/2406.11375

    2024 arXiv

  3. [12]

    Refining llms outputs with iterative consensus ensemble (ice),

    M. Omar, B. S. Glicksberg, G. N. Nadkarni, and E. Klang, “Refining llms outputs with iterative consensus ensemble (ice),” 2024, published online Dec 28

    2024

  4. [13]

    Reducing hallucination in structured outputs via retrieval-augmented generation,

    P. B ´echard and O. M. Ayala, “Reducing hallucination in structured outputs via retrieval-augmented generation,” 2024, published online April 11

    2024

  5. [14]

    React: Synergizing reasoning and acting in language models,

    S. Yao, J. Zhao, D. Yu et al., “React: Synergizing reasoning and acting in language models,” 2022, published online Oct 5. [Online]. Available: http://arxiv.org/abs/2210.03629

    2022 arXiv

  6. [15]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans et al. , “Chain-of-thought prompting elicits reasoning in large language models,” 2022, published online Jan

    2022

  7. [16]

    A systematic survey of prompt engineering in large language models: Techniques and applications,

    P. Sahoo, A. K. Singh, S. Saha, V . Jain, S. Mondal, and A. Chadha, “A systematic survey of prompt engineering in large language models: Techniques and applications,” 2024, published online Feb 5. [Online]. Available: http://arxiv.org/abs/2402.07927

    2024 arXiv

  8. [17]

    Self-consistency improves chain of thought reasoning in language models,

    X. Wang, J. Wei, D. Schuurmans et al. , “Self-consistency improves chain of thought reasoning in language models,” 2022, published online March 21. [Online]. Available: http://arxiv.org/abs/2203.11171

    2022 arXiv

  9. [18]

    Logicot: Logical chain-of-thought instruction-tuning,

    H. Liu, Z. Teng, L. Cui, C. Zhang, Q. Zhou, and Y . Zhang, “Logicot: Logical chain-of-thought instruction-tuning,” 2023, published online May 20. [Online]. Available: http://arxiv.org/abs/2305.12147

    2023 arXiv

  10. [19]

    Automatic chain of thought prompting in large language models,

    Z. Zhang, A. Zhang, M. Li, and A. Smola, “Automatic chain of thought prompting in large language models,” 2022, published online Oct 7. [Online]. Available: http://arxiv.org/abs/2210.03493

    2022 arXiv

  11. [20]

    Tree of thoughts: Deliberate problem solving with large language models,

    S. Yao, D. Yu, J. Zhao et al. , “Tree of thoughts: Deliberate problem solving with large language models,” 2023, published online May 17. [Online]. Available: http://arxiv.org/abs/2305.10601

    2023 arXiv

  12. [21]

    Chain-of-symbol prompting elicits planning in large language models,

    H. Hu, H. Lu, H. Zhang, Y .-Z. Song, W. Lam, and Y . Zhang, “Chain-of-symbol prompting elicits planning in large language models,” 2023, published online May 17. [Online]. Available: http://arxiv.org/abs/2305.10276

    2023 arXiv

  13. [22]

    Chain-of-table: Evolving tables in the reasoning chain for table understanding,

    Z. Wang, H. Zhang, C.-L. Li et al. , “Chain-of-table: Evolving tables in the reasoning chain for table understanding,” 2024, published online Jan 9. [Online]. Available: http://arxiv.org/abs/2401.04398

    2024 arXiv

  14. [23]

    Thread of thought: Unraveling chaotic contexts,

    Y . Zhou, X. Geng, T. Shen et al. , “Thread of thought: Unraveling chaotic contexts,” 2023, published online Nov 15. [Online]. Available: http://arxiv.org/abs/2311.08734

    2023 arXiv

  15. [24]

    Challenging big-bench tasks and whether chain-of-thought can solve them,

    M. Suzgun, N. Scales, N. Sch ¨arli et al. , “Challenging big-bench tasks and whether chain-of-thought can solve them,” 2022, published online Oct 17. [Online]. Available: http://arxiv.org/abs/2210.09261

    2022 arXiv

  16. [25]

    Reasoning with language model prompting: A survey,

    S. Qiao, Y . Ou, N. Zhang et al. , “Reasoning with language model prompting: A survey,” 2022, published online Dec 19. [Online]. Available: http://arxiv.org/abs/2212.09597

    2022 arXiv

  17. [26]

    Beyond the imitation game: Quantifying and extrapolating the capabilities of language models,

    A. Srivastava, A. Rastogi, A. Rao et al. , “Beyond the imitation game: Quantifying and extrapolating the capabilities of language models,” 2022, published online June 9. [Online]. Available: http: //arxiv.org/abs/2206.04615

    2022 arXiv

  18. [27]

    Available: http://arxiv.org/abs/2201.11903

    [Online]. Available: http://arxiv.org/abs/2201.11903

    arXiv

  19. [28]

    H. Chen, S. Wu, X. Quan, R. Wang, M. Yan, and J. Zhang, “Mcc-kd: 9 TABLE VI SUBDOMAIN QUESTIONS IN THE GPQA- MAIN DATASET AND THE CORRESPONDING HUMAN PERFORMANCE Domain Questions Expert Accuracy Non-Expert Accuracy Expertise Gap Molecular Biology 62 75.8 37.1 38.7 Physics (gen...

    2023 arXiv

  20. [29]

    Evaluating large language models trained on code,

    M. Chen, J. Tworek, H. Jun et al. , “Evaluating large language models trained on code,” 2021, published online July 7. [Online]. Available: http://arxiv.org/abs/2107.03374

    2021 arXiv

  21. [30]

    Blimp: The benchmark of linguistic minimal pairs for english,

    A. Warstadt, A. Parrish, H. Liu et al. , “Blimp: The benchmark of linguistic minimal pairs for english,” Transactions of the Association for Computational Linguistics , vol. 8, pp. 377–92, 2020

    2020

  22. [31]

    Cogbench: A large language model walks into a psychology lab,

    J. Coda-Forno, M. Binz, J. X. Wang, and E. Schulz, “Cogbench: A large language model walks into a psychology lab,” 2024, published online Feb 28. [Online]. Available: http://arxiv.org/abs/2402.18225

    2024 arXiv

  23. [32]

    Measuring progress on scalable oversight for large language models,

    S. R. Bowman, J. Hyun, E. Perez et al. , “Measuring progress on scalable oversight for large language models,” 2022, published online Nov 4. [Online]. Available: http://arxiv.org/abs/2211.03540

    2022 arXiv

  24. [33]

    Problems with cosine as a measure of embedding similarity for high frequency words,

    K. Zhou, K. Ethayarajh, D. Card, and D. Jurafsky, “Problems with cosine as a measure of embedding similarity for high frequency words,” 2022, published online May 10. [Online]. Available: http://arxiv.org/abs/2205.05092

    2022 arXiv

  25. [34]

    Mpnet: Masked and permuted pre-training for language understanding,

    K. Song, X. Tan, T. Qin, J. Lu, and T.-Y . Liu, “Mpnet: Masked and permuted pre-training for language understanding,” 2020, published online April 20. [Online]. Available: http://arxiv.org/abs/2004.09297

    2020 arXiv

  26. [35]

    User’s guide to correlation coefficients,

    H. Akoglu, “User’s guide to correlation coefficients,” Turkish Journal of Emergency Medicine , vol. 18, pp. 91–93, 2018

    2018

  27. [36]

    s1: Simple test- time scaling,

    N. Muennighoff, Z. Yang, W. Shi et al. , “s1: Simple test- time scaling,” 2025, published online Jan 31. [Online]. Available: http://arxiv.org/abs/2501.19393

    2025 arXiv

  28. [37]

    Do not think that much for 2+3=? on the overthinking of o1-like llms,

    X. Chen, J. Xu, T. Liang et al. , “Do not think that much for 2+3=? on the overthinking of o1-like llms,” 2024, published online Dec 30. [Online]. Available: http://arxiv.org/abs/2412.21187 VIII. A CKNOWLEDGMENT None

    2024 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.