Pith. sign in

REVIEW 5 major objections 6 minor 28 references

Reasoning Beyond the Obvious: Evaluating Divergent and Convergent Thinking in LLMs for Financial Scenarios

T0 review · 5 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that LLM reasoning splits into two measurable abilities — generating novel futures versus deducing the correct outcome — and that its new financial benchmark, ConDiFi, shows fluent models are often neither novel nor…

desk verdict A promising benchmark idea undermined by an unvalidated GPT-4o oracle and several internal inconsistencies; worth a serious referee, not acceptance as-is. read the letter →

arxiv 2507.18368 v1 pith:CGON4SIZ submitted 2025-07-24 cs.AI

Zhuang Qiang Bok , Watson Wei Khong Chua This is my paper · ORCID
classification cs.AI
keywords LargeLanguageModelsConvergentThinkingDivergentFinancialReasoningCreativityEvaluationBenchmarksMulti-hopLLM-as-a-Judge
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that current LLM reasoning benchmarks are lopsided: they reward factual recall and step-by-step logic but ignore the creative half of professional financial judgment, where an analyst must invent plausible futures under uncertainty. To make that gap measurable, the authors build ConDiFi, a benchmark pairing 607 open-ended prompts that ask a model to generate branching timelines of how a company's situation might evolve with 990 adversarial multiple-choice questions that require multi-hop deduction to pick the one coherent timeline. Across 14 models, the two halves come apart: GPT-4o, fluent and strong on standard tests, sits in the lower half on Novelty and near the bottom on Actionable, while DeepSeek-R1 and Cohere Command A produce the most novel and investable scenarios, and Llama-family models top the exact-match convergent questions. If the paper is right, ConDiFi gives finance a way to measure strategic foresight as a capability distinct from fluent explanation, and shows that a model's training and fine-tuning choices shape that capability independently of raw fluency.

What carries the argument

The load-bearing object is ConDiFi, a two-part benchmark, and the unit that carries the divergent argument is the branching timeline. Each response to the 607 financial scenarios is parsed as a directed tree and scored on five dimensions: Plausibility, Novelty, Elaboration, and Actionable are judged by GPT-4o under a penalty catalogue that docks generic filler, unsupported figures, repetition, and fatal logical flaws, while Richness is computed automatically from graph statistics — branching factor, maximum and mean path length, and number of leaf paths — combined into a score in [1, 10]. For the convergent half, the machinery is the adversarial generation pipeline: GPT-4o builds 990 four-option timeline MCQs using six templates (Historical $\beta$-Swap, Numeric Trip-Wire, Policy Game, Cross-Section Confuser, Reg-Legal Trap, Adversarial Self-Play) so that each distractor violates exactly one convergent criterion — factor alignment, temporal coherence, or logical entailment — and two rounds of self-refinement then harden the questions; performance is measured by the Convergent Correctness Score, $\mathrm{CCS} = (1/N) \sum_{i=1}^{N} [\mathrm{Pred}_i = \mathrm{GT}_i]$. This two-part structure does the argumentative work: it is what lets the paper claim that a model can be a strong convergent reasoner and a weak divergent one, or the reverse.

What would settle it

Take a random subset of the 607 divergent prompts, strip the model names, and have a panel of professional financial analysts score the timelines on the same Novelty and Actionable rubric; if the analysts' model-level rankings do not track GPT-4o's scores, the paper's divergent results do not stand. Separately, since GPT-4o both wrote the convergent questions and fixed their ground truth, have analysts mark the labeled 'correct' timeline in a sample of the 990 MCQs; a nontrivial disagreement rate would mean the CCS numbers measure agreement with one model's causal assumptions rather than with expert financial reasoning.

Watch

Extended reading notes

Core claim

On its own terms, the paper's claim is that divergent and convergent thinking are separable, measurable dimensions of LLM financial reasoning, and that ConDiFi reveals a systematic asymmetry: fluency and plausibility do not imply novelty or strategic utility. The evidence is the model rankings: DeepSeek-R1 and Cohere Command A lead the divergent set with overall scores at or near 8.0, with DeepSeek-R1 highest on Novelty (8.45) and Actionable (8.90), while GPT-4o scores 6.65 and 6.02 on those same dimensions; on the convergent side, Llama-4 Maverick posts the top Convergent Correctness Score of 72.32% on the hardest refinement round, with o1 and Llama-4 Scout close behind, and the spread across models (41.61% to 72.32%) shows the question set separates models rather than saturating them. The paper further claims that the pattern of correlation among the five divergent dimensions constitutes a behavioral fingerprint of each model, that these fingerprints differ systematically with training approach (DeepSeek-R1 is an outlier, Llama-family models cluster together), and that this structure can guide ensemble design. The bottom line the authors draw is that financial evaluation should not collapse to a single reasoning score; it should measure both axes.

Load-bearing premise

The benchmark stands or falls on the assumption that GPT-4o's judgments are a valid proxy for expert financial opinion — it writes the convergent questions, supplies their correct answers, refines them to make them harder, scores every divergent timeline on four of five dimensions, and is itself one of the models being ranked — and although the paper flags the need for human audits, it provides no human validation of those divergent scores.

Editorial extensions

If this is right

  • Model selection for financial deployment should separate the two axes: a system meant to generate investment theses should be chosen on Novelty and Actionable, not on fluency or standard QA accuracy, because the paper's results show those qualities can diverge sharply.
  • Training choices visibly shape cognitive style: DeepSeek-R1's outlier correlation structure and the clustering of Llama-family models suggest that fine-tuning methodology, not just scale, decides whether a model leans toward divergent or convergent reasoning.
  • The adversarial refinement procedure is a reusable difficulty dial: average convergent accuracy fell from 85.5% on the original questions to 65.6% after two rounds, so the same pipeline can produce items that stretch frontier models instead of saturating them.
  • If the rankings hold, general-purpose models are not automatically safe choices for strategic financial tasks: the benchmark's headline asymmetry means plausible-sounding output can coexist with low novelty and low actionability in the very models that dominate conventional benchmarks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The pipeline the paper leaves implicit is a two-stage team: let a top divergent model generate candidate scenario trees, then let a top convergent model stress-test and select among them — the division of labor the correlation results suggest would beat either model alone.
  • The fluency–novelty gap probably transfers beyond equities to other forecasting-heavy domains (macro strategy, geopolitics, supply-chain risk), and the branching-timeline rubric could be ported there directly as a test of that conjecture.
  • The paper itself reports that Richness tracks human ratings only weakly ($\rho \approx 0.56$); a natural fix its own data point toward is reweighting the four graph components or capping breadth so the structural metric rewards deep causal chains rather than wide-but-shallow trees.
  • A sensitivity check the paper does not run is rebuilding the whole pipeline with a different model standing in for GPT-4o as question generator and judge; if model rankings survive the swap, the benchmark's claims would no longer rest on one model's judgment.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper introduces ConDiFi, a benchmark that jointly evaluates divergent and convergent thinking in LLMs on financial scenarios. The divergent component consists of 607 macro-financial prompts; models generate branching timelines that are scored on Plausibility, Novelty, Elaboration, Actionable, and a graph-based Richness metric. The convergent component consists of 990 multi-hop adversarial multiple-choice questions with timelines as options, scored as the proportion exactly matching a generated ground truth. Fourteen models are evaluated, and the paper reports rankings in which, for example, GPT-4o underperforms on Novelty and Actionability while DeepSeek-R1 and Cohere Command A rank highly on divergent dimensions. The central conclusion is that ConDiFi reveals asymmetries: fluency and plausibility do not imply novelty or strategic utility. A number of additional analyses probe intra-model correlations, inter-model distances, PCA, question difficulty, and error categories.

Significance. The conceptual separation of divergent and convergent reasoning in a financial domain is a worthwhile direction, and the paper has several concrete strengths: the adversarial pipelines for MCQ generation are thoughtfully specified; the Richness metric is a deterministic, interpretable structural measure; and the evaluation covers 14 models across multiple families. If the benchmark were properly validated, it could be a useful complement to existing reasoning benchmarks. However, the current empirical claims are conditional on an unvalidated measurement instrument, and the manuscript contains internal inconsistencies that affect the reported results. The paper's significance is therefore not yet established; the benchmark has potential, but the evidence presented does not currently support the conclusions.

major comments (5)
  1. [§3.2, §4.1, §4.2, §6(2)] The evaluation is circular in a way that affects every result. GPT-4o generates the convergent MCQs and their correct answers (§3.2), refines them (§3.2), scores all divergent timelines on Plausibility, Novelty, Elaboration, and Actionable (§4.2), and is also one of the evaluated models (§4.1). No human validation is provided for the convergent ground truth or for the four GPT-4o-scored dimensions; the only human correlation reported (§6(3)) is for Richness, which is a deterministic graph statistic rather than a GPT-4o judgment, and that correlation is moderate (ρ≈0.56). The authors explicitly concede in §6(2) that the automated evaluation 'inherently inherits the model's own biases and reasoning artifacts' and that 'human audits remain essential for robustness,' but no such audit is reported. As a result, the model rankings, the CCS numbers, and the central asymmetry claims may reflect GPT-4o's preferences and reasoning style rather than the quality of financial reasoning. This is load-bearing for the paper's main conclusion and must be addressed, preferably with human expert validation of both the convergent ground truth and a substantial sample of divergent scores.
  2. [§3.2 and Table 4] The sample count is internally inconsistent. The text states that 607 distinct scenarios were evaluated across 14 models to create 9,380 samples, but 607 × 14 = 8,498, not 9,380. Table 4 also reports N = 9,380. If 9,380 is the correct number, then the number of scenarios should be 670, not 607; if 607 is correct, then the statistics reported with N = 9,380 are incorrect. Either way, the discrepancy must be resolved and all affected numbers recomputed.
  3. [§3.1, §3.2, and Appendix B] The claimed temporal contamination control is contradicted by examples in the paper. Section 3.1 says divergent scenarios are 'dated 1 May 2025 or after,' and §3.2 says the sources are 'dated post-training cutoff (after May 2025).' However, Appendix B includes scenarios with explicit 2024 dates, such as the '2024 Global Motor Insurance Market Report,' a scenario referencing stock prices 'as of Dec. 30, 2024,' a scenario about 'President-elect Trump' from the 2024 election period, and a Shiba Inu scenario that mentions a 2021 price. These examples are not post-May-2025, so the anti-contamination rationale fails for at least some prompts in the benchmark. The authors should either replace such scenarios, provide a precise accounting of which scenarios are actually post-May-2025, or substantially soften the contamination claims.
  4. [Table 2] The definitions of Actionable and Elaboration appear to be swapped. The row labeled 'Actionable' states that it 'Assesses the level of detail in each node... and overall tree structure,' while the row labeled 'Elaboration' states that it 'Evaluates whether the timeline yields specific investment takeaways—e.g., tickers, sectors, asset classes, or hedging triggers.' These descriptions match the opposite labels: detail and tree structure are elaboration, while investment takeaways are actionability. This inversion affects the interpretation of all results that refer to Actionable and Elaboration, including the model rankings in Table 3 and the discussion in §5.1. The table should be corrected and the surrounding text checked for consistency.
  5. [§3.2, §4.2, and Table 9] The number of refinement rounds is inconsistent. Section 3.2 says 'We performed two rounds of refinement to produce our final dataset.' Section 4.2, however, refers to 'the three convergent thinking dataset (original dataset, and refined dataset from each of the three refinement rounds).' Table 9 reports three columns: Original, Refinement 1, and Refinement 2. This should be clarified: either there are two refinement rounds (so the three columns are original, first refinement, second refinement) or the text in §4.2 is inaccurate. The distinction matters because the paper's difficulty analysis depends on the refinement trajectory.
minor comments (6)
  1. [Abstract and Table 1] The abstract refers to 'Cohere Command R+,' but the model evaluated in Table 1 is 'Cohere-command-r-08-2024' (shortened to cohere_command_r), which is not the same as 'Command R+.' The model naming should be made consistent to avoid confusion.
  2. [Table 9] The row label 'mistal_lg' is a typo and should read 'mistral_lg' to match Table 1.
  3. [Table 8 and Table 6] The distances reported in Table 8 appear to contradict Table 6. Table 8 lists 'deepseek_r1, phi4' with distance 0.302, while Table 6, Part 1, shows the phi4-row deepseek distance as 2.04. Similarly, Table 8 lists 'deepseek_r1, cohere_command_a' with distance 0.230, but Table 6's split presentation does not show cross-block distances, making the value unverifiable. The distance tables should be recomputed and presented in a consistent single matrix or at least cross-checked.
  4. [§3.2 and Appendix C] The word 'adverserial' is misspelled 'adverserial' in several places, including the list of adversarial pipelines and the Appendix C prompt; it should be 'adversarial.'
  5. [§6(3)] The statement that Richness correlates with human ratings at ρ≈0.56 lacks detail: the number of human raters, the number of timelines rated, and the rating rubric are not reported. This information is needed to interpret the single human-validation point in the paper.
  6. [Figure 7 and §5.2] The 'Hard,' 'Moderate,' and 'Easy' tiers for question answerability are described qualitatively in the text but no formal boundaries are given in terms of the number of models answering correctly. The figure should either include these boundaries or the text should specify them explicitly.

Circularity Check

0 steps flagged · score 0.0 of 10

No formal circularity: the divergent and convergent measurements use GPT-4o as generator/judge and as participant, but no claim reduces by construction to its inputs; the admitted oracle limitation is a validity risk, not a circular derivation.

full rationale

The paper constructs both benchmark halves with GPT-4o (§3.2, §4.2) and includes GPT-4o in the evaluated model set (Table 1, §4.1). This creates an evaluator–participant overlap. However, a circularity finding requires a specific reduction (an equation that equals its input, or a fitted parameter relabeled as a prediction), and none is present. The divergent scores are outputs of a prompted judge applied to independently generated timelines, not algebraic consequences of the scoring rubric; the rankings (e.g., DeepSeek-R1 above GPT-4o on Novelty) are contingent empirical judgments. The convergent CCS is a matching rate against a GPT-4o-generated answer key, but other models outperform GPT-4o on that key, so the result is not forced by construction. The external anchors—post-May-2025 news sources, the textbook-factor list, and the graph-based Richness computation—provide independent content. The paper itself flags the key limitation in §6(2): 'such automated evaluation inherently inherits the model's own biases and reasoning artifacts. Human audits remain essential for robustness.' That concession is a validity/correctness caveat, not evidence that any prediction is identical to its input. No load-bearing self-citation chain or smuggled ansatz appears. Score 0 reflects the absence of formal circularity; the oracle problem should be addressed with human validation, but it is outside the circularity definition used here.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claims rely on GPT-4o as a valid generator and judge, on the claim that scenarios are post-training-cutoff, and on hand-chosen metrics. The paper does not provide independent benchmarks, human validation, or a released dataset.

free parameters (3)
  • Richness normalization thresholds (branching factor cap 3, max path depth cap 10, mean path depth cap 7, breadth log…
    Hand-chosen in Section 4.2 and used to define the Richness score; changing them changes model rankings on that axis.
  • Equal weights in Richness composite (0.25 each) = 0.25 per component
    The four graph components are weighted equally with no empirical justification; this is a free choice affecting the final metric.
  • Adversarial Self-Play plausibility tolerance = 0.05
    Hand-chosen in Section 3.2 for keeping distractors within 0.05 plausibility of the correct timeline; affects difficulty of convergent questions.
assumptions (3)
  • domain assumption GPT-4o is a valid judge of Plausibility, Novelty, Elaboration and Actionable for financial scenario timelines.
    Stated in Section 4.2; no human validation is provided for these metrics, and the model being judged includes GPT-4o itself.
  • domain assumption The selected scenarios are dated after May 2025 and are therefore outside the pretraining data of the evaluated models.
    Claimed in Section 3.1 and Section 6, but contradicted by Appendix B, which contains 2024 events; if false, contamination mitigation fails.
  • domain assumption The correct answers in the convergent MCQs are objectively correct by factor alignment, temporal coherence, and logical entailment.
    Labels are generated by GPT-4o in Section 3.2 without human adjudication; correctness is the generator's own judgment.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Reasoning Beyond the Obvious: Evaluating Divergent and Convergent Thinking in LLMs for Financial Scenarios." pith.science (2026). https://pith.science/paper/CGON4SIZ

@misc{pith2026250718368,
  author       = {Pith},
  title        = {Pith review of: Reasoning Beyond the Obvious: Evaluating Divergent and Convergent Thinking in LLMs for Financial Scenarios},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CGON4SIZ}},
  note         = {Machine review of arXiv:2507.18368}
}
read the original abstract

Most reasoning benchmarks for LLMs emphasize factual accuracy or step-by-step logic. In finance, however, professionals must not only converge on optimal decisions but also generate creative, plausible futures under uncertainty. We introduce ConDiFi, a benchmark that jointly evaluates divergent and convergent thinking in LLMs for financial tasks. ConDiFi features 607 macro-financial prompts for divergent reasoning and 990 multi-hop adversarial MCQs for convergent reasoning. Using this benchmark, we evaluated 14 leading models and uncovered striking differences. Despite high fluency, GPT-4o underperforms on Novelty and Actionability. In contrast, models like DeepSeek-R1 and Cohere Command R+ rank among the top for generating actionable, insights suitable for investment decisions. ConDiFi provides a new perspective to assess reasoning capabilities essential to safe and strategic deployment of LLMs in finance.

Figures

Figures reproduced from arXiv: 2507.18368 by the authors.

Figure 1
Figure 1. Pipeline for Dataset Generation for Divergent [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Pipeline for Dataset Generation for Convergent [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Average Scores Per Model (Grouped by Model). Black [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Scores Distribution of Divergent Thinking Dimen [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Frobenius Norm of Inter-Model Correlations [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: PCA of Model Correlation Structures mistral_sm, o1 and llama3_8b scoring in the middle, and cohere_command_a scoring at the bottom as an extreme outlier. The other models, including deepseek_r1 have very similar scores. Our best guess is that PC2 represents some kind o…
Figure 7
Figure 7. Figure 7: Breakdown of Question Answerability by the dif [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

28 extracted references · 13 canonical work pages

  1. [1]

    Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al

    Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. Beyond the Imitation Game: Quantifying and extrap- olating the capabilities of language models . Transactions on Machine Learning Research, 2023. https://arxiv.org/abs/2206.04615

    arXiv 2023

  2. [2]

    Measuring Massive Multitask Language Understand- ing

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring Massive Multitask Language Understand- ing. In Proceedings of the International Conference on Learning Representations (ICLR), 2021. https://arxiv.org/abs/2009.03300

    arXiv 2021

  3. [3]

    Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2018. https: //arxiv.org/abs/1803.05457

    arXiv 2018

  4. [4]

    Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2381–2391, Brussels, Belgium, 2018. Association for Computational Linguistics. https://aclanthology...

    work page 2018

  5. [5]

    Olson, Yoshua Bengio, and Karim Jerbi

    Antoine Bellemare-Pepin, François Lespinasse, Philipp Thölke, Yann Harel, Kory Mathewson, Jay A. Olson, Yoshua Bengio, and Karim Jerbi. Divergent Creativity in Humans and Large Language Models. arXiv preprint arXiv:2405.13012, 2024

    arXiv 2024

  6. [6]

    TheoremQA: A Theorem-driven Question Answering Dataset

    Wenhu Chen, Ming Yin, Max Ku, Pan Lu, Yixin Wan, Xueguang Ma, Jianyu Xu, Xinyi Wang, and Tony Xia. TheoremQA: A Theorem-driven Question Answering Dataset. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7889–7901. Association for Computational Linguistics, Singapore, 2023

    work page 2023

  7. [7]

    MastermindEval: A Simple But Scalable Reasoning Benchmark

    Jonas Golde, Patrick Haller, Fabio Barth, and Alan Akbik. MASTERMINDEVAL: A Simple but Scalable Reasoning Benchmark. arXiv preprint arXiv:2503.05891,

    work page Pith review arXiv

  8. [8]

    Con- structing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps

    Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Con- structing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps. In Proceedings of the 28th International Conference on Computational Lin- guistics (COLING), pages 6609–6625. International Committee on Computational Linguistics, Barcelona, Spain (Online), 2020

    work page 2020

Show all 28 references
  1. [9]

    Human Creativity in the Age of LLMs: Randomized Experiments on Divergent and Convergent Thinking

    Harsh Kumar, Jonathan Vincentius, Ewan Jordan, and Ashton Anderson. Human Creativity in the Age of LLMs: Randomized Experiments on Divergent and Convergent Thinking. In Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI ’25) . ACM, New York, NY, USA, ...

    2025

  2. [10]

    Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, An- drew M. Dai, Jakob Uszkoreit, Quoc Le, and Sl...

    2019

  3. [11]

    Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate

    Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Shuming Shi, and Zhaopeng Tu. Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate. arXiv preprint arXiv:2305.19118, 2024

    2024 arXiv

  4. [12]

    Benchmarking Language Model Creativity: A Case Study on Code Generation

    Yining Lu, Dixuan Wang, Tianjian Li, Dongwei Jiang, and Daniel Khashabi. Benchmarking Language Model Creativity: A Case Study on Code Generation. arXiv preprint arXiv:2407.09007, 2024

    2024 arXiv

  5. [13]

    GSM-Symbolic: Understanding the Limitations of Math- ematical Reasoning in Large Language Models

    Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar. GSM-Symbolic: Understanding the Limitations of Math- ematical Reasoning in Large Language Models. arXiv preprint arXiv:2410.05229, 2024

    arXiv

  6. [14]

    MS MARCO: A Human Generated MAchine Reading COmprehension Dataset

    Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. MS MARCO: A Human Generated MAchine Reading COmprehension Dataset. arXiv preprint arXiv:1611.09268, 2016

    2016 arXiv

  7. [15]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A Graduate- Level Google-Proof Q&A Benchmark. arXiv preprint arXiv:2311.12022, 2023

    2023 arXiv

  8. [16]

    Nay, Kshitij Gupta, and Aran Komatsuzaki

    Tomohiro Sawada, Daniel Paleka, Alexander Havrilla, Pranav Tadepalli, Paula Vidas, Alexander Kranias, John J. Nay, Kshitij Gupta, and Aran Komatsuzaki. ARB: Advanced Reasoning Benchmark for Large Language Models.arXiv preprint arXiv:2307.13692, 2023

    2023 arXiv

  9. [17]

    Siegel, Michael Tang, Ruoxi Sun, Jinsung Yoon, Sercan Ö

    Hongjin Su, Howard Yen, Mengzhou Xia, Weijia Shi, Niklas Muennighoff, Han- yu Wang, Haisu Liu, Quan Shi, Zachary S. Siegel, Michael Tang, Ruoxi Sun, Jinsung Yoon, Sercan Ö. Arık, Danqi Chen, and Tao Yu. BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retr...

    2025

  10. [18]

    BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models

    Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks , 2021

    2021

  11. [20]

    COFCA: A Step-wise Counterfactual Multi-hop QA Benchmark

    Jian Wu, Linyi Yang, Zhen Wang, Manabu Okumura, and Yue Zhang. COFCA: A Step-wise Counterfactual Multi-hop QA Benchmark. arXiv preprint arXiv:2402.11924, 2024

    2024 arXiv

  12. [21]

    Cohen, Ruslan Salakhutdinov, and Christopher D

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Pr...

    2018

  13. [22]

    NAT- URALREASONING: Reasoning in the Wild with 2.8M Challenging Questions

    Weizhe Yuan, Jane Yu, Song Jiang, Karthik Padthe, Yang Li, Ilia Kulikov, Kyunghyun Cho, Dong Wang, Yuandong Tian, Jason Weston, and Xian Li. NAT- URALREASONING: Reasoning in the Wild with 2.8M Challenging Questions. arXiv preprint arXiv:2502.13124, 2025

    2025

  14. [23]

    Renze and E

    M. Renze and E. Guven. Self-reflection in LLM Agents: Effects on Problem-solving Performance. arXiv preprint arXiv:2405.06682, 2024

    2024 arXiv

  15. [24]

    monetary tightening leads to recession, triggering political backlash

    Andrew K. Lampinen, Ishita Dasgupta, Stephanie C. Y. Chan, Kory Matthewson, Michael Henry Tessler, Antonia Creswell, James L. McClelland, Jane X. Wang, and Felix Hill. Can Language Models Learn from Explanations in Context? arXiv preprint arXiv:2204.02329, 2022. Reasoning Beyo...

    2022 arXiv

  16. [26]

    Scoring Framework • 5 ± 0.5 = average sell-side note • Reserve 9–10 for the top 5% of timelines you have ever seen • Reserve 1–2 for timelines that would mislead an investor Rate strictly and apply the penalties below even if only one branch is affected

  17. [27]

    market reacts

    Penalty Catalogue • Generic filler node (e.g., "market reacts", "tensions rise", "empty buzzwords"):-1 per 3 occurrences • Unsupported or impossible macro figure/date : -2 each • Repeating more than 30% of baseline timeline ideas : -2 total

  18. [28]

    Dimension Rubric (1–10) • Plausibility — 10 = macro-logical and historically grounded; 5 = plausible but thin; <3 = breaks cause-effect or factual reality • Novelty — Average of idea originality and interaction originality; 10 = reveals counterintuitive second-/third-order eff...

  19. [29]

    id": "T0

    Fatal-flaw CAP (before scoring) Set every dimension to 3 or below if any branch contains: • Impossible outcome (e.g., Fed adopts gold standard in 2025) • Date reversal (effect precedes cause) • Verbatim repetition of >30% of scenario paths Example Scenario: German auto giant V...

    2024

  20. [2025]

    (To be published at ICLR 2025 Workshop on Reasoning and Planning for LLMs)

    2025

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.