Pith. sign in

REVIEW 5 major objections 6 minor 1 cited by

On Path to Multimodal Historical Reasoning: HistBench and HistAgent

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper introduces HistBench, a 414-question expert-written benchmark for multimodal historical reasoning, and HistAgent, a history-specialized agent that outperforms general LLMs and agents on it.

desk verdict HistBench is a genuine new benchmark and HistAgent is a reasonable system, but the headline scores rest on an incomplete evaluation protocol and need correction before they can be trusted. read the letter →

arxiv 2505.20246 v3 pith:L3JRWGM4 submitted 2025-05-26 cs.AI cs.CL

Jiahao Qiu , Fulian Xiao , Yimin Wang , Yuchen Mao , Yijia Chen , Xinzhe Juan , Shu Zhang , Siran Wang
show 91 more authors
Xuan Qi Tongcheng Zhang Zixin Yao Jiacheng Guo Yifu Lu Charles Argon Jundi Cui Daixin Chen Junran Zhou Shuyao Zhou Zhanpeng Zhou Ling Yang Shilong Liu Hongru Wang Kaixuan Huang Xun Jiang Yuming Cao Yue Chen Yunfei Chen Zhengyi Chen Ruowei Dai Mengqiu Deng Jiye Fu Yunting Gu Zijie Guan Zirui Huang Xiaoyan Ji Yumeng Jiang Delong Kong Haolong Li Jiaqi Li Ruipeng Li Tianze Li Zhuoran Li Haixia Lian Mengyue Lin Xudong Liu Jiayi Lu Jinghan Lu Wanyu Luo Ziyue Luo Zihao Pu Zhi Qiao Ruihuan Ren Liang Wan Ruixiang Wang Tianhui Wang Yang Wang Zeyu Wang Zihua Wang Yujia Wu Zhaoyi Wu Hao Xin Weiao Xing Ruojun Xiong Weijie Xu Yao Shu Yao Xiao Xiaorui Yang Yuchen Yang Nan Yi Jiadong Yu Yangyuxuan Yu Huiting Zeng Danni Zhang Yunjie Zhang Zhaoyu Zhang Zhiheng Zhang Xiaofeng Zheng Peirong Zhou Linyan Zhong Xiaoyin Zong Ying Zhao Zhenxin Chen Lin Ding Xiaoyu Gao Bingbing Gong Yichao Li Yang Liao Guang Ma Tianyuan Ma Xinrui Sun Tianyi Wang Han Xia Ruobing Xian Gen Ye Tengfei Yu Wentao Zhang Yuxi Wang Xi Gao Mengdi Wang
This is my paper
classification cs.AIcs.CL
keywords historicalreasoningmultimodalbenchmarkLLMagentsdomain-specificOCRformanuscriptsmultilingualsourcesexpert-authoredHistBench
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

HistBench is a new benchmark of 414 expert-authored history questions, spanning 29 languages, that are designed to resist simple web retrieval and require interpreting primary sources such as manuscripts, inscriptions, maps, audio, and video. On it, the paper reports that frontier LLMs with web search answer only 14.5–18.6% correctly on the first attempt. The paper then introduces HistAgent, a GPT-4o-based agent whose specialized OCR, translation, image-provenance, and scholarly-literature tools raise first-attempt accuracy to 27.54% and two-attempt accuracy to 36.47%, while staying competitive at 60.00% on the general GAIA benchmark. A sympathetic reader would take the central claim to be that historical reasoning is a distinct, currently unmet AI capability, and that domain-specific tool orchestration, not a stronger base model, is the route to improving it.

What carries the argument

The load-bearing machinery is the pairing of HistBench's review pipeline with HistAgent's Manager Agent loop. HistBench filters questions through three stages—format screening, LLM solvability testing in which questions that two strong LLMs can solve without source material are removed, and expert historical review—so the remaining 414 questions require evidence beyond memorized knowledge. HistAgent is orchestrated by a Manager Agent that runs a CodeAct-style loop: it emits Python code calling specialist sub-agents for web browsing, image provenance, literature search, OCR, speech transcription, translation, file processing, and video frame extraction, validates each result against sources, and assembles a cited final answer. The same base model, GPT-4o, powers both HistAgent and the main baseline, so the reported gains are attributed to the tool set and orchestration rather than to model scale.

What would settle it

Have historians independently score all 414 HistBench responses, including the 314 not currently human-validated, and compare their labels to the LLM judge; if the human accuracy on those 314 differs substantially from the LLM judge's, the reported 27.54% and 36.47% figures, and the comparisons built on them, would change.

Watch

Extended reading notes

Core claim

The paper's central claim is that existing LLMs and generalist agents cannot do historical reasoning, and that the gap is addressable by a domain-specific agent. The evidence is a benchmark, HistBench: 414 questions written by more than 40 historians, split into exact-match and multiple-choice formats, with six evaluation dimensions and three difficulty levels, spanning 29 languages and all major world regions. On this benchmark, GPT-4o with online search scores 18.60% pass@1, DeepSeek-R1 scores 14.49%, Grok 3 scores 17.63%, and an open-source generalist Deep Research agent scores 20.29% pass@1 and 25.12% pass@2; HistAgent, using the same GPT-4o base model, scores 27.54% pass@1 and 36.47% pass@2. On the 56-question history subset of Humanity's Last Exam, HistAgent reaches 28.57% pass@1 versus 17.86% for the generalist agent and 8.93% for GPT-4o with web search, and on GAIA it reaches 60.00% pass@1. The intended conclusion is that specialized tools—handwriting OCR, translation with provenance, reverse image search, and scholarly retrieval—change agent performance more than the base model does.

Load-bearing premise

The reported numbers rest on the assumption that an LLM judge's Correct/Incorrect labels on all 414 questions are accurate enough, since only 100 of those labels were checked by human experts.

Editorial extensions

If this is right

  • If HistAgent's results hold, history becomes a testbed where agent architecture matters more than base-model choice, since the same GPT-4o improves by 11.35 points over a generalist agent built on the same model.
  • A benchmark that removes questions solvable by two strong LLMs sets a high difficulty floor, so future models will need source-grounded reasoning rather than retrieval to score well.
  • The 60.00% score on GAIA implies domain-specific customization does not sacrifice general task competence, at least at this level of integration.
  • The level-wise breakdown offers a diagnostic for where agents fail, such as HistAgent scoring higher on Level 3 (32.89% pass@1) than on Level 2 (23.84% pass@1).

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the LLM judge's error rate is not uniform across the 314 unvalidated questions, the reported absolute accuracies and the gaps over baselines could shift; a full human rescore of HistBench is the direct way to test this.
  • The HistAgent design—specialized perception tools plus literature-first retrieval—is a template for other source-heavy humanities fields such as art history, legal history, and epigraphy, where the same OCR, translation, and provenance bottlenecks recur.
  • Because HistBench was filtered to remove questions solvable by current strong LLMs, its scores should be read as a capability floor for source-based reasoning, not as an estimate of everyday historical question-answering performance.
  • As stronger multimodal models appear, re-running HistAgent on them could show whether the architecture's gains compound with base-model improvements, a question the paper leaves open.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper introduces HistBench, a benchmark of 414 expert-authored multimodal history questions spanning 29 languages and multiple periods/regions, and HistAgent, a GPT-4o-based agent with OCR, translation, reverse-image-search, and scholarly literature-search tools. The authors report HistAgent pass@1/pass@2 of 27.54%/36.47% on HistBench, claim significant gains over online LLMs (GPT-4o 18.60%, DeepSeek-R1 14.49%, Grok 3 17.63%) and over ODR-smolagents (20.29%/25.12%), and report 28.57% on the HLE history subset and 60.00% on the GAIA validation set. The main evidence is Table 5, with additional cross-benchmark results in Tables 6 and 7.

Significance. If the results are reproducible, HistBench is a useful resource for evaluating AI in the humanities: the dataset is comparatively large, multilingual, source-grounded, and includes structured metadata, while HistAgent offers a plausible architecture for domain-specific agents. Strengths include public release of code and data, a same-base-model comparison against ODR-smolagents on HistBench, evaluation on the HLE history subset and GAIA, and detailed descriptions of the agent tools and workflows. However, the headline metric is not fully specified, several claims are stronger than Table 5 supports, and the GAIA table appears to use a different base model from the main experiments. The overall contribution is therefore conditional on fixing the evaluation protocol and aligning the claims with the evidence.

major comments (5)
  1. [§5.2, Metric for HistBench] The definition of HistBench accuracy is mathematically incomplete. The text says accuracy is the percentage of the 414 questions for which the response is judged correct by both the LLM judge and professional validation, but only 100 randomly sampled questions receive human expert validation. No procedure is given for assigning the human-validation component to the remaining 314 questions. As written, the 27.54% and 36.47% figures are not reproducible, because the label source for 314 of the 414 questions is unspecified. Please either report accuracy as LLM-judge-only accuracy on all 414 questions with the human agreement rate on the 100-sample subset, or provide a clear statistical procedure for extrapolating the human validation to the full set.
  2. [Table 5 vs Abstract/§5.3.1] The abstract and §5.3.1 claim that HistAgent 'significantly outperform[s] LLMs with online search and generalist agents' and 'outperforms all baselines,' but Table 5 shows o3:online achieving pass@1 of 32.37% versus HistAgent's 27.54%, so HistAgent does not outperform all baselines at pass@1. The claim is supportable only at pass@2 or for the same-base-model comparison against ODR-smolagents. Please rephrase the headline claim to specify pass@2, or restrict the 'outperforms all baselines' statement to the pass@2 metric, and explain why pass@1 is not the primary comparison.
  3. [Table 7 / §5.3.3 / Abstract] There is a direct inconsistency in the GAIA evaluation. The abstract and §5.3.3 state that HistAgent based on GPT-4o achieves 60.00% on GAIA, but Table 7 lists the HistAgent row as using Claude 3.7 Sonnet and the ODR-smolagents row as using o1, not GPT-4o. If Table 7 is correct, the claim that HistAgent 'based on GPT-4o' reaches 60.00% is unsupported, and the same-base-model fairness guarantee stated in §5.1 is violated for the GAIA comparison. Please clarify which base model produced the reported GAIA numbers and rerun the comparison with matched base models, or explicitly label Table 7 as a separate configuration.
  4. [Appendix E.2, Judge Prompt] The LLM judge prompt contains contradictory instructions. It says to 'Treat missing key points or any substantive error or omission as incorrect' but later says to 'Tolerate substantive deviations from the correct answer.' These instructions cannot both be applied consistently, and the leniency clause could systematically inflate correctness labels. Because all unvalidated HistBench scores and the HLE history-subset scores rely on this judge, the contradiction is load-bearing. Please remove the contradictory clause and report the agreement rate between the LLM judge and human validation on the 100-question sample.
  5. [§3.3.3, Stage 2 and §4] The benchmark construction and agent design are not fully separated. HistBench questions are filtered by whether 'more than two existing models' solve them without supporting materials, but the exact models, prompting protocol, and number of excluded questions are not specified. At the same time, HistAgent's tool suite (OCR, reverse image search, scholarly search) is closely matched to HistBench's question dimensions and modalities, which creates a home-field advantage. The paper should provide a concrete external or held-out check of the domain-specific claim—for example, human-validated results on the HLE history subset, or an ablation that disables HistAgent's domain tools to quantify how much of the 11.35-point gain over ODR-smolagents comes from architecture rather than from tool access tailored to HistBench.
minor comments (6)
  1. [Appendix A.3, Table 9] The caption says the dataset covers 29 languages, but the table lists 26 entries, and the counts sum to 415 rather than 414. Please reconcile the language count and the total number of questions.
  2. [§5.2] There is a typo: 'professtional validation' should be 'professional validation.'
  3. [§2.4] There is a typo: 'Gloval History Databank' should be 'Global History Databank.'
  4. [§3.2.1 vs §3.4.1] Section 3.2.1 says multiple-choice questions can have 'some of the options' designated as correct, while Section 3.4.1 says each multiple-choice question has 'one correct answer and three plausible distractors.' Please align these descriptions.
  5. [Appendix E.1] The compute resources section states 'M1 chips, 512 MB, 20 minutes for 1 question.' This is unclear: it does not specify whether 512 MB is memory or disk, whether this is per question or per run, and how many chips were used. Please expand the description.
  6. [§5.3.3, Table 7] The prose says 'Our HistAgent, based on the GPT-4o, answers 99 questions correctly,' but Table 7 labels the HistAgent row as Claude 3.7 Sonnet. Please ensure the text and table describe the same configuration.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: HistAgent's outperformance is supported by external benchmarks (HLE history subset, GAIA) and by expert-authored HistBench ground truth; the main weaknesses are methodological, not circular.

full rationale

HistAgent's central outperformance claim does not reduce to its inputs. HistBench is constructed by the same team and HistAgent's toolset (OCR, translation, reverse image search, scholarly search) is aligned with the benchmark's modalities, but that is home-field advantage, not a definitional identity: the benchmark's ground-truth answers are authored by more than 40 expert contributors, and HistAgent's advantage is also measured on two external benchmarks not built by the authors—the HLE history subset (HistAgent 28.6% vs ODR-smolagents 17.9% vs GPT-4o+web 8.9%) and GAIA validation (60.0% vs 55.15%). The self-citations in Related Work (CRISPR-GPT [2], Long-Term Memory [7]) are illustrative examples and are not load-bearing for any result. The main weakness is methodological rather than circular: Section 5.2 defines HistBench accuracy as 'Correct by both the LLM judge and the author (100 samples selected)' for all 414 questions, yet only 100 questions receive human validation; the label source for the other 314 is unspecified, and the judge prompt's instruction to 'Tolerate substantive deviations' plus 'Treat missing key points or any substantive error or omission as incorrect' is internally contradictory. These issues affect absolute-score interpretation and reproducibility, but they apply symmetrically to all compared systems, so the relative outperformance claim does not reduce by construction. Appendix B's acknowledged limitations (closed questions, contributor cognition) further indicate the benchmark is not presented as a formal derivation. I therefore find no circular step; score 2 reflects minor non-load-bearing self-citations.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

No mathematical derivations or physical constants appear. The central claims rest on benchmark design choices and evaluation-protocol assumptions. I list the hand-set thresholds and implicit evaluation assumptions that most affect the reported numbers.

free parameters (3)
  • benchmark solvability exclusion threshold = two of three LLMs
    Questions solved by more than two strong LLMs during Stage 2 are removed; this hand-set cutoff determines the benchmark's difficulty and therefore the baselines.
  • per-question tool invocation budget = unreported
    The paper says a fixed budget is imposed for fair comparison but never gives the number; pass@1/pass@2 depend on this unstated limit.
  • human validation sample size = 100 of 414
    Author validation is performed on 100 random samples, yet accuracy is reported on all 414; the representativeness of 100 is an unstated design choice.
assumptions (4)
  • domain assumption Contributor-authored answers represent correct historical ground truth.
    The benchmark's review process checks rigor but does not provide an external adjudication standard for disputed historical questions.
  • domain assumption An LLM judge can reliably detect semantic equivalence between agent outputs and ground-truth answers across languages and formats.
    LLM judging is applied to all 414 questions; agreement with human labels on 100 samples is mentioned but no kappa or error rate is reported.
  • ad hoc to paper The two-model solvability filter removes questions that are solvable by retrieval or shallow prompting.
    This criterion is specific to HistBench and is not independently validated for the final 414 questions.
  • ad hoc to paper The 100-question human validation sample is representative of the full benchmark.
    No stratification details are provided, and the combination rule for human and LLM labels over 414 questions is unspecified.

how reviews work

0 comments
Cite this review

Pith. "Pith review of On Path to Multimodal Historical Reasoning: HistBench and HistAgent." pith.science (2026). https://pith.science/paper/L3JRWGM4

@misc{pith2026250520246,
  author       = {Pith},
  title        = {Pith review of: On Path to Multimodal Historical Reasoning: HistBench and HistAgent},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/L3JRWGM4}},
  note         = {Machine review of arXiv:2505.20246}
}
read the original abstract

Recent advances in large language models (LLMs) have led to remarkable progress across domains, yet their capabilities in the humanities, particularly history, remain underexplored. Historical reasoning poses unique challenges for AI, involving multimodal source interpretation, temporal inference, and cross-linguistic analysis. While general-purpose agents perform well on many existing benchmarks, they lack the domain-specific expertise required to engage with historical materials and questions. To address this gap, we introduce HistBench, a new benchmark of 414 high-quality questions designed to evaluate AI's capacity for historical reasoning and authored by more than 40 expert contributors. The tasks span a wide range of historical problems-from factual retrieval based on primary sources to interpretive analysis of manuscripts and images, to interdisciplinary challenges involving archaeology, linguistics, or cultural history. Furthermore, the benchmark dataset spans 29 ancient and modern languages and covers a wide range of historical periods and world regions. Finding the poor performance of LLMs and other agents on HistBench, we further present HistAgent, a history-specific agent equipped with carefully designed tools for OCR, translation, archival search, and image understanding in History. On HistBench, HistAgent based on GPT-4o achieves an accuracy of 27.54% pass@1 and 36.47% pass@2, significantly outperforming LLMs with online search and generalist agents, including GPT-4o (18.60%), DeepSeek-R1(14.49%) and Open Deep Research-smolagents(20.29% pass@1 and 25.12% pass@2). These results highlight the limitations of existing LLMs and generalist agents and demonstrate the advantages of HistAgent for historical reasoning.

Figures

Figures reproduced from arXiv: 2505.20246 by the authors.

Figure 1
Figure 1. Performance of LLMs and Agents on HistBench. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Difficulty level definitions across six structured evaluation dimensions. [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Multi-Stage Question Review Pipeline for HistBench [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: Geographic coverage of HistBench. Blue re [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: The architecture of HistAgent, an agent for historical reasoning. The system takes multimodal user inputs [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: Overview of Literature Search Agent. It is specifically designed for academic searching, including a priority [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. AgentDistill: Training-Free Agent Distillation with Generalizable MCP Boxes

    cs.AI 2025-06 reject novelty 5.0 of 10

    AgentDistill distills agent capabilities without any training by having a teacher generate reusable MCP tool boxes that small-model students invoke at inference time.

Reference graph

Works this paper leans on

35 extracted references · 8 canonical work pages · cited by 1 Pith paper

  1. [1]

    Introducing deep research

    OpenAI. Introducing deep research

  2. [2]

    Crispr-gpt: An llm agent for automated design of gene-editing experiments.arXiv preprint arXiv:2404.18021, 2024

    Kaixuan Huang, Yuanhao Qu, Henry Cousins, William A Johnson, Di Yin, Mihir Shah, Denny Zhou, Russ Altman, Mengdi Wang, and Le Cong. Crispr-gpt: An llm agent for automated design of gene-editing experiments.arXiv preprint arXiv:2404.18021, 2024

    arXiv 2024

  3. [3]

    Chemcrow: Augmenting large-language models with chemistry tools.arXiv preprint arXiv:2304.05376, 2023

    Andres M Bran, Sam Cox, Oliver Schilter, Carlo Baldassari, Andrew D White, and Philippe Schwaller. Chemcrow: Augmenting large-language models with chemistry tools.arXiv preprint arXiv:2304.05376, 2023

    arXiv 2023

  4. [4]

    Gaia: a benchmark for general ai assistants

    Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: a benchmark for general ai assistants. InThe Twelfth International Conference on Learning Representations, 2023

    2023

  5. [5]

    Humanity’s last exam.arXiv preprint arXiv:2501.14249, 2025

    Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, et al. Humanity’s last exam.arXiv preprint arXiv:2501.14249, 2025

    arXiv 2025

  6. [6]

    Phybench: Holistic evaluation of physical perception and reasoning in large language models

    Shi Qiu, Shaoyang Guo, Zhuo-Yang Song, Yunbo Sun, Zeyu Cai, Jiashen Wei, Tianyu Luo, Yixuan Yin, Haoxu Zhang, Yi Hu, et al. Phybench: Holistic evaluation of physical perception and reasoning in large language models. arXiv preprint arXiv:2504.16074, 2025

    arXiv 2025

  7. [7]

    Long term memory: The foundation of ai self-evolution.arXiv preprint arXiv:2410.15665, 2024

    Xun Jiang, Feng Li, Han Zhao, Jiaying Wang, Jun Shao, Shihao Xu, Shu Zhang, Weiling Chen, Xavier Tang, Yize Chen, et al. Long term memory: The foundation of ai self-evolution.arXiv preprint arXiv:2410.15665, 2024

    arXiv 2024

  8. [8]

    Autoagent: A fully-automated and zero-code framework for llm agents

    Jiabin Tang, Tianyu Fan, and Chao Huang. Autoagent: A fully-automated and zero-code framework for llm agents. arXiv e-prints, pages arXiv–2502, 2025

    2025

Show all 35 references
  1. [9]

    Owl: Optimized workforce learning for general multi-agent assistance in real-world task automation, 2025

    Mengkang Hu, Yuhang Zhou, Wendong Fan, Yuzhou Nie, Bowei Xia, Tao Sun, Ziyu Ye, Zhaoxuan Jin, Yingru Li, Zeyu Zhang, Yifeng Wang, Qianshuo Ye, Ping Luo, and Guohao Li. Owl: Optimized workforce learning for general multi-agent assistance in real-world task automation, 2025

    2025

  2. [10]

    ‘smo- lagents‘: a smol library to build great agentic systems

    Aymeric Roucher, Albert Villanova del Moral, Thomas Wolf, Leandro von Werra, and Erik Kaunismäki. ‘smo- lagents‘: a smol library to build great agentic systems. https://github.com/huggingface/smolagents, 2025

    2025

  3. [11]

    Towards a holistic framework for multimodal llm in 3d brain ct radiology report generation.Nature Communications, 16(1):2258, 2025

    Cheng-Yi Li, Kao-Jung Chang, Cheng-Fu Yang, Hsin-Yu Wu, Wenting Chen, Hritik Bansal, Ling Chen, Yi-Ping Yang, Yu-Chun Chen, Shih-Pin Chen, et al. Towards a holistic framework for multimodal llm in 3d brain ct radiology report generation.Nature Communications, 16(1):2258, 2025

    2025

  4. [12]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y . K. Li, Y . Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024

    2024

  5. [13]

    Ds-agent: Automated data science by empowering large language models with case-based reasoning, 2024

    Siyuan Guo, Cheng Deng, Ying Wen, Hechang Chen, Yi Chang, and Jun Wang. Ds-agent: Automated data science by empowering large language models with case-based reasoning, 2024

    2024

  6. [14]

    War and peace (waragent): Large language model-based multi-agent simulation of world wars.arXiv preprint arXiv:2311.17227, 2023

    Wenyue Hua, Lizhou Fan, Lingyao Li, Kai Mei, Jianchao Ji, Yingqiang Ge, Libby Hemphill, and Yongfeng Zhang. War and peace (waragent): Large language model-based multi-agent simulation of world wars.arXiv preprint arXiv:2311.17227, 2023

    2023 arXiv

  7. [15]

    Battleagent: Multi-modal dynamic emulation on historical battles to complement historical analysis.arXiv preprint arXiv:2404.15532, 2024

    Shuhang Lin, Wenyue Hua, Lingyao Li, Che-Jui Chang, Lizhou Fan, Jianchao Ji, Hang Hua, Mingyu Jin, Jiebo Luo, and Yongfeng Zhang. Battleagent: Multi-modal dynamic emulation on historical battles to complement historical analysis.arXiv preprint arXiv:2404.15532, 2024

    2024 arXiv

  8. [16]

    Scibench: Evaluating college-level scientific problem-solving abilities of large language models.arXiv preprint arXiv:2307.10635, 2023

    Xiaoxuan Wang, Ziniu Hu, Pan Lu, Yanqiao Zhu, Jieyu Zhang, Satyen Subramaniam, Arjun R Loomba, Shichang Zhang, Yizhou Sun, and Wei Wang. Scibench: Evaluating college-level scientific problem-solving abilities of large language models.arXiv preprint arXiv:2307.10635, 2023

    2023 arXiv

  9. [17]

    Medical large language model benchmarks should prioritize construct validity.arXiv preprint arXiv:2503.10694, 2025

    Ahmed Alaa, Thomas Hartvigsen, Niloufar Golchini, Shiladitya Dutta, Frances Dean, Inioluwa Deborah Raji, and Travis Zack. Medical large language model benchmarks should prioritize construct validity.arXiv preprint arXiv:2503.10694, 2025

    2025 arXiv

  10. [18]

    Benchmarking large language models via random variables.arXiv preprint arXiv:2501.11790, 2025

    Zijin Hong, Hao Wu, Su Dong, Junnan Dong, Yilin Xiao, Yujing Zhang, Zhu Wang, Feiran Huang, Linyi Li, Hongxia Yang, et al. Benchmarking large language models via random variables.arXiv preprint arXiv:2501.11790, 2025

    2025 arXiv

  11. [19]

    Unveiling challenges for llms in enterprise data engineering.arXiv preprint arXiv:2504.10950, 2025

    Jan-Micha Bodensohn, Ulf Brackmann, Liane V ogel, Anupam Sanghi, and Carsten Binnig. Unveiling challenges for llms in enterprise data engineering.arXiv preprint arXiv:2504.10950, 2025

    2025

  12. [20]

    Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models.Advances in Neural Information Processing Systems, 36:44123–44279, 2023

    Neel Guha, Julian Nyarko, Daniel Ho, Christopher Ré, Adam Chilton, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel Rockmore, Diego Zambrano, et al. Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models.Advances in Neural ...

    2023

  13. [21]

    Legalbench-rag: A benchmark for retrieval-augmented generation in the legal domain.arXiv preprint arXiv:2408.10343, 2024

    Nicholas Pipitone and Ghita Houir Alami. Legalbench-rag: A benchmark for retrieval-augmented generation in the legal domain.arXiv preprint arXiv:2408.10343, 2024

    2024 arXiv

  14. [22]

    Swe-bench: Can language models resolve real-world github issues?arXiv preprint arXiv:2310.06770, 2023

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues?arXiv preprint arXiv:2310.06770, 2023

    2023 arXiv

  15. [23]

    Evaluating large language models trained on code

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021

    2021 arXiv

  16. [24]

    Program synthesis with large language models.arXiv preprint arXiv:2108.07732, 2021

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models.arXiv preprint arXiv:2108.07732, 2021

    2021 arXiv

  17. [25]

    Pubmedqa: A dataset for biomedical research question answering.arXiv preprint arXiv:1909.06146, 2019

    Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W Cohen, and Xinghua Lu. Pubmedqa: A dataset for biomedical research question answering.arXiv preprint arXiv:1909.06146, 2019

    1909 arXiv

  18. [26]

    Large language models encode clinical knowledge.Nature, 620(7972):172–180, 2023

    Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. Large language models encode clinical knowledge.Nature, 620(7972):172–180, 2023

    2023

  19. [27]

    Financeqa: A benchmark for evaluating financial analysis capabilities of large language models.arXiv preprint arXiv:2501.18062, 2025

    Spencer Mateega, Carlos Georgescu, and Danny Tang. Financeqa: A benchmark for evaluating financial analysis capabilities of large language models.arXiv preprint arXiv:2501.18062, 2025

    2025 arXiv

  20. [28]

    Large language models’ expert-level global history knowledge benchmark (hist-llm).Advances in Neural Information Processing Systems, 37:32336–32369, 2024

    Jakob Hauser, Daniel Kondor, Jenny Reddish, Majid Benam, Enrico Cioni, Federica Villa, James Bennett, Daniel Hoyer, Pieter Francois, Peter Turchin, et al. Large language models’ expert-level global history knowledge benchmark (hist-llm).Advances in Neural Information Processin...

    2024

  21. [29]

    Cambridge University Press, 2011

    Daniel Woolf.A Global History of History. Cambridge University Press, 2011

    2011

  22. [30]

    Arnold.History: A V ery Short Introduction

    John H. Arnold.History: A V ery Short Introduction. Oxford University Press, 2000

    2000

  23. [31]

    Browser use: Enable ai to control your browser, 2024

    Magnus Müller and Gregor Žuni ˇc. Browser use: Enable ai to control your browser, 2024

    2024

  24. [32]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  25. [33]

    Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  26. [34]

    Introducing openai o3 and o4-mini, 2025

    OpenAI. Introducing openai o3 and o4-mini, 2025. Accessed 15 May 2025

    2025

  27. [35]

    Grok 3 beta — the age of reasoning agents, 2025

    xAI. Grok 3 beta — the age of reasoning agents, 2025. Accessed 15 May 2025. 26

    2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.