Pith. sign in

REVIEW 3 major objections 6 minor 32 references

Reviewed by Pith at T0; open to challenge.

T0 means a machine referee read the full paper against a public rubric. The mark states how deep the mechanical check went, never who wrote it. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

Cheaper LLM Judges Match Frontier Models — But Bias Matters More

2026-07-10 02:41 UTC pith:O2J3UUYQ

load-bearing objection Cheaper LLM judges match frontier models on citation-quality F1, but the 'statistically indistinguishable' claim on factual support rests on CI overlap, not equivalence testing. the 3 major comments →

arxiv 2607.08700 v1 pith:O2J3UUYQ submitted 2026-07-09 cs.CL

Do You Need a Frontier Model as a Citation Verifier? Benchmarking Rubric LLMs for Deep-Research Source Attribution

Ethan Leung , Elias Lumer , Corey Feld , Austin Huber , Vamse Kumar Subbiah , Kevin Paul This is my paper
classification cs.CL
keywords LLM-as-judgecitation qualityreinforcement learning from verifiable rewardsreward model calibrationsource attributionfactual support evaluationrubric-based evaluationfalse positive rate
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper asks a practical question that sits at the intersection of two trends: reinforcement learning pipelines increasingly use LLM judges as reward models, and citation quality in deep-research systems is a prime candidate for this kind of automated scoring. The authors build an adversarial benchmark of 624 attribution-citation pairs (1,248 rubric decisions) with human-reviewed gold labels, then test 8 LLM judges from three model families on two dimensions: whether a cited source is topically relevant to a claim, and whether it factually supports that claim. Two findings carry the argument. First, cheaper judges are competitive with frontier models on standard accuracy metrics — GPT-5-mini leads on source relevance (F1 = 0.908), Claude Opus 4.6 leads on factual support (F1 = 0.750), and on factual support no model is statistically distinguishable from any other. Second, and more consequentially, judges with similar accuracy scores differ substantially in how they err: some over-reject good citations (high false negative rate), some over-accept bad ones (high false positive rate), and most are systematically stricter than the gold labels. A single F1 number hides this directional bias, but when the judge serves as the reward signal in a training loop, that bias becomes the thing the model is optimized for. A strict judge trains the model to under-cite; a lenient one trains it to fabricate. The practical conclusion is that calibrating the judge — understanding its bias profile — is a prerequisite for using citation rubrics as reward signals, and this calibration does not require the most expensive model available.

Core claim

The central discovery is that scalar accuracy metrics like F1 are necessary but insufficient for selecting an LLM judge to serve as a reward model. Two judges can post nearly identical F1 scores while having opposite bias profiles — one rejecting too many good citations, the other accepting too many bad ones — and these biases, invisible to F1, are exactly what a downstream reinforcement learning loop would amplify. The paper demonstrates this concretely on a citation-quality benchmark where most judges are systematically stricter than gold labels on source relevance, and false negative rates on factual support range from 0.18 to 0.47 across models at similar accuracy levels.

What carries the argument

The benchmark decomposes citation quality into two LLM-judged dimensions — source relevance (is the cited source topically on-point?) and factual support (does the source actually back the claim?) — plus a deterministic link-accessibility check. Each attribution-citation pair is scored independently on each dimension by a single LLM call producing a binary pass/fail with rationale. Gold labels come from a six-judge council with human adjudication of the 378 non-unanimous decisions. The bias profile of each evaluated judge is then characterized by three metrics beyond F1: pass-rate drift (how much the judge's overall pass rate deviates from gold), false positive rate (bad citations accepted),

Load-bearing premise

The gold labels against which all judges are measured were produced by a council of six LLM judges, with human review only of the cases where the council disagreed. Two of the eight evaluated judges also served on this council. If the council's own biases shaped which cases were flagged for human adjudication, the gold labels may share biases that the evaluated judges also share, making all judges look more accurate than they truly are.

What would settle it

If a fully independent gold-label set — produced by trained human annotators with no LLM council involvement — yielded different disagreement patterns or pass rates, the bias profiles reported here could shift, potentially changing which judges appear calibrated and which appear biased.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • RLVR pipelines that use LLM judges as reward models should report directional bias metrics (FPR, FNR, pass-rate drift) alongside accuracy, not just F1 or agreement scores
  • Judge selection for reward modeling should be per-criterion rather than per-model: the best judge for source relevance may not be the best for factual support
  • The dominant failure mode on factual support is over-rejection of good citations, not acceptance of fabricated ones — calibration effort should target judge strictness rather than fabrication detection
  • Ensembling across disagreeing judges, or selecting judges with high pairwise agreement, may reduce reward noise without requiring a stronger or more expensive model

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the gold-label council's systematic biases shaped which cases were flagged as disagreements (and thus received human scrutiny), the gold labels may inherit council biases shared by the evaluated judges, inflating apparent agreement — particularly for the two judges that also served on the council
  • The finding that directional bias matters more than scalar accuracy for reward signal quality suggests a broader principle: any automated evaluator used as a training target should be profiled for bias direction before deployment, not just agreement rate
  • The per-criterion judge selection finding implies that multi-dimensional rubrics may benefit from heterogeneous judge ensembles — different models for different criteria — rather than a single judge applied uniformly across all dimensions

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This paper studies whether cheaper, off-the-shelf LLMs can serve as reliable rubric-based judges for citation quality in deep-research systems, a setting where the judge effectively acts as a reward model in RLVR training. The authors construct an adversarial benchmark of 624 attribution-citation pairs across 25 topic domains, produce gold labels via a 6-LLM council with human adjudication of disagreements, and evaluate 8 judges from 3 model families on source relevance and factual support. The central findings are: (1) cheaper judges are competitive on F1, with GPT-5-mini leading on source relevance and no model statistically distinguishable on factual support; and (2) judges differ substantially in pass-rate drift, FPR, and FNR at similar F1 levels, meaning scalar F1 obscures directional biases that a downstream RL loop would reinforce. The experimental design is reasonable and the directional-bias analysis is the most valuable contribution. However, the claim of statistical indistinguishability on factual support rests on overlapping confidence intervals rather than proper equivalence testing, and the gold-label construction involves a partial overlap between the labeling council and the evaluated judges. These issues are addressable but load-bearing for the central claims.

Significance. The paper addresses a practically important question: whether expensive frontier models are necessary as reward signals for citation-quality RLVR, or whether cheaper judges suffice. The directional-bias analysis (pass-rate drift, FPR, FNR) is a genuine contribution — it demonstrates that scalar F1 masks reward-relevant asymmetries, and the mapping of FPR/FNR to specific reward-hacking mechanisms (over-rewarding bad citations vs. signal sparsity from under-rewarding good ones) is well-motivated. The benchmark construction with adversarial edits across 19 strategies and human adjudication of all 378 disagreement cases is thorough. The finding that the dominant factual-support failure mode is over-rejection of correct citations rather than acceptance of fabricated ones is actionable for reward design. The work would benefit from machine-checked reproducibility (code/data release), which is not mentioned.

major comments (3)
  1. [§4.2 and Abstract] The claim that judges are 'statistically indistinguishable' on factual support rests entirely on the observation that 'every 95% confidence interval overlaps' (Table 2). CI overlap is not a valid test of equivalence. The F1 range spans 0.649–0.750 (a 0.101 gap), and with only ~115 positive cases (18.4% of 624), the bootstrap CIs are wide (e.g., GPT-OSS-120B: [.56,.72], Claude Opus 4.6: [.68,.82]). Overlapping CIs in this regime can easily fail to separate models that genuinely differ. The paper does not report pairwise difference CIs, equivalence testing (e.g., TOST), or a power analysis. This is load-bearing because the factual-support dimension is the harder and more RLVR-critical one: if the benchmark lacks power to distinguish models, 'cheaper judges remain competitive across both dimensions' is not demonstrated — it is merely not refuted. The authors should either (a) report a power
  2. [§3.3 and §4.1] Two of the 8 evaluated judges (GPT-5-mini and Claude Opus 4.6) also served on the 6-LLM gold-label council. The authors acknowledge this overlap (§4.1) and argue the human-adjudicated subset (378 cases) offers a council-independent comparison. However, the gold labels for the 870 unanimous cases are entirely LLM-generated with no independent human verification beyond 'confirmation on review.' More critically, if the council's systematic biases shaped which cases were flagged as disagreements (and thus received intensive human scrutiny), the gold labels may inherit council biases that the evaluated judges also share, inflating apparent agreement on the unanimous subset. The paper should report accuracy metrics computed exclusively on the 378 human-adjudicated cases (where gold labels are fully independent of the council) as a robustness check, and discuss whether the council overlap could
  3. [§3.1 and §6] The benchmark is a single long-form document (acknowledged in §6 as a limitation). With 624 pairs drawn from 25 topics within one document, topic-level clustering could inflate or deflate judge performance in ways that a multi-document benchmark would not. The authors note domain diversity reduces domain-knowledge confounds, but they do not report per-topic or per-strategy F1 breakdowns (§5.2 reports detection rates by strategy but not F1 by topic). Without this, it is unclear whether the aggregate F1 numbers are stable across topics or driven by a few easy/hard clusters. This matters for the generalizability of the 'cheaper judges are competitive' claim. A per-topic F1 table or at minimum a variance decomposition across topics would substantially strengthen the paper.
minor comments (6)
  1. [Table 2] The cost column is a base-10 log index relative to the cheapest model, but the absolute cost per decision is only shown in Figure 3 (in USD). Including the absolute cost in Table 2 or a footnote would improve readability.
  2. [Figure 3] The pass-rate drift is mentioned in the caption but not visually encoded in the figure itself. Adding color or marker size to indicate drift direction/magnitude would make the cost-quality-drift tradeoff immediately interpretable without cross-referencing text.
  3. [§4.3] The statement 'no single labeling judge reliably matched the human label, with per-model agreement ranging from 38% to 85% on source relevance and 43% to 67% on factual support' is important but presented without identifying which judges achieved which rates. A supplementary table would help readers assess whether the evaluated judges that also served on the council had systematically higher agreement.
  4. [Appendix A] The authors note the relevance rubric overlaps with factual support ('Does it provide supporting information for the claims') and identify this as the most frequent source of inter-judge disagreement. This is a rubric design flaw that could be fixed by rewording the relevance criterion to ask only about topical relatedness. The authors mention tightening the separation as a 'concrete direction' but do not attempt it. A revised rubric prompt tested on a subset would strengthen the contribution.
  5. [§5.2] The detection rates by adversarial strategy are reported as ranges ('roughly 86% for the subtlest edits to near 100% for negations') without a table. Given 19 strategies, a full table of detection rates per strategy would make the difficulty analysis more useful.
  6. [General] The paper uses future-dated references (e.g., 2026 publications) and model identifiers (e.g., GPT-5-mini, Claude Opus 4.6). If these are real, the API identifiers and pricing assumptions should be verified for reproducibility. If any are hypothetical or projected, this should be disclosed.

Circularity Check

0 steps flagged

No significant circularity: the paper is an empirical benchmark with externally falsifiable metrics; the only self-overlap (two judges on the gold-label council) is acknowledged and partially addressed via the human-adjudicated subset.

full rationale

The paper is an empirical benchmarking study, not a derivation chain. Its central claims (cheaper judges are competitive on F1; judges differ in directional bias at similar F1) are measured against gold labels and are externally falsifiable. The gold labels are produced by a council of 6 LLM judges with human adjudication of disagreements (Section 3.3). Two of the 8 evaluated judges (GPT-5-mini and Claude Opus 4.6) also served on the gold-label council, which the authors acknowledge (Section 4.1): 'Two of the 8 judges (GPT-5-mini and Claude Opus 4.6) also contributed to the gold-label council, but because they span the cost range and the human-adjudicated subset offers a council-independent comparison, this overlap does not drive the cost and quality findings.' This is a methodological concern about potential label contamination, not a circularity where a prediction reduces to its inputs by construction. The metrics (F1, kappa, FPR, FNR) are computed from confusion counts against gold labels, not derived from the judges' own outputs in a self-referential way. The human-adjudicated subset (378 cases) provides a partially independent check. The self-citation to Onweller et al. [2026] is for the evaluation pipeline, not a load-bearing uniqueness theorem or ansatz. No step in the paper's reasoning reduces by definition to its inputs. The concern about statistical power (overlapping CIs vs. equivalence testing) is a correctness risk, not circularity. Score 1 reflects the minor self-overlap that is acknowledged but not load-bearing for the central claim.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 1 invented entities

The paper introduces one new entity (the benchmark) and relies on several domain assumptions about score binarization, human adjudication reliability, and single-document sufficiency. The free parameters (threshold, edit rate, topic selection) are standard choices but not independently justified.

free parameters (3)
  • Score binarization threshold = 0.5
    Judge scores are binarized at 0.5 (Section 4.1). This threshold is not tuned or justified; it's a standard default but affects all downstream metrics.
  • Adversarial edit rate = ~60%
    Roughly 60% of attributed claims receive adversarial edits (Section 3.1). This ratio shapes the gold pass rates (18.4% factual support) and thus the difficulty profile of the benchmark.
  • Topic selection = 25 topics
    The 25 topic domains are manually selected (Section 3.1). Domain choice affects which knowledge the judges need, potentially biasing results toward domains where models have better training coverage.
axioms (4)
  • domain assumption LLM judge scores can be binarized at 0.5 to produce meaningful pass/fail labels
    Section 4.1: 'Both judge scores and gold labels are binarized at a threshold of 0.5, where a score >= 0.5 is treated as pass.' This assumes the continuous scores are calibrated such that 0.5 is a meaningful boundary, which is not verified.
  • domain assumption Human adjudication of LLM council disagreements produces reliable gold labels
    Section 3.3: the 378 non-unanimous decisions receive 'intensive adjudication' and the 870 unanimous ones are 'confirmed on review.' This assumes human reviewers can reliably assess citation quality for 25 diverse domains, including calculus, neurology, and quantum computing.
  • ad hoc to paper A single long-form document is a sufficient test bed for citation quality evaluation
    Section 3.1: the benchmark is 'a single long-form report spanning 25 topic domains.' The authors acknowledge this limitation in Section 6 ('These findings are limited to a single adversarial document') but the central claims are drawn from this single document.
  • domain assumption The 19 adversarial edit strategies comprehensively cover realistic citation failure modes
    Appendix B lists 19 strategies. The paper assumes these cover the space of citation errors that deep-research systems make, but there is no taxonomy comparison or coverage analysis against real-world citation failures.
invented entities (1)
  • Deep-Research Citation Benchmark no independent evidence
    purpose: Adversarial long-form dataset of 624 attribution-citation pairs with gold labels for 1,248 rubric decisions
    The benchmark is introduced by this paper and not independently validated or released. Its gold labels are partly LLM-generated (council of 6 judges). No external reproduction exists.

pith-pipeline@v1.1.0-glm · 15954 in / 2839 out tokens · 320258 ms · 2026-07-10T02:41:51.267820+00:00 · methodology

0 comments
read the original abstract

Reinforcement learning increasingly relies on an LLM judge to score each rubric criterion, and that judge acts as the reward model during training. Before such a signal can be trusted, we need to know how capable the judge must be and how biased it is. We study this calibration question for citation quality in deep-research systems, where a search-grounded LLM must support each claim it writes with a cited source. Citation quality is a structured rubric task in which each attribution-citation pair is judged along two dimensions that require an LLM, source relevance and factual support. On an adversarial long-form benchmark, we score 8 off-the-shelf LLM judges from 3 model families against gold labels over 1,248 rubric decisions, all of which were human-reviewed and 378 of which were hard cases adjudicated from judge disagreements. Cheaper judges remain competitive across both dimensions, with GPT-5-mini attaining the strongest source-relevance pass-class F1 at 0.908 ($\kappa$=0.636), while on factual support the judges are statistically indistinguishable (overlapping confidence intervals), so no single model dominates. At comparable F1, the judges still differ substantially in pass-rate drift, false positive rate, and false negative rate. Scalar F1 obscures this directional bias, yet it is exactly what a downstream reinforcement learning loop would reinforce. Calibrating the judge is therefore a prerequisite for using citation rubrics as reward signals, and our results show that this calibration does not require the most expensive available model.

Figures

Figures reproduced from arXiv: 2607.08700 by Austin Huber, Corey Feld, Elias Lumer, Ethan Leung, Kevin Paul, Vamse Kumar Subbiah.

Figure 1
Figure 1. Figure 1: Construction pipeline for the Deep-Research Citation Benchmark. 25 topics are selected and clean overviews are generated with attributed sources. About 60% of the attributed claims are then adversarially edited using 19 base strategies, with some claims receiving more than one, while the rest are left clean. Gold labels are produced by a council of 6 LLM judges that independently run the source attribution… view at source ↗
Figure 2
Figure 2. Figure 2: Pass-class F1 for 8 judges on source relevance and factual support. No single judge dominates both dimensions. • Pass-rate drift ∆ = rjudge −rgold, the signed difference between the judge and the gold pass rate, indicating systematic leniency (∆ > 0) or strictness (∆ < 0). The cost and quality figure uses a class-balanced average F1. For each dimension we average the pass-class and fail-class F1 (macro-F1)… view at source ↗
Figure 3
Figure 3. Figure 3: Judge quality (average class-balanced F1), estimated cost per decision on a logarithmic axis, and pass-rate drift. Cost does not track quality. Low-cost judges are among the most compet￾itive, and cost alone does not predict reward signal quality. adjudicated subset as an ablation isolates judge behavior on genuinely ambiguous citations. Most disagreements were near-consensus rather than deep splits, with … view at source ↗
Figure 4
Figure 4. Figure 4: Judge accuracy on the full benchmark compared with the 378 multi-judge disagreement cases. Rankings shift substantially on the adjudicated subset, so full-set F1 is an incomplete proxy for reliability on the most ambiguous citations. 5 Discussion 5.1 Directional Bias Most judges are stricter than the gold labels on source relevance. All 8 models produce predicted pass rates below the gold pass rate of 79.3… view at source ↗
Figure 5
Figure 5. Figure 5: False positive rate (FPR) and false negative rate (FNR) per judge per dimension. Judges with similar F1 can differ substantially in directional bias, a property invisible to scalar F1 but directly relevant to what an RLVR training loop reinforces. 5.2 Difficulty by Adversarial Strategy To locate the factual-support difficulty, we linked each adversarially edited claim to its strategy (Appendix B) and measu… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

32 extracted references · 32 canonical work pages · 13 internal anchors

  1. [1]

    Scaling Laws for Reward Model Overoptimization

    Leo Gao and John Schulman and Jacob Hilton , title =. International Conference on Machine Learning , year =. 2210.10760 , archivePrefix =

    work page internal anchor Pith review Pith/arXiv arXiv

  2. [2]

    Towards Understanding Sycophancy in Language Models

    Mrinank Sharma and Meg Tong and Tomasz Korbak and David Duvenaud and Amanda Askell and Samuel R. Bowman and others , title =. International Conference on Learning Representations , year =. 2310.13548 , archivePrefix =

    work page internal anchor Pith review Pith/arXiv arXiv

  3. [3]

    Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

    Lianmin Zheng and Wei-Lin Chiang and Ying Sheng and Siyuan Zhuang and Zhanghao Wu and Yonghao Zhuang and Zi Lin and Zhuohan Li and Dacheng Li and Eric P. Xing and Hao Zhang and Joseph E. Gonzalez and Ion Stoica , title =. Advances in Neural Information Processing Systems , year =. 2306.05685 , archivePrefix =

    work page internal anchor Pith review Pith/arXiv arXiv

  4. [4]

    Ragas: Automated Evaluation of Retrieval Augmented Generation

    Shahul Es and Jithin James and Luis Espinosa-Anke and Steven Schockaert , title =. Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics , year =. 2309.15217 , archivePrefix =

    work page internal anchor Pith review Pith/arXiv arXiv

  5. [5]

    ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems

    Jon Saad-Falcon and Omar Khattab and Christopher Potts and Matei Zaharia , title =. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics , year =. 2311.09476 , archivePrefix =

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  6. [6]

    Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages =

    Tianyu Gao and Howard Yen and Jiatong Yu and Danqi Chen , title =. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages =. 2023 , eprint =

    work page 2023

  7. [7]

    FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation

    Sewon Min and Kalpesh Krishna and Xinxi Lyu and Mike Lewis and Wen-tau Yih and Pang Wei Koh and Mohit Iyyer and Luke Zettlemoyer and Hannaneh Hajishirzi , title =. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , year =. 2305.14251 , archivePrefix =

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  8. [8]

    2026 , eprint =

    Utkarsh Tyagi and Xingang Guo and MohammadHossein Rezaei and Daniel George and Anas Mahmoud and Jackson Lee and Bing Liu and Yunzhong He , title =. 2026 , eprint =

    work page 2026

  9. [9]

    2026 , eprint =

    Anas Mahmoud and MohammadHossein Rezaei and Zihao Wang and Anisha Gunjal and Bing Liu and Yunzhong He , title =. 2026 , eprint =

    work page 2026

  10. [10]

    2026 , eprint =

    MohammadHossein Rezaei and Anas Mahmoud and Zihao Wang and Utkarsh Tyagi and Advait Gosai and Razvan-Gabriel Dumitru and Aakash Sabharwal and Bing Liu and Yunzhong He , title =. 2026 , eprint =

    work page 2026

  11. [11]

    2026 , howpublished =

    Jessica Li , title =. 2026 , howpublished =

    work page 2026

  12. [12]

    2026 , howpublished =

    Vivek Trivedy and Jake Broekhuizen and Harrison Chase and Niko Grupen and Gabe Pereyra and Spencer Poff and Julio Pereyra , title =. 2026 , howpublished =

    work page 2026

  13. [13]

    2022 , eprint =

    Reiichiro Nakano and Jacob Hilton and Suchir Balaji and Jeff Wu and Long Ouyang and Christina Kim and Christopher Hesse and Shantanu Jain and Vineet Kosaraju and William Saunders and others , title =. 2022 , eprint =

    work page 2022

  14. [14]

    2022 , eprint =

    Jacob Menick and Maja Trebacz and Vladimir Mikulik and John Aslanides and Francis Song and Martin Chadwick and Mia Glaese and Susannah Young and Lucy Campbell-Gillingham and Geoffrey Irving and Nat McAleese , title =. 2022 , eprint =

    work page 2022

  15. [15]

    Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection

    Akari Asai and Zeqiu Wu and Yizhong Wang and Avirup Sil and Hannaneh Hajishirzi , title =. International Conference on Learning Representations , year =. 2310.11511 , archivePrefix =

    work page internal anchor Pith review Pith/arXiv arXiv

  16. [16]

    Let's Verify Step by Step

    Hunter Lightman and Vineet Kosaraju and Yura Burda and Harri Edwards and Bowen Baker and Teddy Lee and Jan Leike and John Schulman and Ilya Sutskever and Karl Cobbe , title =. International Conference on Learning Representations , year =. 2305.20050 , archivePrefix =

    work page internal anchor Pith review Pith/arXiv arXiv

  17. [17]

    Prometheus: Inducing Fine-grained Evaluation Capability in Language Models

    Seungone Kim and Jamin Shin and Yejin Cho and Joel Jang and Shayne Longpre and Hwaran Lee and Sangdoo Yun and Seongjin Shin and Sungdong Kim and James Thorne and Minjoon Seo , title =. International Conference on Learning Representations , year =. 2310.08491 , archivePrefix =

    work page internal anchor Pith review Pith/arXiv arXiv

  18. [18]

    2023 , eprint =

    Collin Burns and Pavel Izmailov and Jan Hendrik Kirchner and Bowen Baker and Leo Gao and Leopold Aschenbrenner and Yining Chen and Adrien Ecoffet and Manas Joglekar and Jan Leike and Ilya Sutskever and Jeff Wu , title =. 2023 , eprint =

    work page 2023

  19. [19]

    CiteME: Can Language Models Accurately Cite Scientific Claims?

    Ori Press and Andreas Hochlehnert and Ameya Prabhu and Vishaal Udandarao and Ofir Press and Matthias Bethge , title =. Advances in Neural Information Processing Systems , year =. 2407.12861 , archivePrefix =

    work page internal anchor Pith review Pith/arXiv arXiv

  20. [20]

    Fung and Qingyun Wang , title =

    Yee Man Choi and Xuehang Guo and Yi R. Fung and Qingyun Wang , title =. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages =

    work page

  21. [21]

    2026 , eprint =

    Hailey Onweller and Elias Lumer and Austin Huber and Pia Ramchandani and Vamse Kumar Subbiah and Corey Feld , title =. 2026 , eprint =

    work page 2026

  22. [22]

    2026 IEEE Conference on Artificial Intelligence (CAI) , pages=

    Tool and agent selection for large language model agents in production: A survey , author=. 2026 IEEE Conference on Artificial Intelligence (CAI) , pages=. 2026 , organization=

    work page 2026

  23. [23]

    arXiv preprint arXiv:2603.16862 , year=

    Chronos: Temporal-aware conversational agents with structured event retrieval for long-term memory , author=. arXiv preprint arXiv:2603.16862 , year=

    work page arXiv

  24. [24]

    arXiv preprint arXiv:2601.06007 , year=

    Don't Break the Cache: An Evaluation of Prompt Caching for Long-Horizon Agentic Tasks , author=. arXiv preprint arXiv:2601.06007 , year=

    work page arXiv

  25. [25]

    Is Grep All You Need? How Agent Harnesses Reshape Agentic Search

    Is grep all you need? how agent harnesses reshape agentic search , author=. arXiv preprint arXiv:2605.15184 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  26. [26]

    Miranda and Alisa Liu and Nouha Dziri and others , title =

    Nathan Lambert and Jacob Morrison and Valentina Pyatkin and Shengyi Huang and Hamish Ivison and Faeze Brahman and Lester James V. Miranda and Alisa Liu and Nouha Dziri and others , title =. 2024 , eprint =

    work page 2024

  27. [27]

    2022 , eprint =

    Yuntao Bai and Saurav Kadavath and Sandipan Kundu and Amanda Askell and Jackson Kernion and Andy Jones and others , title =. 2022 , eprint =

    work page 2022

  28. [28]

    G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment

    Yang Liu and Dan Iter and Yichong Xu and Shuohang Wang and Ruochen Xu and Chenguang Zhu , title =. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , year =. 2303.16634 , archivePrefix =

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  29. [29]

    RARR: Researching and Revising What Language Models Say, Using Language Models

    Luyu Gao and Zhuyun Dai and Panupong Pasupat and Anthony Chen and Arun Tejasvi Chaganty and Yicheng Fan and Vincent Y. Zhao and Ni Lao and Hongrae Lee and Da-Cheng Juan and Kelvin Guu , title =. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics , year =. 2210.08726 , archivePrefix =

    work page internal anchor Pith review Pith/arXiv arXiv

  30. [30]

    Bernd Bohnet and Vinh Q. Tran and Pat Verga and Roee Aharoni and Daniel Andor and Livio Baldini Soares and Massimiliano Ciaramita and Jacob Eisenstein and Kuzman Ganchev and Jonathan Herzig and Kai Hui and Tom Kwiatkowski and Ji Ma and Jianmo Ni and others , title =. 2022 , eprint =

    work page 2022

  31. [31]

    Long Ouyang and Jeff Wu and Xu Jiang and Diogo Almeida and Carroll L. Wainwright and Pamela Mishkin and Chong Zhang and Sandhini Agarwal and Katarina Slama and Alex Ray and John Schulman and Jacob Hilton and Fraser Kelton and Luke Miller and Maddie Simens and Amanda Askell and Peter Welinder and Paul Christiano and Jan Leike and Ryan Lowe , title =. 2022 ...

    work page 2022

  32. [32]

    Smith and Hannaneh Hajishirzi , title =

    Nathan Lambert and Valentina Pyatkin and Jacob Morrison and LJ Miranda and Bill Yuchen Lin and Khyathi Chandu and Nouha Dziri and Sachin Kumar and Tom Zick and Yejin Choi and Noah A. Smith and Hannaneh Hajishirzi , title =. 2024 , eprint =

    work page 2024