Pith. sign in

REVIEW 3 major objections 5 minor 15 references

ORCA-bench: How Ready Are Language Model Agents for Oncall?

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Across five frontier agents, best RCA accuracy is 25.3% on Medium-difficulty tasks and 10.0% on Hard—today's agents are not ready for oncall.

desk verdict ORCA-bench is a genuinely useful RCA benchmark with credible low agent scores; the main caveat is heavy GPT-5.4 involvement in ground truth and judging, but the conclusion that agents aren't oncall-ready survives. read the letter →

arxiv 2607.28545 v2 pith:7F7Z2WXU submitted 2026-07-30 cs.CL cs.AIcs.SE

Albert Gong , Kyuseong Choi , Abhineet Agarwal , Jason Schechner , Ryan Huang , Raj Agrawal , Anish Agarwal , Raaz Dwivedi This is my paper · ORCID
classification cs.CLcs.AIcs.SE
keywords rootcauseanalysisLLMagentsbenchmarksitereliabilityengineeringtelemetrymicroservicesincidentresponsehallucination
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces ORCA-bench, a root-cause-analysis benchmark that places general-purpose coding agents in a production-style oncall setting: a live microservice system with six days of metrics, logs, and traces, plus full source code, and a user-facing report that may be vague or mistimed. The central claim is that today's frontier agents are not ready for oncall: the best among five achieves 25.3% RCA accuracy on Medium-difficulty tasks and 10.0% on Hard, where RCA accuracy requires naming every plausible root cause. Hallucination of an implausible root cause ranges from 7% to 40%, and removing source-code access degrades all metrics. The authors argue these numbers lower-bound the real production gap, because the testbed is public, static, and smaller than real systems. A sympathetic reader would care because it tests diagnostic reasoning over noisy evidence, not just code editing.

What carries the argument

The central object is the ORCA-bench task pipeline and scoring rubric built on the OpenTelemetry Astronomy Shop, a 19-microservice demo system whose Prometheus metrics, OpenSearch logs, and Jaeger traces are exposed through a Grafana API while the agent holds terminal access to the source code. Tasks vary user-issue specificity (Easy/Medium/Hard), time-to-detection from 15 minutes to 24 hours, report-time style, and co-occurring fault scenarios; ground truth is a set of plausible root causes derived from expert-validated feature-flag rubrics with symptom evidence in metrics, logs, and traces. Scoring uses a GPT-5.4 judge that assigns 0-3 per rubric and aggregates to RCA accuracy, depth, and hallucination rate, with human rescoring of 40 tasks giving Cohen's $\kappa_w=0.90$ agreement.

What would settle it

Take a stratified random sample of about 200 of the 1,079 tasks and have two independent SREs rebuild the plausible-root-cause sets and rescore agent reports by hand; if human-LLM agreement on plausibility or per-rubric scores falls well below the reported $\kappa_w=0.90$, the headline accuracy and hallucination numbers are not stable.

Watch

Extended reading notes

Core claim

ORCA-bench's core claim is that frontier coding agents cannot yet perform production-grade root cause analysis. On 1,079 tasks built from a six-day schedule of 11 feature-flag incidents in the OpenTelemetry Astronomy Shop, the strongest agent (Claude Sonnet 4.6) achieves 25.3% RCA accuracy on Medium-difficulty tasks and 10.0% on Hard (Claude Opus 4.7), where a task is scored correct only if the agent names every plausible root cause. Hard tasks, where the user report is only 'users are reporting site issues', have an average of 4.41 plausible causes, and accuracy collapses to 1.1% for the weakest model. Removing source-code access drops RCA accuracy by 9 to 16 percentage points across models, so code is load-bearing. Since the environment is a curated 50 GB public demo investigated in isolation, the paper positions the measured gap as a lower bound on readiness for real production reliability.

Load-bearing premise

The load-bearing premise is that the plausible-root-cause sets and judge scores are valid, but those come largely from a single LLM, GPT-5.4, with only 40 of 1,079 tasks checked by humans; if that model's plausibility judgments are systematically wrong, the headline percentages would shift.

Editorial extensions

If this is right

  • If the benchmark is right, no evaluated frontier agent can be trusted to handle oncall: the best miss at least one plausible root cause in about three-quarters of Medium tasks and 90% of Hard tasks.
  • Vague user reports plus concurrent faults are the hard part; accuracy collapses as the plausible-cause set grows, so progress requires exhaustive hypothesis enumeration, not just faster telemetry queries.
  • Source-code access is essential to current agents: removing it drops RCA accuracy by 9-16 points and raises hallucination, so agents use code to disambiguate telemetry.
  • Agent telemetry skills are a bottleneck: 26-40% of telemetry calls error or return empty, and even the best model fails to cite ground-truth metrics, logs, or traces on a large share of tasks.
  • The numbers are a lower bound for production: the testbed is public, small, static, and single-task, so real systems should be harder and the required engineering investment larger.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is to give agents persistent memory across incidents and an action loop to test mitigations; the paper identifies both as untried levers that may narrow the gap.
  • The all-or-nothing accuracy metric punishes partial diagnosis; an agent that finds one correct cause among several scores zero, though the RCA-depth metric already gives partial credit, so operational usability may be better than the headline suggests.
  • Because ground truth and judging rely heavily on GPT-5.4, one could mine real postmortems at similar report granularity to see whether agent rankings replicate outside a public demo codebase.
  • The code-access ablation implies that on private, idiosyncratic codebases outside pretraining, the effective drop from removing or weakening code may be larger than the 9-16 points measured here.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces ORCA-bench, a benchmark that places general-purpose coding agents in a production-fidelity oncall RCA setting built on the OpenTelemetry Astronomy Shop. It exposes live Prometheus, Jaeger, and OpenSearch telemetry through Grafana plus full source-code access, and defines 1,079 tasks that vary report specificity (Easy/Medium/Hard), time-to-detection, and co-occurring fault scenarios. Ground-truth symptoms are curated from the schedule and feature-flag rubrics, while the plausible root-cause answer sets are generated by GPT-5.4 in stage B5; a 40-task ORCA-bench Verified subset has every ground-truth label and per-model score hand-confirmed. A GPT-5.4 LLM judge is validated against human scoring on that subset (Cohen's kappa_w = 0.90). Across five frontier agents the paper reports best RCA Accuracy of 25.3% on Medium and 10.0% on Hard tasks, hallucination rates from 7% to 40%, and a performance drop when source-code access is removed. The authors conclude that frontier agents are not ready for oncall and that the measured gap is a lower bound on the investment needed for production reliability.

Significance. If the reported numbers hold up, this is a valuable and timely contribution. The benchmark is the first in this line to combine real telemetry interfaces with source-code access, systematically vary report specificity and TTD, and ship public code and data. The per-rubric scoring scheme, the human-judge agreement study, the inclusion of control tasks, and the unusually candid limitations section are all strengths. The infrastructure is reusable, and the central claim is falsifiable: future agents can be re-run on the same harness and compared on the headline metrics. The main risk is that the exact percentages depend on GPT-5.4-generated plausible root-cause sets that are human-verified for only 40 of 1,079 tasks, with judge agreement also estimated only on that subset. The qualitative conclusion would likely survive modest label noise, but the precise Medium/Hard accuracy numbers and the cross-model ordering need stronger validation or a sensitivity analysis.

major comments (3)
  1. [Sec. 3 (B4/B5), App. E, Sec. 4 (M1)] The load-bearing part of the headline numbers is the validity of the ground-truth answer sets, and this is established only very thinly. For 1,039 of 1,079 tasks, the set of plausible root causes is produced by GPT-5.4 in stage B5, prompted to "Be inclusive" (App. E), while the user-facing issue is also generated by GPT-5.4 (B4). Because M1 requires the agent to name every listed plausible root cause, an over-inclusive answer set mechanically lowers RCA accuracy, and an under-inclusive set inflates hallucination rates. The 40-task Verified subset and the human-judge agreement (Table 3, Sec. 4) are genuine evidence, but they do not bound the error distribution on the remaining tasks, particularly Hard tasks with an average of 4.41 ground-truth root causes. Since the judge is also GPT-5.4, high human-judge agreement on 40 tasks does not rule out systematic alignment between the generator and the judge. I ask for a sensitivity analysis: independently re-annotate a larger stratified sample of tasks (especially Hard/vague reports), and recompute M1 and hallucination under conservative and liberal label sets (e.g., human-only, intersection of GPT-5.4 and human, or perturbed by adding/removing borderline candidates). Without this, the exact percentages 25.3% and 10.0% are not fully supported, even though the qualitative conclusion may survive.
  2. [Sec. 4, Table 3, Sec. 5] The validation statistics in Table 3 are internally inconsistent with the stated Verified subset size. The text says one author re-scored reports on the 40 tasks in ORCA-bench Verified, and Sec. 5 says there are 32 incident tasks in this subset; with five models this would yield at most 160 (or 200 if controls are included) model-task scores, but Table 3 reports N=191 for All, with Easy 47, Medium 72, Hard 72. Please state the unit of analysis explicitly (model-task, model-rubric, or per-rubric score) and reconcile the totals; this is needed to know exactly what the kappa_w=0.90 is computed over.
  3. [Abstract, Sec. 5 (Finding #3), Tables I.2/I.6] The abstract and contributions state that removing source-code access degrades every metric, but the appendix tables contain counterexamples. For example, DeepSeek-V4-Pro's control-task RCA depth is 77.9±3.0 with code+telemetry (Table I.2) and 81.0±2.8 with telemetry only (Table I.6), and several telemetry citation rates are higher in the telemetry-only condition (e.g., Claude Opus 4.7 Metrics (any): 65.3 vs 69.5; Logs (all): 3.1 vs 4.9). The main incident metrics (RCA accuracy, hallucination) do degrade, but the universal claim should be qualified to the metrics shown in Fig. 6.
minor comments (5)
  1. [Sec. 3, B2] The text says "ORCA-bench comprises 1076 diverse and challenging tasks" and "195 of the 1076 total tasks are control tasks," but the abstract and other sections state 1,079 tasks; since 884 incident + 195 control = 1,079, please make the count consistent throughout.
  2. [Sec. 4, E3, Fig. 1] Hallucination rate is defined as the fraction of non-empty tasks, but Fig. 1's caption gives N=884 tasks for the hallucination plot; please clarify whether empty reports are excluded from the denominator and whether N=884 is the number of incident tasks or the number of non-empty reports.
  3. [Sec. 5, Fig. 4] There are several typos in the failure-mode discussion: "user-facing isuse" should be "user-facing issue," "consistently quantity" should be "consistently quantify," and "co-occuring" should be "co-occurring".
  4. [Abstract, Sec. 5] The abstract says the gap "remains even with Claude Fable 5," but Claude Fable 5 was evaluated only on the 32 incident tasks in ORCA-bench Verified (Sec. 5, Fig. K.1); please state this qualification explicitly in the abstract or in the contribution list.
  5. [Abstract] The phrase "order of magnitudes" should be "orders of magnitude".

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: ORCA-bench is an empirical benchmark; the same-model (GPT-5.4) label generation and judging is a disclosed validity limitation, not a circular reduction.

full rationale

ORCA-bench's central claims are measurements of five external agents in a fixed environment, not derivations from its own inputs. The answer sets are produced in stage B5 by GPT-5.4 under an explicitly inclusive prompt, and the judge is also GPT-5.4, so the labels and scores are not independent of the labeler; however, no quantity is fitted to agent outputs, no parameter is renamed as a prediction, and no load-bearing self-citation or imported uniqueness theorem appears. The paper discloses the 40-task ORCA-bench Verified subset where ground truth and scores are hand-confirmed (Sec. 3, Sec. 4, Tab. 3). The relation that M1 requires naming every listed plausible root cause means that larger GPT-5.4-generated plausible sets mechanically make M1 harder, but this is a label-validity and correctness risk rather than a circular derivation, because the labels are fixed before evaluation and do not encode the agents' behavior. The hard-task average of 4.41 ground-truth root causes is a construction property of the benchmark, and the observed performance collapse is an empirical finding whose magnitude could in principle have been large or small. The LLM-generated ground truth for 1,039 of 1,079 tasks is a legitimate limitation to weigh under correctness risk, but it does not make the headline accuracy numbers equivalent to the paper's inputs by construction.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim depends on the validity of LLM-generated ground truth and judge scoring for mostly unverified tasks, the representativeness of the Astronomy Shop, and several hand-chosen evaluation thresholds. These are domain assumptions and design choices rather than fitted parameters or invented entities.

free parameters (4)
  • Incident-time tolerance (±10 minutes)
    Hand-chosen tolerance in the judge prompt (App. G.1.1); it changes per-rubric scores and therefore RCA depth and accuracy.
  • Telemetry grace period (5 minutes)
    Agents may query telemetry 5 minutes after the detection time (Sec. 3, stage B1); this widens the information available and affects scores.
  • Quiet-window margin (10 minutes)
    The control-task judge shrinks the quiet window by 10 minutes on each side (App. G.1.2); it changes control-task accuracy.
  • Time-to-detection grid (15 min, 1 h, 8 h, 24 h)
    Chosen TTD values in stage B3 (Sec. 3); a task-design parameter that the paper does not derive.
assumptions (4)
  • domain assumption The OpenTelemetry Astronomy Shop and its feature-flag incidents are a representative proxy for production oncall conditions.
    The benchmark's transferability and lower-bound claims rest on this; the system is smaller, static, and public, as the authors acknowledge.
  • ad hoc to paper GPT-5.4 generated plausible root-cause sets are valid ground truth for the tasks outside the 40-task Verified subset.
    Stage B5 (Sec. 3) uses GPT-5.4 plausibility verdicts to define answer sets; only 40 tasks are manually checked by two authors.
  • domain assumption The LLM judge's agreement with one human rater on the Verified subset generalizes to all tasks.
    Section 4 reports quadratic-weighted Cohen's kappa of 0.90 on 191 per-rubric scores, but the judge is then applied to the full benchmark.
  • domain assumption A single prompt template and the Terminus-2 harness fairly represent each model's capability.
    The authors state in the limitations that only one prompt template and one harness were used; all model conclusions are conditional on that configuration.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ORCA-bench: How Ready Are Language Model Agents for Oncall?." pith.science (2026). https://pith.science/paper/7F7Z2WXU

@misc{pith2026260728545,
  author       = {Pith},
  title        = {Pith review of: ORCA-bench: How Ready Are Language Model Agents for Oncall?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7F7Z2WXU}},
  note         = {Machine review of arXiv:2607.28545}
}
abstract

Large language models can write, patch, and search code, but oncall root cause analysis (RCA) demands something different: reasoning over noisy metrics, logs, traces, and source code, starting from ambiguous user-facing reports, often hours after the incident began. We introduce ORCA-bench, a benchmark that puts general-purpose coding agents in a production-fidelity oncall setting. ORCA-bench pairs a live OpenTelemetry-instrumented microservice system--exposing six days of metrics, logs, and traces through real telemetry interfaces (Prometheus, Jaeger, and OpenSearch via Grafana) and full source-code access--with 1,079 RCA tasks that systematically vary report specificity, time-to-detection, and co-occurring fault scenarios. Ground-truth symptoms are curated and signed off by expert SREs, and our LLM-as-judge is independently re-scored by humans (Cohen's $\kappa_w=0.90$). Across five frontier agents, the best RCA Accuracy is 25.3% on Medium-difficulty tasks (the realistic-input setting) and 10.0% on Hard--a gap that remains even with Claude Fable 5. The weakest model hallucinates an implausible root cause in 40% of incident reports, and removing source-code access degrades every metric. Crucially, these are performances on a curated 50 GB / six-day testbed with tasks investigated in isolation on a system whose code and instrumentation are public. Since real production systems are order of magnitudes larger, more dynamic, and more idiosyncratic, the gap we report is a lower bound on the engineering investment required before frontier coding agents can be safely entrusted with production reliability. We release the public set at https://hub.harborframework.com/datasets/orca-bench/orca-bench.

Figures

Figures reproduced from arXiv: 2607.28545 by the authors.

Figure 1
Figure 1. RCA accuracy measures whether agents identified [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Construction pipeline. We construct ORCA-bench in six stages. and manually verify a subset of 40 tasks to obtain ORCA-bench Verified. See Sec. 3 for details. Incident start time [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Task definition. The SRE agent is provided a user-facing issue at the report time (an offset after the incident start time) and begins investigation at the detection time (a time-to-detection after the incident start time). B1. Environment. Our starting point is the OpenTelemetry Astronomy Shop, a microservice-based distributed system intended to illustrate the implementation of OpenTelemetry in a near real-world en… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Excerpts from a low-scoring and high-scoring incident RCA report on the same task. See App. J for the full, unabridged reports. 0% 50% 100% Control Control Control Control Control Easy Easy Easy Easy Easy Medium Medium Medium Medium Medium Hard Hard Hard Hard Hard RCA …
Figure 5
Figure 5. Figure 5: RCA quality by issue specificity. We use GPT-5.4 as the LLM judge and report mean ± 1 standard error across the 195 control, 288 easy, 316 medium, 280 hard tasks. Two failure modes are agents get distracted by background noise and cannot exhaustively find all root caus…
Figure 6
Figure 6. Figure 6: RCA quality with and without code access. We use GPT-5.4 as the LLM judge using the metrics defined in Sec. 4 and report mean ± 1 standard error across all N = 884 incident tasks. (a) Opus 4.7 2% 16% 70% 12% 10% 58% 22% 34% 10% 61% 87% 56% 22% (b) GLM-5 13% 20% 57% 10%…
Figure 7
Figure 7. Figure 7: Agent behavior during investigation. Node area is proportional to the marginal share of commands in that category. Directed edges are transition probabilities P(next | prev) between con￾secutive commands. For readability, we omit edges whose transition probability P(ne…
Figure 8
Figure 8. Figure 8: Telemetry retrieval efficiency and accuracy. We report mean ± 1 standard error across all N = 1079 tasks, all telemetry commands, and all tasks with ground-truth metrics/logs/traces for (a), (b), and (c), respectively. Empty means the call ran successfully but returned…

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

15 extracted references · 8 canonical work pages

  1. [1]

    Long code arena: a set of benchmarks for long-context code models

    Egor Bogomolov, Aleksandra Eliseeva, Timur Galimzyanov, Evgeniy Glukhov, Anton Shapkin, Maria Tigina, Yaroslav Golubev, Alexander Kovrigin, Arie Van Deursen, Maliheh Izadi, et al. Long code arena: a set of benchmarks for long-context code models. arXiv preprint arXiv:2406.11612, 2024

    arXiv 2024

  2. [2]

    Sregym: A live benchmark for ai sre agents with high-fidelity failure scenarios

    Jackson Clark, Yiming Su, Saad Mohammad Rafid Pial, Yifang Tian, Lily Gniedziejko, Hans-Arno Jacobsen, Yinfang Chen, and Tianyin Xu. Sregym: A live benchmark for ai sre agents with high-fidelity failure scenarios. arXiv preprint arXiv:2605.07161, 2026

    arXiv 2026

  3. [3]

    Itbench: Evaluating ai agents across diverse real-world it automation tasks

    Saurabh Jha, Rohan R Arora, Yuji Watanabe, Takumi Yanagawa, Yinfang Chen, Jackson Clark, Bhavya Bhavya, Mudit Verma, Harshit Kumar, Hirokuni Kitahara, et al. Itbench: Evaluating ai agents across diverse real-world it automation tasks. In International Conference on Machine Learning, pages 27134--27197. PMLR, 2025

    work page 2025

  4. [4]

    Swe-bench: Can language models resolve real-world github issues? In International Conference on Learning Representations, volume 2024, pages 54107--54157, 2024

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? In International Conference on Learning Representations, volume 2024, pages 54107--54157, 2024

    work page 2024

  5. [5]

    Merrill, Alexander G

    Mike A. Merrill, Alexander G. Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Yeon Shin, Thomas Walshe, E. Kelly Buchanan, Junhong Shen, Guanghao Ye, Haowei Lin, Jason Poulos, Maoyu Wang, Marianna Nezhurina, Jenia Jitsev, Di Lu, Orfeas Menis Mastromichalakis, Zhiwei Xu, Zizhao Chen, Yue Liu, Robert Zhang, Leon Liangyu Chen, An...

    arXiv 2026

  6. [6]

    Rcaeval: A benchmark for root cause analysis of microservice systems with telemetry data

    Luan Pham, Hongyu Zhang, Huong Ha, Flora Salim, and Xiuzhen Zhang. Rcaeval: A benchmark for root cause analysis of microservice systems with telemetry data. In Companion Proceedings of the ACM on Web Conference 2025, pages 777--780, 2025

    work page 2025

  7. [7]

    Building ai agents for autonomous clouds: Challenges and design principles

    Manish Shetty, Yinfang Chen, Gagan Somashekar, Minghua Ma, Yogesh Simmhan, Xuchao Zhang, Jonathan Mace, Dax Vandevoorde, Pedro Las-Casas, Shachee Mishra Gupta, et al. Building ai agents for autonomous clouds: Challenges and design principles. In Proceedings of the 2024 ACM Symposium on Cloud Computing, pages 99--110, 2024

    work page 2024

  8. [8]

    Openrca: Can large language models locate the root cause of software failures? In The Thirteenth International Conference on Learning Representations, 2025

    Junjielong Xu, Qinan Zhang, Zhiqing Zhong, Shilin He, Chaoyun Zhang, Qingwei Lin, Dan Pei, Pinjia He, Dongmei Zhang, and Qi Zhang. Openrca: Can large language models locate the root cause of software failures? In The Thirteenth International Conference on Learning Representations, 2025

    work page 2025

Show all 15 references
  1. [9]

    Swe-bench multimodal: Do ai systems generalize to visual software domains? In The Thirteenth International Conference on Learning Representations, 2025

    John Yang, Carlos E Jimenez, Alex L Zhang, Kilian Lieret, Joyce Yang, Xindi Wu, Ori Press, Niklas Muennighoff, Gabriel Synnaeve, Karthik R Narasimhan, et al. Swe-bench multimodal: Do ai systems generalize to visual software domains? In The Thirteenth International Conference o...

    2025

  2. [10]

    Swe-smith: Scaling data for software engineering agents

    John Yang, Kilian Lieret, Carlos Jimenez, Alexander Wettig, Kabir Khandpur, Yanzhe Zhang, Binyuan Hui, Ofir Press, Ludwig Schmidt, and Diyi Yang. Swe-smith: Scaling data for software engineering agents. Advances in Neural Information Processing Systems, 38, 2026

    2026

  3. [11]

    Judging llm-as-a-judge with mt-bench and chatbot arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems, 36: 0 46595--46623, 2023

    2023

  4. [12]

    Graders should cheat: privileged information enables expert-level automated evaluations

    Jin Peng Zhou, S \'e b Arnold, Nan Ding, Kilian Q Weinberger, Nan Hua, and Fei Sha. Graders should cheat: privileged information enables expert-level automated evaluations. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 16583--...

    2025

  5. [13]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  6. [14]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  7. [15]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.