Pith. sign in

REVIEW 4 major objections 4 minor 41 references

This paper argues that agentic AI evaluation validity degrades multiplicatively across task generation, simulation, and automated judgment, so benchmark scores without human reliability checks can retain less than 5% of valid signal.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Agentic AI evaluation validity is bounded by the product of task-generation, simulator, and judge reliability, leaving most current automated benchmarks with less than 30% valid signal.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection Worth engaging for its synthesis and prescriptions; the central multiplicative bound and the under-5% claims do not survive close reading, but the paper is an honest, useful critique. the 4 major comments →

arxiv 2608.00794 v2 pith:FJVMDCI5 submitted 2026-08-01 cs.AI

Measurement Without Validity: The Compounding Reliability Problem in Agentic AI Evaluation

William Caban This is my paper
classification cs.AI
keywords inter-rater reliabilityagentic AI evaluationconstruct validityLLM-as-a-Judgebenchmark qualitysimulation calibrationcompounding validity model
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Agentic AI evaluation pipelines turn raw agent behavior into a benchmark score through three stages: generating tasks, simulating users or worlds, and having human or LLM judges score the results. The paper's central claim is that each stage loses some fraction of the signal the evaluation is supposed to measure, and these losses multiply rather than add. It formalizes this as V_total ≤ V_1 × V_2 × V_3, where each V_i is the validity retained at one stage, and estimates that automated grading without human inter-rater reliability leaves V_3 near 0.05. Drawing on published measurements, it argues that major benchmarks such as WebArena, SWE-bench, and τ-bench retain under 5% of valid signal, while the human-verified version WebArena Verified retains about 80%. If the model is right, reporting a benchmark score without validity-layer evidence is not measurement but a number.

Core claim

The paper's core discovery is the compounding-validity inequality V_total ≤ V_1 × V_2 × V_3: the overall construct validity of an agentic evaluation pipeline is bounded above by the product of task-generation validity, simulation-calibration validity, and judgment validity. Using published estimates, the paper finds that a pipeline retaining 70% validity at each stage is at most 34% valid against the intended construct, and that a survey of 55 published agentic evaluation papers finds 82% use structurally mismatched, incomplete, or absent inter-rater reliability metrics. The paper reports that automated substring grading agrees with human annotation at κ = 0.049, essentially chance level, an

What carries the argument

The central object is the three-layer compounding validity bound V_total ≤ V_1 × V_2 × V_3. V_1 is task-generation validity, defined as one minus the total variation distance between the generated task distribution and the ideal construct distribution; V_2 is simulation-calibration validity, defined as the intraclass correlation ICC(A,1) between simulated and real agent outcomes; V_3 is judgment validity, defined as the structurally correct inter-rater reliability statistic (Fleiss's κ or Krippendorff's α) for the rating design. The bound carries the argument by converting the intuition that failures multiply into a computable upper limit, and by supporting a four-step estimation protocol (S

Load-bearing premise

The load-bearing premise is that the three layer measures behave like independent signal-to-noise fractions whose product bounds a single total validity, and that a rater-agreement statistic can serve as a validity measure—both stated in the paper as a conceptual model, not a proved proposition.

What would settle it

Measure automated-grader agreement against trained human annotators on a large, representative sample of a major benchmark such as SWE-bench or τ-bench. If grader-human agreement exceeds κ ≈ 0.5, the V_3 floor of 0.05 does not generalize. Also test the independence assumption by measuring error correlation across the three layers; negative correlation could violate the product bound.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Any benchmark score reported without IRR metrics, simulation-calibration evidence, and metric-selection justification is incomplete; interpreting the score requires reporting the three validity layers alongside it.
  • For six widely used benchmarks, the model gives V_total < 0.05 when no human IRR is reported, so an 82% score on WebArena original should not be read as 82% of the intended capability.
  • Adding rigorous human annotation with two fixed raters (as in WebArena Verified) raises the bound roughly 16-fold, from 0.049 to 0.805, making the judgment layer the highest-leverage intervention.
  • Simulation and judge calibration vary by demographic and linguistic group, so unstratified evaluation overstates validity for Standard American English users and understates it for AAVE and other non-SAE users.
  • Even under an optimistic automated-grader validity of V_3 = 0.50, both τ-bench and SWE-bench remain below the paper's V_total = 0.30 pipeline-redesign threshold.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the V_3 floor generalizes, automated-only leaderboards are mostly ranking measurement noise; a human-verified subset re-run could reorder current leaderboards substantially.
  • Because positively correlated failures tighten the bound, pipelines that use one LLM family for task generation, simulation, and judging are even less valid than the headline product suggests.
  • The same multiplicative structure likely applies to other multi-stage AI measurement chains, such as retrieval-augmented generation pipelines or agentic safety audits that combine generation, execution, and scoring stages.
  • The paper's Section 9 protocol is directly testable: applying it to any major benchmark would produce a public V_total estimate and either confirm or bound the generality of the 0.05 V_3 floor.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper argues that agentic AI evaluation pipelines suffer from compounding validity failures across task generation (V1), simulation calibration (V2), and automated judgment (V3). It proposes the bound Vtotal ≤ V1 × V2 × V3, presents a structured survey of 55 papers claiming that about 82% misuse or omit inter-rater reliability metrics, derives eight prescriptions for valid evaluation, and applies the model to six benchmarks. The manuscript is transparent that Eq. (1) is a conceptual model rather than a proved theorem, includes a sensitivity analysis (Table 4), and provides open data and code for the survey and reliability check. The central applied conclusion is that benchmarks without human IRR validation retain less than 5% of valid signal, a claim that depends on assigning V3 = 0.05 to several benchmarks.

Significance. If the central quantitative claims held, the paper would fill a genuine gap by importing psychometric validity concepts into agentic evaluation and offering a deployable diagnostic framework. The strengths are real: the prescriptions are concrete and actionable, the four-rater reliability check on the coding instrument is a useful practice, the data and code are shared, and the threats-to-validity section is unusually candid. However, the headline numbers are not yet established. The model conflates reliability with validity, the multiplicative bound is explicitly not derived, and the empirical V3 floor is applied to graders of a structurally different kind. These are load-bearing issues because the manuscript's main conclusions—less than 5% valid signal, and benchmarks below the redesign threshold—follow directly from them.

major comments (4)
  1. [§7.1, Eq. (1); §5.3; §6.2] V3 is operationally defined as an IRR metric (e.g., κ or α), but Section 5.3 correctly states that agreement on a distorted signal is not validity. Defining V3 as IRR therefore makes Eq. (1) a bound on reliability, not on the 'overall construct validity of the evaluation pipeline' claimed in §7.1. The model needs a separate grader-vs-gold-standard component, or the text must consistently present V3 as a necessary-but-not-sufficient reliability coefficient.
  2. [Appendix B.3; §7.1] The inequality is stated as 'a conceptual model, not a proved mathematical proposition,' and Vtotal is not independently defined. Without an operational definition of Vtotal, the claimed bound is close to a definition of the product. The motivating SNR argument assumes that each Vi is an independent normalized signal-to-noise ratio; this is asserted, not derived. Since all quantitative conclusions depend on Eq. (1), the authors should supply a formal derivation and an independent definition of Vtotal, or explicitly reframe the result as a heuristic with stated conditions.
  3. [§7.2, Table 3; §10.4] The V3 = 0.05 floor from Gurram's substring-matching κ = 0.049 is applied to SWE-bench, OSWorld, and WebArena, which use deterministic test-suite execution or rule-based checkers rather than LLM substring grading. The manuscript itself says in §10.4 that generalization to deterministic graders is 'an open empirical question.' For these benchmarks the 'less than 5% valid signal' conclusion does not follow; a plausible V3 = 0.9 would give SWE-bench Vtotal ≈ 0.53 and put it above the redesign threshold. Additionally, the 0.049 value is labeled a 'floor' but used as an upper-bound point estimate; if it is a lower bound, the claimed upper bound is not licensed. Table 3 needs per-grader-type estimates or a restricted claim.
  4. [§6.2, Table 2] The central empirical claim that about 82% of papers misuse or omit IRR comes from a purposive, non-representative sample of 55 papers, with no confidence interval or sensitivity analysis. The authors acknowledge selection bias, but the abstract and §7.2 cite '82%' as a precise prevalence. Please report binomial uncertainty and show how the figure changes under alternative sampling frames or category definitions.
minor comments (4)
  1. [Abstract; Figure 2] The abstract says a pipeline retaining 70% validity at each stage is 'at most 34% valid,' which is 0.7^3, while the stated range 0.22–0.54 comes from the broader V1/V2/V3 ranges in Figure 2. Clarify that these are different scenarios.
  2. [Table 4; §10.4] The phrase 'even under the most optimistic V3 = 0.50 scenario' is overstated, since V3 = 0.50 is just an intermediate point; a reader could assign V3 > 0.50. Rephrase to 'under the specified sensitivity values.'
  3. [§5.1; §10.1] V2 is operationalized as 1 − ECE/100 rather than the ICC(A,1) used in the definition. The approximation is acknowledged, but consider reporting both quantities when primary data are available.
  4. [§6.2] The sentence 'approximately 10 (18%)' mixes a point count with a rounded percentage; give exact counts in the text or move the full distribution to Appendix A to avoid apparent imprecision.

Circularity Check

2 steps flagged

The headline Vtotal bound is asserted rather than derived, and the 'less than 5% valid signal' finding is the V3=0.05 input restated as an output; the survey evidence itself is independent.

specific steps
  1. self definitional [Section 7.1, Eq. (1); Appendix B.3]
    "The overall construct validity of the evaluation pipeline is bounded above by: Vtotal ≤ V1 × V2 × V3 (1) ... We state the following as a conceptual model, not a proved mathematical proposition."

    Vtotal is never given an independent operational definition; the only formal content assigned to 'overall construct validity' is the inequality itself. Appendix B.3 concedes there is no derivation ('conceptual model, not a proved mathematical proposition'). Eq. (1) is therefore a stipulation of what Vtotal means rather than a bound derived from the definitions in B.1-B.2. Every downstream number in the paper (0.34, 0.027, 0.049, 'less than 5%') is the product of the layer inputs, so the 'compounding' result cannot fail: it is true by construction.

  2. fitted input called prediction [Section 7.2, Table 3, 'Three findings'; Section 10.4]
    "For benchmarks with no reported human IRR, we use the empirically measured automated grader validity from Gurram[25]: substring matching achieves κ = 0.049 against human annotation on comparable tasks (chance-level). This is the empirical V3 floor for automated-only evaluation. ... every benchmark without human IRR (V3 ≈ 0.05) produces Vtotal < 0.05 regardless of V1 and V2: less than 5% of valid signal survives the judgment layer alone."

    The finding 'Vtotal < 0.05 for every benchmark without human IRR' is a logical consequence of assigning V3 = 0.05 to those benchmarks; it holds regardless of V1 and V2 by construction, so it is not an empirical discovery about SWE-bench, OSWorld, or τ-bench. The paper itself concedes in §10.4 that generalizing the single-study κ=0.049 to deterministic test-suite graders 'is an open empirical question,' yet Table 3 treats it as a floor for exactly those graders. The headline 'less than 5% valid signal' is the input value renamed as an output.

full rationale

The paper's empirical survey (82% IRR misuse, the κ/α failure-mode distribution, V1 and V2 evidence) is drawn from external sources and is not circular; there are no load-bearing self-citations. However, the central quantitative result is not derived from the definitions: Vtotal is never independently defined, so Eq. (1) is a conceptual stipulation (acknowledged in Appendix B.3), and the Table 3/4 Vtotal values are arithmetic products of the chosen Vi. The 'less than 5%' conclusion is forced by assigning V3=0.05 to every benchmark without reported human IRR, an extrapolation the paper itself flags as an open empirical question for deterministic graders. These two steps make the headline numerical claims partially circular/definitional, even though the survey evidence and V1/V2 findings retain independent content. The score reflects partial, not total, circularity.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 1 invented entities

The model depends on borrowed estimates for V1 and V2, a single borrowed constant for the V3 floor, hand-assigned ranges that do not consistently produce the abstract's headline range, and an unproved multiplicative decomposition. The most load-bearing assumptions are the product form of the bound and the treatment of IRR as a validity factor.

free parameters (5)
  • V1 point/range estimates = 0.60-0.80; per-benchmark 0.62, 0.72, 0.59, 0.97
    Taken from Zhu et al. validity-failure rates; not measured directly; drives the compounding results.
  • V2 estimates (simulator calibration) = 0.55-0.90; SAE = 0.88, AAVE = 0.80
    Approximated as 1 - ECE/100 from Seshadri et al.; the paper acknowledges this approximation in Section 10.1.
  • V3 floor for unvalidated automated graders = 0.05 (kappa = 0.049)
    Single study (Gurram 2026) used as the floor for all benchmarks without human IRR; not measured per benchmark.
  • V3 range for Figure 2 = 0.50-0.75
    Hand-assigned range that appears to generate the abstract's 0.22-0.54 claim, although the stated lower product is 0.165.
  • Proposed reliability thresholds = alpha >= 0.70 and alpha >= 0.80
    Labeled 'we propose'; no decision-theoretic derivation, while alpha >= 0.67 is taken from Krippendorff.
axioms (5)
  • ad hoc to paper The three layer validity coefficients are independent normalized signal-to-noise ratios whose product bounds total validity.
    Stated in Appendix B.3 as a conceptual model, not a proved mathematical proposition; this is the core unproved premise of Equation (1).
  • domain assumption An ideal construct distribution D_C* exists and induces a task distribution for total-variation comparison.
    Section 7.1 and B.1 define V1 via TV distance to an unobservable D_C*.
  • domain assumption IRR or agreement coefficients can serve as one factor of construct validity.
    V3 is defined as an IRR value, yet Section 5.3 states high agreement on distorted inputs is not validity; the paper does not resolve this tension.
  • domain assumption V2 can be approximated by 1 - ECE/100 from a published simulator-versus-human study.
    Section 10.1 acknowledges this is an approximation that holds only under uniform miscalibration.
  • ad hoc to paper The kappa = 0.049 grader-versus-human agreement from one study generalizes to all unvalidated agentic benchmarks.
    Applied in Table 3 to benchmarks not measured in Gurram 2026; the sensitivity analysis varies V3 but does not validate the floor per benchmark.
invented entities (1)
  • V_total (overall pipeline construct validity) no independent evidence
    purpose: Single number claimed to bound the valid signal remaining after task generation, simulation, and judgment layers.
    No independent operationalization is provided; V_total is only characterized by the inequality Vtotal <= V1 x V2 x V3, so the 'bound' is effectively the model definition.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Measurement Without Validity: The Compounding Reliability Problem in Agentic AI Evaluation." pith.science (2026). https://pith.science/paper/FJVMDCI5

@misc{pith2026260800794,
  author       = {Pith},
  title        = {Pith review of: Measurement Without Validity: The Compounding Reliability Problem in Agentic AI Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FJVMDCI5}},
  note         = {Machine review of arXiv:2608.00794}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Agentic AI evaluation pipelines produce benchmark scores that justify deployment decisions, safety certifications, and regulatory compliance claims. No formal framework has yet characterized how validity degrades across the stages of these pipelines. We present a three-layer compounding validity model, V_total <= V_1 x V_2 x V_3, that captures multiplicative degradation across task generation (V_1), human-simulator calibration (V_2), and automated judgment (V_3). Under empirically grounded estimates, a pipeline retaining 70% validity at each stage is at most 34% valid against the intended construct (range 0.22--0.54). We validate the model against a structured survey of 55 published agentic evaluation papers, finding that approximately 82% apply structurally mismatched, incomplete, or absent inter-rater reliability (IRR) metrics---a pattern consistent with systematic V_3 collapse. We further identify empirical evidence of V_1 failures (task validity flaws in 7 of 10 popular benchmarks) and V_2 miscalibration (up to 9 percentage points inter-simulator variance, with systematic demographic disparities for non-Standard American English speakers). We derive eight prescriptions grounded in psychometric science and domain-stratified reliability thresholds (ICC>=0.70; alpha >= 0.67/0.70/0.80 by consequence level) that practitioners and benchmark authors can apply immediately. The framework provides a tractable knowledge-based tool for diagnosing and correcting evaluation pipeline validity before deployment decisions are made.

Figures

Figures reproduced from arXiv: 2608.00794 by William Caban.

Figure 1
Figure 1. Figure 1: IRR metric selection. Left: decision tree based on rater design, matrix com [PITH_FULL_IMAGE:figures/full_fig_p018_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Three-layer agentic evaluation pipeline with validity estimates. Each layer [PITH_FULL_IMAGE:figures/full_fig_p020_2.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

41 extracted references · 18 canonical work pages · 3 internal anchors

  1. [1]

    L. J. Cronbach, P. E. Meehl, Construct validity in psychological tests, Psychological Bulletin 52 (1955) 281–302. doi:10.1037/h0041570

    work page doi:10.1037/h0041570 1955

  2. [2]

    Y. L. Liu, S. L. Blodgett, J. C. K. Cheung, Q. V. Liao, A. Olteanu, Z. Xiao, ECBD: Evidence-centered benchmark design for NLP, in: Proceedings of the 62nd Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), 2024, pp. 16349–16365. doi:10.18653/v1/2024.acl-long.861

    doi:10.18653/v1/2024.acl-long.861 2024

  3. [3]

    K. J. Meimandi, G. Aránguiz-Dias, G. R. Kim, L. Saadeddin, A. Grif- fith, M. J. Kochenderfer, The measurement imbalance in agentic AI 36 evaluation undermines industry productivity claims, arXiv preprint arXiv:2506.02064, 2025

    work page arXiv 2025

  4. [4]

    Prandi, V

    M. Prandi, V. Suriani, F. Pierucci, M. Galisai, D. Nardi, P. Bisconti, Bench-2-CoP: Can we trust benchmarking for EU AI compliance?, arXiv preprint arXiv:2508.05464, 2025

    Pith/arXiv arXiv 2025

  5. [5]

    Cohen, A coefficient of agreement for nominal scales, Educa- tional and Psychological Measurement 20 (1960) 37–46

    J. Cohen, A coefficient of agreement for nominal scales, Educa- tional and Psychological Measurement 20 (1960) 37–46. doi:10.1177/ 001316446002000104

    work page 1960

  6. [6]

    J. L. Fleiss, Measuring nominal scale agreement among many raters, Psychological Bulletin 76 (1971) 378–382. doi:10.1037/h0031619

    doi:10.1037/h0031619 1971

  7. [7]

    Krippendorff, Content Analysis: An Introduction to Its Methodology, 2nd ed., Sage Publications, Thousand Oaks, CA, 2004

    K. Krippendorff, Content Analysis: An Introduction to Its Methodology, 2nd ed., Sage Publications, Thousand Oaks, CA, 2004

    work page 2004

  8. [8]

    J. R. Landis, G. G. Koch, The measurement of observer agreement for categorical data, Biometrics 33 (1977) 159–174. doi:10.2307/2529310

    doi:10.2307/2529310 1977

  9. [9]

    Mitchell, S

    M. Mitchell, S. Wu, A. Zaldivar, P. Barnes, L. Vasserman, B. Hutchinson, E. Spitzer, I. D. Raji, T. Gebru, Model cards for model reporting, in: Proceedings of the Conference on Fairness, Accountability, and Transparency, 2019, pp. 220–229. doi:10.1145/3287560.3287596

    arXiv 2019

  10. [10]

    R. Dhar, D. S. Villegas, A. Karamolegkou, A. Schiavone, Y. Yuan, X. Chen, J. Li, S. Frank, L. De Grazia, M. Swain, et al., EvalCards: A framework for standardized evaluation reporting, arXiv preprint arXiv:2511.21695, 2025. 37

    arXiv 2025

  11. [11]

    Hayes-Roth, D

    F. Hayes-Roth, D. A. Waterman, D. B. Lenat, Building Expert Systems, Addison-Wesley, Reading, MA, 1983

    work page 1983

  12. [12]

    B. G. Buchanan, E. H. Shortliffe (Eds.), Rule-Based Expert Systems: The MYCIN Experiments of the Stanford Heuristic Programming Project, Addison-Wesley, Reading, MA, 1984

    work page 1984

  13. [13]

    A. Z. Jacobs, H. Wallach, Measurement and fairness, in: Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, 2021, pp. 375–385. doi:10.1145/3442188.3445901

    arXiv 2021

  14. [14]

    James, Counting on consensus: Selecting the right inter-annotator agreement metric for NLP annotation and evaluation, arXiv preprint arXiv:2603.06865, 2026

    J. James, Counting on consensus: Selecting the right inter-annotator agreement metric for NLP annotation and evaluation, arXiv preprint arXiv:2603.06865, 2026

    arXiv 2026

  15. [15]

    Y. Zhu, T. Jin, Y. Pruksachatkun, A. Zhang, S. Liu, S. Cui, S. Kapoor, S. Longpre, K. Meng, R. Weiss, et al., Establishing best practices for building rigorous agentic benchmarks, arXiv preprint arXiv:2507.02825, 2025

    Pith/arXiv arXiv 2025

  16. [16]

    Seshadri, S

    P. Seshadri, S. Cahyawijaya, A. Odumakinde, S. Singh, S. Goldfarb- Tarrant, Lost in simulation: LLM-simulated users are unreliable proxies for human users in agentic evaluations, in: The Fourteenth International Conference on Learning Representations, 2026. ArXiv:2601.17087

    arXiv 2026

  17. [17]

    Zhuge, C

    M. Zhuge, C. Zhao, D. R. Ashley, W. Wang, D. Khizbullin, Y. Xiong, Z. Liu, E. Chang, R. Krishnamoorthi, Y. Tian, Y. Shi, V. Chandra, J. Schmidhuber, Agent-as-a-judge: Evaluate agents with agents, in: 38 Proceedings of the 42nd International Conference on Machine Learning,

    work page

  18. [18]

    Models used: NVIDIA Nemotron-Ultra- 550B, Google Gemma-4-31B, Alibaba Qwen3-80B-A3B

    OpenRouter, Openrouter: A unified api for large language models, https://openrouter.ai, 2026. Models used: NVIDIA Nemotron-Ultra- 550B, Google Gemma-4-31B, Alibaba Qwen3-80B-A3B

    work page 2026

  19. [19]

    Yao, et al.,τ-bench: A benchmark for tool-agent-user interaction in real-world domains, arXiv preprint arXiv:2406.12045, 2024

    S. Yao, et al.,τ-bench: A benchmark for tool-agent-user interaction in real-world domains, arXiv preprint arXiv:2406.12045, 2024

    Pith/arXiv arXiv 2024

  20. [20]

    Xie, et al., OSWorld: Benchmarking multimodal agents for open-ended tasks in real computer environments, arXiv preprint arXiv:2404.07972, 2024

    T. Xie, et al., OSWorld: Benchmarking multimodal agents for open-ended tasks in real computer environments, arXiv preprint arXiv:2404.07972, 2024

    Pith/arXiv arXiv 2024

  21. [21]

    C. E. Jiménez, et al., SWE-bench: Can language models resolve real- world GitHub issues?, arXiv preprint arXiv:2310.06770, 2024

    Pith/arXiv arXiv 2024

  22. [22]

    Mialon, et al., GAIA: A benchmark for general AI assistants, arXiv preprint arXiv:2311.12983, 2024

    G. Mialon, et al., GAIA: A benchmark for general AI assistants, arXiv preprint arXiv:2311.12983, 2024

    Pith/arXiv arXiv 2024

  23. [23]

    S. Zhou, F. F. Xu, H. Zhu, X. Zhou, R. Lo, A. Sridhar, X. Cheng, Y. Bisk, D. Fried, U. Alon, G. Neubig, WebArena: A realistic web environment for building autonomous agents, in: The Twelfth International Conference on Learning Representations, 2024. URL:https://openreview.net/ forum?id=oKn9c6ytLx

    work page 2024

  24. [24]

    Barres, H

    V. Barres, H. Dong, S. Ray, X. Si, K. Narasimhan,τ 2-bench: Evaluating conversational agents in a dual-control environment, arXiv preprint arXiv:2506.07982, 2025. 39

    Pith/arXiv arXiv 2025

  25. [25]

    Evaluating Tool-Using Language Agents: Judge Reliability, Propagation Cascades, and Runtime Mitigation in AgentProp-Bench

    B. Gurram, Evaluating tool-using language agents: Judge reliability, propagation cascades, and runtime mitigation in AgentProp-Bench, arXiv preprint arXiv:2604.16706, 2026. Under review

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  26. [26]

    El Hattami, M

    A. El Hattami, M. Thakkar, N. Chapados, C. Pal, WebArena Ver- ified: Reliable evaluation for web agents, in: Scaling Environments for Agents Workshop, NeurIPS 2025, 2025. URL:https://openreview. net/forum?id=94tlGxmqkN

    work page 2025

  27. [27]

    S. Fan, X. Ye, Y. Huo, Z.-Y. Chen, Y. Guo, S. Yang, W. Yang, S. Ye, J. Chen, H. Chen, X. Cong, Y. Lin, AgentProcessBench: Diagnosing step- levelprocessqualityintool-usingagents, arXivpreprintarXiv:2603.14465,

    Pith/arXiv arXiv

  28. [28]

    S. Han, G. Titericz Junior, T. Balough, W. Zhou, Judge’s verdict: A com- prehensive analysis of LLM judge capability through human agreement, arXiv preprint arXiv:2510.09738, 2025

    arXiv 2025

  29. [29]

    Movva, P

    R. Movva, P. W. Koh, E. Pierson, Annotation alignment: Comparing LLM and human annotations of conversational safety, in: Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024. ArXiv:2406.06369

    Pith/arXiv arXiv 2024

  30. [30]

    Chiang, L

    W.-L. Chiang, L. Zheng, Y. Sheng, A. N. Angelopoulos, T. Li, D. Li, H. Zhang, B. Zhu, M. Jordan, J. E. Gonzalez, I. Stoica, Chatbot arena: An open platform for evaluating LLMs by human preference, arXiv preprint arXiv:2403.04132, 2024. 40

    Pith/arXiv arXiv 2024

  31. [31]

    Ouyang, J

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al., Training language models to follow instructions with human feedback, in: Advances in Neural Information Processing Systems, volume 35, 2022, pp. 27730–27744. ArXiv:2203.02155

    Pith/arXiv arXiv 2022

  32. [32]

    Dialectal Toxicity Detection: Evaluating LLM-as-a-Judge Consistency Across Language Varieties

    F. Faisal, M. M. Rahman, A. Anastasopoulos, Dialectal toxicity detection: Evaluating LLM-as-a-judge consistency across language varieties, in: Findings of the Association for Computational Linguistics: EMNLP 2025, 2025. ArXiv:2411.10954; citation key reflects arXiv preprint date (Nov 2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  33. [33]

    Spiliopoulou, R

    E. Spiliopoulou, R. Fogliato, H. Burnsky, T. Soliman, J. Ma, G. Horwood, M. Ballesteros, Play favorites: A statistical method to measure self-bias in LLM-as-a-judge, arXiv preprint arXiv:2508.06709, 2025

    Pith/arXiv arXiv 2025

  34. [34]

    Wataoka, T

    K. Wataoka, T. Takahashi, R. Ri, Self-preference bias in LLM-as-a-judge, in: NeurIPS 2024 Safe Generative AI Workshop, 2024. ArXiv:2410.21819

    Pith/arXiv arXiv 2024

  35. [35]

    Expert Evaluation and the Limits of Human Feedback in Mental Health AI Safety Testing

    K. Jafari, P. U. N. Rust, D. Eddy, R. Fraser, N. Vasan, D. Djordjevic, A. Dadlani, M. Lamparth, E. Kim, M. J. Kochenderfer, Expert evaluation and the limits of human feedback in mental health AI safety testing, arXiv preprint arXiv:2601.18061, 2026. Under review

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  36. [36]

    Wohlin, P

    C. Wohlin, P. Runeson, M. Höst, M. C. Ohlsson, B. Regnell, A. Wesslén, Experimentation in Software Engineering, Springer, 2012. doi:10.1007/ 978-3-642-29044-2. 41

    work page 2012

  37. [37]

    Jedlitschka, M

    A. Jedlitschka, M. Ciolkowski, D. Pfahl, Reporting experiments in software engineering, in: F. Shull, J. Singer, D. I. Sjøberg (Eds.), Guide to Advanced Empirical Software Engineering, Springer, 2008, pp. 201–228. doi:10.1007/978-1-84800-044-5_8. Appendix A. Literature Scan: Coding Criteria and Category Sum- mary This appendix documents the methodology an...

    doi:10.1007/978-1-84800-044-5_8 2008

  38. [38]

    IRR metric reported: Cohen’sκ, Fleiss’ κ, Krippendorff’s α, ICC (form specified or unspecified), percentage agreement only, or no metric reported

    work page

  39. [39]

    Rater design: number of raters; whether rater identity was fixed across items or varied (rotating pool, crowdsourced, or multiple independent runs of the same model)

    work page

  40. [40]

    Measurement scale: binary, nominal (unordered categories), ordinal or ternary (ordered but discrete), or continuous

    work page

  41. [41]

    Correct” = structurally valid metric with explicit rationale. “Failure mode

    Structural validity: whether the reported metric’s structural assump- tions were satisfied by the rater design and measurement scale. Coding 42 applied the following rules: Cohen’sκ for > 2raters or rotating identity → mismatch; Fleiss’ κ for ordinal or continuous scale→ partial mismatch; percentage agreement without chance correction→ incom- plete; no IR...

    work page 2025

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.