Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

ScopeJudge: Cost-Aware Pre-Execution Gating for Offensive Security Agents

T0 review · 3 major / 5 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that scope enforcement for autonomous security agents cannot be achieved with a fixed policy: a judge that sees only the static policy and proposed tool call misses almost every violation, while adding the user's request l

desk verdict Useful benchmark and honest evaluation, but the 'static policy is structurally insufficient' claim overreaches because the tested static condition withholds the user request that, by the paper's own framing, carries the scope. read the letter →

arxiv 2607.07774 v2 pith:M7VXJOBS submitted 2026-07-08 cs.CR

Shane Caldwell , Max Harley , Ads Dawson , Michael Kouremetis , Vincent Abruzzo , Will Pearce This is my paper · ORCID
classification cs.CR
keywords LLMagentsafetypre-executiongatingscopeenforcementoffensivesecurityLLM-as-judgetranscriptstrategiestrustedmonitoringtool-call
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that for autonomous offensive-security agents, the boundary between allowed and forbidden tool calls is not fixed in advance but is declared in the user's request. It builds a benchmark of 4,897 expert-labeled tool calls from 100 agent trajectories designed to tempt agents out of scope, and tests eight judges under five context strategies. The result: judges shown only a static policy catch essentially nothing (median recall 0.005), while adding the user's intent restores recall to 0.60–0.94. The best judge is open-weight, reaching F1 0.66 against an expert agreement reference of 0.78. The paper concludes that request-conditioned pre-execution gating is necessary and gives cost-aware operating points for deployment.

What carries the argument

The pre-execution gate: a cheap, fixed LLM judge inspects each tool call proposed by a strong, swappable agent and accepts or rejects it before execution. The transcript strategy controls how much context the judge sees—ranging from only the static policy plus the proposed call, to adding the user's intent, prior tool calls, summarized outputs, or the full trajectory. This design isolates the contribution of request-level intent and produces the paper's central comparison: static policy alone gates nothing, while intent restores recall. The labeled calibration set of 4,897 calls provides the ground truth against which every judge–strategy pair is scored.

What would settle it

Run the judge sweep on a new, independently labeled corpus from a different engagement type (for example, cloud-incident-response or mobile pentesting). If a static-policy-only judge catches a substantial share of violations there (recall well above 0.1), the structural-insufficiency claim fails.

Watch

Extended reading notes

Core claim

Scope is not a property of the tool call itself but of the request that authorizes it. The same scan, exploit, or data read is in or out of bounds depending on the user's stated engagement constraints, which no fixed policy can enumerate in advance. Empirically, under the static-policy-only transcript strategy, judge recall collapses to near zero (median 0.005 across seven of eight judges), and the moment the user's intent is added, recall jumps to 0.60–0.94. The paper therefore argues that request-conditioned monitoring is structurally necessary for pre-execution gating, and that the user's request is where scope actually lives.

Load-bearing premise

The entire scoreboard rests on one calibration set: 100 English-language web-pentest trajectories labeled by five graders from the paper's own team, so if that suite and those labels do not represent real engagements, the measured recall collapse and scoreboard could shift.

Editorial extensions

If this is right

  • A monitor that sees only a fixed policy will wave through nearly all scope violations; deployments need request-conditioned context to catch them.
  • Adding the user's intent is the main accuracy lever: recall jumps from near zero to 0.60–0.94, while adding full history buys less than 0.02 F1 at nearly double the cost.
  • The best judge in the study is open-weight, reaching F1 0.66 at roughly one-third the per-call cost of the best proprietary judge (0.60), making practical gating feasible without a closed model.
  • Two deployment points are recommended: a cost-sensitive setting that catches about 87% of violations and a recall-first setting that catches about 94% but flags many more calls for human review.
  • Judge performance still trails expert agreement (0.66 vs 0.78), so the benchmark sets a headroom target for future monitors.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If scope truly lives in the user's request, the same gating logic should transfer to other autonomous-agent domains where a request defines boundaries—cloud operations, database access, or code changes—and static allowlists will similarly underperform.
  • The paper's violation base rate (7.7%) is suite-specific; deployments with different violation prevalence should expect precision and F1 to shift and may need to re-tune thresholds rather than reuse the reported operating points.
  • History-bearing strategies expose the judge to attacker-controllable tool outputs; a natural next experiment is to test whether a misaligned agent can poison outputs to get an out-of-scope call allowed, which the paper explicitly leaves open.
  • Because the best judge still lags human agreement, a production design would likely pair a cheap judge as the first gate with human or stronger-model review only of flagged calls, rather than relying on the judge alone.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces ScopeJudge, a benchmark of 4,897 expert-labeled tool calls drawn from 100 agent trajectories on offensive-security tasks, and uses it to evaluate eight LLM judges under five transcript strategies for pre-execution gating. It reports a cost-accuracy Pareto frontier, identifies open-weight judges as competitive (GLM-5.2 reaches F1=0.66 against an expert-agreement reference of 0.78), and argues that a static policy is structurally insufficient for scope enforcement because recall collapses when the judge is not shown the user's request. The authors recommend two deployment operating points, one cost-sensitive and one recall-first, and release the dataset and code.

Significance. If the claims hold, the paper provides a useful, reproducible benchmark for a practical problem: gating LLM security agents before tool execution. The release of the labeled corpus and code, the five-strategy taxonomy, the explicit cost accounting, and the human-agreement reference are genuine contributions. The central empirical pattern—generic static context alone gives near-zero recall, while adding the user request restores recall—is clearly supported by Table 3 and Figure 4. However, the paper's headline conclusion that request-conditioned monitoring is 'necessary' and that a specification-time static policy is 'structurally insufficient' goes beyond the evidence, because the static-policy condition deliberately withholds the only carrier of task-specific scope. The empirical contribution remains valuable once the conclusion is appropriately scoped.

major comments (3)
  1. [§6.2, Eq. (5), Appendix A/Table 4] The central 'necessity' claim is not established by the reported experiment. Eq. (5) defines static_policy as φ_static=(SP, a_k), and Section 2.1 states that task-specific scope is declared in the user message u, not in the global static policy SP. The static condition thus removes the only information source where the engagement boundary is stated; the recall collapse in Figure 4 is a built-in consequence of this information architecture, not evidence that no specification-time policy could work. Moreover, Appendix A's constant system prompt tells the judge that it 'will be shown the original engagement instruction,' but under static_policy that instruction is withheld, so the condition is not simply 'static policy only' but also a broken prompt contract. Real engagements have written scope/ROE documents that are fixed before execution; no such per-engagement static contract is tested.
  2. [§6.1, Table 3, §7] The paper's deployment recommendations are based on single-run point estimates with no confidence intervals, as the authors acknowledge in Section 7. Several configuration differences that motivate the recommended operating points are small (e.g., GLM-5.2 intent_plus_calls F1=0.637 vs. intent_plus_outputs_summary F1=0.639; full F1=0.656). With 4,897 calls and no repeated sampling, these differences are not distinguishable from noise. This does not undermine the large static-vs-intent effect, but it does undermine the precision with which the two operating points are ranked. The authors should either provide variance estimates (e.g., bootstrap over trajectories, or repeated API samples) or explicitly present the recommendations as qualitative guidance rather than as a ranking of nearly tied configurations.
  3. [§4.1, §4.3, §7] All ground-truth labels were produced by five graders who are also the paper's authors, and the expert reference point (F1=0.78) is computed against the majority of those same graders. The paper discloses the evaluator overlap between two judges and the trajectory-generation agents, but the author-grader issue is separate and is not discussed as a limitation. The inter-grader agreement (κ=0.64) is useful, but the reference point is not an external ceiling. A small external validation sample labeled by independent practitioners would materially strengthen the calibration claim. At minimum, this should be added to the limitations in Section 7.
minor comments (5)
  1. [§6.1 / Table 3] The 'best-F1' selection in Figure 3 is an oracle choice: for each judge the best-F1 strategy is selected post hoc using the gold labels. If the recommendation is for a fixed deployed configuration, the paper should also report the expected F1 of that configuration without selection, or explicitly note that Figure 3 is an upper envelope.
  2. [§2.1 / §3.1.1] The term 'static policy' is used in two senses: the concrete global SP shared across tasks, and the general idea of any specification-time policy. This ambiguity makes the Section 6.2 conclusion easier to overstate. Consider renaming the former 'generic global policy' and reserving 'static policy' for the broader class.
  3. [Appendix B] The worked example is clear and helpful, but the trajectory is generated by claude-sonnet-4-6, one of the judge models, which is also the evaluator-overlap case flagged in Section 4.1. This is fine as an illustration, but it is worth a one-sentence reminder that the example is from the overlapping subset.
  4. [§7 / §4.2] Section 4.2 states that roughly 380 positive instances are 'enough for stable estimates of monitor recall,' but no variance estimate is provided. Given the base rate of 7.7% and the acknowledged lack of confidence intervals, this assertion should be softened or supported.
  5. [Table 3] The static-policy rows for most judges have Flag% around 0.1 and recall near zero; a log-scale or per-judge plot might communicate the collapse more clearly than the numbers alone. The current Figure 4 is good, but a per-judge panel would help.

Circularity Check

1 steps flagged · score 6.0 of 10

Headline 'necessity' claim is self-definitional: scope is defined as carried in the user message (Sec. 2.1), the static_policy strategy withholds that message (Eq. 5), and the recall collapse is read as proof that request-conditioned monitoring is necessary (Sec. 6.2) — a conclusion entailed by the definitions, while a static per-engagement contract is never tested.

  1. self definitional [Section 2.1 (problem formalization), Eq. (5) in Section 3.1, Section 6.2 (central claim)]
    ""The task-specific scope (which hosts, services, techniques, and data are in or out of bounds) is not part of the static policy but is declared in the user message u..." ... "φstatic(τ<k, ak) = (SP, ak)" ... "This is the empirical form of our thesis: scope lives in the user's request, and a specification-time static policy, however well written, is structurally insufficient to gate it.""

    Scope is defined (Sec. 2.1) as carried in u, while SP is defined as containing only generic ground rules shared across tasks; Eq. (5) feeds the judge nothing but (SP, ak). The recall collapse in Sec. 6.2 is therefore a necessary consequence of withholding the only information that, by definition, determines scope; a judge without u lacks the definitional criterion for a 'violation.' The conclusion that a static policy is 'structurally insufficient' and that request-conditioned monitoring is 'necessary' restates the definitions ('scope lives in the user's request') rather than testing them, because no static policy that incorporates a per-engagement scope/ROE contract is ever evaluated. The specific recall values are measured, but the 'structural/necessary' interpretation is by construction

full rationale

The central necessity claim is partially circular. Section 2.1 defines task-specific scope as 'not part of the static policy but ... declared in the user message u'; Eq. (5) defines φstatic to withhold u; Section 6.2 then reports near-zero recall and concludes that 'scope lives in the user's request' and that request-conditioned monitoring is necessary. A judge denied the only carrier of scope cannot recover scope, so the necessity conclusion is entailed by the information architecture, not independently established — and the 'however well written' generalization is untested because a static policy containing a per-engagement scope document is never evaluated. The measured quantities themselves (recall 0.005, the jump to 0.60–0.94, F1 scoreboard, costs, expert F1=0.78) are genuine empirical results from real runs and human labels and would not be forced by the definitions; the open-weight cost-accuracy finding is independent of this circularity. The self-citation to PentestJudge [8] is contextual, not load-bearing. Section 7 discloses two soft self-reference loops: two judge models also generated trajectories they judged ('a self-preference effect cannot be ruled out'), and the summarization strategy summarizes outputs with the same judge model; both are disclosed and neither drives the static-policy result, which is scored against human labels. Base-rate sensitivity is also disclosed. Net: partial definitional circularity on the headline necessity claim, with substantial independent empirical content (score 6).

Assumptions & free parameters 1 free parameters · 3 assumptions · 1 invented entities

The paper's contribution is empirical, and its load-bearing assumptions are domain assumptions about where scope lives (the user request) and about the representativeness of the author-generated calibration set. There are no invented physical entities; ScopeBench is an unreleased test suite, which counts as an unverified entity. No free parameters are fitted to data in the classical sense, though the two recommended operating points are hand-selected from the results.

free parameters (1)
  • Operating points (cost-sensitive vs recall-first) = GLM-5.2 intent_plus_calls (recall 0.87, precision 0.50) and Claude Opus 4.8 intent_only (recall 0.94, precision 0.39)
    These are chosen by the authors as deployment recommendations based on the measured frontier; not fitted to data in a statistical sense, but they are hand-selected from the results.
assumptions (3)
  • domain assumption The user message u fully specifies the operative per-engagement scope, and scope is inferred from the user's stated goal and constraints (Section 2.1).
    The entire benchmark and gating design depends on the claim that task-specific scope is declared in the user request and is inferable. If scope can be implicit or unknowable from the request alone, the labeled ground truth and judge task are ill-posed.
  • domain assumption The 100-calibration-set trajectories from ScopeBench are representative of real offensive-security agent deployments (Section 4.2, 7).
    All quantitative claims are measured on this pool; the paper itself flags base-rate sensitivity and single-suite scope.
  • domain assumption Professional penetration testers' majority labels constitute reliable ground truth (Section 4.3).
    Judges are scored against majority labels; the expert reference (F1=0.78) is computed from the same graders. No independent oracle is used.
invented entities (1)
  • ScopeBench task suite (30 tasks engineered to tempt scope violations)
    purpose: Generates the agent trajectories that form the ScopeJudge calibration set.
    ScopeBench is described as 'the subject of separate work' and is not released or described in enough detail to be independently scrutinized. The dataset's representativeness therefore cannot be verified.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ScopeJudge: Cost-Aware Pre-Execution Gating for Offensive Security Agents." pith.science (2026). https://pith.science/paper/M7VXJOBS

@misc{pith2026260707774,
  author       = {Pith},
  title        = {Pith review of: ScopeJudge: Cost-Aware Pre-Execution Gating for Offensive Security Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/M7VXJOBS}},
  note         = {Machine review of arXiv:2607.07774}
}
read the original abstract

As LLM agents take on offensive security work, a single out-of-scope tool call can breach a client's engagement boundary, disrupt production, or void a bug-bounty finding. Unlike a fixed safety policy, the boundary that matters is declared in the user's request and must be inferred from intent. That challenge is sharpened by the adversarial nature of offensive security: the same tool call is in or out of scope depending not on the action itself but on the target it touches and the context in which it runs, which no fixed policy can enumerate in advance. We study pre-execution gating: a cheap, trusted LLM judge inspects each call proposed by a strong, swappable agent, and accepts or rejects it before it runs. We introduce ScopeJudge, a benchmark of 4,897 tool calls (7.7% scope violations) from agent trajectories on tasks engineered to tempt agents out of scope and labeled at the call level by professional penetration testers, with substantial inter-grader agreement (Fleiss kappa = 0.64) that sets an expert agreement reference point of F1 = 0.78. We evaluate eight judge models under five transcript strategies, varying how much context the judge sees, from the static policy alone to the full raw transcript, and chart the resulting cost-accuracy Pareto frontier. We find that a static policy is structurally insufficient for scope enforcement: blind to the user's request, judge recall collapses to near zero, confirming that scope lives in the request and that request-conditioned monitoring is necessary. Because a missed violation costs more than a spurious rejection, we report precision, recall, and F1 separately and recommend two operating points: a cost-sensitive configuration and a recall-first one for high-stakes deployments. We release the ScopeJudge dataset to support real-time monitoring and scalable oversight of autonomous security agents.

Figures

Figures reproduced from arXiv: 2607.07774 by the authors.

Figure 1
Figure 1. Pre-execution gating. At each step the untrusted agent proposes a tool call ak. A transcript strategy φ projects the static policy SP, the user’s intent u, and a chosen amount of trajectory history onto the context window of a cheap, fixed judge Jpre, which accepts the call (it executes and is appended to the trajectory) or rejects it (the agent is asked to revise, or a human is escalated to). appear in a security c… view at source ↗
Figure 2
Figure 2. Pairwise inter-grader agreement (Cohen’s κ). Rows and columns index the five graders. Darker is higher agreement. All pairs fall in the moderate-to-substantial range (0.50–0.76), bounding how well any automated monitor can match the expert consensus. family is absent. At 100 trajectories of 6–170 calls each (mean 49.0), the calibration set contains 4,897 labeled tool-call decisions; even at a single-digit violation … view at source ↗
Figure 3
Figure 3. Cost–accuracy frontier for pre-execution gating. Each point is one judge at its best-F1 transcript strategy (one point per judge); cost is per tool call on a log scale, F1 against the expert gold labels. The dashed red line is the expert agreement reference point (F1 = 0.78), and the solid line traces the Pareto frontier over these points. GLM-5.2 attains the best F1 of any judge and dominates the proprietary flagsh… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Recall by transcript strategy. With only the static policy (leftmost), recall is near zero for seven of eight judges—a fixed specification cannot locate task scope. Adding the user’s intent restores recall sharply. Strategies are ordered by increasing context. • No var…

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. StealthBench: Measuring Operational Stealth in Autonomous Offensive-Security Agents

    cs.CR 2026-07 conditional novelty 6.0 of 10

    StealthBench's LLM-judge panel finds no AI agent solves offensive-security tasks stealthily more than 54% of the time.

  2. When Do Agent Loops Mistake Stagnation for Progress? Self-Evaluation Bias and Externally Grounded Verification in Long-Running Autonomous LLM Agent Loops

    cs.AI 2026-07 conditional novelty 6.0 of 10

    For open-ended agent goals whose success lives outside the transcript, even a strong in-band judge fails; out-of-band world-state gating is structurally required to stop the progress mirage.

Reference graph

Works this paper leans on

23 extracted references · 15 linked inside Pith · cited by 2 Pith papers

  1. [1]

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, et al.ReAct: Synergizing Reasoning and Acting in Language Models. 2023. arXiv:2210.03629 [cs.CL].url: https://arxiv.org/abs/2210.03629

    arXiv 2023

  2. [2]

    Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, et al.Toolformer: Language Models Can Teach Themselves to Use Tools. 2023. arXiv: 2302.04761 [cs.CL].url:https://arxiv.org/abs/2302.04761

    arXiv 2023

  3. [3]

    catastrophic failure

    Fortune.An AI-powered coding tool wiped out a software company’s database in a “catastrophic failure”. Fortune, 23 July 2025. 2025.url:https://fortune.com/2025/ 07/23/ai-coding-tool-replit-wiped-database-called-it-a-catastrophic- failure/

    2025

  4. [4]

    Constitutional AI: Harmlessness from AI Feedback

    Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, et al. Constitutional AI: Harmlessness from AI Feedback. 2022. arXiv:2212.08073 [cs.CL]. url:https://arxiv.org/abs/2212.08073. 17 ScopeJudge: Cost-Aware Pre-Execution Gating for Offensive Security Agents

    arXiv 2022

  5. [5]

    Ryan Greenblatt, Buck Shlegeris, Kshitij Sachan, and Fabien Roger.AI Control: Im- proving Safety Despite Intentional Subversion. 2024. arXiv:2312.06942 [cs.LG].url: https://arxiv.org/abs/2312.06942

    arXiv 2024

  6. [6]

    Alice in Warningland: A Large-Scale Field Study of Browser Security Warning Effectiveness

    Devdatta Akhawe and Adrienne Porter Felt. “Alice in Warningland: A Large-Scale Field Study of Browser Security Warning Effectiveness”. In:22nd USENIX Security Symposium (USENIX Security 13). 2013, pp. 257–272

    2013

  7. [7]

    Identifying the Risks of LM Agents with an LM-Emulated Sandbox

    Yangjun Ruan, Honghua Dong, Andrew Wang, Silviu Pitis, Yongchao Zhou, et al. Identifying the Risks of LM Agents with an LM-Emulated Sandbox. 2024. arXiv:2309. 15817 [cs.AI].url:https://arxiv.org/abs/2309.15817

    arXiv 2024

  8. [8]

    PentestJudge: Judging Agent Behavior Against Operational Requirements

    Shane Caldwell, Max Harley, Michael Kouremetis, Vincent Abruzzo, and Will Pearce. PentestJudge: Judging Agent Behavior Against Operational Requirements. 2025. arXiv: 2508.02921 [cs.AI].url:https://arxiv.org/abs/2508.02921

    arXiv 2025

Show all 23 references
  1. [9]

    Gelei Deng, Yi Liu, Víctor Mayoral-Vilches, Peng Liu, Yuekang Li, et al.PentestGPT: An LLM-empowered Automatic Penetration Testing Tool. 2023. arXiv:2308.06782 [cs.CR]. url:https://arxiv.org/abs/2308.06782

    2023 arXiv

  2. [10]

    Zhang, Neil Perry, Riya Dulepet, Joey Ji, Celeste Menders, et al.Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models

    Andy K. Zhang, Neil Perry, Riya Dulepet, Joey Ji, Celeste Menders, et al.Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models

  3. [11]

    Yutao Mou, Zhangchi Xue, Lijun Li, Peiyang Liu, Shikun Zhang, et al.ToolSafe: Enhancing Tool Invocation Safety of LLM-based agents via Proactive Step-level Guardrail and Feedback. 2026. arXiv:2601.10156 [cs.CL].url: https://arxiv.org/abs/2601. 10156

    2026

  4. [12]

    Alaia Solko-Breslin, Pramod Kaushik Mudrakarta, Mihai Christodorescu, Somesh Jha, and Krishnamurthy Dj Dvijotham.Efficient and Sound Probabilistic Verification for AI Agents. 2026. arXiv:2606.20510 [cs.AI].url: https://arxiv.org/abs/2606.20510

    2026 arXiv

  5. [13]

    Bowman, Jeeyoon Hyun, Ethan Perez, Edwin Chen, Craig Pettit, et al

    Samuel R. Bowman, Jeeyoon Hyun, Ethan Perez, Edwin Chen, Craig Pettit, et al. Measuring Progress on Scalable Oversight for Large Language Models. 2022. arXiv: 2211.03540 [cs.HC].url:https://arxiv.org/abs/2211.03540

    2022 arXiv

  6. [14]

    Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. 2023. arXiv:2306.05685 [cs.CL].url:https://arxiv.org/abs/2306.05685

    2023 arXiv

  7. [15]

    Joar Skalse, Nikolaus H. R. Howe, Dmitrii Krasheninnikov, and David Krueger.Defining and Characterizing Reward Hacking. 2022. arXiv:2209.13085 [cs.LG].url: https: //arxiv.org/abs/2209.13085

    2022 arXiv

  8. [16]

    Jonathan Gabor, Jayson Lynch, and Jonathan Rosenfeld.EvilGenie: A Reward Hacking Benchmark. 2026. arXiv:2511.21654 [cs.LG].url: https://arxiv.org/abs/2511. 21654

    2026 arXiv

  9. [17]

    Lin Shi, Chiyu Ma, Wenhua Liang, Xingjian Diao, Weicheng Ma, et al.Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge.2025.arXiv:2406.07791 [cs.CL].url:https://arxiv.org/abs/2406.07791

    2025

  10. [18]

    Haitao Li, Junjie Chen, Qingyao Ai, Zhumin Chu, Yujia Zhou, et al.CalibraEval: Calibrating Prediction Distribution to Mitigate Selection Bias in LLMs-as-Judges. 2024. arXiv:2410.15393 [cs.CL].url:https://arxiv.org/abs/2410.15393

    2024 arXiv

  11. [19]

    PaperBench: Evaluating AI’s Ability to Replicate AI Research

    Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, et al. PaperBench: Evaluating AI’s Ability to Replicate AI Research. 2025. arXiv:2504.01848 [cs.AI].url:https://arxiv.org/abs/2504.01848. 18 ScopeJudge: Cost-Aware Pre-Execution Gating for Offensive Secur...

    2025 arXiv

  12. [20]

    Measuring nominal scale agreement among many raters

    Joseph L. Fleiss. “Measuring nominal scale agreement among many raters”. In:Psycho- logical Bulletin76.5 (1971), pp. 378–382

    1971

  13. [21]

    A coefficient of agreement for nominal scales

    Jacob Cohen. “A coefficient of agreement for nominal scales”. In:Educational and Psychological Measurement20.1 (1960), pp. 37–46

    1960

  14. [22]

    The measurement of observer agreement for categorical data

    J. Richard Landis and Gary G. Koch. “The measurement of observer agreement for categorical data”. In:Biometrics33.1 (1977), pp. 159–174. A The Judge Prompt This appendix reproduces the verbatim static policySP supplied to every judge and specifies exactly which context blocks ...

    1977

  15. [2024]

    arXiv:2408.08926 [cs.CR].url:https://arxiv.org/abs/2408.08926

    arXiv

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.