Pith. sign in

REVIEW 3 major objections 6 minor 12 references

Trusted sabotage monitors overfit the model family they were calibrated on, so published detection numbers overstate safety against a different lineage.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 03:59 UTC pith:KAXFEDBW

load-bearing objection Solid measurement of a real cross-lineage transfer failure in trusted sabotage monitors; the residual interaction is well controlled, and the main soft spot is attempt-level safety rather than powered realized sabotage. the 3 major comments →

arxiv 2607.06596 v1 pith:KAXFEDBW submitted 2026-07-06 cs.CR cs.LG

Calibration-Family Overfit: Why Trusted Sabotage Monitors Don't Transfer Across Lineages

Lucas Pinto This is my paper
classification cs.CR cs.LG
keywords AI controltrusted monitoringsabotage detectioncross-family transfercalibration overfitinteraction decompositiondefer-to-trusted
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Trusted monitoring is a main defense against a misaligned untrusted model: a cheaper trusted model scores actions for sabotage and the most suspicious get audited or deferred. Papers usually measure a monitor against one or two untrusted models and report that accuracy as a property of the monitor. This paper asks whether the number is partly a property of the pairing. It holds the untrusted policy family as the controlled axis, fits a monitor on family A's transcripts, applies it to family B, and decomposes the cross-family ranking into attacker obviousness, monitor capability, and a residual own-family advantage called the interaction. On code-backdoor transcripts that residual is large and positive after the main effects are removed, survives matching for how blatant the attacks are, holds for trained monitors up to 34B, and appears across open and frontier families from multiple vendors. Geometrically each lineage's sabotage points in a different scoring direction, so a head trained on one lineage ranks another poorly. At a fixed 1% audit budget an off-lineage monitor catches far fewer attack attempts than an in-lineage one at the same cost. The practical claim is that control evaluations should publish full cross-family transfer matrices rather than single pairings, and should recalibrate with labeled sabotage from the lineage actually faced.

Core claim

After subtracting how obvious each family's sabotage is and how capable each monitor is, a residual own-family advantage remains: a monitor ranks its own lineage's sabotage substantially above an equally detectable attack from a different lineage. On a strict leak-free basis the interaction is +0.172 on four open-weight families and stays positive when frontier attackers from multiple vendors are added, when attacks are matched on blatancy proxies, and when a trained transcript-reading monitor is used. The gap is geometric: each lineage occupies a different scoring direction, and rotating the head toward the target recovers most of the lost detection while a matched random rotation recovers

What carries the argument

The interaction term from a two-way decomposition of the cross-family AUROC matrix: M[A,B] = grand mean + row_A + col_B + interaction[A,B], with the mean diagonal interaction reporting the residual own-family advantage after row (monitor capability) and column (attacker obviousness) main effects are removed. The same residual appears as a diagonal coefficient in a placement-value regression net of length, success, and task fixed effects.

Load-bearing premise

That detecting attack attempts (honest-versus-attack transcripts, even when the backdoor never fires) is a legitimate primary safety metric, even though clean realized backdoors are near zero for the open families and a full realized-sabotage matrix cannot be powered.

What would settle it

Build a substrate where every family produces enough clean both-success backdoors to fill a full realized-sabotage transfer matrix; if the own-family interaction collapses to zero on those successful backdoors after the same main-effect subtraction and blatancy matching, the operational claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper argues that trusted sabotage-monitor accuracy is partly a property of the calibration–evaluation family pairing, not of the monitor alone. On ControlArena code-backdoor transcripts it builds N×N AUROC matrices (calibrate on A, test on B), decomposes them into row (monitor capability), column (attacker obviousness), and residual diagonal interaction, and reports a positive own-family interaction of +0.172 (95% CI [+0.158, +0.185]) on four open-weight families under a strict leak-free feature basis. The interaction survives difficulty and dual-proxy blatancy matching, embedder/head-class ablations, a LoRA capability staircase to 34B, a second code substrate, and multi-vendor frontier expansions; a geometric account (misaligned sabotage directions w_A, geodesic rotation recovers most of the gap) is given. At a 1% audit budget under budget-enforced defer-to-trusted, the in- vs off-lineage catch-rate gap is +21.7 points on attack attempts. A four-step protocol (report full matrices, detect off-lineage drift, recalibrate with labeled sabotage, treat pooling as bounded) is proposed.

Significance. If the result holds, it is a genuine measurement-validity contribution for AI control: single-pairing monitor numbers are diagonal cells, and the off-diagonal that deployment faces is systematically lower after the confounds a careful evaluation already controls for. Strengths include a clean two-way residual (not a circular fitted constant), task-clustered bootstraps, leak-free primary basis, label-shuffle and pseudo-family negative controls, dual blatancy proxies, exact permutation tests on large family matchings, a causal rotation intervention, and an explicit four-step protocol each step of which is a measured result. The geometric cosine-to-AUROC law and the failure of benign-only CORAL/mean-shift are particularly useful for separating concept shift from covariate shift. The work is reproducible in design (split, provenance, analysis scripts named) and should change how control evaluations report monitor accuracy.

major comments (3)
  1. [§2, §4.4, §7, Abstract] §2 Success filtering and §4.4 / §7 Success framing: the primary operational claim (and the +21.7-point catch-rate gap in Table 3b) is powered on unfiltered honest-versus-attack attempt detection. Both-success realized backdoors are ~0% for the pre-registered four-open roster (Table 1) and only thinly powered on frontier lineages; on the realized-sabotage slice AUROC saturates near 1.0 for both in- and off-lineage heads while Cohen’s d remains large. The manuscript flags this, but the abstract and §5 protocol still read as if the safety gap is established for sabotage that fires. Either (i) power a higher-yield realized-sabotage matrix on a substrate that produces clean backdoors across families, or (ii) restate the headline catch-rate and protocol claims strictly as attempt-detection results and move realized-sabotage language to a robustness subsection with the saturation caveat made fi
  2. [§4.4, Abstract, Table 3] §4.4 per-family view and §7 “rankable lineages”: at the 1% budget the mean gap is carried almost entirely by Mistral (own 77.6% vs foreign 30.7%); Llama/Qwen/Gemma sit near the budget floor on both own and foreign heads. The interaction is correctly net of main effects, but the deployment translation (“roughly halves attack-attempt catch”) is not uniform. The abstract and Table 3 should report the per-family catch-rate table (or an explicit rankable-lineage restriction) alongside the mean gap so readers do not treat +21.7 points as applying to every lineage a deployer might face.
  3. [§4.2, §7, Abstract, §1] §4.2 and §7 Scope: every prompt-only family-conditioned construction (few-shot, rubric-distilled, retrieval-augmented) fails the competence gate on Llama, so the established claim is restricted to trained monitors. That scoping is correct but appears late. The title, abstract, and introduction currently read as applying to “trusted sabotage monitors” as practitioners deploy them (fixed-prompt / few-shot). State the trained-monitor scope in the abstract and early in §1, and treat the prompted failure as a first-class negative result rather than a limitation appendix item.
minor comments (6)
  1. [Table 2 / 2b] Table 2 vs Table 2b: the main text still mixes transductive and leak-free numbers; make leak-free the only number in the body tables and move transductive to an appendix lower-bound note to avoid double-reading.
  2. [Figure 4] Figure 4 caption says the curves use the transductive basis while the headline pair is leak-free; label both curves or replot on the primary basis.
  3. [§4.5] §4.5 GPT-5 same-vendor cell and lower attack yield (325 vs ~600) are noted; a one-line sensitivity that drops the GPT-5 row/column would strengthen the eight- and nine-family claims.
  4. [§4.3] SHADE-Arena is correctly reported as underpowered; consider moving the full SHADE subsection to an appendix and keeping only a one-paragraph “suggestive non-code” note in the main text so the code-substrate claim stays sharp.
  5. [§3] Notation: define the interaction estimand (mean diagonal residual) with an explicit equation early in §3 rather than only in prose.
  6. [A.1] Minor: “ControlArena” / “bigcodebench” capitalization and the arXiv id in the header are fine; check that all pre-registration dates in the roster registry match the appendix text.

Circularity Check

0 steps flagged

No significant circularity: the interaction residual, catch-rate gap, and controls are empirical measurements, not quantities forced by definition or self-citation.

full rationale

The paper's load-bearing claim is an empirical two-way residual: after building the calibrate-on-A / evaluate-on-B AUROC matrix and subtracting additive row (monitor capability) and column (attacker obviousness) main effects, the mean diagonal interaction is positive (+0.172 leak-free on four open families, CIs excluding zero). That residual is not defined to equal a fitted constant later re-labeled as a prediction; it is computed from held-out task-clustered transcripts and can be (and is) driven to zero under negative controls the paper reports (label-shuffle null ≈ −0.001; random pseudo-family partitions +0.000). Fixed-prompt monitors, which have no calibration-family ranking axis, show a null diagonal bonus by construction of the metric track—an honest null, not a circular positive. Difficulty- and blatancy-matched re-estimations, LoRA monitors up to 34B, a second code substrate, and frontier multi-vendor matrices are independent re-measurements, not re-uses of a fitted parameter. The geometric cosine–AUROC link and geodesic rotation recoveries are mechanism evidence for linear heads (aligned directions transfer better is expected once directions differ); they are controlled by matched random rotations recovering nothing and by benign-only CORAL failing, and they do not force the headline interaction. There is no self-citation chain, no uniqueness theorem imported from the same author, and no ansatz smuggled via prior work. Related-work citations are external. The attempt-level framing and underpowered realized-sabotage matrix are scope/limitation issues, not circularity. Score 0.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 2 invented entities

The result is almost entirely empirical. Free parameters are ordinary experimental choices (PCA dimension, regularization, LoRA rank, audit budget q). Domain assumptions are the standard AI-control framing (trusted monitor, fixed audit budget, passive stealth scaffolds). No new physical or mathematical entities are postulated; the 'interaction' is ordinary two-way ANOVA residual and the 'sabotage direction' is the gradient of a fitted linear head.

free parameters (4)
  • PCA components = 50 (primary)
    Fixed at 50 for the primary embedding head; also ablated to 10. Choice affects absolute AUROC but the interaction remains positive.
  • logistic regularization C = 1.0
    Fixed at 1.0; interaction invariant across three orders of magnitude of L2 strength.
  • audit budget q = 0.01
    Primary operating point set at 1% (also reported at 2% and 5%); standard control-protocol choice, not fitted to maximize the gap.
  • LoRA rank / epochs / seeds
    Training hyperparameters held fixed across the capability staircase; three seeds reported for stability.
axioms (4)
  • domain assumption Attack attempts (unfiltered honest-vs-attack labels) are a legitimate primary detection target even when both-success realized backdoors are rare.
    Stated in Section 2 Success filtering and defended in Section 7; required because open-weight families produce near-zero clean backdoors.
  • domain assumption Passive distribution shift under fixed stealth scaffolds is the relevant threat model (no adaptive attack, collusion, or best-of-N).
    Explicitly scoped in the Introduction and Section 2; adaptive attacks are left to other work.
  • standard math Two-way additive decomposition (grand mean + row + column + interaction) correctly isolates calibration overfit from obviousness and capability.
    Standard ANOVA residual; also recovered by a single conditional placement regression with task fixed effects (Section 3).
  • domain assumption Task-ID hash split (60/40) and task-clustered bootstrap correctly handle dependence between honest and attack samples on the same task.
    Stated in Sections 2–3; prevents same-task leakage from masquerading as transfer.
invented entities (2)
  • calibration-family overfit (own-family interaction residual) independent evidence
    purpose: Name and quantify the residual diagonal advantage after main-effect removal.
    Defined operationally as mean diagonal of the residual matrix; not a new physical object, just a named estimand.
  • family-specific sabotage direction w_A independent evidence
    purpose: Geometric explanation of the transfer gap via cosine alignment of head gradients.
    Simply the gradient of the fitted linear decision function; recoverable from labeled data of each family.

pith-pipeline@v1.1.0-grok45 · 30294 in / 3363 out tokens · 31759 ms · 2026-07-11T03:59:27.445051+00:00 · methodology

0 comments
read the original abstract

Trusted monitoring is a central defense in AI control: a cheaper trusted model scores an untrusted model's actions for sabotage, and the most suspicious are audited or deferred. Such monitors are evaluated against one or two untrusted models, and the accuracy is reported as a property of the monitor. We ask whether it is partly a property of the pairing. We make the untrusted policy family the controlled axis: we fit a monitor on family A's transcripts, apply it to family B, and decompose the cross-family AUROC into how obvious each family's sabotage is, how capable each monitor is, and the residual own-family advantage after both are removed: the interaction. On code-backdoor transcripts the interaction is positive and survives the dominant confounds: +0.172 (95% CI [+0.158, +0.185]) on four open-weight families on a strict leak-free basis, holding across an eight-family matrix spanning four frontier vendors, blatancy-matched controls, a trained transcript-reading monitor up to 34B parameters, and a second code benchmark. The gap is geometric: each lineage's sabotage occupies a different scoring direction, and rotating the monitor's scoring direction toward the target lineage recovers most of the lost detection while a matched random rotation recovers nothing. At a 1% audit budget under defer-to-trusted, an off-lineage monitor catches 19% of attack attempts where an in-lineage monitor catches 41% at the same audit cost, a +21.7-point gap. Control evaluations should report cross-family transfer matrices: a single-pairing accuracy overstates the safety a monitor delivers against a lineage it was not calibrated on. We give a four-step protocol that acts on the gap, with each step a measured result.

Figures

Figures reproduced from arXiv: 2607.06596 by Lucas Pinto.

Figure 1
Figure 1. Figure 1: Cross-family monitor AUROC, four open-weight families (diagonal = in-lineage). [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Monitor-capability staircase: the cross-family interaction stays positive (CI exclud [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: The interaction replicates across cuts, monitor classes, and substrates (95% CIs). [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Realized catch rate under a defer-to-trusted protocol at a 1% audit budget, as [PITH_FULL_IMAGE:figures/full_fig_p012_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Cross-family monitor AUROC across all eight families (four open-weight plus [PITH_FULL_IMAGE:figures/full_fig_p014_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Frontier-only sub-matrix: four strong models from four independent vendors, the [PITH_FULL_IMAGE:figures/full_fig_p015_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: In-lineage (own-family head) versus mean foreign-head AUROC per family, all [PITH_FULL_IMAGE:figures/full_fig_p017_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Off-lineage transfer AUROC rises with the cosine alignment between the calibra [PITH_FULL_IMAGE:figures/full_fig_p018_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: The head angle is a causal lever (four-open primary roster). Left: rotating calibra [PITH_FULL_IMAGE:figures/full_fig_p018_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Capability gap: weaker-model calibration roughly halves frontier detection (left); [PITH_FULL_IMAGE:figures/full_fig_p020_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Six-family cross-family AUROC (two independent frontier vendors, Sonnet and [PITH_FULL_IMAGE:figures/full_fig_p022_11.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

12 extracted references · 2 canonical work pages · 2 internal anchors

  1. [1]

    How does information access affect LLM monitors’ ability to detect sabotage? arXiv preprint arXiv:2601.21112,

    Rauno Arike, Raja Mehta Moreno, Rohan Subramani, Shubhorup Biswas, and Francis Rhys Ward. How does information access affect LLM monitors’ ability to detect sabotage? arXiv preprint arXiv:2601.21112,

    arXiv

  2. [2]

    CoT Red-Handed: Stress testing chain-of-thought monitoring.arXiv preprint arXiv:2505.23575,

    Benjamin Arnav et al. CoT Red-Handed: Stress testing chain-of-thought monitoring.arXiv preprint arXiv:2505.23575,

    arXiv

  3. [3]

    Async control: Stress-testing asynchronous control measures for LLM agents.arXiv preprint arXiv:2512.13526,

    Asa Cooper Stickland, Jan Michelfeit, Arathi Mani, Charlie Griffin, Ollie Matthews, Tomek Korbak, Rogan Inglis, Oliver Makins, and Alan Cooney. Async control: Stress-testing asynchronous control measures for LLM agents.arXiv preprint arXiv:2512.13526,

    arXiv

  4. [4]

    AI control: Improving safety despite intentional subversion.arXiv preprint arXiv:2312.06942,

    Ryan Greenblatt, Buck Shlegeris, Kshitij Sachan, and Fabien Roger. AI control: Improving safety despite intentional subversion.arXiv preprint arXiv:2312.06942,

    Pith/arXiv arXiv

  5. [5]

    Measuring coding challenge compe- tence with APPS.arXiv preprint arXiv:2105.09938,

    Dan Hendrycks, Steven Basart, Saurav Kadavath, et al. Measuring coding challenge compe- tence with APPS.arXiv preprint arXiv:2105.09938,

    Pith/arXiv arXiv

  6. [6]

    Knight, and Zifan Wang

    Neil Kale, Chen Bo Calvin Zhang, Kevin Zhu, Ankit Aich, Paula Rodriguez, Scale Red Team, Christina Q. Knight, and Zifan Wang. Reliable weak-to-strong monitoring of LLM agents. arXiv preprint arXiv:2508.19461,

    Pith/arXiv arXiv

  7. [7]

    Ensemble Monitoring for AI Control: Diverse Signals Outweigh More Compute

    Eugene Koran, Yejun Yun, Samantha Tetef, Benjamin Arnav, and Pablo Bernabeu-P ´erez. Ensemble monitoring for AI control: Diverse signals outweigh more compute.arXiv preprint arXiv:2605.15377,

    work page internal anchor Pith review Pith/arXiv arXiv

  8. [8]

    Evaluating control protocols for untrusted AI agents.arXiv preprint arXiv:2511.02997, 2025a

    Jon Kutasov, Chloe Loughridge, Yuqi Sun, Henry Sleight, Buck Shlegeris, Tyler Tracy, and Joe Benton. Evaluating control protocols for untrusted AI agents.arXiv preprint arXiv:2511.02997, 2025a. Jonathan Kutasov, Yuqi Sun, Paul Colognese, Teun van der Weij, Linda Petrini, Chen Bo Calvin Zhang, John Hughes, Xiang Deng, Henry Sleight, Tyler Tracy, Buck Shleg...

    arXiv

  9. [9]

    Classifier context rot: Monitor performance degrades with context length.arXiv preprint arXiv:2605.12366,

    Sam Martin and Fabien Roger. Classifier context rot: Monitor performance degrades with context length.arXiv preprint arXiv:2605.12366,

    Pith/arXiv arXiv

  10. [10]

    Cross-LLM generalization of behavioral backdoor detection in AI agent supply chains.arXiv preprint arXiv:2511.19874,

    Arun Chowdary Sanna. Cross-LLM generalization of behavioral backdoor detection in AI agent supply chains.arXiv preprint arXiv:2511.19874,

    arXiv

  11. [11]

    CIAware-Bench: Benchmarking Control Intervention Awareness Across Frontier LLMs

    Joachim Schaeffer, Thomas Jiralerspong, Alexander Panfilov, Guillaume Lajoie, Jonas Geip- ing, Yoshua Bengio, and Roland S. Zimmermann. CIAware-Bench: Benchmarking control intervention awareness across frontier LLMs.arXiv preprint arXiv:2606.11063, 2026a. Joachim Schaeffer et al. Attack selection reduces safety in concentrated AI control settings against ...

    work page internal anchor Pith review Pith/arXiv arXiv

  12. [12]

    BigCodeBench: Benchmarking code generation with diverse function calls and complex instructions.arXiv preprint arXiv:2406.15877,

    Terry Yue Zhuo et al. BigCodeBench: Benchmarking code generation with diverse function calls and complex instructions.arXiv preprint arXiv:2406.15877,

    Pith/arXiv arXiv