Pith. sign in

REVIEW 3 major objections 5 minor 13 references

Single-run agentic-RL numbers are dominated by data-draw variance, not evaluation noise, and on hard cells the spread is bimodal.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 18:54 UTC pith:4CFZLSVT

load-bearing objection An honest, useful reliability study of agentic CUA-RL; the core warning about single-run deltas holds, but the variance-budget numbers are provisional because they rest on a verifier-defined proxy with thin validity evidence. the 3 major comments →

arxiv 2607.17136 v1 pith:4CFZLSVT submitted 2026-07-19 cs.SE cs.AIcs.HCcs.LG

Teach it to stop, not just to click

Barada Sahu (Cabal AI) , Shivesh Pandey (Para AI) This is my paper
classification cs.SE cs.AIcs.HCcs.LG
keywords agentic computer-use RLsingle-run reliabilityvariance decompositionbimodalityverifier-guided repairon-policy self-distillationk-seed reportingdifficulty gradient
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that single-run reporting in agentic computer-use reinforcement learning is unreliable because success rates are dominated by variance upstream of evaluation—chiefly which training data is drawn and run-to-run nondeterminism—while evaluation noise is near zero and training-seed effects are small. Using verifier-guided repair of a 35B computer-use agent across five oracle-graded environments, the authors show that on the hardest repair cell the run-to-run distribution is bimodal, so a single run has roughly a 30% chance of landing in the failure mode and mean±std is the wrong summary. On that footing they establish two positive findings: repairability is graded by how constrained the corrective action is—a single fixed token installs reliably (done-detection 0.97±0.06), while open-ended corrections (spatial-coordinate clicks, generative fills) are only partial—and the frame-level repair transfers to end-to-end task success only when the corrective action is the task's sole remaining blocker (LinkedIn 8/20 vs 0/15, Fisher p=0.006). The paper contributes a measured reliability methodology—variance decomposition, bimodality testing, and a k-seed protocol released as a library—and reports two of its own over-claims caught only by replication.

Core claim

The central discovery is that a repaired computer-use policy's success rate is dominated by variance upstream of evaluation: a variance-components decomposition across a crossed data-draw×seed grid finds evaluation variance effectively zero and training-seed effects small (≤10%), with the remaining spread split between the data draw and run-to-run nondeterminism, the data draw's share rising to 48% on the hardest cell. On that cell the run-to-run distribution is bimodal (Hartigan dip p=0.07), so a single run has about a 30% chance of the failure mode, and mean±std misleads. The paper then establishes a two-tier difficulty gradient in repairability: a single fixed token (done detection) insta

What carries the argument

The completion verifier is a second, more capable vision-language model that judges from the same screenshot whether the task is complete and what the corrective next action is; it serves as hint, teacher, reward, and gate across four correction channels. The weight update is segment-aggregated on-policy self-distillation (SA-OPSD), which combines a GRPO-style policy term that down-weights bad actions with advantage-gated behavior cloning of the verifier's corrected action—the active ingredient that installs actions the base policy never samples. The reliability analysis is carried by a variance-components decomposition (Henderson crossed-ANOVA on a data-draw×seed grid with bootstrap CIs) an

Load-bearing premise

The entire frame-level measurement chain assumes the completion verifier's judgment—which frames need correction and what the corrective action is—is a valid oracle; if that judgment is biased for these environments, the repairability rates and variance decomposition are measuring the verifier's own opinion, and only the end-to-end LinkedIn result (8/20 vs 0/15) is independent of the verifier.

What would settle it

Run the same data-draw×seed grid measuring end-to-end oracle-graded task success on LinkedIn (the verifier-independent metric) instead of verifier-labeled emission rates; if the variance decomposition on that metric shows evaluation or training-seed variance dominating—or if the single-run sign-flip rate drops materially—the central claim about upstream variance is an artifact of the frame-level proxy.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Any single-run number in agentic computer-use RL should be treated as unreliable: in a variance regime comparable to this testbed, a single-run improvement of the size commonly published has roughly a one-third chance of having the wrong sign.
  • Repairability is graded by the constraint of the corrective action: a single fixed token (done detection) installs reliably, while spatial-coordinate clicks and generative field-fills are only partially and unreliably installed; the ordering replicates on a second, architecturally distinct policy.
  • Frame-level repair transfers to end-to-end task success only when the corrective action is the sole remaining blocker; when it is one of several required steps, or when the repaired behavior is not the bottleneck, the repair does not lift task success.
  • To stabilize an agentic-RL number, control the data draw first and the seed second, and average over k runs; evaluation variance is near zero, so re-evaluating a fixed checkpoint is deterministic.
  • The data-draw share of variance grows with task difficulty—dominant (48%) on the hardest cell—so which fresh training batch you draw matters most exactly where the repair is marginal.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The bimodality result implies that failure-mode claims ('this repair cannot be done') are likely low-tail draws from a distribution whose mean lies above base; any single-run negative in this regime should be re-run across seeds before being taken as a boundary.
  • Because the frame-level rates are built on the verifier's own judgment (frame selection, teacher label, and metric all come from the same verifier), the load-bearing numbers are the oracle-graded end-to-end results; an independent calibration of the verifier's corrective-action labels against human annotation would strengthen the frame-level chain.
  • The finding that independent data draws reveal instability that within-batch subsampling hides suggests that reproducibility standards for agentic RL should require fresh data draws, not just fresh seeds—otherwise the very instability being measured can be engineered away.
  • The transfer boundary—repair helps only when the repaired action is the sole blocker—suggests that end-to-end benchmarks for agentic repair should be designed so that each repaired failure is the bottleneck; otherwise frame-level improvements will not show up as task success.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies verifier-guided repair of a 35B computer-use agent (Holo3-35B-A3B) across five oracle-graded web-mirror environments. It reports a two-tier difficulty gradient in repairability: a fixed-token correction (done-detection) installs reliably (0.97±0.06), while open-ended corrections (spatial-coordinate clicks, generative field-fill) are only partially and unreliably installed. It further reports an end-to-end transfer result—repair improves oracle-graded task success only when the corrective action is the sole remaining blocker (LinkedIn 8/20 vs 0/15, Fisher p=0.006). The paper's central methodological claim is that agentic computer-use RL results are dominated by variance upstream of evaluation: a variance-components decomposition finds evaluation variance ≈0, training-seed effects ≤10%, and a dominant data-draw share (48%) on the hardest cell, with a bimodal run-to-run distribution (Hartigan dip p=0.07). The paper retracts two of its own earlier single-run over-claims, releases a library for k-seed reporting, and provides a stress test showing that single-run improvements of the size common in the field would have the wrong sign ~1/3 of the time in a comparable variance regime.

Significance. If the variance-decomposition and bimodality claims were established on a verifier-independent success metric, this would be a valuable methodological contribution: it demonstrates concrete failures of single-run reporting in agentic RL, validates a k-seed protocol out-of-sample, and provides a reusable library. The use of DB-level oracles, leakage-free held-out splits, explicit retractions of over-claims, and a stress test that externalizes the stakes are exemplary practices. However, the central variance budget and bimodality are measured on a frame-level proxy whose decision frames, teacher labels, and evaluation metric are all defined by the same completion verifier. The verifier's validity probe has only two completion-positive frames, and the only verifier-independent end-to-end result (LinkedIn) lacks seed/data-draw replication. Thus the broad claims about the unreliability of published task-success numbers rest on an unvalidated proxy. The paper's honesty and apparatus are strengths, but the evidence as it stands does not fully support the headline conclusions.

major comments (3)
  1. [§6 'Bimodality, and the seeds a claim needs'] The Hartigan dip test gives p=0.07 on k=10 seeds, which is not statistically significant at the 0.05 level. The abstract and conclusion nevertheless state the distribution is bimodal and that a single run has ~30% chance of the failure mode. The two-component mixture is preferred by ΔBIC=13, but with a marginal dip test and k=10 in a single cell, the bimodality evidence is suggestive rather than conclusive. Please report the result as suggestive, present the per-seed scatter (as in Fig. 2) with a clear statement of the dip-test p-value and its implication, and soften any claim that 'mean ± std is the wrong summary' unless the mixture model is given with appropriate uncertainty.
  2. [§5 Table 3 and 'Held-out measurement'] The bootstrap 95% CIs are computed over pooled seed×frame hits. Decision frames from the same held-out trajectory are correlated even with a run-level split, so pooling frames across trajectories and seeds treats them as independent and likely understates uncertainty. For example, grounding has per-seed rates 0.81/0.24/0.86/0.00/0.76 but a pooled CI of [0.44,0.63], which seems too tight for such a spread. The mean ± std over seeds is more informative for the instability claim. Please compute CIs that respect clustering within trajectories, or report the bootstrap distribution of the seed-level std, or remove the pooled CIs from the headline table.
  3. [§6 Table 6] Table 6 reports point estimates of variance components but no bootstrap CIs, despite the text claiming 'parametric-bootstrap CIs.' With only D=3 independent data draws, the data-draw component is estimated from three points; the point estimate σ_data=0.26 (48% share) could have a very wide CI. Without reported CIs, the reader cannot assess whether the data-draw dominance is robust or a small-sample artifact. Please include bootstrap CIs for σ_data, σ_resid, and the variance shares, or state if they were omitted for space; if they cannot be computed, the central quantitative claim is not falsifiable.
minor comments (5)
  1. [Title page] The author affiliation line reads 'Cabal AIShivesh Pandey'; this appears to be a typo for 'Cabal AI' and 'Shivesh Pandey'. Please fix.
  2. [§6 'Falsification 1'] The sample-efficiency curve description reports three data-seed means but does not give the seed construction method (how data draws differ, how many trajectories per draw). Please add a sentence describing the data-seed protocol so the result is reproducible.
  3. [§8 Limitations] The phrase 'Appropriate for a mechanism study, not a data-collection one' is vague. Please specify which mechanism is being studied and why the verifier-defined frame-level rates are the right instrument for that mechanism.
  4. [§2 Metric] The sentence 'evaluated adapter-on vs. adapter-off on the same model load' is unclear: does 'same model load' mean the base model weights are shared and only the adapter changes, or something else? Please clarify.
  5. [Table 5] The LinkedIn row uses an em dash in 'type note —sole blocker' while other rows use parentheses. Please make consistent.

Circularity Check

3 steps flagged

Headline variance and bimodality claims rest on a verifier-defined frame-level proxy that is the training target by construction; only the single LinkedIn cell is verifier-independent.

specific steps
  1. self definitional [§2 'On verifier circularity' (Setup)]
    "One completion verifier plays three roles that a skeptic will note are not independent: it selects the decision frames (those where it recommends the corrective action), it supplies the training label (teacher action), and its action is what the emission metric counts. Measured this way, the frame-level rates answer a deliberately narrow question—can a weight update install the verifier’s own judgment into the policy?—and nothing broader."

    The paper's headline variance decomposition (Table 6) and difficulty gradient (Table 3) are computed on exactly these frame-level emission rates. Because the outcome is defined as emitting the teacher's own recommended action on frames the teacher selected, the measured rates and their run-to-run variance are self-consistency statistics of the verifier–student loop, not task-success reliability. The abstract and conclusion nevertheless rest the 'success rate dominated by upstream variance' claim on this proxy.

  2. fitted input called prediction [Abstract; §6 'A measured variance budget']
    "a repaired policy's success rate is dominated by upstream variance: a variance-components decomposition across three cells (crossed data-draw × seed grid, bootstrap CIs) finds evaluation variance negligible (σ_eval ≈ 0) ... data draw's share rising to dominant (48%) on the hardest cell."

    Table 6's three cells are Fiverr (token emission), Shopify (near-coordinate emission), and Indeed (far-coordinate emission) — all frame-level rates defined by the verifier's own corrective-action judgment, not oracle-graded task success. The only verifier-independent success measurement (LinkedIn 8/20 vs 0/15, Table 5) is a single cell with asymmetric rollouts and no data-draw × seed grid. Thus the variance-budget conclusion about 'success rate' is an extrapolation from the fitted proxy rather than a measured property of task success.

  3. self definitional [§6 'A measured variance budget']
    "Two invariants hold across every cell: σ_eval≈0, so the noise is never in evaluation (re-evaluating a fixed checkpoint is deterministic—confirmed on all re-eval pairs)"

    The evaluation is a deterministic function of a fixed checkpoint and fixed held-out frames, so σ_eval=0 is true by construction of the metric, not an empirical discovery about agentic-RL evaluation noise. Presenting it as a measured 'invariant' that supports the single-run unreliability claim overstates a definitional property of the scoring procedure.

full rationale

The paper is unusually transparent: §2 explicitly names the verifier circularity and §8 repeats that frame-selection, teacher labels, and the emission metric share one source. It also provides one genuinely verifier-independent result (the oracle-graded LinkedIn transfer, 8/20 vs 0/15). However, the paper's central quantitative contributions — the 48% data-draw share, σ_eval≈0, the Hartigan dip p=0.07 bimodality, and the ~1/3 wrong-sign stress test — are all computed on the frame-level corrective-action emission rate, which by construction measures whether a weight update installs the verifier's own judgment on frames the verifier chose. That is a legitimate self-consistency study, but the abstract and conclusion translate it into a claim about 'a repaired policy's success rate' and 'agentic computer-use RL numbers' without an oracle-graded variance decomposition. The oracle-graded LinkedIn cell is too sparse (20 challenger / 15 base rollouts, one environment) to carry the variance budget, and no data-draw × seed grid is run on it. Hence the headline reliability claims partially reduce to the verifier-defined proxy; the acknowledged circularity limits but does not eliminate this reduction. Score 6 reflects one or more central 'predictions' that are, in part, constructed from their own training/evaluation target.

Axiom & Free-Parameter Ledger

4 free parameters · 7 axioms · 0 invented entities

The paper's quantitative claims rest on a small measured grid (D=3 data draws × k=8 seeds for variance components, k=5 seeds for headline rates, k=10 for bimodality) and on the completion verifier as the source of frames, labels, and the metric. The variance components and mixture parameters are fitted values, not independent benchmarks. The verifier's validity is checked on only 16 frames with 2 positives, and the environments/backend are proprietary.

free parameters (4)
  • σ_data (Indeed, hardest cell) = 0.26 (48% of variance)
    Fitted from D=3 independent data draws × k=8 seeds; central to the claim that the data draw dominates on the hardest cell.
  • σ_resid (Indeed) = 0.22
    Fitted run-to-run residual variance from the crossed grid; used in the claim of bimodality and the wrong-sign stress test.
  • Bimodal mixture modes/weights (Indeed) = modes ≈0.09/0.80, weights ≈0.4/0.6
    Fitted to k=10 seeds; used to state 'a single run has roughly a 30% chance of the failure mode.'
  • σ≈0.3 (coordinate regime) = ~0.3
    Used in the stress test (wrong sign 33–44% for +7.7 improvement); measured from the Indeed cell.
axioms (7)
  • domain assumption The completion verifier's judgment is a valid oracle for corrective action on decision frames.
    Tested on only 16 frames with 2 positive completion cases (Table 7); used throughout §5 and §6 to define frames, labels, and the metric.
  • domain assumption Oracle-graded environments (DB state + audit log) provide ground truth for task success.
    Section 2 and §5; the LinkedIn transfer result depends on this.
  • domain assumption The five repair environments are representative of agentic CUA failure types.
    Section 3 claims the failures are not cherry-picked; only eight app families tested, HubSpot excluded.
  • domain assumption Base success is near zero across apps, so non-zero repaired rates are real effects.
    Table 2, six trials per app; HubSpot excluded due to infra fast-fail; small sample for the universality of zero base.
  • standard math Henderson crossed-ANOVA and Hartigan dip test are appropriate for this small-sample variance decomposition.
    Used in §6 on D=3×k=8 and k=10 grids; the paper does not cite the specific ANOVA method or the dip test.
  • domain assumption Warm-served load-once serving removes cold-load variance.
    Section 5 and Table 5; the authors found identical configs gave 5/5 then 0/5 before the fix.
  • ad hoc to paper HubSpot exclusion due to infrastructure fast-fail is not selection bias.
    Section 3: all six rollouts fast-failed at ~45s, leaving no decision trajectory.

pith-pipeline@v1.3.0-alltime-deepseek · 13034 in / 14416 out tokens · 127807 ms · 2026-08-01T18:54:16.418894+00:00 · methodology

0 comments
read the original abstract

Agentic computer-use RL is reported in single runs, and those numbers mislead. Using verifier-guided repair of a 35B computer-use agent (CUA) across five oracle-graded environments, we show a repaired policy's success rate is dominated by upstream variance: a variance-components decomposition across three cells (crossed data-draw $\times$ seed grid, bootstrap CIs) finds evaluation variance negligible ($\sigma_{\mathrm{eval}} \approx 0$) and the training-seed effect small everywhere ($\leq 10\%$); instead it splits between the data draw and run-to-run nondeterminism, the data draw's share rising to dominant ($48\%$) on the hardest cell. There the run-to-run distribution is bimodal (Hartigan dip $p=0.07$, $k=10$), so a single run has roughly a 30% chance of the failure mode and mean$\pm$std is the wrong summary. On that footing, two findings hold. First, repairability is two-tier in how constrained the corrective action is: a single fixed token installs reliably (done-detection $0.97\pm0.06$), while open-ended corrections are only partial -- spatial-coordinate clicks (grounding $0.53\pm0.35$) and a generative field-fill ($0.14\pm0.04$). Second, the frame-level repair transfers to task success only when the corrective action is the task's sole remaining blocker (LinkedIn 8/20 vs. base 0/15, Fisher $p=0.006$). We caught two of our own over-claims -- a sample-efficiency curve and a 'grounding cannot be bought' boundary -- only by replicating across seeds; a stress test makes the stakes external: a single-run improvement of the size this field publishes would have the wrong sign roughly one-third of the time in a comparable regime. We release a library (cua_reliability) for routine k-seed reporting. The apparatus is, to our knowledge, the first multimodal segment-aggregated on-policy self-distillation (SA-OPSD) update on a real 35B CUA policy.

Figures

Figures reproduced from arXiv: 2607.17136 by Barada Sahu (Cabal AI), Shivesh Pandey (Para AI).

Figure 1
Figure 1. Figure 1: Held-out corrective-action rate (five seeds, bootstrap [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Per-seed held-out rate (𝑘=5); the bar is the mean. The discrete-token repair is stable across seeds; the spatial-coordinate repairs are not, and one grounding seed lands essentially at zero—the low tail that produced the earlier “0/14” boundary claim. The protocol, validated out-of-sample. Oracle grading; deterministic resets; leakage-free run￾level held-out splits; and mean ± std over 𝑘 training runs per … view at source ↗
Figure 3
Figure 3. Figure 3: Training seeds needed (per condition) to resolve an effect of size [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

13 extracted references · 4 linked inside Pith

  1. [1]

    Agarwal, N

    R. Agarwal, N. Vieillard, Y. Zhou, et al. On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes. ICLR, 2024

    2024

  2. [2]

    GUI-GENESIS: Automated Synthesis of Efficient Environments with Verifiable Rewards for GUI Agent Post-Training

    Cao et al. GUI-GENESIS: Automated Synthesis of Efficient Environments with Verifiable Rewards for GUI Agent Post-Training. arXiv:2602.14093, 2026

    arXiv 2026

  3. [3]

    GUI-Shepherd: Reliable Process Reward and Verification for Long-Sequence GUI Tasks

    Chen et al. GUI-Shepherd: Reliable Process Reward and Verification for Long-Sequence GUI Tasks. arXiv:2509.23738, 2025

    arXiv 2025

  4. [4]

    Agentic Reward Modeling: Verifying GUI Agent via Online Proactive Interaction

    Cui et al. Agentic Reward Modeling: Verifying GUI Agent via Online Proactive Interaction. arXiv:2602.00575, 2026

    Pith/arXiv arXiv 2026

  5. [5]

    DeepSeekMath: Pushing the Limits of Mathematical Reasoning (Group Relative Policy Optimization)

    DeepSeek-AI. DeepSeekMath: Pushing the Limits of Mathematical Reasoning (Group Relative Policy Optimization). arXiv:2402.03300, 2024

    Pith/arXiv arXiv 2024

  6. [6]

    Henderson, R

    P. Henderson, R. Islam, P. Bachman, J. Pineau, D. Precup, and D. Meger. Deep Reinforcement Learning That Matters. AAAI, 2018

    2018

  7. [7]

    Refined Policy Distillation: From VLA Generalists to RL Experts

    J\"ulg et al. Refined Policy Distillation: From VLA Generalists to RL Experts. arXiv:2503.05833, 2025

    Pith/arXiv arXiv 2025

  8. [8]

    OS-Themis: A Scalable Critic Framework for Generalist GUI Rewards

    Li et al. OS-Themis: A Scalable Critic Framework for Generalist GUI Rewards. arXiv:2603.19191, 2026

    arXiv 2026

  9. [9]

    MagicGUI-RMS: A Multi-Agent Reward Model System for Self-Evolving GUI Agents

    Li et al. MagicGUI-RMS: A Multi-Agent Reward Model System for Self-Evolving GUI Agents. arXiv:2601.13060, 2026

    arXiv 2026

  10. [10]

    Agarwal, M

    R. Agarwal, M. Schwarzer, P. S. Castro, A. Courville, and M. G. Bellemare. Deep Reinforcement Learning at the Edge of the Statistical Precipice. NeurIPS, 2021

    2021

  11. [11]

    The Art of Building Verifiers for Computer Use Agents

    Rosset et al. The Art of Building Verifiers for Computer Use Agents. arXiv:2604.06240, 2026

    Pith/arXiv arXiv 2026

  12. [12]

    Adaptive Milestone Reward for GUI Agents

    Zheng et al. Adaptive Milestone Reward for GUI Agents. arXiv:2602.11524, 2026

    arXiv 2026

  13. [13]

    VLA-OPD: Bridging Offline SFT and Online RL for Vision-Language-Action Models via On-Policy Distillation

    Zhong et al. VLA-OPD: Bridging Offline SFT and Online RL for Vision-Language-Action Models via On-Policy Distillation. arXiv:2603.26666, 2026

    arXiv 2026