Pith. sign in

REVIEW 4 major objections 4 minor 37 references

This paper claims a small, learned reward model can replace hand-written rule checks when training computer-use agents, matching or exceeding rule-based supervision in online RL while costing far less.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-07-31 23:54 UTC pith:3SZUNEXQ

load-bearing objection A serious, well-engineered reward-framework paper whose headline RL claim depends on a nonstandard mean-over-checkpoints protocol; referee it, but push for final-checkpoint reporting. the 4 major comments →

arxiv 2607.23263 v1 pith:3SZUNEXQ submitted 2026-07-25 cs.AI

SeekJudge: A Practical Reward Framework for Reinforcement Learning in Computer-Use Agents

Yang Wan , Zhenhao Zhang , Jierui Wang , Linchao Zhu This is my paper
classification cs.AI
keywords computer-use agentsreward modelreinforcement learningGUI trajectory evaluationmodel-based judgingstep-level rewardslong-horizon tasksdistillation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that judging whether a GUI-agent trajectory fulfills its instruction can be done by a learned judge rather than by hand-coded rules, and that this learned judge works as an RL reward signal. It argues the key is to decompose judging into localization, finding the frame with the decisive evidence, and extraction, reading that frame accurately, instead of pouring all screenshots into one forward pass. With that decomposition, a single open 9B model matches or surpasses native rule-based supervision on downstream test success, at a fraction of the cost and with a context small enough for long trajectories.

Core claim

SeekJudge's central claim is that model-based trajectory judging is now a practical drop-in for rule-based evaluation in computer-use RL: used as a reward, it matches or surpasses the environment's native rule verifier on held-out test goals, where previous model judges left a gap. The discovery behind it is that multi-image judging dilutes the decisive evidence, so the task should be split into locating the decisive image and then reading that image alone, one at a time. This decomposition enables a small 9B model to beat closed-source large-model judges and to offer step-level judgments, cheap inference, and a context footprint that scales to long trajectories.

What carries the argument

The core object is the Seek-Analyze loop over a condensed timeline. A Condense agent and a Ground agent compress the trajectory into compact text entries; a stateful Seek agent decides whether evidence is sufficient and, when it is not, names one step and one image to inspect; a stateless Analyze agent reads that single screenshot and returns the decisive detail as text. All four roles share one 9B backbone trained through a seed-calibrated distillation pipeline, and the dense outputs (seven trajectory-level dimension scores, nine step-level classes) are reduced by a fitted regressor to a scalar reward for group-relative policy optimization. A rollout-overlapped, zero-client-state reward ser

Load-bearing premise

The headline comparison depends on a nonstandard RL protocol — test success averaged over all evaluations during only 75 training steps with episodes capped at 15 training and 25 test steps, plus one backbone/environment cell left pending in Table 2 — so if standard final-checkpoint or longer-horizon reporting changes the ranking, the 'match or surpass' claim may not hold.

What would settle it

Re-run the RL comparison under standard final-checkpoint reporting with longer episode caps and fill the missing backbone/environment cell; if SeekJudge's test success does not match or exceed the rule-based reward under that protocol, the central claim collapses.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If SeekJudge is right, RL for computer-use agents no longer depends on version-frozen parsers and hard-coded answers; a learned reward can track human intention and survive app updates and drifting online content.
  • Step-level judgments supply a denser training signal than a single pass/fail bit, and the granularity ablation shows continuous plus per-step rewards improve test success over a Boolean reward.
  • The small per-call context lets judging scale to trajectories of hundreds of steps that would overflow a single-pass judge's context window.
  • The asynchronous reward server cuts the stall the reward model adds to RL rollouts, making model-based reward practical inside the training loop.
  • A single 9B open model makes judging orders of magnitude cheaper than closed-source large-model judges, lowering the cost barrier for RL experiments.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the localization/extraction decomposition is the true source of gain, the same two-stage design could transfer to other long-context vision-language judging tasks, such as document QA or video understanding, where a single decisive frame is buried in a long observation stream.
  • The paper's component ablation is run on the base model but not on the trained specialist, so attributing the full pipeline gain to the Seek-Analyze loop is an extrapolation; a direct ablation on the distilled model would test this.
  • A standard-protocol re-run, using final-checkpoint test success and longer episode caps, would be a stringent external check of the claim that model-based reward matches rule-based supervision.
  • The exclusion of goal-level dimensions from the scalar reward follows from the group-relative structure of the RL objective, suggesting that other multi-turn RL settings should similarly drop objective-fixed dimensions to avoid adding only offsets.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes SeekJudge, a multi-agent reward framework for computer-use agents that decomposes trajectory judging into localization (Condense/Ground/Seek agents) and extraction (Analyze agent), all sharing a single 9B open-source backbone. It introduces CUAStepBench, a human-annotated benchmark with dense step-level labels across 177 applications, and reports offline F1 scores on three reward benchmarks and online RL results (GRPO) on Chrome, Impress, and OS domains under two actor backbones. The central claim is that SeekJudge is the first practical model-based reward to match or surpass native rule-based supervision in online RL, measured by downstream test success on held-out goals, while being cheaper and supporting longer trajectories.

Significance. If the central claim survives scrutiny, this is a meaningful step toward replacing brittle rule verifiers with practical model-based judges in CUA reinforcement learning. The paper ships an open-source 9B model, a new human-annotated benchmark with dense step labels, a cost/latency analysis, and a reward-server architecture. The decoupled localization/extraction design and the step-level output are plausible and well-motivated. However, the headline comparison is sensitive to the reporting protocol (mean over training checkpoints) and to incomplete/negative cells in the main table. The offline F1 evaluation also involves calibration on the evaluation benchmarks and a benchmark built with the authors' own judge, which limits the strength of the offline claims. The online RL comparison against external rule verifiers is not intrinsically circular, which is a credit to the design, but the load-bearing RL claim needs more standard and complete evidence.

major comments (4)
  1. [Section 4.2, Appendix D, Table 2] The reported test success is the mean over all test evaluations during training (steps 15/30/45/60/75, three passes each), not the final or best checkpoint. This is nonstandard and load-bearing: Section 4.3 argues rule-based training reward overfits, implying test curves are plausibly non-monotonic. Averaging over checkpoints can invert a final-checkpoint comparison in either direction. Since the claim is about downstream success rate, which convention attaches to a deployed final model, please report final and best checkpoint numbers with learning curves, plus run-to-run variance for every Table 2 cell. Without this, the headline 'match or surpass' claim is not established.
  2. [Table 2] The abstract claims SeekJudge 'matches or surpasses' native rule-based supervision, but the completed cells show one deficit and one missing run: Qwen3VL-8B on Impress (48.41 vs 49.28) and the Qwen-OS cell is pending. Of the five completed cells, four surpass the rule baseline and one does not. The claim should be qualified to the completed settings, or the missing cell should be filled and the deficit explained. This is not a cosmetic issue because the paper's own Section 4.3 uses this table to support the broad claim.
  3. [Section 3.2, Appendix B, Table 7] The offline F1 results are reported using a score regression fitted jointly on all three evaluation benchmarks (Table 7, row 1). The leakage probe in Table 7 shows the diagonal advantage is small, but it does not eliminate the fact that the calibration has seen the test distributions; the probe is a sensitivity check, not a leakage-control method. In addition, CUAStepBench was constructed by filtering trajectories with the authors' own judge's uncertainty (Section 4.1), so offline F1 on it is not a fully independent test of judge quality. These issues do not affect the online RL comparison against rule verifiers, but they weaken the offline claims and the statement about 'first to clear the Rule baseline.' Please report an independent calibration split or reframe the offline results as indicative rather than definitive.
  4. [Section 4.2, Appendix D] The paper states that 'the whole paper uses only two parameters' for calibration, but the reward pipeline also depends on the per-step nine-class rubric constants, the w_step weight, and the seed-calibrated criteria and seek prompts. These are tunable and no sensitivity analysis is provided for w_step or the rubric constants. Since the RL result depends on the scalar reward formed by the outcome score plus w_step times the per-step scores, a sensitivity analysis of these choices is needed to show that the headline comparison is not the result of a particular calibration.
minor comments (4)
  1. [Figure 1(a) caption] The caption says '(dashed = test set)' and implies learning curves, but no learning curves are shown in the paper. Either add the curves or revise the caption to match Table 2.
  2. [Table 3] For baselines without native step-level judgments, step labels are obtained via SeekJudge's step extraction procedure. This may bias the step-level comparison; please clarify whether the step prompt was applied identically and whether any baseline-specific adaptation was made.
  3. [Section 4.2 / Table 2] The pending Qwen-OS cell should be flagged in the abstract or conclusions as well as in the table caption, since readers may overlook the incompleteness.
  4. [Appendix E] The cost model charges cache hits at zero cost for repeated prefixes. While this matches the deployment logs, it is a potentially favorable assumption for methods with long reused contexts; a sensitivity table under a no-cache or partial-cache assumption would strengthen the cost comparison.

Circularity Check

1 steps flagged

Offline F1 is calibrated on the same benchmarks, but the headline online-RL comparison is against external rule verifiers and is not circular.

specific steps
  1. fitted input called prediction [Section 4.2 (offline calibration) and Appendix B, Table 7]
    "We score every offline benchmark with a single shared calibration, applying the same fit to a trained specialist and to an untrained base model alike, obtained by the procedure of Section 3.2. ... Our reported setting fits jointly on all three. ... A diagonal cell is the highest-risk reading, since the regressor has seen the target distribution."

    The reported offline F1 is obtained after fitting a gradient-boosted regressor and a threshold on the same three benchmarks whose human labels define the reported F1. Thus the binary verdict used for offline evaluation is not a parameter-free prediction of those labels: the calibration has already seen the target answers. The paper explicitly acknowledges this in Appendix B by calling the same-benchmark cell the highest-risk reading. This is the fitted-input-called-prediction pattern, but only partial: Table 7's off-diagonal transfers are within a few F1 points and the central online-RL claim is measured by external rule verifiers, so the headline comparison does not reduce by construction.

full rationale

The paper's central claim — that SeekJudge is the first practical model-based reward to match or surpass native rule-based supervision in online RL — is checked against the environments' native rule verifiers on held-out test goals (Table 2, Section 4.2), so that comparison is externally anchored and not internally circular. The rule-based baseline's training reward is the same rule metric, while SeekJudge's reward comes from a separately trained judge, so the comparison is meaningful. The main caveat found is in the offline benchmark evaluation: the score-regressor/threshold calibration is fitted on the same three benchmarks whose human labels are then used to compute the reported F1. The paper discloses this and runs a leakage probe (Table 7) showing the diagonal fit does not meaningfully beat off-diagonal transfer, so this is a calibration-in-evaluation weakness rather than a forced identity. Other concerns raised by the reader — CUAStepBench tasks selected using the authors' own judge's uncertainty, seed-calibrated prompts tuned on about 20 human trajectories, and the mean-over-training-checkpoints test-success protocol — are selection-bias or reporting-protocol issues, not circularity: none of them makes the claimed prediction equal to its own input. No load-bearing self-citation chain or imported uniqueness theorem appears. Overall, the central result is self-contained against external verifiers; the offline F1 step has partial in-sample calibration but is explicitly controlled for.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

No new physical or ontological entities are introduced; the four agents are software roles sharing one model. The central assumptions are that human verdicts are ground truth, closed-source teacher labels are reliable, and a single screenshot can hold decisive evidence. The calibration regressor, step constants, and prompts are fitted or hand-chosen, so they are counted as free parameters.

free parameters (4)
  • Offline score calibration (XGBoost regressor + binary threshold) = Not specified; fit to human labels on all three benchmarks jointly
    Maps seven 0-100 dimension scores to a scalar/verdict; the paper calls it 'few fitted parameters', but gradient boosting fits many splits and the threshold is tuned on the target benchmark labels (Section 3.2, Appendix B, Table 7).
  • RL reward calibration (second XGBoost fit and w_step weight) = Not specified; fit on base-model rollouts; w_step hand-fixed
    Maps dimension scores and per-step class constants into a trajectory reward with fixed weight w_step; the fit is data-dependent, though goal-fixed dimensions are excluded for a principled GRPO-group reason (Section 4.2, Appendix D).
  • Per-step nine-class rubric constants = Preset constants fixed by human rubric
    Each of the nine step classes is mapped to a hand-chosen constant that shapes the trajectory reward (Appendix D).
  • Seed-calibrated criteria and seek prompts = Text prompts optimized on ~20 human trajectories via Claude Code
    Both prompts are tuned against a small human-labeled seed set; this is prompt fitting rather than a scalar parameter, included for completeness (Section 3.2).
axioms (5)
  • domain assumption Human trajectory verdicts are the correct ground truth for judging task completion.
    The benchmark and calibration define success by human labels (Sections 3.2, 4.1).
  • domain assumption Closed-source teacher models (DeepSeek V3.2, Gemini 3.0 Flash, Codex/GPT-5.5) produce reliable dense labels for distillation.
    Distillation data is generated by these models after prompt search on a small human seed (Section 3.2).
  • domain assumption The decisive evidence for a trajectory verdict can be localized to a single screenshot and extracted at high fidelity.
    The Seek-Analyze loop design assumes a single image (or few) holds the decisive detail (Section 3.1, Figure 2b).
  • domain assumption Multi-image contexts monotonically dilute VLM judging performance even when the decisive image is included.
    Established by the authors' own 123-case experiment in Section 4.5 and used to motivate the entire decomposition.
  • domain assumption Rule-based verifiers are a fair 'native' baseline and their held-out test success is an unbiased measure of task success.
    The headline comparison treats rule rewards as the reference; if rule test success is itself noisy or leaky, the comparison is affected.

pith-pipeline@v1.3.0-alltime-deepseek · 16952 in / 15101 out tokens · 136288 ms · 2026-07-31T23:54:03.479483+00:00 · methodology

0 comments
read the original abstract

Deciding whether a trajectory actually fulfills its instruction governs how we measure computer-use agents on long-horizon graphical-user-interface tasks and how we train them with reinforcement learning. This judgment has long relied on rule-based evaluation, which struggles to align with human intention and goes stale when an app updates or its online content drifts. Existing model-based judges attempt to address these problems but still leave a performance gap to the rule-based evaluation. We propose the \textbf{SeekJudge} framework, in which four role-specialized agents, a Condense, a Ground, a Seek and an Analyze agent, reach a verdict through a Seek--Analyze loop over the trajectory. A seed-calibrated distillation pipeline trains one specialized $9$B model to serve as the shared backbone for all four agents. Measured by downstream success rate on held-out RL test goals, SeekJudge is the first practical model-based reward to match or surpass native rule-based supervision in online RL. Beyond accuracy, SeekJudge provides step-level judgments, runs far cheaper than a closed-source large model, and keeps a small per-call context that scales to much longer trajectories. We further contribute a general architectural improvement to the reward server that speeds up judging in RL. Together these make model-based reward a practical drop-in for rule-based supervision in CUA reinforcement learning.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

37 extracted references · 12 linked inside Pith

  1. [1]

    Proceedings of the 34th International Conference on Machine Learning (ICML) , series =

    World of Bits: An Open-Domain Platform for Web-Based Agents , author =. Proceedings of the 34th International Conference on Machine Learning (ICML) , series =

  2. [2]

    Advances in Neural Information Processing Systems (NeurIPS) , year =

    Mind2Web: Towards a Generalist Agent for the Web , author =. Advances in Neural Information Processing Systems (NeurIPS) , year =

  3. [3]

    International Conference on Learning Representations (ICLR) , year =

    WebArena: A Realistic Web Environment for Building Autonomous Agents , author =. International Conference on Learning Representations (ICLR) , year =

  4. [4]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL) , year =

    VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks , author =. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL) , year =

  5. [5]

    Advances in Neural Information Processing Systems (NeurIPS) , year =

    OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments , author =. Advances in Neural Information Processing Systems (NeurIPS) , year =

  6. [6]

    International Conference on Learning Representations (ICLR) , year =

    AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents , author =. International Conference on Learning Representations (ICLR) , year =

  7. [7]

    arXiv preprint arXiv:2501.12326 , year =

    UI-TARS: Pioneering Automated GUI Interaction with Native Agents , author =. arXiv preprint arXiv:2501.12326 , year =

    Pith/arXiv arXiv

  8. [8]

    arXiv preprint arXiv:2606.29537 , year =

    OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks , author =. arXiv preprint arXiv:2606.29537 , year =

    Pith/arXiv arXiv

  9. [9]

    2026 , eprint=

    WebGym: Scaling Training Environments for Visual Web Agents with Realistic Tasks , author=. 2026 , eprint=

    2026

  10. [10]

    Advances in Neural Information Processing Systems (NeurIPS) , year =

    On the Effects of Data Scale on UI Control Agents , author =. Advances in Neural Information Processing Systems (NeurIPS) , year =

  11. [11]

    arXiv preprint arXiv:2605.25624 , year =

    CUA-Gym: Scaling Verifiable Training Environments and Tasks for Computer-Use Agents , author =. arXiv preprint arXiv:2605.25624 , year =

    Pith/arXiv arXiv

  12. [12]

    arXiv preprint arXiv:2507.02825 , year =

    Establishing Best Practices for Building Rigorous Agentic Benchmarks , author =. arXiv preprint arXiv:2507.02825 , year =

    Pith/arXiv arXiv

  13. [13]

    xlang.ai , year =

    Introducing OSWorld-Verified , author =. xlang.ai , year =

  14. [14]

    Second Conference on Language Modeling (COLM) , year =

    An Illusion of Progress? Assessing the Current State of Web Agents , author =. Second Conference on Language Modeling (COLM) , year =

  15. [15]

    arXiv preprint arXiv:2602.10356 , year =

    Autonomous Continual Learning of Computer-Use Agents for Environment Adaptation , author =. arXiv preprint arXiv:2602.10356 , year =

    Pith/arXiv arXiv

  16. [17]

    2026 , eprint=

    PRO-CUA: Process-Reward Optimization for Computer Use Agents , author=. 2026 , eprint=

    2026

  17. [18]

    arXiv preprint arXiv:2606.02031 , year =

    OpenWebRL: Demystifying Online Multi-turn Reinforcement Learning for Visual Web Agents , author =. arXiv preprint arXiv:2606.02031 , year =

    Pith/arXiv arXiv

  18. [19]

    arXiv preprint arXiv:2509.02544 , year =

    UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learning , author =. arXiv preprint arXiv:2509.02544 , year =

    Pith/arXiv arXiv

  19. [20]

    arXiv preprint arXiv:2604.06126 , year =

    Gym-Anything: Turn any Software into an Agent Environment , author =. arXiv preprint arXiv:2604.06126 , year =

    Pith/arXiv arXiv

  20. [21]

    arXiv preprint arXiv:2508.04700 , year =

    SEAgent: Self-Evolving Computer Use Agent with Autonomous Learning from Experience , author =. arXiv preprint arXiv:2508.04700 , year =

    Pith/arXiv arXiv

  21. [22]

    arXiv preprint arXiv:2508.15144 , year =

    Mobile-Agent-v3: Fundamental Agents for GUI Automation , author =. arXiv preprint arXiv:2508.15144 , year =

    Pith/arXiv arXiv

  22. [23]

    arXiv preprint arXiv:2508.09123 , year =

    OpenCUA: Open Foundations for Computer-Use Agents , author =. arXiv preprint arXiv:2508.09123 , year =

    arXiv

  23. [24]

    arXiv preprint arXiv:2504.08942 , year =

    AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectories , author =. arXiv preprint arXiv:2504.08942 , year =

    arXiv

  24. [25]

    arXiv preprint arXiv:2603.19191 , year =

    OmniGUIRewardBench: A Cross-Platform Benchmark for GUI Outcome Reward Models , author =. arXiv preprint arXiv:2603.19191 , year =

    arXiv

  25. [26]

    arXiv preprint arXiv:2510.18596 , year =

    CUARewardBench: A Benchmark for Evaluating Reward Models on Computer-using Agent , author =. arXiv preprint arXiv:2510.18596 , year =

    arXiv

  26. [27]

    arXiv preprint arXiv:2604.06240 , year =

    The Art of Building Verifiers for Computer Use Agents , author =. arXiv preprint arXiv:2604.06240 , year =

    Pith/arXiv arXiv

  27. [28]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages =

    OS-Oracle: A Comprehensive Framework for Cross-Platform GUI Critic Models , author =. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages =. 2026 , note =

    2026

  28. [29]

    International Conference on Learning Representations (ICLR) , year =

    Let's Verify Step by Step , author =. International Conference on Learning Representations (ICLR) , year =

  29. [30]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL) , year =

    Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations , author =. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL) , year =

  30. [31]

    arXiv preprint arXiv:2505.23762 , year =

    ZeroGUI: Automating Online GUI Learning at Zero Human Cost , author =. arXiv preprint arXiv:2505.23762 , year =

    Pith/arXiv arXiv

  31. [32]

    International Conference on Learning Representations (ICLR) , year =

    WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learning , author =. International Conference on Learning Representations (ICLR) , year =

  32. [33]

    Advances in Neural Information Processing Systems (NeurIPS) , year =

    DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning , author =. Advances in Neural Information Processing Systems (NeurIPS) , year =

  33. [34]

    First Conference on Language Modeling (COLM) , year =

    Autonomous Evaluation and Refinement of Digital Agents , author =. First Conference on Language Modeling (COLM) , year =

  34. [35]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL) , year =

    WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models , author =. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL) , year =

  35. [36]

    Advances in Neural Information Processing Systems (NeurIPS) , year =

    Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena , author =. Advances in Neural Information Processing Systems (NeurIPS) , year =

  36. [37]

    Advances in Neural Information Processing Systems (NeurIPS) , year =

    WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents , author =. Advances in Neural Information Processing Systems (NeurIPS) , year =

  37. [38]

    2025 , eprint =

    Expanding Computation Spaces of LLMs at Inference Time , author =. 2025 , eprint =

    2025