REVIEW 4 major objections 6 minor
OpenForgeRL: Train Harness-native Agents in Any Environment
T0 review · 4 major / 6 minor · reviewed 2026-08-01 · deepseek-v4-flash
Pith's one-line read OpenForgeRL claims that a recording proxy plus remote per-rollout containers can turn any agent harness into trainable RL data.
desk verdict A genuinely useful systems contribution with broad empirical validation, but the Eq. (1) credit-assignment assumption needs an ablation and the benchmark results need seeds before publication. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the proxy-plus-orchestrator pair. The proxy wraps the harness's inference server and records every prompt-response pair the harness generates, including calls made by internal subagents and context managers, reconstructing each rollout as an unordered trajectory of (H(s_t), a_t) pairs. The orchestrator launches each rollout in its own remote container, so harness rollouts scale elastically and are fully decoupled from the training GPUs. Together they convert a stateful, multi-process harness into the flat (state, action, reward) sample format expected by standard RL training code, with terminal reward propagated to every step as gamma^(T-t) times r_T (gamma=1.0)
What would settle it
Take a harness whose internal planner emits many prompt-response pairs before a single successful action, train two identical RL runs on the same tasks—one rewarding every recorded pair with the terminal reward and one rewarding only the final action—and compare final benchmark performance; roughly equal results would support the proxy-equivalent MDP assumption, while a reliable gap in favor of the final-action-only run would falsify it.
Extended reading notes
Core claim
The paper's core proposal is that a harness's inference process can be treated as a black-box MDP and made trainable by instrumentation rather than by reimplementation. A proxy sits in front of the model server, intercepts every generation request the harness issues, and records the resulting prompt-response pairs as trajectory steps (H(s_t), a_t). When a rollout ends, the terminal success reward is assigned to every recorded step with the discount gamma^(T-t), normally gamma=1.0. Because each rollout runs in its own remote container managed by an orchestrator, the trainer never needs to understand the harness's internal subagents, planners, or context management; it just receives standard (
Load-bearing premise
The load-bearing premise is that every prompt-response pair the proxy records—including calls made by harness-internal subagents, planners, and context managers—is a genuine decision step that deserves the full terminal reward discounted with gamma=1.0; if those internal calls are not actually decision steps, RL credit assignment is training on noise.
Editorial extensions
If this is right
- Agents can be trained end-to-end in the exact harness and environment they are deployed in, eliminating the train-deploy mismatch caused by simplified training reimplementations.
- The same infrastructure works across text-based tool-use, browser-use, and computer-use agents, since only the sandbox container changes when the harness or environment changes.
- RL on top of SFT improves agentic reliability—self-verification, tool coverage, and multi-step plan completion—using only a few hundred RL tasks, and these gains transfer to harnesses not seen in training.
- Harness choice becomes a measurable training variable: simpler harnesses are easier to learn, and training across multiple harnesses helps more than training on a single one.
- Open teams can reproduce and scale this recipe with a standard RL codebase and a cloud provider, because each rollout is an independent container and there is no special coupling to the trainer.
Reading between the lines
- The paper's Eq. (1) treats every recorded prompt-response pair as a decision step receiving the full terminal reward; if a harness's internal subagent or planner makes calls that are not true policy decisions, RL credit assignment would train on noise. A natural ablation is to reward only the outermost agent's actions and compare.
- The observation that some harnesses are much harder to learn than others suggests co-adaptation: one could treat harness prompts, tool descriptions, and control flows as trainable parameters alongside the policy, rather than as fixed infrastructure.
- Since error recovery remains weak after RL, a testable extension is to add dedicated recovery-oriented rewards or curriculum tasks that force the agent to retry after failed commands, which the paper's data pipeline could generate automatically.
- The 'any harness x any environment' claim could be stress-tested by training on a harness whose internal control flow is stochastic or generates many subagent calls, and checking whether trajectory reconstruction still yields stable RL improvements.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. OpenForgeRL proposes an infrastructure for RL training of harness-based agents: a proxy intercepts all model calls made by an agent harness (e.g., OpenClaw, Codex, Kimi-Agent), records them as prompt-response pairs, and reconstructs them as RL trajectories; a Kubernetes orchestrator runs each rollout in a remote container. The paper validates this on text-based claw/tool-use agents (OpenForge-Claw, Qwen3-30B-A3B) and multimodal GUI agents (OpenForge-GUI, Qwen3-VL-8B), reporting improvements over similar-size open baselines on ClawEval, QwenClawBench, MCPAtlas, OSWorld-Verified, Online-Mind2Web, and WebVoyager. It also analyzes how harness choice and RL affect agent behavior, finding that simpler harnesses are easier to learn, RL improves self-verification and tool coverage, and error recovery remains weak. The central claim is that the proxy-based trajectory reconstruction makes any harness and environment trainable with standard RL codebases such as veRL.
Significance. If the empirical claims hold, OpenForgeRL is a practically valuable contribution: it addresses a real infrastructure gap by enabling end-to-end RL in the actual harnesses used at deployment, avoiding train-deploy mismatch, and it demonstrates results across a wider range of environments than prior work. The paper is strong in its engineering details: the appendices carefully document training setups, evaluation protocols, data synthesis, and even the exact action spaces, and the planned open-source release would allow others to reproduce and extend the work. The behavioral analysis (Section 5) is a useful addition, going beyond scores to examine harness-dependent learning difficulty and generalization. However, the validity of the RL signal under the proxy-equivalent MDP assumption is not established, and the empirical comparisons lack the statistical grounding needed to fully support the headline benchmark claims.
major comments (4)
- [§3.2, Eq. (1)] The proxy-equivalent MDP assumption is load-bearing but unvalidated. Eq. (1) assigns every recorded prompt-response pair, including harness-internal subagent/planner/context-manager calls, a reward r_t = γ^(T−t)·r_T with γ=1.0. This means the policy gradient trains on all internal calls as if they were environment-level decisions, giving full terminal credit to non-decision steps. The paper itself acknowledges in §3.2 that partial rollouts can inject misleading signal and defers better credit assignment to future work, yet this same broadcast reward is used in every reported RL run. Consequently, the benchmark gains in Tables 2–4 cannot distinguish genuine policy improvement from artifacts such as length bias or credit attributed to internal calls that did not causally produce the outcome. I request an ablation that isolates this assumption—for example, training with only terminal-step r
- [Tables 2 and 3] All reported benchmark numbers are single runs with no error bars, multiple seeds, or significance tests. This is particularly concerning in Table 2, where the gain of OpenForge-Claw(SFT+RL) over the SFT checkpoint on ClawEval pass3 is 31.7 vs. 21.7, but the difference over the strong baseline Qwen3-Coder-30B-A3B-Instruct (30.4) is small; without variance estimates the headline improvements may be within noise. Table 3 also compares models under different step caps (#Steps column): OpenForge-GUI is evaluated with 30 steps while several baselines use 50 or 100 steps. Because GUI success rates are sensitive to step budget, this is a confound. The MCPAtlas result is additionally on a reduced 89-task subset, which the paper transparently states, but it makes the '28.1' number not directly comparable to other MCPAtlas leaderboard values. I ask for error bars or multiple seeds on the main resu
- [§B.2] The SFT data filtering uses GPT-5.4 as a judge without the test-and-refine environment verifier, while RL tasks use the full verifier. The paper states this is affordable because SFT success signals are used only for one-time filtering, but the SFT trajectories are the initialization for RL, so label noise in SFT can propagate. The manuscript does not assess the judge's accuracy against the verifier, nor the effect of SFT label noise on the final RL policy. I request a small empirical check—e.g., judge agreement on a sample of RL-verified tasks—or a discussion of why SFT label noise is benign for the downstream RL result.
- [§5.2, Table 5] The generalization-to-unseen-harness analysis reports improvements of +3.3 and +4.6 on OpenClaw and Codex for a ZeroClaw-only model. These are single-run deltas over the base model and are small relative to the variance typical of these benchmarks. The paper interprets them as evidence of cross-harness transfer, but without repeated evaluations or confidence intervals the claim is not firmly supported. The same applies to the comparison between ZeroClaw-only and multi-harness training (46.0 vs. 48.5), which is a single trial. Please provide multiple evaluation runs or otherwise quantify uncertainty for the key numbers in Table 5.
minor comments (6)
- [§3.1] The trajectory τ is described as an 'unordered collection' but is then written as an ordered sequence. Please correct to 'ordered sequence' or 'list'.
- [§3.2, Eq. (1)] Notation is inconsistent: the text defines prompt-response pairs as (H(s_t), a_t) but Eq. (1) uses (s_H, a) without the time index. Align the notation.
- [§B.2, §B.3] Typos: 'suitbale' and 'afforadable' (§B.2), 'distrations' (§B.3), 'anf' (§B.3). Also, the phrase 'reserve only the last turn for more than 3 consecutive identical actions' is ambiguous—presumably it means remove repeated actions except the last, but the writing should be clarified.
- [Figure 1] The right panel labels 'OpenForge-RL' in the legend, but the corresponding model is called 'OpenForge-GUI' elsewhere. Also, the figure is dense and the reader's eye is drawn to the left panel; consider splitting or enlarging.
- [Table 3 caption] The header 'OnlineMind2Web' should be 'Online-Mind2Web' for consistency with the text.
- [§4.1, Table 2] The metric names 'pass3' and 'pass@3' are used side by side; clarify whether these are the same or different metrics (e.g., 'pass at 3 attempts' vs. 'pass@3 sampling').
Circularity Check
No significant circularity: the paper's central claims are benchmark outcomes measured on external evaluators, and no derivation step reduces to its own inputs or to a load-bearing self-citation.
full rationale
OpenForgeRL is an empirical systems paper; its headline results are pass rates on external benchmarks (ClawEval, QwenClawBench, MCPAtlas, OSWorld-Verified, Online-Mind2Web, WebVoyager) using official evaluation protocols. Equation (1), r_t = gamma^(T-t)*r_T with gamma=1.0, is a training credit-assignment choice, not a fitted constant renamed as a prediction: the terminal reward comes from task verifiers or LLM judges, and the reported benchmark gains are not entailed by this reward rule alone. The paper's self-citations to Orchard, OpenWebRL, and Synthetic-Computer-Use supply infrastructure, data pipelines, and asset pools; they are cited as code/data sources, not as uniqueness theorems or as the origin of the benchmark numbers. The paper itself flags the partial-rollout credit-assignment limitation in Section 3.2 ('Designing better credit assignment ... left to future work'), which is a correctness/robustness concern rather than a circular reduction. No step defines X in terms of Y, fits a parameter and then predicts a closely related quantity, or imports a forced choice through an unverified self-citation chain. The evaluation is self-contained against external benchmarks, so there is no significant circularity.
Assumptions & free parameters
free parameters (4)
- RL hyperparameters (LR, KL, entropy, batch/group, timeout) =
Claw: lr 1e-6, kl 0.001, entropy 0, batch 8, group 8, timeout 900s; Computer: 5e-7/0.01/0, 8/8, 600s; Browser: 1e-6/0.0/
- Trajectory reward decay γ =
1.0
- Browser task dedup threshold =
0.55 (embedding similarity)
- Evaluation step/time caps =
30 max steps for OSWorld/Mind2Web/WebVoyager; 20 max RL browser turns; 600-900s rollout timeouts
assumptions (5)
- standard math MDP formulation and GRPO advantage estimation are assumed correct.
- ad hoc to paper Proxy-recorded io-pairs (H(s_t), a_t) form a valid MDP trajectory; all harness-internal model calls are legitimate training steps and can receive r_t = γ^(T−t) r_T.
- domain assumption Synthetic verifiers and LLM judges score task success correctly; SFT tasks are filtered by GPT-5.4 judge without test-and-refine.
- domain assumption Training and evaluation task distributions do not overlap; Claw tasks seeded from ZClawBench/ClawHub are disjoint from ClawEval/QwenClawBench.
- domain assumption Orchard's Kubernetes orchestration works as described and is correctly inherited.
Cite this review
Pith. "Pith review of OpenForgeRL: Train Harness-native Agents in Any Environment." pith.science (2026). https://pith.science/paper/AVUDWPYU
@misc{pith2026260721557,
author = {Pith},
title = {Pith review of: OpenForgeRL: Train Harness-native Agents in Any Environment},
year = {2026},
howpublished = {\url{https://pith.science/paper/AVUDWPYU}},
note = {Machine review of arXiv:2607.21557}
}
read the original abstract
Modern AI agents rely on elaborate inference harnesses such as Claude Code, Codex, and OpenClaw to drive multi-turn reasoning, tool use, and access to external systems. While powerful, these complex harnesses also make agents hard to train end-to-end with open infrastructure, whose SFT/RL stacks cannot natively express stateful, multi-process harness inference. To address this, we present OpenForgeRL, an open-source framework for training harness-based agents end-to-end in diverse environments. OpenForgeRL achieves this with a lightweight proxy that serves the harness's model calls while recording them as training data for a standard RL codebase (e.g., veRL), and a Kubernetes orchestrator that runs each rollout in its own remote container, together enabling training on any harness in any environment at scale. By decoupling training and inference, OpenForgeRL allows researchers to easily train, study, and improve agents directly in the real harnesses and environments they are deployed with. We validate our framework across diverse, complex harnesses and environments, spanning tool/claw-based agents and multimodal GUI browser- and computer-use agents. Using only hundreds to a few thousand tasks, OpenForgeClaw reaches 31.7 pass^3 and 55.9 pass@3 on ClawEval and 33.7 on QwenClawBench. OpenForgeGUI reaches 37.7 on OSWorld-Verified, 63.0 on Online-Mind2Web, and 72.3 on WebVoyager. Both outperform open baselines of similar size on nearly all benchmarks, and in the GUI setting match or surpass models several times larger. Beyond benchmarks, we analyze how harness choice (e.g., ZeroClaw, OpenClaw, Codex) and RL shape agent behavior. We find that some harnesses are substantially harder to learn than others, and that RL improves agentic reliability, such as self-verification, tool coverage, and completing multi-step plans, though critical abilities such as error recovery remain weak.
Figures
Figures from the paper (2 more)
Reviewed August 1, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.