{"id":"075b3ae1-840f-4cc8-803b-6adc9b6b19c2","arxiv_id":"2607.21557","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Training agents inside their real deployment harnesses—via a lightweight proxy and containerized rollouts—substantially lifts tool-use and GUI benchmark performance over similarly sized open models.","lead":"OpenForgeRL is an open-source training framework that lets AI agents be trained directly inside the same inference harnesses (Claude Code, OpenClaw, browser-use scaffolds) they run in when deployed. Using a proxy to record model calls and cloud containers for rollouts, it reports large gains over open baselines on claw/tool-use and GUI benchmarks with only hundreds to a few thousand tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (1)'s proxy-equivalent MDP is unvalidated: broadcasting terminal reward to every recorded harness call, including subagent/planner/context-manager calls, may train on non-decisions and needs a filtered/terminal-only ablation.","rationale":"The reader's weakest_assumption correctly identifies Eq. (1) as the load-bearing modeling premise. I agree that this is the place where the central claim is least secure. The framework's core promise is that a proxy can turn harness inference into standard RL data; if the recorded decision steps do not correspond to real decisions, the RL signal is corrupted regardless of how well the Kubernetes orchestrator scales or how strong the final benchmark numbers are. I considered other possible concerns—single-run results without error bars, possible benchmark contamination from synthesized tasks, and code/data not yet released—but these affect the strength of the empirical validation rather than the validity of the training mechanism itself. The proxy-equivalent MDP assumption is more fundamental: if it fails, the reported RL improvements could stem from spurious length/verbosity correlations or from reinforcing harness-internal chatter rather than from improved task-level decision-making. The paper's own §3.2 discussion of partial-rollout credit assignment suggests the authors are aware that reward-to-go assignment is not trivially correct, yet they do not test whether the same issue affects complete trajectories with internal calls. Because the assumption is plausible and may well be valid—policy-gradient methods can in principle assign terminal rewards to every generated token—this does not warrant rejecting the paper. It does warrant a condition that the authors validate Eq. (1) with an ablation that separates genuine task actions from harness-internal calls. The proposed concrete test would settle whether the concern lands. Therefore the reader's CONDITIONAL verdict remains appropriate; no adjustment is needed.","tokens_in":24447,"tokens_out":7719,"duration_ms":81132,"concrete_test":"Instrument the proxy to tag each recorded call with its harness role (main-loop action, subagent, planner, context manager). From the same SFT checkpoint, train OpenForge-Claw under three reward variants: (a) Eq. (1) as published; (b) reward/advantage computed only on calls that directly issue environment/tool actions, with internal calls masked out of the RL loss; and (c) terminal-only reward (r_T at the final step, zeros elsewhere). Hold group size, seeds, and training steps fixed. Evaluate on ClawEval pass@1/pass@3 and QwenClawBench. If (b) or (c) matches or exceeds (a), the broadcast credit in Eq. (1) is not load-bearing; if (a) is required for the gains, the paper must demonstrate that the extra credit is not noise by inspecting which recorded calls actually drive the improvement.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires the proxy's recorded (H(s_t), a_t) pairs to be a valid RL decision sequence. Eq. (1) assigns every recorded prompt-response pair the reward r_t = γ^(T−t)·r_T, with γ=1.0, so every step receives the full terminal reward. But harness-internal model calls—subagents, planners, context managers—are not necessarily the policy's environment-level actions; they are part of the harness's control flow. The proxy records them indistinguishably from genuine task actions. If this equivalence fails, the policy gradient is trained on noise: internal calls that did not causally produce the outcome receive full credit or blame, and the effective objective is distorted (e.g., with γ=1.0 and r_t = r_T for every t, a successful trajectory's total reward is proportional to its number of recorded calls, implicitly rewarding verbosity; if rewards are averaged over steps, credit is diluted by internal calls). The paper never validates this equivalence or ablates it. The only related statement is in §3.2, where partial rollouts are discarded because a correct prefix receiving a negative reward can mislead, and the authors say \"designing better credit assignment ... is a promising direction ... left to future work.\" That is an acknowledgment that reward-to-go assignment is not automatically sound. Since the same broadcast reward is used in all reported RL runs, the benchmark gains in Tables 2–4 cannot by themselves distinguish genuine learning from artifacts of this assumption.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"OpenForgeRL proposes an infrastructure for RL training of harness-based agents: a proxy intercepts all model calls made by an agent harness (e.g., OpenClaw, Codex, Kimi-Agent), records them as prompt-response pairs, and reconstructs them as RL trajectories; a Kubernetes orchestrator runs each rollout in a remote container. The paper validates this on text-based claw/tool-use agents (OpenForge-Claw, Qwen3-30B-A3B) and multimodal GUI agents (OpenForge-GUI, Qwen3-VL-8B), reporting improvements over similar-size open baselines on ClawEval, QwenClawBench, MCPAtlas, OSWorld-Verified, Online-Mind2Web, and WebVoyager. It also analyzes how harness choice and RL affect agent behavior, finding that simpler harnesses are easier to learn, RL improves self-verification and tool coverage, and error recovery remains weak. The central claim is that the proxy-based trajectory reconstruction makes any harness and environment trainable with standard RL codebases such as veRL.","tokens_in":24830,"tokens_out":4477,"duration_ms":47298,"significance":"If the empirical claims hold, OpenForgeRL is a practically valuable contribution: it addresses a real infrastructure gap by enabling end-to-end RL in the actual harnesses used at deployment, avoiding train-deploy mismatch, and it demonstrates results across a wider range of environments than prior work. The paper is strong in its engineering details: the appendices carefully document training setups, evaluation protocols, data synthesis, and even the exact action spaces, and the planned open-source release would allow others to reproduce and extend the work. The behavioral analysis (Section 5) is a useful addition, going beyond scores to examine harness-dependent learning difficulty and generalization. However, the validity of the RL signal under the proxy-equivalent MDP assumption is not established, and the empirical comparisons lack the statistical grounding needed to fully support the headline benchmark claims.","major_comments":[{"comment":"The proxy-equivalent MDP assumption is load-bearing but unvalidated. Eq. (1) assigns every recorded prompt-response pair, including harness-internal subagent/planner/context-manager calls, a reward r_t = γ^(T−t)·r_T with γ=1.0. This means the policy gradient trains on all internal calls as if they were environment-level decisions, giving full terminal credit to non-decision steps. The paper itself acknowledges in §3.2 that partial rollouts can inject misleading signal and defers better credit assignment to future work, yet this same broadcast reward is used in every reported RL run. Consequently, the benchmark gains in Tables 2–4 cannot distinguish genuine policy improvement from artifacts such as length bias or credit attributed to internal calls that did not causally produce the outcome. I request an ablation that isolates this assumption—for example, training with only terminal-step r","section":"§3.2, Eq. (1)"},{"comment":"All reported benchmark numbers are single runs with no error bars, multiple seeds, or significance tests. This is particularly concerning in Table 2, where the gain of OpenForge-Claw(SFT+RL) over the SFT checkpoint on ClawEval pass3 is 31.7 vs. 21.7, but the difference over the strong baseline Qwen3-Coder-30B-A3B-Instruct (30.4) is small; without variance estimates the headline improvements may be within noise. Table 3 also compares models under different step caps (#Steps column): OpenForge-GUI is evaluated with 30 steps while several baselines use 50 or 100 steps. Because GUI success rates are sensitive to step budget, this is a confound. The MCPAtlas result is additionally on a reduced 89-task subset, which the paper transparently states, but it makes the '28.1' number not directly comparable to other MCPAtlas leaderboard values. I ask for error bars or multiple seeds on the main resu","section":"Tables 2 and 3"},{"comment":"The SFT data filtering uses GPT-5.4 as a judge without the test-and-refine environment verifier, while RL tasks use the full verifier. The paper states this is affordable because SFT success signals are used only for one-time filtering, but the SFT trajectories are the initialization for RL, so label noise in SFT can propagate. The manuscript does not assess the judge's accuracy against the verifier, nor the effect of SFT label noise on the final RL policy. I request a small empirical check—e.g., judge agreement on a sample of RL-verified tasks—or a discussion of why SFT label noise is benign for the downstream RL result.","section":"§B.2"},{"comment":"The generalization-to-unseen-harness analysis reports improvements of +3.3 and +4.6 on OpenClaw and Codex for a ZeroClaw-only model. These are single-run deltas over the base model and are small relative to the variance typical of these benchmarks. The paper interprets them as evidence of cross-harness transfer, but without repeated evaluations or confidence intervals the claim is not firmly supported. The same applies to the comparison between ZeroClaw-only and multi-harness training (46.0 vs. 48.5), which is a single trial. Please provide multiple evaluation runs or otherwise quantify uncertainty for the key numbers in Table 5.","section":"§5.2, Table 5"}],"minor_comments":[{"comment":"The trajectory τ is described as an 'unordered collection' but is then written as an ordered sequence. Please correct to 'ordered sequence' or 'list'.","section":"§3.1"},{"comment":"Notation is inconsistent: the text defines prompt-response pairs as (H(s_t), a_t) but Eq. (1) uses (s_H, a) without the time index. Align the notation.","section":"§3.2, Eq. (1)"},{"comment":"Typos: 'suitbale' and 'afforadable' (§B.2), 'distrations' (§B.3), 'anf' (§B.3). Also, the phrase 'reserve only the last turn for more than 3 consecutive identical actions' is ambiguous—presumably it means remove repeated actions except the last, but the writing should be clarified.","section":"§B.2, §B.3"},{"comment":"The right panel labels 'OpenForge-RL' in the legend, but the corresponding model is called 'OpenForge-GUI' elsewhere. Also, the figure is dense and the reader's eye is drawn to the left panel; consider splitting or enlarging.","section":"Figure 1"},{"comment":"The header 'OnlineMind2Web' should be 'Online-Mind2Web' for consistency with the text.","section":"Table 3 caption"},{"comment":"The metric names 'pass3' and 'pass@3' are used side by side; clarify whether these are the same or different metrics (e.g., 'pass at 3 attempts' vs. 'pass@3 sampling').","section":"§4.1, Table 2"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid systems/empirical contribution, but the central RL credit-assignment assumption needs direct validation before the benchmark results can be interpreted as evidence of genuine policy learning. The single-run comparisons and step-cap mismatches in Tables 2–3 are fixable with more evaluation, but they are currently load-bearing. I would encourage the editor to request at least the credit-assignment ablation and error bars on the main tables before accepting."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the two things to know. First, this is a real infrastructure contribution: a proxy plus Kubernetes orchestrator that lets you train any harness-based agent with standard RL stacks, demonstrated across claw/text and GUI/computer/browser domains on six benchmarks. Second, the headline numbers look plausible and internally consistent, but the paper's core modeling premise — treating every recorded harness model call as an RL decision step with full terminal reward broadcast (Eq. 1, gamma=1.0) — is not validated or ablated. That is the main thing to press on.\n\nWhat's genuinely new: the integration breadth. Proxy interception and remote containerized rollouts exist in Orchard, veRL, and concurrent Polar (SWE only). This paper shows end-to-end RL/GRPO across multiple harnesses (ZeroClaw, OpenClaw, Codex, ReACT, Kimi-Agent, Molmo-Web) and environment types, with a behavioral analysis of harness choice and RL effects. The SFT+RL gains over SFT are consistent across all three domains, and the data-synthesis pipeline is documented in enough detail (costs, verifier policy, task distributions) to reproduce.\n\nWhat's solid: the appendices are unusually thorough — hyperparameters, training curves, action spaces, evaluation protocols, and the decision to skip verifier-based filtering only for SFT while keeping test-and-refine for RL tasks is a sensible cost/quality tradeoff. Self-citations to Orchard/OpenWebRL are infrastructure-relevant and not load-bearing for the benchmark results. No red flags like p-hacking or circularity.\n\nSoft spots, in proportion:\n1. Eq. (1) is the load-bearing assumption. Broadcasting the terminal reward to every recorded prompt-response pair, including harness-internal subagent/planner/context-manager calls, with gamma=1.0, means total reward is proportional to trajectory length. The paper itself notes in §3.2 that partial rollouts can mislead and 'better credit assignment' is left to future work — that's an admission that reward-to-go isn't automatically sound. A filtered or terminal-only ablation would strengthen the causal claim substantially.\n2. Single-run benchmark scores without error bars or seeds. Given training is only 80–100 steps, multiple seeds are feasible and expected for this venue.\n3. Evaluation/training harness mismatches: ClawEval is evaluated with the official ReACT loop while training used mixed harnesses; GUI baselines use their own harnesses. This confounds harness and step budget in some comparisons, though the authors partly address this in Section 5.\n4. Code/data not yet released, so the infrastructure claims aren't independently checkable yet.\n\nBottom line: this deserves a serious referee. The framework is timely, the empirical scope is wide, and the weaknesses are fixable with an ablation and seeds. I'd bring it to reading group and cite it if I worked on agent training infrastructure.","headline":"A genuinely useful systems contribution with broad empirical validation, but the Eq. (1) credit-assignment assumption needs an ablation and the benchmark results need seeds before publication.","tokens_in":25297,"tokens_out":2670,"would_cite":true,"duration_ms":23857,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"OpenForgeRL claims that a recording proxy plus remote per-rollout containers can turn any agent harness into trainable RL data.","keywords":["harness-based agents","reinforcement learning","proxy trajectory reconstruction","containerized rollouts","RL training infrastructure","tool-use agents","GUI agents","agentic reliability"],"falsifier":"Take a harness whose internal planner emits many prompt-response pairs before a single successful action, train two identical RL runs on the same tasks—one rewarding every recorded pair with the terminal reward and one rewarding only the final action—and compare final benchmark performance; roughly equal results would support the proxy-equivalent MDP assumption, while a reliable gap in favor of the final-action-only run would falsify it.","tokens_in":24314,"feed_emoji":"🤖","tokens_out":6926,"duration_ms":67280,"temperature":0.7,"pith_summary":"OpenForgeRL is an attempt to make end-to-end training of harness-based agents practical for open research. The paper's central claim is that a lightweight proxy which intercepts and records every model call made by an agent harness, combined with a container orchestrator that runs each rollout in its own remote sandbox, converts the harness's stateful multi-process inference into ordinary state-action-reward samples that any standard RL codebase can consume. If correct, this removes the train-deploy mismatch that forces open efforts to reimplement simplified harnesses, and lets agents be optimized in the exact harness and environment they will be deployed in. The paper validates this on tool-use and multimodal GUI agents with only hundreds to a few thousand tasks: OpenForge-Claw reaches 31.7 on ClawEval and 33.7 on QwenClawBench, and OpenForge-GUI reaches 37.7 on OSWorld-Verified, 63.0 on Online-Mind2Web, and 72.3 on WebVoyager, beating similar-size open baselines on nearly all benchmarks. It also reports that RL mainly improves agentic reliability—self-verification, tool coverage, multi-step plan completion—while error recovery remains weak, and that some harnesses are substantially harder to learn than others.","feed_headline":"One proxy turns any agent harness into RL training data","feed_subtitle":"Recording proxy plus per-rollout containers let standard RL stacks train in the exact harness and environment.","key_machinery":"The load-bearing mechanism is the proxy-plus-orchestrator pair. The proxy wraps the harness's inference server and records every prompt-response pair the harness generates, including calls made by internal subagents and context managers, reconstructing each rollout as an unordered trajectory of (H(s_t), a_t) pairs. The orchestrator launches each rollout in its own remote container, so harness rollouts scale elastically and are fully decoupled from the training GPUs. Together they convert a stateful, multi-process harness into the flat (state, action, reward) sample format expected by standard RL training code, with terminal reward propagated to every step as gamma^(T-t) times r_T (gamma=1.0)","core_discovery":"The paper's core proposal is that a harness's inference process can be treated as a black-box MDP and made trainable by instrumentation rather than by reimplementation. A proxy sits in front of the model server, intercepts every generation request the harness issues, and records the resulting prompt-response pairs as trajectory steps (H(s_t), a_t). When a rollout ends, the terminal success reward is assigned to every recorded step with the discount gamma^(T-t), normally gamma=1.0. Because each rollout runs in its own remote container managed by an orchestrator, the trainer never needs to understand the harness's internal subagents, planners, or context management; it just receives standard (","pith_inferences":["The paper's Eq. (1) treats every recorded prompt-response pair as a decision step receiving the full terminal reward; if a harness's internal subagent or planner makes calls that are not true policy decisions, RL credit assignment would train on noise. A natural ablation is to reward only the outermost agent's actions and compare.","The observation that some harnesses are much harder to learn than others suggests co-adaptation: one could treat harness prompts, tool descriptions, and control flows as trainable parameters alongside the policy, rather than as fixed infrastructure.","Since error recovery remains weak after RL, a testable extension is to add dedicated recovery-oriented rewards or curriculum tasks that force the agent to retry after failed commands, which the paper's data pipeline could generate automatically.","The 'any harness x any environment' claim could be stress-tested by training on a harness whose internal control flow is stochastic or generates many subagent calls, and checking whether trajectory reconstruction still yields stable RL improvements."],"forward_implications":["Agents can be trained end-to-end in the exact harness and environment they are deployed in, eliminating the train-deploy mismatch caused by simplified training reimplementations.","The same infrastructure works across text-based tool-use, browser-use, and computer-use agents, since only the sandbox container changes when the harness or environment changes.","RL on top of SFT improves agentic reliability—self-verification, tool coverage, and multi-step plan completion—using only a few hundred RL tasks, and these gains transfer to harnesses not seen in training.","Harness choice becomes a measurable training variable: simpler harnesses are easier to learn, and training across multiple harnesses helps more than training on a single one.","Open teams can reproduce and scale this recipe with a standard RL codebase and a cloud provider, because each rollout is an independent container and there is no special coupling to the trainer."],"fun_headline_variants":["Train any agent harness with RL via a recording proxy","RL for any harness: just record and containerize","OpenForgeRL: transform harness calls into training data","Turn any environment into RL training with a proxy"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that every prompt-response pair the proxy records—including calls made by harness-internal subagents, planners, and context managers—is a genuine decision step that deserves the full terminal reward discounted with gamma=1.0; if those internal calls are not actually decision steps, RL credit assignment is training on noise.","fun_headline_variants_meta":{"raw":{"variants":["Train any agent harness with RL via a recording proxy","RL for any harness: just record and containerize","OpenForgeRL: transform harness calls into training data","Turn any environment into RL training with a proxy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000165,"raw_usage":{"total_tokens":1170,"prompt_tokens":907,"completion_tokens":263,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":651,"completion_tokens_details":{"reasoning_tokens":200}},"tokens_in":651,"tokens_out":263,"duration_ms":3612,"temperature":1.0,"reasoning_tokens":200,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T07:03:55.042138+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a harness whose internal planner emits many prompt-response pairs before a single successful action, train two identical RL runs on the same tasks—one rewarding every recorded pair with the terminal reward and one rewarding only the final action—and compare final benchmark performance; roughly equal results would support the proxy-equivalent MDP assumption, while a reliable gap in favor of the final-action-only run would falsify it.","supporting_citations":[],"review_version":1}