{"id":"421462e6-12fe-48ae-bc50-6019441b9e12","arxiv_id":"2605.24202","paper_version":2,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"Multi-agent RL on LLM workflows improves base models depending on workflow, task, and scale, with Isolated-Policy reaching higher peaks but more terminal accuracy cliffs than Shared-Policy.","lead":"The paper finds that end-to-end RL training of multi-agent LLM workflows usually improves accuracy over base models, but the gains depend jointly on workflow type, task, and model scale rather than policy sharing alone. A smart generalist might read it to understand practical design choices for building reliable multi-agent AI systems on reasoning tasks.","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.3","headline":"Gradient-dynamics explanation for failure patterns rests on unverified assumptions about per-role gradient amplification","rationale":"Reader's weakest assumption correctly flags the limited matrix (3 workflows × 2 tasks × 3 scales) as insufficient for general claims. The additional load-bearing issue is that the mechanistic explanation itself is not directly tested, which is required to move from 'we observe different failure signatures' to 'topology routes training pressure through these channels.' Empirical 'usually improves' result can stand with caveats; the explanatory part needs the gradient check.","tokens_in":1864,"tokens_out":338,"duration_ms":32664,"concrete_test":"Re-run the Voting and Orch-Workers Isolated-Policy trainings while logging per-role policy-gradient L2 norms (or cosine similarities across parallel agents) at each update; if the predicted amplification for same-role agents on shared prompts is absent or does not temporally precede the terminal accuracy cliff, the gradient-dynamics account does not explain the observed patterns.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that policy-sharing effects are explained by 'role-level gradient dynamics induced by workflow topology and policy routing' (parallel same-role agents amplifying gradients under Isolated-Policy; asymmetric per-step mass under Shared-Policy) is load-bearing for the assertion that gains 'depend jointly on workflow, task, and scale, not on policy sharing alone.' No indication is given that gradients were measured, that per-role gradient norms were logged, or that ablations decoupled topology from reward sparsity or prompt overlap. Without those, the mechanistic account remains post-hoc interpretation of accuracy curves rather than a tested causal link.","agreement_with_reader":"partial"},"referee_report":{"model":"grok-4.3","summary":"The paper empirically studies when end-to-end RL training of multi-agent LLM workflows improves over base models. It compares Shared-Policy (single policy for all roles) versus Isolated-Policy (per-role parameters) training across Eval-Opt, Voting, and Orch-Workers workflows on math and code tasks at three scales (0.6B, 1.7B, 4B). Key claims are that gains depend jointly on workflow, task, and scale rather than policy sharing alone; Isolated-Policy reaches higher peak accuracy but more often exhibits terminal accuracy cliffs, while Shared-Policy redistributes failures into different patterns; and these patterns are explained by role-level gradient dynamics induced by workflow topology and policy routing (parallel amplification under Isolated-Policy; asymmetric per-step mass under Shared-Policy).","tokens_in":1974,"tokens_out":484,"duration_ms":28038,"significance":"If the empirical map and mechanistic account hold, the work is significant for mapping conditional tradeoffs in multi-agent RL for LLMs and for attempting to link observed accuracy patterns to workflow-induced gradient effects. The experimental matrix spanning multiple workflows, tasks, and scales is a clear strength, as is the focus on failure modes rather than only success cases. This could guide design choices in multi-agent LLM systems.","major_comments":[{"comment":"The mechanistic explanation in the abstract (role-level gradient dynamics from workflow topology and policy routing, with parallel same-role agents amplifying gradients under Isolated-Policy and asymmetric per-step mass under Shared-Policy) is load-bearing for the central claim that gains 'depend jointly on workflow, task, and scale, not on policy sharing alone.' No indication is given that per-role gradient norms were measured, that ablations decoupled topology from reward sparsity or prompt overlap, or that gradient dynamics were logged; without these the account remains post-hoc interpretation of accuracy curves rather than a tested causal link.","section":"gradient-dynamics explanation (abstract and analysis section)"}],"minor_comments":[{"comment":"The abstract states that multi-agent RL 'usually improves over base models' but provides no quantitative effect sizes, error bars, or details on how 'improvement' and 'failure' are operationalized across the experimental matrix.","section":"abstract"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive review and for highlighting both the strengths of the experimental matrix and the need for precision around the mechanistic account. We address the major comment below.","responses":[{"response":"We agree the gradient-dynamics discussion is interpretive rather than directly tested. The manuscript reports no per-role gradient norm measurements, no ablations separating topology from reward sparsity or prompt overlap, and no logged gradient trajectories. The core empirical claim—that performance gains depend jointly on workflow, task, and scale—is established by the accuracy results across the full matrix of workflows (Eval-Opt, Voting, Orch-Workers), tasks (math, code), and scales (0.6B–4B); this evidence stands independently of the proposed mechanism. The role-level gradient account is presented as a post-hoc explanation that aligns with the observed patterns of terminal cliffs under Isolated-Policy and redistributed failures under Shared-Policy. We will revise the abstract and analysis section to (i) explicitly label the gradient dynamics as an interpretive hypothesis consistent with the accuracy curves and workflow structure, (ii) separate it from the empirical claims, and (iii) note the absence of direct gradient or ablation evidence. This change will be made in the next revision.","revision_made":"yes","referee_comment":"The mechanistic explanation in the abstract (role-level gradient dynamics from workflow topology and policy routing, with parallel same-role agents amplifying gradients under Isolated-Policy and asymmetric per-step mass under Shared-Policy) is load-bearing for the central claim that gains 'depend jointly on workflow, task, and scale, not on policy sharing alone.' No indication is given that per-role gradient norms were measured, that ablations decoupled topology from reward sparsity or prompt overlap, or that gradient dynamics were logged; without these the account remains post-hoc interpretation of accuracy curves rather than a tested causal link."}],"tokens_in":1496,"tokens_out":400,"duration_ms":27921,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main takeaway is that multi-agent RL on LLM workflows usually beats the base models, yet the gains hinge on the specific workflow, task, and model size rather than on whether policies are shared or isolated. Isolated training can hit higher peaks but risks sharper drops at the end; shared training spreads the failure modes differently. That empirical pattern across Eval-Opt, Voting, Orch-Workers, math and code tasks, and three scales is the concrete contribution.\n\nWhat the work does well is run a reasonably broad matrix and tie the accuracy curves to workflow topology and routing. The claim that policy sharing just reroutes training pressure instead of providing blanket stability is a useful design point for people building these systems.\n\nThe soft spot is the mechanistic story. The abstract attributes the failure patterns to role-level gradient amplification under isolated policies and asymmetric mass under shared ones, but nothing in the provided text shows that gradients were actually logged or that ablations separated topology from reward structure or prompt overlap. Without those measurements the explanation stays post-hoc. If the full paper has the per-role norm logs or the decoupling experiments, that would tighten it; otherwise the central causal link rests on curve interpretation.\n\nThis is the kind of paper that belongs in a reading group for people working on agentic pipelines. It is not a foundational result, but the experimental scope makes it worth citing if you are choosing between shared and isolated setups on similar tasks. It deserves a serious referee because the question is practical and the matrix is non-trivial, even if the gradient account needs more direct evidence.","headline":"The paper maps multi-agent RL tradeoffs across workflows and scales with a gradient-dynamics account, but that account looks interpretive rather than directly tested.","tokens_in":2467,"tokens_out":388,"would_cite":false,"duration_ms":14440,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Multi-agent RL improves LLM workflows depending jointly on workflow, task, and scale rather than policy sharing alone.","keywords":["multi-agent RL","LLM workflows","policy sharing","gradient dynamics","workflow topology","reinforcement learning","agent roles","accuracy degradation"],"falsifier":"A new experiment at an untested scale or task where Isolated-Policy no longer shows terminal cliffs or where Shared-Policy produces no distinct failure patterns would falsify the dependence on workflow, task, and scale.","tokens_in":2769,"feed_emoji":"","tokens_out":794,"duration_ms":33935,"temperature":0.7,"pith_summary":"The paper establishes that applying reinforcement learning to train roles in multi-agent LLM systems typically raises accuracy above base models, yet the size of the gains and the stability of training vary with the workflow structure, the nature of the task, and the size of the underlying models. Isolated-Policy training, where each role has separate parameters, often achieves the highest accuracies but is more likely to encounter a sharp drop in performance late in training. Shared-Policy training, by contrast, avoids some of those drops but produces different kinds of failures because the single policy gets pulled toward the role that receives the most gradient updates. The authors link these outcomes to how the workflow's routing of prompts and the sharing of parameters shape the flow of gradients during training. This matters for practitioners who want to know whether investing in multi-agent RL will pay off for their particular setup.","feed_headline":"Multi-agent RL improves LLM accuracy conditionally on workflow and scale","feed_subtitle":"Isolated policies reach higher peaks but risk cliffs; shared policies shift failures through role-specific gradient flows.","key_machinery":"Role-level gradient dynamics induced by workflow topology and policy routing, which control how training pressure is distributed across roles in Shared-Policy versus Isolated-Policy setups.","core_discovery":"End-to-end RL training of multi-agent LLM workflows improves over base models in a manner that depends jointly on workflow, task, and scale, not on policy sharing alone. Isolated-Policy training tends to reach higher peak accuracy yet falls off a terminal accuracy cliff more often because parallel same-role agents on shared prompts amplify per-role gradients. Shared-Policy training does not eliminate failure but redistributes it into qualitatively different patterns because asymmetric per-step gradient mass causes the shared policy to be captured by the dominant role. These patterns are explained by role-level gradient dynamics induced by workflow topology and policy routing.","pith_inferences":["Practitioners should evaluate both policy configurations at their target scale and task instead of assuming shared policies are always more stable.","The gradient mechanisms imply that altering prompt routing or adding role-specific regularization could reduce the observed failure modes.","Similar tradeoffs may appear in non-LLM multi-agent RL settings where roles interact through shared environments.","Future experiments at larger scales could test whether the scale dependence persists or if failures become less pronounced."],"forward_implications":["Multi-agent RL gains vary by workflow (Eval-Opt, Voting, Orch-Workers), so different topologies produce different improvement levels.","Isolated-Policy amplifies gradients in parallel roles, leading to terminal degradation in Voting and Orch-Workers.","Shared-Policy causes capture by the dominant role due to asymmetric gradient mass, producing task-specific failure signatures.","These effects appear across math and code tasks and model scales from 0.6B to 4B.","Policy sharing routes training pressure differently rather than providing uniform stability."],"fun_headline_variants":["Multi-agent RL gains depend on workflow task and model scale","Isolated policies peak higher but often hit accuracy cliffs in RL","Shared policy training shifts failure patterns through gradient mass","Role level gradients drive RL outcomes in multi-agent LLM workflows"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"That the tested combinations of workflows, tasks, and scales capture the general conditions under which multi-agent RL succeeds or fails and that the gradient dynamics explain the failures.","fun_headline_variants_meta":{"raw":{"variants":["Multi-agent RL gains depend on workflow task and model scale","Isolated policies peak higher but often hit accuracy cliffs in RL","Shared policy training shifts failure patterns through gradient mass","Role level gradients drive RL outcomes in multi-agent LLM workflows"]},"model":"grok-4.3","cost_usd":0.007202,"raw_usage":{"total_tokens":3378,"prompt_tokens":779,"num_sources_used":0,"completion_tokens":57,"cost_in_usd_ticks":72024500,"prompt_tokens_details":{"text_tokens":779,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":2542,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":779,"tokens_out":57,"duration_ms":27090,"temperature":1.0,"reasoning_tokens":2542,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-30T15:43:28.643240+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A new experiment at an untested scale or task where Isolated-Policy no longer shows terminal cliffs or where Shared-Policy produces no distinct failure patterns would falsify the dependence on workflow, task, and scale.","supporting_citations":[],"review_version":1}