{"id":"ad9e7785-4fef-4369-954e-dc346c5d884d","arxiv_id":"2605.30461","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A consensus-based distributed algorithm for constrained MARL with separable dynamics achieves linear scalability and bounded constraint violations through state-augmented policies and dual variable agreement.","lead":"This paper develops a scalable method for multi-agent reinforcement learning under global constraints by having each agent learn a policy that includes a dual variable for constraint feedback and then using local communication to agree on that variable. Smart generalists might read it because it solves a key bottleneck in applying RL to large-scale systems like energy grids where independent agents cannot coordinate on shared limits without central control.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Consensus error bound may require rounds scaling with graph diameter for fixed violation tolerance, undermining linear execution scaling on sparse graphs","rationale":"Reader correctly flags separability as necessary for independent policies, but the load-bearing step for the coordination claim itself is the consensus-to-violation translation under realistic large-N graphs. The proposed test directly checks whether the number of rounds stays bounded independently of N.","tokens_in":1813,"tokens_out":318,"duration_ms":20116,"concrete_test":"Implement the consensus iteration on a path graph of N=1024 agents with random initial multipliers in [0,10]; measure rounds until max pairwise disagreement < 0.01. If rounds exceed 200 for this N while remaining <20 for N=32, the per-step cost grows and the linear scalability claim weakens.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim rests on the statement that bounded consensus error on multipliers translates to bounded constraint violation that decreases with connectivity and rounds. For separable dynamics this translation is plausible via sensitivity of the Lagrangian, but the paper's mild connectivity assumption does not preclude graphs (e.g., paths or trees) whose spectral gap shrinks as 1/N or worse. In such cases the number of neighbor-to-neighbor rounds needed to drive multiplier disagreement below the threshold that keeps violation acceptable grows with diameter; each execution step then incurs non-constant communication, contradicting the linear-in-N execution claim. The proof sketch in the abstract does not appear to quantify this dependence or restrict the graph family beyond connectivity.","agreement_with_reader":"partial"},"referee_report":{"model":"grok-4.3","summary":"The paper proposes a distributed constrained MARL algorithm for agents with separable dynamics but global resource constraints. Each agent learns an offline policy augmented with a dual variable (Lagrange multiplier) encoding constraint feedback; at execution, agents perform neighbor-to-neighbor consensus on the multipliers. The central claim is that, under mild connectivity assumptions, the resulting consensus error is bounded and this bound translates into a bounded constraint violation that improves with graph connectivity and the number of consensus rounds. The method is asserted to scale linearly in both training and execution (unlike CTDE), with experiments on a smart-grid demand-response task showing that consensus is required for feasible solutions while independent learning produces degenerate policies.","tokens_in":1957,"tokens_out":480,"duration_ms":17997,"significance":"If the claimed translation from bounded multiplier consensus error to bounded constraint violation holds with explicit dependence only on local graph properties that preserve linear scaling, the approach would offer a practical route to large-scale constrained MARL that avoids the quadratic cost of centralized training. The empirical contrast between consensus-enabled feasibility and independent-learning degeneracy is a useful demonstration. The combination of state augmentation with distributed dual consensus is a novel synthesis for this separable-dynamics setting.","major_comments":[{"comment":"Abstract: the proof sketch asserts that bounded consensus error on multipliers translates to bounded constraint violation that decreases with graph connectivity and consensus rounds, yet provides no explicit dependence of the required number of rounds on graph diameter or spectral gap. Under the stated 'mild connectivity' assumption, graphs such as paths or trees (spectral gap O(1/N)) would require rounds scaling with diameter to keep violation below a fixed tolerance, contradicting the linear-in-N execution claim. This dependence is load-bearing for the scalability assertion.","section":"Abstract"},{"comment":"Abstract and experimental section: the manuscript reports that consensus is 'essential for feasibility' on the smart-grid task and that the method scales to thousands of agents, but supplies neither the full derivation of the error-to-violation bound, experiment hyperparameters, nor error bars on the reported constraint violations or demand-fulfillment metrics. Without these, the empirical support for the central claim cannot be assessed at the level required for a soundness judgment.","section":"Abstract"}],"minor_comments":[],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive and detailed comments. We address each major point below, indicating planned revisions where appropriate to improve clarity and completeness.","responses":[{"response":"The abstract summarizes the high-level result; the full proof in the appendix derives the consensus error bound explicitly in terms of the number of rounds k and the spectral gap (second-smallest eigenvalue of the normalized Laplacian). The constraint violation is then bounded proportionally to this error. We agree the dependence is important. The 'mild connectivity' assumption is intended to cover graphs with spectral gap bounded below by a positive constant independent of N (e.g., expanders), allowing fixed k for any target violation tolerance and thereby preserving linear scaling. For path or tree graphs the gap scales as O(1/N) and more rounds would be required, which we acknowledge would impact the claim. We will revise the abstract and add a dedicated paragraph clarifying the precise graph conditions needed for linear scaling. This constitutes a partial revision.","revision_made":"partial","referee_comment":"[Abstract] Abstract: the proof sketch asserts that bounded consensus error on multipliers translates to bounded constraint violation that decreases with graph connectivity and consensus rounds, yet provides no explicit dependence of the required number of rounds on graph diameter or spectral gap. Under the stated 'mild connectivity' assumption, graphs such as paths or trees (spectral gap O(1/N)) would require rounds scaling with diameter to keep violation below a fixed tolerance, contradicting the linear-in-N execution claim. This dependence is load-bearing for the scalability assertion."},{"response":"We will ensure the revised manuscript supplies all requested details. The complete derivation of the error-to-violation bound already appears in the appendix; we will move a concise version into the main text for accessibility. A new 'Experimental Details' subsection will list all hyperparameters (learning rates, consensus rounds per step, network sizes, training episodes, etc.). Error bars are included on the figures; we will add explicit discussion of them in the text and report numerical values with standard deviations. These changes will allow full assessment of the empirical claims.","revision_made":"yes","referee_comment":"[Abstract] Abstract and experimental section: the manuscript reports that consensus is 'essential for feasibility' on the smart-grid task and that the method scales to thousands of agents, but supplies neither the full derivation of the error-to-violation bound, experiment hyperparameters, nor error bars on the reported constraint violations or demand-fulfillment metrics. Without these, the empirical support for the central claim cannot be assessed at the level required for a soundness judgment."}],"tokens_in":1517,"tokens_out":554,"duration_ms":27493,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main takeaway is that agents can learn a single policy conditioned on both local state and a dual variable, then run lightweight neighbor consensus on that dual at runtime to coordinate on a global constraint. This keeps training fully decentralized and independent while still producing feasible joint behavior, which independent learners cannot achieve on their own.\n\nThe paper does this cleanly. The smart-grid demand-response experiments show the practical problem: without consensus every agent defers consumption indefinitely to meet the capacity limit, a degenerate outcome. With consensus the agents converge to a shared multiplier and satisfy both the grid constraint and actual demand, scaling to thousands of agents where CTDE methods already fail. That contrast is direct evidence that the coordination step matters.\n\nThe soft spot is the communication scaling. The abstract states that consensus error is bounded under mild connectivity and that the bound on violation improves with connectivity and rounds. But mild connectivity includes path or tree graphs whose diameter grows with N. On those graphs the number of rounds needed to drive multiplier disagreement below the level that keeps violation acceptable can grow with diameter, so per-step communication is no longer constant. The proof sketch does not appear to restrict the graph family or quantify this dependence, which leaves the linear-in-N execution claim open to the stress-test concern.\n\nThe work is aimed at people building constrained MARL for resource allocation settings. It deserves a serious referee because the motivation is concrete, the empirical demonstration is useful, and the technical move is straightforward to evaluate even if the graph dependence needs tightening.","headline":"The paper shows state augmentation plus local consensus on duals can enforce global constraints in separable-dynamics MARL at large scale, but the linear execution claim is not yet secure on sparse graphs.","tokens_in":2452,"tokens_out":385,"would_cite":false,"duration_ms":22821,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"State augmentation with neighbor consensus on Lagrange multipliers enables scalable constrained MARL under separable dynamics.","keywords":["constrained MARL","state augmentation","consensus optimization","Lagrange multipliers","separable dynamics","distributed reinforcement learning","smart grid demand response","multi-agent systems"],"falsifier":"Observing whether constraint violations remain bounded when the communication graph is disconnected or when one agent's transition depends on another's action.","tokens_in":2710,"feed_emoji":"🤖","tokens_out":444,"duration_ms":22229,"temperature":0.7,"pith_summary":"The paper establishes that agents with independent dynamics can still coordinate to meet shared constraints by augmenting each policy with a shared dual variable that tracks the constraint and then averaging that variable locally with neighbors. A sympathetic reader would care because standard independent learning leaves agents unable to allocate the shared resource correctly, often leading to either violations or useless solutions like never using the resource. The authors prove that the local averaging keeps the dual variables close enough across agents that the resulting joint behavior violates the constraint by only a bounded amount, and this bound improves with denser communication graphs or extra averaging steps. This keeps both training and runtime linear in the number of agents rather than quadratic, enabling experiments with thousands of agents on a demand-response task where the method succeeds and baselines fail.","feed_headline":"Consensus on dual variables scales constrained MARL to thousands of agents","feed_subtitle":"Local averaging of constraint signals produces feasible solutions for large agent teams where independent learning does not.","key_machinery":"State-augmented policies conditioned on a dual variable, combined with distributed consensus over those dual variables via neighbor-to-neighbor communication.","core_discovery":"The central claim is that lightweight neighbor-to-neighbor consensus over Lagrange multipliers suffices for globally coordinated constraint enforcement while preserving the scalability of independent training. Each agent learns a single augmented policy offline, conditioned on both its local state and a dual variable encoding constraint feedback. During execution, agents reach agreement on this dual variable through local communication alone. Under mild connectivity assumptions, the consensus error among agents' multipliers is bounded, translating to a bounded constraint violation that decreases with graph connectivity and the number of consensus rounds.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["Dual consensus scales constrained MARL","Local dual consensus coordinates agent teams","Multiplier consensus enforces feasible MARL solutions","State augmentation with dual consensus for MARL"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"Each agent's state transition depends only on its own action and state, not on those of other agents.","fun_headline_variants_meta":{"raw":{"variants":["Dual consensus scales constrained MARL","Local dual consensus coordinates agent teams","Multiplier consensus enforces feasible MARL solutions","State augmentation with dual consensus for MARL"]},"model":"grok-4.3","cost_usd":0.007068,"raw_usage":{"total_tokens":3223,"prompt_tokens":738,"num_sources_used":0,"completion_tokens":48,"cost_in_usd_ticks":70678000,"prompt_tokens_details":{"text_tokens":738,"audio_tokens":0,"image_tokens":0,"cached_tokens":64},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":2437,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":738,"tokens_out":48,"duration_ms":18498,"temperature":1.0,"reasoning_tokens":2437,"cache_read_input_tokens":64,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-29T08:36:16.014166+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Observing whether constraint violations remain bounded when the communication graph is disconnected or when one agent's transition depends on another's action.","supporting_citations":[],"review_version":1}