{"id":"ae03c4b2-0a3b-4ee5-bdd9-84e692d1af20","arxiv_id":"2506.01639","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Bidirectional SAC combines an explicit forward-KL policy projection with reverse-KL policy refinement and reports up to 30% higher episodic rewards on MuJoCo and Box2D continuous control tasks.","lead":"This paper proposes Bidirectional SAC, a reinforcement learning algorithm that first estimates a good policy using forward KL divergence and then refines it with the usual reverse KL updates. In continuous control tests, it reports faster convergence and up to 30% higher rewards than plain SAC.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The forward-KL projection is mathematically sound, but the VDN-a critic that supplies the Boltzmann marginals is unidentifiable: any additive gauge transformation of the subnetworks leaves the total Q unchanged while changing the estimated marginals, so 'sufficiently learned U_aux' (Appendix…","rationale":"I read the paper in good faith. The paper's central theoretical observation is clean: with a diagonal Gaussian policy, minimizing the forward KL divergence to a Boltzmann distribution yields an explicit projection whose mean and variance are the marginal moments of the Boltzmann distribution. That derivation is internally consistent. The paper should also be credited for clearly identifying the computational obstacle: obtaining those marginals is hard because the Q-function couples action dimensions. The proposed remedy, VDN-a, is where the argument becomes insecure. The copula decomposition in Eq. (11) is mathematically exact, but the network architecture does not enforce the proposed split of labor between the additive subnetworks and the auxiliary copula term. Because the total Q is the only supervised quantity, any additive perturbation of the subnetworks can be exactly compensated by U_aux. Consequently, the subnetworks need not learn ln M^(j) up to a constant; they may learn an arbitrary gauge-shifted version of it. The Appendix C.2.4 proof assumes the very condition that must be learned, and the paper's own Figure 9 shows a failure case. This is not a disagreement with consensus; it is an internal gap between the network parameterization and the claimed marginal recovery. If the marginals are wrong, the forward projection that initializes and regularizes Bidirectional SAC is wrong, so the method's theoretical motivation collapses even if the empirical curves happen to look good. I do not see grounds for rejection outright, because the issue could be resolved by adding an explicit decomposition loss or by providing a controlled experiment showing moment recovery independent of initialization. But the paper as submitted cannot support the central claim without that evidence. The reader's weakest-assumption analysis identified the same VDN-a marginal-recovery dependence; my stress-test refines it into an identifiability failure with a concrete experimental test.","tokens_in":17048,"tokens_out":6771,"duration_ms":83813,"concrete_test":"Run a supervised identifiability check on a fixed state-less 2D target Q(a1,a2) = a1^2 + a2^2 + 3 a1 a2, whose Boltzmann marginals can be computed exactly by 2D quadrature. Train VDN-a with several random initializations to minimize only (Qhat_total - Q)^2 over a dense grid until the total error is negligible (e.g., < 1e-6). Then compute the estimated marginals from Qsub_1 and Qsub_2 and their first two moments. If the inferred means and variances vary across seeds or deviate from the true marginals despite negligible total error, the VDN-a decomposition cannot recover the marginals required for Eq. (5), and Bidirectional SAC's forward anchor is miscalibrated. A passing result would require stable, accurate moments across all seeds and, ideally, across different gauge-regularization choices.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The load-bearing step is not the KL calculus in Section 3.2, which correctly shows that the forward-KL projection of a Boltzmann distribution onto a diagonal Gaussian is given by the marginal means and variances. The fragile step is the claim that VDN-a recovers those Boltzmann marginals. From Sklar's theorem, Eq. (11) decomposes the Boltzmann potential as (1-n)ln Z + sum_j ln M^(j)(a_j) + L_c(a), where L_c is the log-copula interaction term. The VDN-a network represents the total Q as sum_j Qsub_j(s,a_j) + U_aux(s,a). The training loss supervises only the total sum, so the split between the subnetworks and U_aux is unidentifiable: for any functions g_j(a_j), replacing Qsub_j by Qsub_j + g_j and U_aux by U_aux - sum_j g_j leaves the total Q exactly unchanged but changes each exp(Qsub_j), and therefore changes the estimated marginal q_i. Appendix C.2.4's recovery proof begins with 'if exp(Qhat_j) = K_j M^(j)', which is precisely the unidentified property that needs to be established; 'sufficiently learned U_aux' is not a mechanism that forces this decomposition. Figure 9 in the appendix concedes that the approximation fails on bimodal marginals. As a result, the f_i* and Sigma_i* that anchor Bidirectional SAC's initialization and regularization can be miscalibrated even when the total critic is perfect. This makes the central forward-projection claim conditional on an unproved, and as stated non-identifiable, representation; the paper's own conclusion acknowledges that the projection is only approximated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates the use of forward KL divergence in the SAC framework. It derives that, for diagonal Gaussian policies, the forward-KL optimal projection of a Boltzmann target onto the policy family is given by the marginal means and variances of the target distribution (Section 3.2). To compute these marginals efficiently, the paper introduces VDN-a, a single-agent adaptation of Value Decomposition Networks, and proposes two algorithms: Forward SAC, which updates the policy directly via the explicit projection, and Bidirectional SAC, which uses the forward projection to initialize and regularize the standard reverse-KL actor update (Section 4.2). Experiments on MuJoCo and Box2D environments claim that Bidirectional SAC improves sample efficiency and achieves up to a 30% increase in episodic rewards over SAC and other baselines (Section 5.1).","tokens_in":17423,"tokens_out":5571,"duration_ms":50621,"significance":"The explicit forward-KL projection formula in Section 3.2 is a valid and potentially useful observation: for a factorization of the Gaussian policy into independent diagonal components, the optimal projection is exactly the moment-matching solution. The proposed combination of forward and reverse KL in Bidirectional SAC is conceptually appealing, as it offers a concrete way to initialize the actor closer to the target before applying the reverse-KL update. However, the practical value of the method rests on the VDN-a critic's ability to recover the true Boltzmann marginals, and this is not established. Appendix C.2.4 proves marginal recovery only from an unverified 'sufficiently learned' condition, and the decomposition underlying VDN-a is not identifiable. Therefore, the central algorithmic claim is conditional; the theoretical contribution is sound in isolation but not yet connected to the proposed algorithm in a rigorous way.","major_comments":[{"comment":"The claim that the VDN-a critic recovers the marginal Boltzmann distributions is not established. The additive decomposition Q(s,a)=Σ_j Qsub_j(s,a_j)+U_aux(s,a) is subject to a gauge freedom: for arbitrary functions g_j(a_j), replacing Qsub_j by Qsub_j+g_j and U_aux by U_aux-Σ_j g_j leaves the total Q invariant but changes each exp(Qsub_j) and hence all estimated marginals. Thus, a perfectly learned total Q does not uniquely determine the marginal Q components. The recovery proof in Appendix C.2.4 begins with the assumption exp(Qhat_j)=K_j M^(j), which is exactly the property that needs to be shown; the 'sufficiently learned U_aux' condition is a restatement, not a mechanism, of the desired outcome. Since the forward-projection moments f_i^* and Sigma_i^* in Eq. (7) are computed from these marginals, the central anchor of Bidirectional SAC is miscalibrated unless an identifiability constraint is imposed or a separate training objective enforces the decomposition.","section":"Section 3.3 / Appendix C.2.4"},{"comment":"The paper concedes that VDN-a's approximation fails on bimodal marginal structures. The Boltzmann distribution in continuous control can be multimodal, and the paper does not quantify how often or how severely the estimated mean/variance deviate from the true moments in such cases. The statement that 'the first two moments are often still well-captured' is not supported by any numerical evidence. This is a load-bearing concern because the forward projection uses only these moments, and a miscalibrated mean or variance can mislead both the initialization and the regularization in Bidirectional SAC.","section":"Appendix C.2.4, Figure 9"},{"comment":"The empirical claim of 'up to a 30% increase in episodic rewards' is not supported by standard statistical rigor. The learning curves are presented without error bars, and Appendix F does not report the number of independent runs or the variance across seeds. Without this information, the comparison against SAC, A2C, PPO, and DDPG is not verifiable, and the claim of 'significantly outperforms' is not substantiated. The paper should report means and standard deviations over at least 5-10 seeds for all environments and baselines.","section":"Section 5.1, Figure 4"}],"minor_comments":[{"comment":"The abstract and introduction state that the explicit projection holds 'for Gaussian policies'; the derivation in Section 3.2 specifically assumes a diagonal covariance matrix. Please state this assumption in the abstract or at the beginning of Section 3.2 to avoid overgeneralization.","section":"Abstract / Section 3.2"},{"comment":"Equation (12) introduces the integration bound b as a hyperparameter, and Equation (13) uses a Simpson partition count I, but no concrete values are given anywhere in the paper. Please provide these values or a sensitivity analysis.","section":"Equation (12) / Equation (13)"},{"comment":"In Section 3.2, the notation Σ^i_φ is used both for the standard deviation in the Gaussian density and for the variance in the optimal projection formula. This is confusing; please use distinct symbols (e.g., σ for standard deviation and σ^2 for variance).","section":"Section 3.2"},{"comment":"The captions for Figures 8 and 9 contain \"(Assumed: ...)\", which appears to be a placeholder. Please replace with proper captions that state precisely what is plotted (e.g., true vs. estimated marginal densities).","section":"Appendix C.2.4, Figures 8 and 9"},{"comment":"The paper does not compare against modern SAC variants that also improve sample efficiency (e.g., REDQ, DroQ, or distributional SAC). Adding such comparisons would help isolate the contribution of the bidirectional mechanism.","section":"Section 5.1"},{"comment":"In Section 5.2, Figure 5 reports a qualitative comparison of estimated marginals to ground truth for a single state; please provide quantitative metrics (e.g., KL divergence, error in mean/variance) over a set of states to substantiate the claim that the first two moments are captured.","section":"Section 5.2, Figure 5"}],"recommendation":"major_revision","confidential_remarks":"The identifiability issue in the VDN-a decomposition is the main technical obstacle. The forward-KL derivation is solid, but the link between the learned critic and the Boltzmann marginals is not proven. The paper would need either a normalization/constraint that makes the decomposition identifiable (e.g., zero-mean constraints on the subnetworks) or a different method to estimate the marginals, plus a more thorough empirical evaluation with seeds and error bars. As it stands, the paper is probably not ready for acceptance in a leading journal, but it contains a promising core idea that could be revised into a strong contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Let me give it to you straight. The paper's main mathematical observation — that minimizing forward KL to a Boltzmann target over a diagonal Gaussian family yields the marginal means and variances of the target — is correct, but it's a standard moment-matching result in variational inference, not a new discovery. The novel part is the packaging: anchor a reverse-KL SAC update with a forward-KL projection computed from a VDN-a critic. That combination is a reasonable incremental idea, and Section 3.2 is cleanly derived. The copula decomposition in Appendix C is a nice attempt to justify the VDN-a architecture.\n\nThe problem is the load-bearing step: the VDN-a critic's ability to recover Boltzmann marginals. The stress-test note is right. The training loss only supervises the total Q, so the split between subnetworks and the auxiliary network is unidentifiable. Add any function g_j to subnetwork j and subtract sum g_j from U_aux: total Q is unchanged, but the estimated marginals change. Appendix C.2.4's recovery proof simply assumes exp(Qhat_j) = K_j M^(j), which is exactly the property that needs to be established. 'Sufficiently learned U_aux' is not a mechanism that forces that decomposition. The paper's own Figure 9 concedes the approximation fails on bimodal marginals, and the conclusion admits the projection is only approximated. So the central anchor of Bidirectional SAC is conditional on an unproved, and as stated non-identifiable, representation.\n\nThe empirical side is also thin. No seeds, no error bars, no code, no hyperparameters. A 30% episodic reward claim without any variance reporting is hard to evaluate. The unfinished figure captions in Appendix C suggest the preprint is rough.\n\nTo be fair, the paper does several things well. It cites the relevant prior work, including forward-KL policy optimization (Chan et al., V-MPO). It correctly identifies the reverse-KL gradient instability. The derivation of the forward projection in the presence of tanh squashing is a useful detail. And the paper is honest about its own limitations in the conclusion.\n\nWho should read it? Researchers working on KL-based policy updates, especially those interested in making SAC more sample-efficient. It deserves peer review because the core idea is testable and the weaknesses are addressable, but I would not cite it in its current form. If I were handling it, I'd ask for code, seeds, and a rigorous treatment of the VDN-a identifiability issue before accepting.","headline":"The forward-KL projection derivation is correct but standard, and the Bidirectional SAC idea is reasonable, but the VDN-a critic's marginal recovery is unidentifiable and the experiments are too thinly reported to support the 30% claim.","tokens_in":17971,"tokens_out":2472,"would_cite":false,"duration_ms":25052,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Switching the KL direction in SAC turns its policy update into an explicit formula.","keywords":["soft actor-critic","forward KL divergence","reverse KL divergence","maximum entropy reinforcement learning","policy projection","value decomposition networks","continuous control","sample efficiency"],"falsifier":"Train the VDN-a critic on a fixed Q-function whose Boltzmann distribution has a known bimodal marginal; if the mean and variance computed by Simpson integration of the learned marginal deviate from the true values by more than the integration error, the projection anchor is miscalibrated. Selecting a continuous-control task with strong action-dimension coupling and comparing Bidirectional SAC's episodic reward against standard SAC with identical hyperparameters would also test the claim.","tokens_in":16814,"feed_emoji":"🎯","tokens_out":5085,"duration_ms":46882,"temperature":0.7,"pith_summary":"Standard Soft Actor-Critic improves its Gaussian policy by minimizing reverse KL divergence, which has no closed-form solution and is updated by noisy gradient steps. This paper shows that minimizing the forward KL divergence instead yields an explicit optimal projection: the new policy's mean and variance are exactly the mean and variance of the Boltzmann distribution's action marginals. The authors build Bidirectional SAC, which starts each policy update at that exact forward-KL projection and then refines it with reverse KL, preserving SAC's improvement guarantee while removing the wobbly starting point. Experiments on continuous-control benchmarks report faster convergence and up to a 30% increase in episodic reward over standard SAC. The claim matters because it identifies a simple, non-parametric anchor for policy improvement in maximum-entropy reinforcement learning.","feed_headline":"Switching KL direction makes SAC's policy update explicit","feed_subtitle":"Starting every update at the exact mean and variance of the Boltzmann target lifts rewards by up to 30 percent.","key_machinery":"The load-bearing object is the forward-KL projection identity: for a diagonal Gaussian policy, the optimal forward-KL projection of the Boltzmann distribution is obtained by setting each action dimension's mean and variance to the corresponding marginal moments, computed by numerical integration. The VDN-a critic, adapted from value decomposition networks, decomposes the global Q-function into per-action-dimension subnetworks plus an auxiliary interaction network so that each action marginal can be estimated independently. The Bidirectional SAC objective then uses the forward projection as both the initialization and a regularizer for the reverse-KL policy improvement step.","core_discovery":"The paper's central claim is that the intractable projection step in SAC becomes explicit when the KL direction is flipped. For a Gaussian policy with diagonal covariance, the minimizer of $D_{\\mathrm{KL}}(q(\\cdot|s_t)\\|\\pi'(\\cdot|s_t))$ is the product of one-dimensional Gaussians whose mean and variance are the mean and variance of the corresponding marginal of the Boltzmann target $q(a|s_t) \\propto \\exp(Q(s_t,a)/\\alpha)$. The paper further claims that an actor initialized at this projection and then refined by reverse-KL minimization, through the Bidirectional SAC objective $D_{\\mathrm{KL}}(\\pi\\|q) + \\epsilon(\\|f^* - f\\|^2 + \\|\\Sigma^* - \\Sigma\\|^2)$, converges faster and reaches higher reward than standard SAC's purely gradient-based reverse-KL update. A value-decomposition-style critic with per-action-dimension subnetworks and an auxiliary interaction network is proposed to learn the marginals needed for the projection.","pith_inferences":["The authors do not test this, but the same moment-matching projection could initialize other maximum-entropy or trust-region policy algorithms, not just SAC.","Because the method only needs the first two moments of each marginal, its occasional failure on bimodal marginals may not hurt control performance; a direct comparison of policies matched on mean and variance would test this.","The result reframes SAC's reverse-KL gradient step as a noisy estimator of an exact moment-matching step, suggesting that variance reduction in policy updates, rather than a new objective, may be the primary source of the reported gains.","A natural extension the paper leaves implicit is using higher-order marginal moments or non-Gaussian policy families when the Boltzmann target is strongly multimodal."],"forward_implications":["Forward SAC alone obtains an explicit, gradient-free policy projection and outperforms standard SAC on simpler continuous-control tasks.","Bidirectional SAC inherits the reverse-KL guarantee of monotonic Q improvement while starting from a better-conditioned initialization, giving faster convergence and higher asymptotic reward.","The per-dimension marginal decomposition allows the forward projection to be computed with one-dimensional numerical integration, avoiding the curse of dimensionality.","If the VDN-a critic captures the first two moments of each Boltzmann marginal, the same bidirectional update should transfer to other maximum-entropy algorithms that use the same Gaussian policy family."],"supporting_citations":[{"why":"defines the SAC algorithm and the Gaussian policy family that the paper modifies.","marker":"[Haarnoja et al., 2018a]"},{"why":"supplies the reverse-KL policy improvement step and its monotonic-improvement guarantee that Bidirectional SAC retains.","marker":"[Haarnoja et al., 2018b]"},{"why":"documents the instability of reverse-KL gradient optimization, the problem the paper addresses.","marker":"[Chan et al., 2022]"},{"why":"provides the value decomposition architecture that VDN-a adapts for single-agent Q-function learning.","marker":"[Sunehag et al., 2017]"},{"why":"supplies the MuJoCo continuous-control benchmarks used in the comparisons.","marker":"[Todorov et al., 2012]"},{"why":"supplies the Box2D continuous-control benchmarks used in the comparisons.","marker":"[Catto, 2011]"},{"why":"provides Simpson's rule, the numerical integration method used to compute the forward-KL projection.","marker":"[McKeeman, 1962]"},{"why":"motivates the per-dimension integration scheme by showing the curse of dimensionality for high-dimensional numerical integration.","marker":"[Hinrichs et al., 2014]"}],"fun_headline_variants":["Forward KL makes SAC's policy update exact, lifting rewards 30%","Start with forward KL, refine with reverse: Bidirectional SAC +30% rewards","Explicit policy projection in SAC via forward KL – 30% better rewards","Swap KL directions in SAC: explicit updates, 30% higher reward","Bidirectional SAC: explicit start, refined finish, 30% more reward"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The forward-KL anchor is only as good as the critic's estimate of each action dimension's share of the target distribution, and the paper's proof of correctness requires an auxiliary network that has fully learned the interactions between action dimensions.","fun_headline_variants_meta":{"raw":{"variants":["Forward KL makes SAC's policy update exact, lifting rewards 30%","Start with forward KL, refine with reverse: Bidirectional SAC +30% rewards","Explicit policy projection in SAC via forward KL – 30% better rewards","Swap KL directions in SAC: explicit updates, 30% higher reward","Bidirectional SAC: explicit start, refined finish, 30% more reward"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001315,"raw_usage":{"total_tokens":5349,"prompt_tokens":926,"completion_tokens":4423,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":542,"completion_tokens_details":{"reasoning_tokens":4323}},"tokens_in":542,"tokens_out":4423,"duration_ms":30305,"temperature":1.0,"reasoning_tokens":4323,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:36:59.823757+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the VDN-a critic on a fixed Q-function whose Boltzmann distribution has a known bimodal marginal; if the mean and variance computed by Simpson integration of the learned marginal deviate from the true values by more than the integration error, the projection anchor is miscalibrated. Selecting a continuous-control task with strong action-dimension coupling and comparing Bidirectional SAC's episodic reward against standard SAC with identical hyperparameters would also test the claim.","supporting_citations":[{"cited_title":"Box2d: A 2d physics engine for games","cited_arxiv_id":null,"evidence_quote":"supplies the Box2D continuous-control benchmarks used in the comparisons."},{"cited_title":"Algorithm 145: Adaptive numerical integration by simpson's rule","cited_arxiv_id":null,"evidence_quote":"provides Simpson's rule, the numerical integration method used to compute the forward-KL projection."},{"cited_title":"The curse of dimensionality for numerical integration of smooth functions","cited_arxiv_id":null,"evidence_quote":"motivates the per-dimension integration scheme by showing the curse of dimensionality for high-dimensional numerical integration."}],"review_version":1}