{"id":"c2f2705c-1097-4b26-9359-17e9a163acf4","arxiv_id":"2501.19133","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Decorrelated Soft Actor-Critic (DSAC) adds layerwise input decorrelation to discrete SAC and reports faster wall-clock training in 5 of 7 Atari games and better reward in 2, though the gains are partly confounded by per-game hyperparameter choices.","lead":"This paper adds a decorrelation step to the neural networks inside the soft actor-critic reinforcement learning algorithm, aiming to make learning faster and more sample efficient. The authors test it on seven Atari games and report faster wall-clock training in five and better scores in two, but the speedups mostly coincide with smaller batch sizes.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"At matched batch size DSAC's wall-clock advantage largely disappears, so the abstract's speedup claim is not yet attributable to decorrelation.","rationale":"The strongest claim has two parts: a measurement claim that DSAC's best configurations finish 100k steps faster and score higher on 2 of 7 games, and a causal claim that decorrelation causes this. The measurement part is plausible and reproducible from the released code. The causal part requires that the only systematic difference between DSAC and SAC is the decorrelation update, but the paper varies batch size, learning rate, and possibly activation across the comparison. The best evidence against the causal claim is internal: Figure 1's caption states that the faster games are precisely those where DSAC uses batch size 64 versus SAC's 256, and Appendix G's matched-batch-size runs show the time advantage disappears for Seaquest, Alien, PrivateEye, and BankHeist. This means the headline 'faster in five of seven games' is mostly a smaller-batch-size effect. The reward gains do not survive multiple-testing correction and are confounded by per-game learning rates. Figure 2 only shows that DSAC minimizes its own auxiliary loss, not that this loss causes better credit assignment; additionally, wall-clock speed is not sample efficiency, and only 2 of 7 games show reward-per-step gains. I am not arguing the method is fraudulent or useless: the DBP lineage is published, the code is available, and the authors honestly flag the batch-size correlation. But the current experimental design cannot support the abstract's causal wording. A clean matched-batch-size ablation with adequate seeds and multiple-testing correction would settle it: if the advantage survives there, the causal claim is supported; if not, the claim should be weakened to 'decorrelation permits lower batch sizes without performance loss' and the wall-clock speedups should be attributed mainly to batch size. This is exactly the kind of condition the reader attached, so I would keep the CONDITIONAL verdict rather than moving it.","tokens_in":13864,"tokens_out":7422,"duration_ms":75526,"concrete_test":"Re-run Appendix G's matched-batch-size ablation for all seven games with the SAC baseline and DSAC at the same batch size (256), identical per-game learning rates, same network and activation, and at least 20 seeds, toggling only the decorrelation update; report mean wall-clock time and final reward with paired permutation tests and Holm-Bonferroni correction. If DSAC's wall-clock advantage over SAC is not significant in a majority of games at matched batch size, the abstract's 'faster training in five of seven games' claim should be restated as a batch-size effect rather than an effect of network-wide decorrelation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is causal: the decorrelation update (Eq. 3, Algorithm 1) is said to speed up training and improve sample efficiency through better credit assignment. For that to be true, the comparison must isolate decorrelation from the other differences between DSAC and the SAC baseline. Table 2 shows that the five games where DSAC is faster are exactly the games where DSAC uses batch size 64 while SAC uses 256; in ChopperCommand, where DSAC uses the larger batch size, it is 288% slower. The paper acknowledges this in Section 3.2 and attempts a control in Appendix G, but the reported matched-batch-size runs undercut the speed claim: Seaquest, Alien, PrivateEye and BankHeist lose their training-time advantage at identical batch size, and only BattleZone retains it (Figure 10 caption). Thus the 'five of seven games faster' result appears to be largely a consequence of the smaller batch size reducing per-step computation, not of network-wide decorrelation. The reward evidence is also thin: two nominally significant gains out of seven games, with no multiple-testing correction, and each at different learning rates and batch sizes. The decorrelation-loss curves in Figure 2 only show that DSAC minimizes an objective that SAC does not optimize, which is expected and does not demonstrate effective credit assignment. The load-bearing assumption that decorrelation, rather than configuration, causes the reported gains is therefore not secured by the reported experiments.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper integrates decorrelated backpropagation (DBP) into discrete Soft Actor-Critic (SAC), adding per-layer decorrelation matrices that are updated with a separate learning rule alongside the usual RL losses. DSAC is evaluated against a BP-based discrete SAC baseline on seven Atari 100k games. The abstract claims faster training in five of seven games, improved rewards in two games, and a roughly 50% wall-clock reduction, and attributes these gains to network-wide decorrelation improving sample efficiency through more effective credit assignment.","tokens_in":14117,"tokens_out":4118,"duration_ms":41932,"significance":"If validated, the method would be a practically useful, architecture-agnostic add-on for off-policy deep RL, and the released code is a positive contribution. However, the current experimental design does not isolate the decorrelation mechanism from per-game hyperparameter differences, especially batch size. The paper is therefore significant only as a proposal with preliminary evidence; the central causal claim needs substantially stronger support before the results can be accepted.","major_comments":[{"comment":"The primary wall-clock speed-up claim is confounded with batch size. In all five games where DSAC is reported as faster, DSAC uses batch size 64 while SAC uses 256, and in ChopperCommand, the one game where DSAC uses batch size 256 against SAC's 64, DSAC is 288% slower. The Figure 1 caption itself states this pattern. Because wall-clock time per gradient step scales with batch size, these results do not establish that decorrelation, rather than the smaller batch size, causes faster training. The paper should report matched-batch-size wall-clock comparisons for the main results, together with per-step learning curves and statistical tests, and should not claim \"five of seven games faster\" based on the current comparisons.","section":"§3.1, Fig. 1, Table 2"},{"comment":"The matched-batch-size control is qualitative and undercuts the central claim. At identical batch sizes, DSAC loses its training-time advantage in Seaquest, Alien, PrivateEye, and BankHeist, retaining it only in BattleZone, and the reward advantage also disappears for BankHeist. The appendix does not report the number of seeds, error bars, or significance tests for these runs, and the configurations are selected as \"the other batch size\" rather than by a full factorial design. A quantitative controlled comparison with identical hyperparameters except for the presence of the decorrelation update is needed to support the causal attribution to decorrelation.","section":"Appendix G, Fig. 10"},{"comment":"The decorrelation-loss evidence is partly by construction. DSAC explicitly updates R using Eq. (3) to minimize the decorrelation loss in Eq. (5), while the SAC baseline has no such update, so the lower DSAC loss in Figure 2 is expected and does not by itself demonstrate more effective credit assignment or better sample efficiency. The paper needs a mechanistic quantity that is not directly optimized, such as gradient conditioning, effective rank of representations, or an ablation with a mismatched or random decorrelation update, to support the claim that decorrelation improves learning rather than merely that DSAC minimizes its own auxiliary objective.","section":"§3.4, Fig. 2; §2.1, Eq. (3)"},{"comment":"The reward evidence for the headline claim consists of two nominally significant gains (p<0.05, N=10) out of seven games, with no correction for multiple comparisons, and each game uses a separately selected best hyperparameter configuration from the grid search. This selection procedure risks overfitting to final reward and makes the per-game comparisons non-independent. Report corrected p-values or false-discovery-rate control, include seed-level dispersion, and provide a sensitivity analysis with a common hyperparameter setting across games for both algorithms.","section":"§3.1, §2.5, Appendix A"}],"minor_comments":[{"comment":"The paper should clarify early that the main results decorrelate only the policy network, while the hyperparameter search decorrelated the policy and both Q-networks; the text currently oscillates between \"all networks\" and \"policy network,\" which is confusing for reproducibility.","section":"§2.5, Appendix A"},{"comment":"The sample-size formula uses D_r and p without precise definitions; please define the dimensionality per patch and the number of patches before the equation so readers can interpret the downsampling rule.","section":"§2.2, Eq. (7)"},{"comment":"The batch-size confounding acknowledged in the Figure 1 caption and Section 3.2 is a central limitation and should be restated explicitly in the abstract and conclusions, where the causal phrasing currently overstates the evidence.","section":"Fig. 1 caption, §3.2"},{"comment":"Wall-clock times are reported as single values without confidence intervals or seed-level spread; given that the training-time claims are central, per-seed timing distributions should be reported.","section":"Table 2"},{"comment":"The statement that Leaky ReLU performs better than ReLU for the decorrelation procedure is anecdotal; either add supporting results or present it as a design choice without a performance claim.","section":"§2.3.1"},{"comment":"The reference \"Huangi et al. (2018)\" appears to be a typo for \"Huang et al. (2018)\" (Decorrelated Batch Normalization); please correct it and unify the rendering of author names with diacritics throughout the bibliography.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper's abstract and Section 3.1 overstate the result relative to the acknowledged batch-size confound. I recommend major revision rather than rejection because a properly controlled matched-batch-size study may recover a meaningful claim, but the current manuscript as written does not support the central causal attribution to decorrelation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper is worth a look, but the central speedup claim doesn't yet hold up. DSAC applies decorrelated backpropagation to every layer of discrete SAC's policy network, online, with no pretraining or extra encoder. That combination is genuinely new relative to prior decorrelation-in-RL work, which targeted latent encoders, last hidden layers, or offline pretraining. Credit also for reporting wall-clock time, not just environment steps, and for releasing code.\n\nThe paper is honest about the obvious confound: Table 2 shows the five games where DSAC trains faster are exactly those where it uses batch size 64 against SAC's 256. ChopperCommand, where DSAC uses the larger batch, is 288% slower. The authors acknowledge this in Section 3.2 and point to Appendix G for a matched-batch-size control. But that control undercuts the claim: at identical batch size, Seaquest, Alien, PrivateEye, and BankHeist lose their time advantage; only BattleZone retains it. The main-text speedup is therefore mostly a batch-size effect, not decorrelation. The reward evidence is also thin: two nominally significant gains out of seven games, no multiple-testing correction, and per-game learning rates and batch sizes are fitted to final performance. The decorrelation-loss curves in Figure 2 just show DSAC minimizing an objective SAC never optimizes; that is by construction, not evidence of better credit assignment.\n\nTo be fair, the paper does not hide these issues, and the matched-batch-size appendix is an effort in the right direction, even if it is qualitative and lacks significance testing. The idea is coherent and the code is available, so a serious referee could push the authors to isolate decorrelation from configuration. That would mean a cleaner ablation with fixed batch size and learning rate across a larger game set, proper error bars and multiple-testing correction, and comparison to prior decorrelation methods.\n\nMy bottom line: this deserves a serious referee, but the current version does not establish the causal claim. I wouldn't cite it for sample-efficiency gains yet; it's more a methodological proposal with a promising but unresolved confound. Bring it to reading group if you want to discuss how easy it is for hyperparameter search to drive RL results.","headline":"A clean idea and honest reporting, but the speedup claim is confounded by batch size; the matched-batch-size control in Appendix G shows the effect mostly disappears.","tokens_in":14709,"tokens_out":2233,"would_cite":false,"duration_ms":20821,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that adding network-wide input decorrelation to the soft actor-critic algorithm speeds up deep reinforcement learning training in five of seven Atari games, improves final reward in two, and keeps performance level in the…","keywords":["decorrelated backpropagation","soft actor-critic","sample efficiency","Atari 100k","credit assignment","representation learning","deep reinforcement learning"],"falsifier":"Run DSAC and SAC on the seven Atari games with identical batch sizes, identical learning rates, and identical wall-clock budgets, and apply a significance test to the difference in final reward and time-to-threshold; if no significant advantage remains when batch sizes are matched, the paper's attribution of the gains to decorrelation is not supported.","tokens_in":13602,"feed_emoji":"⏱️","tokens_out":5736,"duration_ms":51387,"temperature":0.7,"pith_summary":"Sample efficiency is a major bottleneck in deep reinforcement learning. This paper proposes a simple modification to the soft actor-critic algorithm: before each neural network layer, multiply the layer input by a decorrelation matrix that is updated, in parallel with normal training, to suppress correlations between features. Applied to discrete SAC on seven hard Atari games, this decorrelated SAC (DSAC) trains faster in wall-clock time in five of the seven games, reduces training time by roughly half in the games where it also improves reward (Alien and Seaquest), and keeps performance level in the rest. The authors argue that decorrelation improves representation learning and credit assignment, which is why the agent learns more efficiently.","feed_headline":"Decorrelation speeds up deep RL training in five of seven Atari games","feed_subtitle":"Network-wide input decorrelation also lifts final rewards in two games, by up to 86 percent, and keeps the rest level.","key_machinery":"The central object is the decorrelating matrix R inserted before each layer's forward pass, transforming the raw input z into x = Rz. R is updated by R ← R − η C R, where C is the empirical off-diagonal correlation of x with itself, a rule that minimizes the sum of squared off-diagonal correlation elements (the decorrelation loss) for that layer. The paper applies this to every layer of the policy network in a discrete-action version of SAC; the decorrelation update runs in parallel with the usual SAC loss updates, and in the main results only the policy network is decorrelated.","core_discovery":"On the paper's own terms, the central discovery is that network-wide decorrelation of layer inputs, implemented with decorrelated backpropagation, can be integrated into discrete soft actor-critic as a separate parallel update, and that doing so reduces wall-clock training time in five of seven Atari 100k games (up to 76%), with statistically significant reward gains in two games (Alien +86%, Seaquest +6%) and no significant loss in the others. The paper also reports that the decorrelation loss of the policy network stays near zero throughout training for DSAC while rising beyond $10^{6}$ for the SAC baseline, and that the entropy-temperature loss converges faster, suggesting decorrelation supports the exploration objective. The authors attribute the gains to more effective credit assignment through better representation learning.","pith_inferences":["Editorial extension: The cleanest test of the paper's causal story is a matched-batch-size comparison with significance testing; the paper's own Appendix G only provides a qualitative, un-tested look at equal batch sizes, so the field would benefit from a pre-registered study that holds batch size and compute fixed and varies only the decorrelation update.","Editorial extension: If decorrelation is indeed what enables the smaller batch sizes, then the mechanism may be that decorrelated inputs allow the optimizer to take noisier gradient steps safely; this could be tested by measuring gradient variance with and without decorrelation.","Editorial extension: The decorrelation loss could be used as an intrinsic reward or exploration bonus in algorithms without SAC's built-in entropy term, which would be a different application than the paper's presentation.","Editorial extension: The paper's results are on Atari with discrete actions; continuous control with larger action spaces might see larger benefits because credit assignment is harder, a direction the paper mentions but does not test."],"forward_implications":["If the central claim holds, deep RL agents on image-based tasks can be trained to the same reward in substantially less wall-clock time simply by adding a decorrelation step, without changing the RL update rules.","Because the decorrelation loss stays low during training and rises when the baseline converges, it can act as a signal of how novel the input is, which the paper suggests could guide exploration in other RL algorithms.","The method is applicable to both convolutional and fully-connected layers, so it can be dropped into other actor-critic or value-based RL algorithms beyond SAC.","The observation that decorrelation works best with smaller batch sizes suggests it may reduce the amount of data needed per gradient step, further improving sample efficiency.","Whitening (normalizing input variance as well as correlations) could extend DSAC's gains, since whitening has been shown to improve convergence."],"supporting_citations":[{"why":"Supplies the decorrelated backpropagation algorithm, its decorrelation learning rule, and the computational tricks for convolutional layers.","marker":"(Dalm et al., 2024)"},{"why":"Supplies the discrete SAC variant, the baseline implementation, and the hyperparameter ranges for batch size and learning rate.","marker":"(Christodoulou, 2019)"},{"why":"Provides the original soft actor-critic algorithm that DSAC modifies with decorrelation.","marker":"(Haarnoja et al., 2018b)"},{"why":"Defines the Atari 100k benchmark protocol that the paper uses for its empirical evaluation.","marker":"(Kaiser et al., 2020)"},{"why":"Establishes the iterative decorrelation procedure and the constrained-parameter-inference principle underlying DBP.","marker":"(Ahmad et al., 2023)"},{"why":"Shows that deep RL learns highly correlated features that slow learning, motivating the decorrelation approach.","marker":"(Mavrin et al., 2019b)"},{"why":"Provides the classic observation that correlations in layer inputs slow gradient-descent convergence, the conceptual basis for the method.","marker":"(LeCun et al., 2012)"}],"fun_headline_variants":["DSAC: network decorrelation cuts Atari training time up to 76%","Decorrelated backprop makes SAC train faster on 5 of 7 Atari games","Network-wide decorrelation lifts SAC rewards by 86% on Alien","DSAC: decorrelated SAC achieves 86% reward gain, 50% faster","Decorrelation in SAC: faster Atari training, higher rewards"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim assumes that the speedups and reward gains come from the decorrelation update rule itself, but in five of the seven games DSAC used a smaller batch size than the SAC baseline, so if batch-size differences rather than decorrelation drive the gains, the causal story collapses.","fun_headline_variants_meta":{"raw":{"variants":["DSAC: network decorrelation cuts Atari training time up to 76%","Decorrelated backprop makes SAC train faster on 5 of 7 Atari games","Network-wide decorrelation lifts SAC rewards by 86% on Alien","DSAC: decorrelated SAC achieves 86% reward gain, 50% faster","Decorrelation in SAC: faster Atari training, higher rewards"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000163,"raw_usage":{"total_tokens":1244,"prompt_tokens":947,"completion_tokens":297,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":563,"completion_tokens_details":{"reasoning_tokens":196}},"tokens_in":563,"tokens_out":297,"duration_ms":3574,"temperature":1.0,"reasoning_tokens":196,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T21:11:58.169307+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run DSAC and SAC on the seven Atari games with identical batch sizes, identical learning rates, and identical wall-clock budgets, and apply a significance test to the difference in final reward and time-to-threshold; if no significant advantage remains when batch sizes are matched, the paper's attribution of the gains to decorrelation is not supported.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the decorrelated backpropagation algorithm, its decorrelation learning rule, and the computational tricks for convolutional layers."},{"cited_title":"H., Czechowski, K., Erhan, D., Finn, C., Kozakowski, P., Levine, S., Mohiuddin, A., Sepassi, R., Tucker, G., and Michalewski, H","cited_arxiv_id":null,"evidence_quote":"Defines the Atari 100k benchmark protocol that the paper uses for its empirical evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the iterative decorrelation procedure and the constrained-parameter-inference principle underlying DBP."},{"cited_title":"u ller, K.-R. (2012). Efficient backprop. In Montavon, G., Orr, G. B., and M \\","cited_arxiv_id":null,"evidence_quote":"Provides the classic observation that correlations in layer inputs slow gradient-descent convergence, the conceptual basis for the method."}],"review_version":1}