{"id":"5bb3e280-9c0c-4584-8d11-4b3455d6f508","arxiv_id":"2411.15806","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Replacing the deep critic with a broad learning system trained by ridge regression makes DDPG, SAC, and TD3 train faster on MuJoCo continuous-control benchmarks, with roughly similar or slightly better final rewards.","lead":"This paper replaces the deep neural network critic in DDPG, SAC, and TD3 with a broad learning system that estimates Q-values through ridge regression, while keeping a deep network for the policy. On seven MuJoCo continuous-control tasks, the hybrid agents train faster than their originals, though final reward gains are small or absent on several tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The abstract's 'surpass ... accuracy' claim is unsupported: Table I shows equality (INP, ANT) or a tiny deficit (HUM) for BCDA-DDPG, and no significance testing or per-seed statistics is provided anywhere, including for the TD3/SAC variants.","rationale":"I read the paper as making an empirical claim about wall-clock efficiency and final-task accuracy. The reader's weakest assumption focuses on whether the closed-form ridge-regression critic is stable and expressive; that is a real gap, but the experiments are the primary evidence for the paper, and a stability failure would show up as poor final rewards on the seven tasks. The more directly load-bearing problem is that the paper's own evidence for the accuracy half of the central claim is not merely missing but contradictory: Table I contains equal or slightly worse entries for several tasks, and no statistical testing accompanies any of the tables or curves. For an efficiency-method paper, a null-or-tiny accuracy difference can still be acceptable if the claim is softened to 'comparable accuracy with faster training'; the current abstract and Section III-B claim superiority in accuracy, which Table I does not support. Because this is a claim about the reported experiments rather than about asymptotic theory, it can be settled by re-running the released code and applying a standard paired test. I therefore keep the reader's CONDITIONAL verdict (the efficiency contribution is plausible and the code is public), but I would require the accuracy claim to be either statistically supported or weakened before acceptance. My main concern differs from the reader's weakest_assumption, though the reader's rationale also flags the Table I inconsistency; hence partial agreement.","tokens_in":14523,"tokens_out":10599,"duration_ms":100108,"concrete_test":"Re-run the released repository (github.com/ShironT/bcda) for all seven BCDA-DDPG tasks and for BCDA-TD3/BCDA-SAC on HCH and HUM with at least 10 seeds, recording the per-seed maximum evaluation reward used for Table I. Compute paired bootstrap 95% confidence intervals and a Wilcoxon signed-rank test for each BCDA-vs-baseline pair. If the intervals include zero for INP, ANT, and HUM, and the HCH/PUS/REA differences shrink below the resolution of the plots, then the 'surpass ... accuracy' sentence in the abstract and Section III-B should be amended to 'matches or slightly improves' or removed. Also report training-time variance to assess Table II.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is the abstract's assertion that 'all BLS-enhanced versions surpass their original counterparts in terms of training efficiency and accuracy.' The efficiency half is supported by Table II for BCDA-DDPG (22-68% faster), though without variance. The accuracy half is not supported by the paper's own numbers. Table I reports maximum average reward over 5 trials: BCDA-DDPG equals DDPG on INP (1000 vs 1000) and ANT (995.51 vs 995.51) and is marginally lower on HUM (928.105 vs 928.10); improvements on REA, PUS, and HCH are small (0.03, 3.84, and 52.36 reward units) with no confidence intervals. No per-seed data, error bars, or significance tests are reported for any table or figure. For BCDA-TD3 and BCDA-SAC, the accuracy claim rests entirely on Figs. 4-5, which are unquantified learning curves from 5 seeds, and no training-time table is given for these variants. Since the paper's headline asserts superiority on both dimensions, and the quantitative evidence for the accuracy dimension is either absent or inconsistent with the abstract, the central claim as worded is not established. This is load-bearing because if the accuracy differences are within seed noise (likely given n=5 and the tiny magnitudes), the method's contribution reduces to a speedup at matched reward, which is a weaker claim than the one made.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes BCDA (Broad Critic Deep Actor), a hybrid actor-critic framework in which the critic is a Broad Learning System (BLS) trained by ridge regression over random feature and enhancement nodes, while the actor remains a DNN trained by gradient-based policy optimization. The framework is instantiated on DDPG, TD3, and SAC, and evaluated on MuJoCo continuous control tasks. The authors report that BCDA variants train 22–68% faster than their DNN-critic counterparts (Table II) and, in the abstract, claim that all BLS-enhanced versions \"surpass their original counterparts in terms of training efficiency and accuracy.\" For DDPG, learning curves and Table I show sizable speedups and slightly higher or equal maximum average rewards. For TD3 and SAC, efficiency and accuracy claims rest on learning curves over five seeds. The paper also studies ridge regularization strength, ensemble BCNs, dropout, and incremental learning schemes.","tokens_in":14829,"tokens_out":2862,"duration_ms":28139,"significance":"The core idea—replacing a deep critic with a closed-form ridge-regression BLS critic while keeping a deep actor—is plausible and, if the efficiency numbers are reproducible, practically useful for real-time control. The manuscript ships source code, which aids reproducibility. The strongest evidence is Table II: all seven BCDA-DDPG variants show 22–68% training-time reductions, and these are consistent with the BLS mechanism (closed-form output weights avoid iterative critic backpropagation). However, the accuracy half of the central claim is not established by the reported statistics, and there is no formal analysis of the stability of the non-stationary closed-form critic updates. The contribution, as presented, is therefore best characterized as \"faster training at roughly matched reward,\" which is weaker than the abstract's claim of both efficiency and accuracy superiority.","major_comments":[{"comment":"The abstract's claim that all BLS-enhanced versions \"surpass their original counterparts in terms of ... accuracy\" is not supported by Table I. On INP and ANT, BCDA-DDPG and DDPG are exactly tied (1000 vs. 1000 and 995.51 vs. 995.51), and on HUM, BCDA-DDPG is marginally lower (928.105 vs. 928.10). The remaining differences (REA, PUS, HCH) are small, with no confidence intervals, per-seed values, or significance tests anywhere in the paper. With n=5 and these magnitudes, the accuracy differences are within plausible seed noise, so the stated accuracy superiority is not established.","section":"III-B, Table I"},{"comment":"Equation (2) defines the non-terminal target as y_t = γ_t max_{a_{t+1}} Q(s_{t+1}, a_{t+1}), omitting the reward r_t. The standard target, consistent with the MSBE in Eq. (1), should be r_t + γ_t max_{a_{t+1}} Q(s_{t+1}, a_{t+1}). If this omission is not a typographical error but reflects the implementation, the critic is regressing against targets that discard the immediate reward in all non-terminal transitions, which would make the reported results very surprising. The paper needs to clarify this and, if it is a typo, correct it.","section":"II-B, Eq. (2)"},{"comment":"The robustness hyperparameters (λ = 2^{-20}, ensemble of 5 BCNs, dropout rate 0.2) and the four IL schemes are selected on the same tasks (INP for regularization/ensemble/dropout, REA for IL) whose final results are then reported, with no held-out validation tasks or sensitivity analysis beyond these specific selections. Because the final reported learning curves in Fig. 3 use these chosen settings, the efficiency and reward comparisons are not blind to the tuning process. This post-hoc selection does not invalidate the empirical results, but it weakens the claim that the framework's gains are robust rather than task-specific.","section":"III-C, Fig. 6-8"},{"comment":"The paper provides no analysis—formal or empirical—of the stability of the closed-form ridge-regression critic refit against non-stationary target BCNs. Since the output weights are recomputed by pseudoinverse each update while the targets themselves are moving (Eqs. 9 and 14), the actor gradient depends on the critic Q-values staying grounded; without diagnostics such as target-Q drift, feature-space rank, or Bellman-error monitoring, it is unclear that the reported efficiency gains are not accompanied by silent value-estimation degradation in other regimes. The representational capacity of the fixed 10-feature/500-enhancement BLS for the 393-dimensional HUM state-action space is also unexamined.","section":"II-B, III-A"},{"comment":"The per-task efficiency and accuracy results are reported as means over 5 trials without variance, and the learning curves in Figs. 3–5 show shaded half-standard-deviation regions but no individual seed trajectories. The paper should report per-seed values or confidence intervals for Table I and II, and ideally for the TD3/SAC comparisons. Without this, the \"efficiency improvement\" percentages and reward differences cannot be separated from seed noise, particularly for the small reward gaps in Table I.","section":"III-A, III-B"}],"minor_comments":[{"comment":"The caption says \"in the two controls tasks\" but the table lists all seven control tasks; this should be corrected.","section":"Table II caption"},{"comment":"The label \"DINV\" in Fig. 2 and \"INV\" in Figs. 6-7 are inconsistent with the \"DIN\" and \"INP\" names used in the text and Table I; unify the naming.","section":"Fig. 2 and Figs. 6-7"},{"comment":"The hyperparameters are written as \"2-30\" and \"2-20\"; these are presumably powers of two (2^{-30}, 2^{-20}) and should be typeset consistently.","section":"Section III-A"},{"comment":"The caption says \"in 10000 timesteps\" while the text says 100,000 timesteps; clarify the evaluation horizon.","section":"Table I caption"},{"comment":"Equation (1) contains a stray \"d\" multiplied by the discount factor; this is likely a typo for the terminal indicator or is otherwise undefined and should be cleaned up.","section":"Section II-B, Eq. (1)"},{"comment":"The claim that \"prior studies [12], [13] validated the superiority of the original SAC and TD3 over DDPG\" is not backed by numerical comparisons in this paper; since the paper compares BCDA to the original algorithms, a brief table or plot of DDPG vs. SAC/TD3 on the two selected tasks would help contextualize the TD3/SAC results.","section":"References and baselines"}],"recommendation":"major_revision","confidential_remarks":"The paper's central contribution is the speedup at roughly matched reward, which is plausible and worth publishing if the accuracy claim is toned down and the statistical evidence is strengthened. The Eq. (2) target error must be checked against the released code before acceptance, as it bears directly on the algorithm's correctness. The novelty is incremental but appropriate for a brief paper in TNNLS; the main risk is overclaiming accuracy superiority from n=5 means. I would recommend major revision rather than rejection because the efficiency result in Table II is substantial and consistent, and the theoretical gap (no convergence/stability analysis) is not fatal for an empirically focused paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The real news here is simple and worth knowing: replacing the DNN critic with a BLS critic trained by ridge regression, while keeping the DNN actor, gives 22-68% training-time reductions in Table II across seven MuJoCo tasks. That efficiency result is the paper's genuine contribution, and it is broadly consistent across environments, which makes it credible as a drop-in speedup. The code is public, which helps. This combination—BLS critic plus DNN actor inside DDPG, TD3, and SAC—is not in the prior BLS-RL work I know, so novelty is modest but real.\n\nNow the soft spots, in proportion. The abstract says BLS-enhanced versions \"surpass\" original ones in both efficiency and accuracy. The efficiency half is supported for BCDA-DDPG, though only as mean training time with no variance. The accuracy half is simply not supported by Table I: INP, ANT, and HUM are ties or near-ties, and the differences on REA, PUS, and HCH are small with no error bars or significance tests. For BCDA-TD3 and BCDA-SAC, there is no training-time table at all; the accuracy claim rests on unquantified learning curves from five seeds. So the headline claim, as worded, is not established. The paper would be more honest if it claimed \"comparable accuracy with large speedups.\"\n\nTwo technical issues are worth flagging. Equation (2) defines the target as r_t only in the terminal case and γ max Q otherwise, which omits the immediate reward in the non-terminal case—that is not the standard Bellman target. Equation (8) says W = (A)^+ without multiplying by Y; the correct pseudoinverse formula appears only in Eq. (9). These look like typos, but they matter because the paper's appeal is the closed-form critic update. Also, the robustness hyperparameters (λ, ensemble size, dropout rate, IL node counts) were tuned on the same tasks whose results are reported, so those specific numbers should be treated as anecdotal, not validated.\n\nIs the central argument sound? The efficiency contribution is plausible and useful. The lack of a convergence analysis for bootstrapped ridge-regression refits is a real gap, but for an empirical method paper in RL that is not disqualifying. The paper is written clearly, and the design rationale (BLS for regression-type critic, DNN for policy gradients) is sensible.\n\nWho should read this: anyone working on cheap critic architectures for online continuous control, or on BLS in RL. It deserves a serious referee: I would send it to review, but I would require the authors to fix the equations, add per-seed statistics or confidence intervals, and recalibrate the abstract's accuracy claim. As is, it is a reasonable incremental method paper with an overstated conclusion.","headline":"A useful, incremental hybrid (BLS critic + DNN actor) with a plausible speedup claim, but the abstract's accuracy claim overreaches the paper's own Table I, and two equations have outright typos.","tokens_in":15379,"tokens_out":1624,"would_cite":false,"duration_ms":17743,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Replacing the DNN critic with a broad learning system critic trained by ridge regression speeds up DDPG, TD3, and SAC by up to 68 percent on continuous-control benchmarks.","keywords":["broad critic deep actor","broad learning system","broad reinforcement learning","continuous control","deep reinforcement learning","actor-critic","ridge regression","incremental learning"],"falsifier":"The clearest single check would be to log the critic's Bellman error during BCDA-DDPG training on a sparse-reward continuous-control task; if the error grows rather than shrinks across refits, the closed-form critic is not learning the moving target and the reported speedup would not generalize.","tokens_in":14308,"feed_emoji":"🤖","tokens_out":10016,"duration_ms":85254,"temperature":0.7,"pith_summary":"This paper argues that the critic network in actor-critic reinforcement learning need not be a deep neural network. It proposes a hybrid 'broad critic deep actor' (BCDA) design in which the critic is a broad learning system trained by closed-form ridge regression, while the actor remains a DNN trained by gradient descent. The claim is that swapping DNN critics for this broad critic in DDPG, TD3, and SAC improves training efficiency and accuracy on continuous-control tasks. If the claim holds, actor-critic RL becomes cheaper to run online, which matters for real-time and resource-constrained control applications.","feed_headline":"Broad critic speeds up actor-critic RL by 22 to 68 percent","feed_subtitle":"Closed-form ridge-regression critic updates make DDPG, TD3, and SAC faster without losing final reward.","key_machinery":"The load-bearing component is the broad critic network (BCN): a broad learning system with random feature mapping nodes and enhancement nodes whose only learned weights are the output-layer weights $W_n^m$, computed in closed form by ridge regression, $W_n^m = (A_n^m (A_n^m)^T + \\lambda I)^{-1}(A_n^m)^T Y$, where $A_n^m = [Z^n | H^m]$ concatenates the feature and enhancement nodes. Target values come from a delayed copy (t-BCN) via the Bellman-style rule $y_t = r_t + \\gamma_t \\max_{a_{t+1}} Q(s_{t+1}, a_{t+1})$, and the t-BCN is updated by Polyak averaging. This machinery turns the critic update into a one-shot least-squares fit over the replay buffer rather than iterative backpropagation, which is the mechanism behind the reported speedups; the incremental node-adding rule of BLS is its mechanism for growing capacity when the base critic underfits.","core_discovery":"The central claim is that BLS-based critics can replace DNN critics in standard actor-critic algorithms without losing performance, while training substantially faster. The authors implement the BCDA swap for DDPG, TD3, and SAC: target Q-values are computed from target BCNs, the BCN output weights are refit by ridge regression over the replay buffer, and the actor keeps its original gradient-based update. Their experiments on seven continuous-control benchmark tasks show that all BLS-augmented variants match or slightly exceed the peak rewards of the original algorithms and reduce average training time by 22 to 68 percent. The authors present the framework as generalizable to other actor-critic methods, with robustness added through regularization tuning, an ensemble of five BCNs, and dropout in the enhancement nodes.","pith_inferences":["The speed advantage is likely to be largest when critic updates dominate wall-clock time, so tasks with wide state-action inputs and large replay batches should show the biggest gap; a per-component timing breakdown would test this.","Measuring the BCN critic's mean squared Bellman error during training—something the paper does not report—would reveal whether the closed-form refit is tracking the drifting target or simply fitting a moving regression surface.","Because each ridge-regression refit reweights the entire replay buffer, BCDA may be more sensitive to stale or imbalanced replay data than gradient-descent critics; varying buffer size and priority would settle this.","The incremental node-adding property of BLS suggests a natural route to online adaptation in non-stationary environments, but the paper only evaluates incremental learning on a single fixed task."],"forward_implications":["BCDA-DDPG, BCDA-TD3, and BCDA-SAC train 22 to 68 percent faster than their DNN-critic baselines on the evaluated benchmark tasks, with matching or slightly higher peak rewards.","Because the critic is fit by closed-form regression, the efficiency gain should grow relative to DNN critics as the critic's hidden layers get wider or deeper.","The framework applies to any actor-critic algorithm whose critic solves a Q-regression problem, so similar BLS-critic versions of other off-policy algorithms should inherit the speedup.","Ensembling several broad critics and applying dropout reduces post-convergence reward variance, as demonstrated on the inverted-pendulum task.","If the base critic underfits, incrementally adding feature and enhancement nodes can raise accuracy, as shown on the reacher task, giving a practical tuning path for high-dimensional control."],"supporting_citations":[{"why":"This reference supplies the broad learning system architecture and incremental node-adding rule that the BCN critic is built from.","marker":"[9]"},{"why":"This reference defines the DDPG algorithm and its gradient-descent actor update that BCDA-DDPG preserves.","marker":"[11]"},{"why":"This reference defines the SAC algorithm whose twin critics and entropy objective BCDA-SAC modifies.","marker":"[12]"},{"why":"This reference defines the TD3 algorithm whose twin delayed critics BCDA-TD3 replaces with BCNs.","marker":"[13]"},{"why":"This reference is the prior offline BLS-RL work whose pre-collected-data requirement BCDA removes.","marker":"[16]"},{"why":"This reference introduces the mean squared Bellman error whose target values the BCN fits by ridge regression.","marker":"[20]"},{"why":"This reference supplies the generalized-inverse theory behind the closed-form critic weight update.","marker":"[21]"},{"why":"This reference provides the continuous-control testbed used for all experiments.","marker":"[22]"},{"why":"This reference motivates the ensemble of BCN critics whose median output stabilizes Q-value estimates.","marker":"[27]"},{"why":"This reference supplies dropout, applied to the enhancement nodes as a robustness technique.","marker":"[28]"}],"fun_headline_variants":["BLS critic shaves 22-68% off actor-critic training","Faster continuous control via broad-learning critic","Ridge regression critic speeds up RL without accuracy hit","Hybrid actor-critic: DNN actor, BLS critic, big speedup","Swap the critic for BLS to train RL agents faster"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework assumes that repeatedly refitting the critic's output weights with ridge regression, using target values generated by a delayed copy of the same critic, is a stable learning rule that does not drift or diverge as the targets move during training.","fun_headline_variants_meta":{"raw":{"variants":["BLS critic shaves 22-68% off actor-critic training","Faster continuous control via broad-learning critic","Ridge regression critic speeds up RL without accuracy hit","Hybrid actor-critic: DNN actor, BLS critic, big speedup","Swap the critic for BLS to train RL agents faster"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000219,"raw_usage":{"total_tokens":1426,"prompt_tokens":913,"completion_tokens":513,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":529,"completion_tokens_details":{"reasoning_tokens":426}},"tokens_in":529,"tokens_out":513,"duration_ms":4594,"temperature":1.0,"reasoning_tokens":426,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T13:52:53.811967+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"The clearest single check would be to log the critic's Bellman error during BCDA-DDPG training on a sparse-reward continuous-control task; if the error grows rather than shrinks across refits, the closed-form critic is not learning the moving target and the reported speedup would not generalize.","supporting_citations":[{"cited_title":"Broad learning system: An effective and efficient incremental learning system without the need for deep architecture,","cited_arxiv_id":null,"evidence_quote":"This reference supplies the broad learning system architecture and incremental node-adding rule that the BCN critic is built from."},{"cited_title":"Soft actor -critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor,","cited_arxiv_id":null,"evidence_quote":"This reference defines the SAC algorithm whose twin critics and entropy objective BCDA-SAC modifies."},{"cited_title":"Addressing function approximation error in actor-critic methods,","cited_arxiv_id":null,"evidence_quote":"This reference defines the TD3 algorithm whose twin delayed critics BCDA-TD3 replaces with BCNs."},{"cited_title":"Efficient Incremental Offline Reinforcement Learning With Sparse Broad Critic Approximation,","cited_arxiv_id":null,"evidence_quote":"This reference is the prior offline BLS-RL work whose pre-collected-data requirement BCDA removes."},{"cited_title":"Human -level control through deep reinforcement learning,","cited_arxiv_id":null,"evidence_quote":"This reference introduces the mean squared Bellman error whose target values the BCN fits by ridge regression."},{"cited_title":"Generalized inverse of a matrix and its applications ,","cited_arxiv_id":null,"evidence_quote":"This reference supplies the generalized-inverse theory behind the closed-form critic weight update."},{"cited_title":"Mujoco: A physics engine for model-based control,","cited_arxiv_id":null,"evidence_quote":"This reference provides the continuous-control testbed used for all experiments."},{"cited_title":"Ensemble bootstrapping for q-learning,","cited_arxiv_id":null,"evidence_quote":"This reference motivates the ensemble of BCN critics whose median output stabilizes Q-value estimates."}],"review_version":1}