{"id":"548bd2bc-7939-40e3-bcab-4c70eb77ad66","arxiv_id":"2412.06139","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Proposes bounded exploration, selecting high world-model uncertainty actions from SAC's sampled candidates, and reports mixed, statistically weak results on MuJoCo benchmarks.","lead":"This paper proposes a reinforcement learning exploration strategy that samples candidate actions from the Soft Actor-Critic policy and selects the one that an ensemble of world models considers most uncertain. The authors report higher average rewards than plain SAC on several MuJoCo tasks, but the gains are mostly within statistical noise and the method leaves key parameters unspecified.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section 4.3 admits raw uncertainty differences were 'not apparent,' then adds an unspecified normalization and a closest-to-the-mean selection step; without ablations, the uncertainty signal may not be what drives the reported gains.","rationale":"The reader's weakest assumption identifies the same spot: the uncertainty signal's reliability is not established. I sharpen it: the paper itself reports the raw signal was too weak to separate actions ('difference in uncertainty ... is not apparent'), and the final decision rule contains an unquantified normalization and a closest-to-mean selection step that can override the Gibbs distribution. This is load-bearing because if those extra steps dominate, the mechanism named in the title and abstract is not what produces the reported results. Table 1 is statistically fragile (about three seeds, overlapping error bars, post hoc sliding window on Hopper-v4), the SAC+QU comparison does not isolate uncertainty, and the absence of code/data makes independent checking impossible. This does not prove the idea is wrong; it means the central claim is unsupported as written. A focused ablation with more seeds would settle it. I therefore leave the reader's REJECT verdict unchanged.","tokens_in":8317,"tokens_out":6485,"duration_ms":59516,"concrete_test":"On Swimmer-v4 and HalfCheetah-v4, run at least 10 seeds per condition: (A) the full rule with the exact normalization and S supplied by the authors (if not supplied, the algorithm is underspecified); (B) pure Eq. 3 softmax over raw uncertainties, no normalization and no closest-to-mean post-processing; (C) no uncertainty: sample N candidates from the SAC policy and output the candidate closest to the policy mean; (D) vanilla SAC. Report mean and standard error. If C matches or beats B/A within one standard error, the uncertainty term is not load-bearing. Also report the normalized uncertainty range across candidates to verify that Eq. 3 weights actually deviate from uniform.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that world-model predictive variance (Eq. 2), encoded by the rank-preserving Gibbs distribution (Eq. 3), makes bounded exploration improve SAC. But the decision rule actually executed is not Eq. 3 alone. Section 4.3 states: 'During training, it was noticed that the difference in uncertainty between different actions is not apparent... A normalization was applied before computing the probability. The result distribution was sampled S times. The action closest to the mean of the SAC policy's distribution was outputted.' The normalization formula and S are never specified. If the normalized uncertainties are nearly uniform, the closest-to-mean post-processing dominates, turning the method into action-smoothing or variance reduction rather than uncertainty-driven exploration. In that case the Table 1 improvements cannot be attributed to world-model uncertainty and the central 'bounded exploration' mechanism collapses. No ablation separates Eq. 2, the normalization, and the closest-to-mean rule; the SAC+QU baseline also confounds uncertainty with Q-values. With roughly three seeds, overlapping error bars, and no code/data release, the 6/8 claim is not independently reproducible.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes 'bounded exploration,' a method for Soft Actor-Critic that samples N candidate actions from the SAC policy, estimates the epistemic uncertainty of each candidate using the summed per-dimension variance of an ensemble of world models' next-state predictions (Eq. 2), and then selects an action from a Gibbs-style distribution over those uncertainties (Eq. 3) without modifying the reward function. The authors evaluate this mechanism on MuJoCo environments in both model-free (SAC) and model-based (MVE) settings, comparing against vanilla SAC and a Q-value-plus-uncertainty baseline, and report that bounded exploration achieved the highest score in 6 of 8 experiments. The paper concludes that combining soft exploration with world-model uncertainty improves data efficiency and convergence speed.","tokens_in":8437,"tokens_out":5610,"duration_ms":55212,"significance":"If the central claim were well supported, the idea would be valuable: it offers a way to inject intrinsic-motivation exploration without altering a reward function that may have a strict semantic meaning, and it explicitly addresses the risk of novelty-bonus exploitation. The paper also provides a useful comparison against a Q-value-and-uncertainty baseline and evaluates across multiple continuous-control environments. However, the current evidence does not support the strength of the claim. The method description is incomplete in a load-bearing way (Section 4.3), the statistical support in Table 1 is weak, and the design decisions appear to have been made after observing results on the same benchmark environments. No code, data, or per-seed curves are provided. As a result, the contribution as presented is not reproducible and the mechanism is not isolated from post hoc design choices.","major_comments":[{"comment":"The actual decision rule executed by the agent is not the Gibbs distribution of Eq. (3) alone. The text states that a 'normalization' was applied before computing probabilities, that the resulting distribution was sampled S times, and that 'the action closest to the mean of the SAC policy's distribution was outputted.' The normalization formula and the value of S are never specified. This is a load-bearing omission: if the normalized uncertainties are nearly uniform, the closest-to-the-mean post-processing dominates and the reported gains cannot be attributed to world-model uncertainty. The authors must specify the exact algorithm and provide ablations that separately remove (i) the uncertainty signal, (ii) the normalization, and (iii) the closest-to-the-mean rule.","section":"Section 4.3, Eq. (3)"},{"comment":"The central claim that bounded exploration 'notably improved' performance is not statistically supported. Only about three seeds were used, and most comparisons show overlapping standard deviations; for example, SAC+BE vs. SAC in HalfCheetah is 9747.7 ± 2444.5 vs. 9200.0 ± 2258.5, in Ant it is -102.1 ± 156.2 vs. -151.5 ± 251.8, and in Walker2d it is 269.3 ± 204.0 vs. 172.8 ± 157.4. Moreover, bounded exploration is worse than the baseline in Hopper (475.2 ± 104.4 vs. 776.1 ± 242.8) and in model-based Walker2d (1136.4 ± 1042.9 vs. 1386.7 ± 1501.2). The paper should report more seeds, confidence intervals or significance tests, and per-seed learning curves before claiming a 6-of-8 improvement.","section":"Section 5, Table 1"},{"comment":"The normalization and closest-to-the-mean selection rule were introduced after the authors noticed that raw uncertainty differences were 'not apparent' during training, and the update frequency G = 10 was set by 'experimental trial-and-error.' Because these design choices were made after observing results on the same environments that are later used for the headline comparisons, the reported gains are vulnerable to selection bias. The authors should either pre-specify these choices or evaluate the method on a separate set of environments not used for design decisions.","section":"Section 4.3 and Section 5"},{"comment":"The proposed uncertainty signal is not validated. The paper assumes that the summed per-dimension variance of predicted next-state differences reliably distinguishes informative actions, but Section 4.3 admits that the differences were not apparent before normalization. There is no diagnostic showing that Eq. (2) actually separates high- and low-uncertainty actions at representative states, and no comparison to alternative uncertainty measures. Without such evidence, the mechanism driving the reported results remains unclear.","section":"Section 4.2, Eq. (2)"}],"minor_comments":[{"comment":"Figure 3 states N = 100 candidate actions, while Section 4.1 uses N = 4 in its example; the paper should state the value of N used in the experiments and keep the notation consistent.","section":"Figure 3 and Section 4.1"},{"comment":"Equation (3) uses exp(u_n) with unnormalized uncertainties, but the text says a normalization was applied 'before computing the probability.' Please clarify whether u_n in Eq. (3) is the raw uncertainty or a normalized value, and define what 'rank-preserving Gibbs distribution' means in this context.","section":"Section 4.3, Eq. (3)"},{"comment":"The statement that Hopper-v4 rewards were averaged with a sliding window to 'show a clear trend' is a post hoc smoothing choice. The authors should clarify whether this was pre-specified, and should show unsmoothed per-seed curves as well.","section":"Section 5"},{"comment":"The seed protocol is described only as 'All random seeds (Numpy and Pytorch) were constantly changed at every reset.' This is ambiguous; the paper should specify how many seeds were used per environment, how seeds were assigned, and whether the same seeds were used across methods.","section":"Section 5"},{"comment":"There are numerous formatting issues, including inconsistent citation styles, repeated citation text, and a stray math expression 'fθ(xt+1, rt|st, at)' at the very beginning of the Introduction. These should be cleaned up.","section":"References and Introduction"}],"recommendation":"reject","confidential_remarks":"This is an early-stage empirical report whose main contribution, as written, is not reproducible and whose headline claim is not statistically supported. The missing normalization and sampling parameter S, the post hoc design of the selection rule, and the absence of ablations are load-bearing rather than cosmetic. I recommend rejection, but I would view a substantially revised resubmission favorably if it provides a complete algorithm specification, code or data, ablations separating the uncertainty signal from the closest-to-the-mean rule, and a proper statistical analysis with more seeds. The core idea of using world-model uncertainty to select among policy-sampled actions without changing the reward is worth pursuing."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look if you care about exploration in SAC, but the paper's central claim does not survive contact with its own table.\n\nWhat's new: sampling N actions from the SAC policy and selecting among them via a Gibbs distribution over ensemble world-model variance (Eq. 3) is not in Yao et al. 2021, which adds uncertainty to Q-values. That's a distinct and reasonable idea, and the paper is honest about its limitations — it explicitly says MBRL gains are marginal and that this is a preliminary attempt.\n\nWhat's soft: Section 4.3 is the problem. The authors say raw uncertainty differences were 'not apparent,' then add a normalization and a closest-to-the-mean selection rule, neither of which is specified. The stress-test note is right: without an ablation separating Eq. 2, the normalization, and the closest-to-mean rule, the reported gains cannot be attributed to world-model uncertainty. If normalized uncertainties are near-uniform, the method is effectively action-smoothing. The evaluation is also too thin: roughly three seeds, overlapping error bars on most of the 6/8 wins, and post-hoc sliding-window smoothing for Hopper. No code or data release, and the ensemble training details are missing. The reader's 'circularity burden' score of 6 seems a bit harsh — tuning hyperparameters on the benchmark is normal practice, though the paper should say so — but the missing ablation is a load-bearing gap, not a minor omission.\n\nOverall: the idea is worth exploring, but the paper as written does not support 'notably improved performance.' A serious referee could help the authors tighten this; I would not desk-reject it outright, but I would send it back with a request for ablations, more seeds, and full disclosure of the selection rule.\n\nFor you: maybe bring it to reading group if the topic is hot, but don't cite it yet.","headline":"A genuinely novel selection mechanism undermined by an unspecified post-processing step and statistically weak results.","tokens_in":9045,"tokens_out":2378,"would_cite":false,"duration_ms":21710,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A bounded-exploration mechanism for Soft Actor-Critic that picks actions by world-model ensemble uncertainty improves rewards and sample efficiency in most tested continuous-control benchmarks without modifying the reward function.","keywords":["bounded exploration","Soft Actor-Critic","intrinsic motivation","world model uncertainty","model ensemble","model-based reinforcement learning","continuous control","exploration-exploitation trade-off"],"falsifier":"Run the bounded-exploration pipeline on a task where it helped, such as Swimmer-v4, with the uncertainty values shuffled or replaced by fixed random noise while keeping all other hyperparameters identical; if the agent's reward stays at the bounded-exploration level, the mechanism does not depend on world-model disagreement. A second check is to set the normalization to identity and S to one, removing the closest-to-the-mean pull, and see whether the performance gap persists.","tokens_in":8011,"feed_emoji":"🤖","tokens_out":11370,"duration_ms":98264,"temperature":0.7,"pith_summary":"This paper proposes bounded exploration, a way to add intrinsic-motivation exploration to Soft Actor-Critic (SAC) without touching the reward function. For each state, the agent samples many candidate actions from SAC's stochastic policy, asks an ensemble of world models how much their next-state predictions disagree for each candidate, and uses a Gibbs distribution over that disagreement to choose which action to execute. The authors report that this selection rule improves average return and data efficiency in 6 of 8 MuJoCo experiments, including faster convergence for a model-based SAC extension, and they argue this is a safer alternative to adding uncertainty bonuses to a reward that has a strict numerical meaning. The paper concedes the gains are not universal: in environments that require narrow learned motion sequences such as Hopper and Walker2d, bounded exploration performed worse or indistinguishably.","feed_headline":"Uncertainty-guided picks beat plain SAC in 6 of 8 runs","feed_subtitle":"Action picks driven by world-model disagreement speed learning without touching the reward.","key_machinery":"The carrying object is the rank-preserving Gibbs distribution over world-model uncertainty, $p(a_i^t)=\\frac{\\exp(u_i)}{\\sum_{n=1}^{N}\\exp(u_n)}$, where $u_n$ is the summed per-dimension variance of an ensemble's predicted next states for candidate action $a_n^t$. This distribution converts uncertainty ranks into probabilities, letting uncertainty choose among actions that SAC's stochastic policy already regards as plausible, and it is what distinguishes bounded exploration from methods that add uncertainty to the reward or to the Q-value. The ensemble, trained on normalized state differences, supplies the uncertainty signal; a normalization and a closest-to-the-mean pull were added during training to make the signal usable, but their exact form is not disclosed.","core_discovery":"The central claim is that exploration can be bounded by the soft policy's own distribution while still being driven by world-model uncertainty. SAC proposes a distribution of actions; the agent samples N candidates, feeds each candidate through an ensemble of five world models trained to predict normalized state differences, and computes the uncertainty of each candidate as the summed per-dimension variance of the next-state predictions, $u_n = \\sum_{d=1}^D \\sigma_d^2$. A rank-preserving Gibbs distribution then turns these uncertainty values into selection probabilities, so high-uncertainty actions are more likely to be executed while actions far outside SAC's recommendation are suppressed. During training the authors noticed that raw uncertainty differences were not apparent, so they added a normalization before the Gibbs step and, after sampling the resulting distribution S times, output the action closest to the SAC policy mean; the exact normalization and S value are not reported. On its own terms, the paper establishes that this bounded selector improves averaged reward and convergence speed in several continuous-control tasks and that it does so while leaving the reward function's meaning intact.","pith_inferences":["An implication the paper leaves untested is that the uncertainty signal itself, not the bounded resampling mechanics, causes the improvement; a clean way to check this is to replace the uncertainty values with random noise in the Gibbs selector while keeping all other details fixed.","The same selector design could be attached to any stochastic policy, not only SAC, provided a world-model ensemble is available, which would let the bounded-exploration principle transfer to other continuous-control algorithms.","The equal weighting of state dimensions in the variance sum is a hidden design choice; a learned or task-dependent weighting of prediction discrepancies could make the selector more discriminating in high-dimensional or partially observable environments.","Because the normalization and the S sampling parameter are not reported, reproducing the method requires filling those gaps; fixing S to a single sample and removing the normalization would reveal how much of the gain depends on the pull toward the SAC mean."],"forward_implications":["On tasks such as HalfCheetah, Swimmer, Ant, and Walker2d, adding bounded exploration improves average episodic reward or reduces the number of transitions needed to reach it, relative to vanilla SAC.","Because the reward function is unchanged, the method gives practitioners a way to pursue intrinsic exploration in settings where the reward has a strict meaning and must not be distorted by bonuses.","Equipping a model-based extension of SAC with bounded exploration can speed convergence in at least one tested environment (Ant-v4), even when final rewards are similar.","The method does not generalize uniformly: in Hopper-v4 and Walker2d-v4 the bounded-exploration agent's performance was lower or indistinguishable, suggesting that tasks whose rewards require a narrow learned sequence of motions may not benefit.","The comparison against an action-selection method that adds uncertainty to the Q-value shows the bounded-exploration design can produce competitive or better scores without preferring on-policy actions."],"supporting_citations":[{"why":"Provides the Soft Actor-Critic algorithm whose stochastic policy supplies the candidate actions and serves as the baseline.","marker":"[Haarnoja et al., 2018]"},{"why":"Establishes the practice of using an ensemble of learned dynamics models to quantify model uncertainty, which bounded exploration adapts for action selection.","marker":"[Chua et al., 2018]"},{"why":"Provides the justification that epistemic uncertainty from predictive models is a suitable intrinsic motivation.","marker":"[Depeweg et al., 2018]"},{"why":"Defines curiosity via prediction error in action space, the intrinsic-motivation approach the paper contrasts with its state-space uncertainty selector.","marker":"[Pathak et al., 2017]"},{"why":"The closest prior action-selection method, which adds Q-value and uncertainty, is compared as the 'QU' baseline.","marker":"[Yao et al., 2021]"},{"why":"Introduces model-based value expansion (MVE), the model-based extension on which bounded exploration is tested.","marker":"[Feinberg et al., 2018]"},{"why":"Supplies the short-horizon (H=2) rollout recommendation used in the model-based experiments.","marker":"[Janner et al., 2019]"},{"why":"Provides the training target of normalized state differences used to train the world-model ensemble.","marker":"[Kurutach et al., 2018]"}],"fun_headline_variants":["Uncertainty-bounded exploration lifts SAC to 6 of 8 wins","World-model uncertainty guides SAC to 6/8 best scores","Bounded exploration with uncertainty beats SAC in 6 of 8","SAC's exploration bounded by world-model doubt: 6/8 top scores","Uncertainty-aware action picking speeds SAC, wins 6 of 8"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the spread among an ensemble's predicted next states is a meaningful guide to which action to try; if that spread does not separate good from bad actions, the reported gains would come from the selection tricks rather than from world-model uncertainty.","fun_headline_variants_meta":{"raw":{"variants":["Uncertainty-bounded exploration lifts SAC to 6 of 8 wins","World-model uncertainty guides SAC to 6/8 best scores","Bounded exploration with uncertainty beats SAC in 6 of 8","SAC's exploration bounded by world-model doubt: 6/8 top scores","Uncertainty-aware action picking speeds SAC, wins 6 of 8"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000203,"raw_usage":{"total_tokens":1331,"prompt_tokens":837,"completion_tokens":494,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":453,"completion_tokens_details":{"reasoning_tokens":397}},"tokens_in":453,"tokens_out":494,"duration_ms":5022,"temperature":1.0,"reasoning_tokens":397,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T19:57:56.014257+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the bounded-exploration pipeline on a task where it helped, such as Swimmer-v4, with the uncertainty values shuffled or replaced by fixed random noise while keeping all other hyperparameters identical; if the agent's reward stays at the bounded-exploration level, the mechanism does not depend on world-model disagreement. A second check is to set the normalization to identity and S to one, removing the closest-to-the-mean pull, and see whether the performance gap persists.","supporting_citations":[{"cited_title":", 2018] Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine","cited_arxiv_id":null,"evidence_quote":"Establishes the practice of using an ensemble of learned dynamics models to quantify model uncertainty, which bounded exploration adapts for action selection."},{"cited_title":", 2018] Stefan Depeweg, Jose-Miguel Hernandez-Lobato, Finale Doshi-Velez, and Steffen Udluft","cited_arxiv_id":null,"evidence_quote":"Provides the justification that epistemic uncertainty from predictive models is a suitable intrinsic motivation."},{"cited_title":", 2017] Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell","cited_arxiv_id":null,"evidence_quote":"Defines curiosity via prediction error in action space, the intrinsic-motivation approach the paper contrasts with its state-space uncertainty selector."},{"cited_title":", 2021] Yao Yao, Li Xiao, Zhicheng An, Wan- peng Zhang, and Dijun Luo","cited_arxiv_id":null,"evidence_quote":"The closest prior action-selection method, which adds Q-value and uncertainty, is compared as the 'QU' baseline."},{"cited_title":", 2019] Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine","cited_arxiv_id":null,"evidence_quote":"Supplies the short-horizon (H=2) rollout recommendation used in the model-based experiments."}],"review_version":1}