{"id":"95d160fe-25ad-4241-96e9-8d667cc2b68e","arxiv_id":"2412.06486","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"SimuDICE uses DualDICE weights and model confidence to re-sample synthetic transitions in a tabular world model, improving offline Dyna-Q style policy optimization in small discrete environments.","lead":"SimuDICE is an offline reinforcement learning method that reweights synthetic experiences from a learned world model using distribution correction (DICE) weights and model confidence. In small grid-world tests it matches or beats two simple baselines with fewer planning steps, but it is not tested against modern offline RL methods.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"With λ=1000, Eq. 13–14 makes the DICE term effectively a one-hot over the highest-DICE pair, so the claimed DICE–confidence balance may not be the actual mechanism behind the Taxi gains.","rationale":"The reader's weakest assumption concerns whether the DualDICE correction w^{π/D}, defined via the true environment, remains a valid guide for reweighting samples from a mode-based world model. My concern is distinct but adjacent: even if w is correctly defined and estimated, the specified sampling formula with the chosen λ does not implement the advertised 'balance' between confidence and correction. The extreme softmax temperature converts the formula into argmax selection, so the reported Taxi gains may stem from a trivial priority effect rather than from the DICE+confidence mechanism. This is a load-bearing internal consistency issue: the paper's central conceptual contribution is the balancing of two terms, and the experimental configuration may eliminate one term entirely. I do not see this as a reason to reject outright; the empirical comparison against uniform Dyna could still be valid for a simpler reason, and the paper already reports a CONDITIONAL verdict from the reader. Adding the λ-sweep test is the smallest experiment that would settle whether the stated mechanism is active. My agreement with the reader is partial because they emphasized the alignment of w with world-model dynamics, while I emphasize the effective collapse of the sampling distribution due to λ; both point to the same fragility of the reweighting step, but the concrete failure mode and the test differ. The paper's own limitation section admits sensitivity to the sampling-probability formula, but it does not test the scale parameter λ, which is exactly the parameter that controls whether the formula balances or collapses.","tokens_in":11312,"tokens_out":7348,"duration_ms":85352,"concrete_test":"Run SimuDICE on Taxi (ϵ=0.1, 400 episodes) with λ ∈ {0.001, 0.1, 1, 10, 100, 1000}, keeping all other hyperparameters fixed. For each λ, record (i) the average per-step reward after planning and (ii) the fraction of planning samples drawn from the single (s,a) with maximal w^{π/D}. Additionally, compare against two baselines: P(s,a) ∝ C(s,a) (pure count priority) and P(s,a) = one-hot argmax_w (pure DICE max). If the reward gain appears only near λ=1000 where the top-w pair receives >50% of samples, the balancing claim is falsified and the result is a non-specific focus effect. If performance is stable across λ and the top-w share stays moderate, the claimed mechanism survives.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that SimuDICE improves sample efficiency by 'balancing experiences similar to those frequently encountered with ones that have a distribution mismatch' (Abstract). This balance is implemented by Eq. 13–14, where L(s,a) = C(s,a) + λ·softmax(w^{π/D}(s,a)) and then normalized. With the reported hyperparameter λ=1000 (Section 5), the added term is 1000 times a softmax that sums to 1. For any state-action pair whose softmax value exceeds roughly C(s,a)/1000, the DICE term dominates. In a tabular environment with thousands of (s,a) pairs, the softmax is typically concentrated on a few large w values, so the max softmax is often ≫ 1/|S×A|, making the maximum-w pair receive close to 100% of the sampling mass. Thus the algorithm behaves as 'replay the single highest-DICE transition', not the stated balancing of confidence and correction. This is also internally inconsistent with the text: λ is introduced to 'prevent mode collapse', but increasing λ sharpens the softmax and induces mode collapse. If this reading is correct, the Taxi improvements in Fig. 3 could be reproduced by any priority rule that focuses on one critical transition, undermining the specific DICE-based contribution and the generality of the sample-efficiency claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SimuDICE, an offline model-based RL method for discrete tabular domains. It learns a tabular world model from an offline dataset, estimates the stationary distribution correction w^{π/D} between the target policy and the behavior policy using DualDICE, and uses it to reweight state-action pairs sampled from the world model during planning. The reweighting combines a model-confidence term C(s,a) (normalized dataset frequency) with a λ-regularized softmax of the DICE weights. The policy is updated by Q-learning on the synthetic transitions, and the procedure can be iterated. Experiments on Taxi, CliffWalking, and FrozenLake compare SimuDICE against offline Q-learning and a uniform-sampling Dyna-Q variant, with ablations over planning steps, sampling formulas, and the number of DICE-update iterations. The paper claims improved sample efficiency and fewer required planning steps relative to the tested baselines, with the gains concentrated in Taxi.","tokens_in":11653,"tokens_out":9696,"duration_ms":98943,"significance":"If validated, SimuDICE would provide a simple, interpretable mechanism for combining DICE-based distribution correction with model-based planning in tabular offline RL, and the open-source code and clear algorithmic structure are valuable for reproducibility. The study is honest about its limitations, and the ablations over planning steps and sampling formulas are useful. However, the contribution is currently supported by a narrow experimental base—only one environment shows a clear improvement—and there is a load-bearing inconsistency in the definition of the sampling likelihood that makes the claimed mechanism ambiguous. As it stands, the paper establishes a plausible sampling-priority heuristic rather than a demonstrated general method.","major_comments":[{"comment":"The likelihood function is defined inconsistently. Eq. (13) as typeset reads L(s,a)=C(s,a)+ e^{w(s,a)λ}/(Σ e^{w(s',a')λ}/λ), i.e., C plus λ times a softmax term, whereas Eq. (15), which is called the default formula, reads L_1(s,a)=C(s,a)+ e^{w(s,a)λ}/(λ Σ e^{w(s',a')λ}), i.e., C plus softmax/λ. These differ by a factor of λ² and have opposite behavior for the reported λ=1000: Eq. (13) makes the DICE term dominate and concentrate nearly all sampling mass on the largest-w pair, while Eq. (15) makes the DICE term contribute only 1/λ of the total likelihood mass, so P(s,a) is essentially C(s,a) after normalization. The statement in §4 that λ 'prevent[s] mode collapse' is compatible with neither interpretation: under Eq. (13) a large λ sharpens the softmax and induces mode collapse, and under Eq. (15) the DICE contribution is negligible. This inconsistency is load-bearing because the paper attributes the Taxi improvements to the interaction of C and w, and the reported hyperparameter λ=1000 must be reconciled with the actual default formula before the experimental results can be interpreted.","section":"§4, Eq. (13); §5.2, Eq. (15)"},{"comment":"The central sample-efficiency and 'comparable to existing algorithms' claims are not supported by the presented comparisons. In CliffWalking the paper itself reports 'no significant difference' (Sec. 5.1), and in FrozenLake the only clear gap is that offline Dyna-Q underperforms, with SimuDICE matching offline Q-learning; the consistent advantage appears only in Taxi. No significance tests accompany the word 'significantly.' The two baselines are offline Q-learning and offline Dyna-Q, which is a uniform-sampling variant of SimuDICE itself; no comparison is made to standard offline or offline MBRL algorithms such as CQL, IQL, MOPO, or MOReL. The Abstract's claim of performance 'comparable to existing algorithms' therefore overstates the evidence, and the Section 6 limitations already acknowledge this. This should be fixed either by adding competitive baselines or by explicitly restricting the claim to a comparison against uniform Dyna-style planning.","section":"§5.1, Fig. 3; §6"},{"comment":"The DualDICE component is not described at implementation level. The paper does not state the function class used for ν, the optimizer, the number of gradient steps, or any clipping or normalization applied to w before it is inserted into Eq. (13)/(15). Additionally, the seed count is inconsistent: Section 5 says results are averaged over '500 plays and 20 seeds,' while the Fig. 3 caption says '5 different random seeds.' These details are necessary to reproduce the reported variance bands and to check whether the estimated w values are stable in the small-sample tabular setting.","section":"§5; Algorithm 1, line 5"},{"comment":"The role of the DICE weights is unclear because in the one-iteration experiments they are computed from the initial target policy and are not updated during planning, and Fig. 6 shows that additional iterations do not change performance. This is consistent with the concern that the reweighting mainly enforces self-consistency with the current policy rather than correcting distribution shift toward a better policy. A concrete test would be to compare SimuDICE against a version that re-estimates w during planning and against a baseline that prioritizes transitions with random weights or with C-only weights; without such a control, the mechanism behind the Taxi gains remains ambiguous.","section":"§4, Algorithm 1; §5.2, Fig. 6"}],"minor_comments":[{"comment":"The sentence 'The evaluation was performed on the Taxi environment using the 9 datasets collected from the other environments' appears to be a textual error; the results in Fig. 3 are reported for Taxi, FrozenLake, and CliffWalking, each with three datasets.","section":"§5, first paragraph"},{"comment":"The tabular world model returns only the most frequently observed next state, which removes stochasticity from the model; the statement that this approach 'allows the model to handle stochastic environments' is misleading, particularly for FrozenLake where the environment is stochastic.","section":"§4, Algorithm 2"},{"comment":"In Formula 2 the subtractive term e^{wλ}/(λ Σ e^{w'λ}) can exceed C(s,a), making L_2(s,a) negative; after normalization by the total sum this does not define a valid probability distribution. The ablation should either clip negative likelihoods or otherwise ensure nonnegativity.","section":"§5.2, Eq. (16)"},{"comment":"The word 'significantly' is used to describe Taxi improvements without any statistical test or reported standard error; the authors should report per-seed results and a paired test or bootstrap confidence intervals.","section":"§5.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is borderline for publication. The core idea is simple and potentially interesting for tabular offline MBRL, and the open-source code and ablations are strengths. However, the discrepancy between Eq. (13) and Eq. (15) is load-bearing: if the implemented formula is Eq. (15), the DICE term is nearly negligible at the reported λ=1000 and the mechanism may reduce to empirical-frequency sampling, while if Eq. (13) is used, the reported setting induces near one-hot sampling. The narrow baseline set and the lack of statistical tests further weaken the abstract's claims. These issues are fixable in a revision, so I recommend major revision rather than rejection, provided the authors clarify the actual default formula and restrict their claims to what the experiments support."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a modest but coherent paper. It builds a Dyna-Q planner over a tabular world model and reweights synthetic experience sampling by combining model confidence with DualDICE stationary distribution corrections. That specific combination is new as far as I can tell, and the paper is honest that it is an integration of known components rather than a new theory.\n\nWhat it does well: the Taxi experiments are the load-bearing evidence, and they support the claim that the method beats offline Q-learning and uniform-sampling Dyna-Q in most data regimes, especially with only 10 planning steps. The ablation over sampling formulas is useful, the code is public, and the limitations section is unusually candid: it admits the environments are simple, the sampling formula is sensitive, and the comparison set is weak. That honesty earns real credit.\n\nSoft spots: the baseline set is too weak for the abstract's \"comparable to existing algorithms\" claim. There is no comparison to MOPO, MOReL, or any other offline MBRL method. There are also no significance tests, and there is an internal inconsistency: the text says 20 seeds, the Figure 3 caption says 5. Those are fixable but need fixing.\n\nThe bigger issue is the mechanism. Equations 13-15 add a λ-scaled softmax of DICE weights to a normalized frequency count, with λ=1000. In these datasets, observed state-action frequencies average around 0.01, so the DICE term (at most 0.001 after the division by λ) is either negligible for frequent pairs or, for unobserved pairs, still small relative to typical confidence. The practical sampling distribution is therefore closer to frequency-weighted replay than to the stated \"balancing\" of DICE and confidence. The stress-test worry about one-hot behavior is directionally right: whichever way the counts fall, the equations do not implement the described balance. This does not kill the empirical Taxi result, and the ablation suggests formula 1 genuinely helps, but it means the paper does not demonstrate that DICE is the active ingredient.\n\nOverall: the central empirical claim is credible against the two chosen baselines, but the mechanism story and the generality claim need more work. This paper deserves a serious referee, especially for a workshop or a venue that accepts clean incremental empirical contributions. I would ask the authors to add at least one stronger offline MBRL baseline, report error bars with proper significance testing, and include a λ sweep to show what the sampling distribution actually looks like.","headline":"SimuDICE is a clean, honest integration of DualDICE weights into Dyna-style offline planning; the Taxi gains look real but the claimed DICE-confidence balance is not established, and the baseline set is too thin for the abstract's claims.","tokens_in":707,"tokens_out":729,"would_cite":false,"duration_ms":46520,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that reweighting a tabular world model's sample distribution with DualDICE corrections can match or beat offline Q-learning and uniform Dyna-Q using fewer pre-collected episodes and fewer planning steps.","keywords":["offline reinforcement learning","model-based reinforcement learning","distribution correction estimation","DICE","world model","sample efficiency","Dyna-style planning","tabular MDP"],"falsifier":"Take a discrete environment where the true transitions are multimodal, such that a tabular model's most-frequent-next-state approximation hides uncertainty, and compare the ordering of state-action pairs induced by $w^{\\pi/D}$ computed on the true model versus on the learned tabular model. If the rankings differ materially, SimuDICE's sampling guidance is based on a correction that does not match the model it is drawing from, and the reported sample-efficiency gains would not transfer.","tokens_in":11132,"feed_emoji":"🎲","tokens_out":5743,"duration_ms":54235,"temperature":0.7,"pith_summary":"This paper tries to show that offline model-based reinforcement learning can be made more sample-efficient without changing the dynamics model at all. Instead of sampling stored experiences uniformly during planning, SimuDICE reweights state-action pairs by combining the model's confidence in its own predictions with DualDICE estimates of how much the target policy's state-action distribution differs from the data. On three discrete grid-world tasks, the authors report that this guided sampling matches or outperforms both vanilla offline Q-learning and uniform-sampling Dyna-Q while needing fewer pre-collected episodes and fewer planning steps. The significance, if the claim holds, is that distribution-correction ideas from policy evaluation can double as a cheap exploration signal inside a model-based planner.","feed_headline":"Reweighted world-model sampling matches offline RL with less data","feed_subtitle":"SimuDICE blends model confidence with DICE corrections so planning revisits distribution-mismatched transitions first.","key_machinery":"The load-bearing object is the sampling distribution $P(s,a)$ over the tabular world model's stored transitions, defined by Eq. (14) as the normalized version of $L(s,a) = C(s,a) + \\lambda^{-1} e^{\\lambda w^{\\pi/D}(s,a)} / \\sum_{s',a'} e^{\\lambda w^{\\pi/D}(s',a')}$. $C(s,a)$ is the empirical confidence (normalized occurrence count) of the transition, and $w^{\\pi/D}$ is the DualDICE density ratio $d^\\pi(s,a)/d^D(s,a)$. The mechanism works by making Q-learning updates in the planner visit state-action pairs in an order that balances the model's reliability with the policy's distributional shift.","core_discovery":"The central claim is that the sampling probabilities used by a Dyna-style planner can be set to $P(s,a) \\propto L(s,a)$, where $L(s,a) = C(s,a) + \\lambda^{-1} e^{\\lambda w^{\\pi/D}(s,a)} / \\sum_{s',a'} e^{\\lambda w^{\\pi/D}(s',a')}$, with $C(s,a)$ the normalized frequency of the state-action pair in the dataset (the model's confidence) and $w^{\\pi/D}(s,a)$ the DualDICE stationary distribution correction $d^\\pi(s,a)/d^D(s,a)$. With this reweighting, the planner preferentially revisits transitions that are both trusted by the model and favored by the current target policy, and it backs off toward confident transitions when the correction is unreliable. The paper shows empirically that this yields average per-step rewards comparable to or higher than uniform sampling and offline Q-learning while using less data and fewer planning steps, with the gains largest in Taxi, the largest of the three tested environments.","pith_inferences":["The reweighting formula only reorders observed transitions; it cannot create genuinely new states, so a generative world model paired with the same correction would likely need a stronger confidence safeguard than the empirical frequency $C(s,a)$ used here.","DualDICE weights are computed against the fixed dataset at each iteration, but the target policy changes during planning; an alternating update that recomputes corrections after each planning round could matter in environments with stronger distribution shift than the three tested.","A testable extension is to make the regularization parameter $\\lambda$ adaptive to the discrepancy between $C$ and $w^{\\pi/D}$, rather than fixing it at 1000 as in the reported experiments."],"forward_implications":["Matching baseline performance with fewer pre-collected episodes means offline practitioners can get usable policies from smaller datasets in tabular domains.","Fewer planning steps per iteration reduce the number of synthetic transitions generated, which lowers the chance that an imperfect model produces hallucinated states.","The formula's robustness across epsilon-greedy data-collection policies suggests that DICE-guided reweighting can adapt when the behavioral policy is noisy.","Because the ablation found no benefit from multiple reweighting iterations, the method can be run as a single reweighting pass, which is computationally cheaper."],"supporting_citations":[{"why":"Supplies the DualDICE stationary distribution correction $w^{\\pi/D}$ that drives the reweighting of the world model's samples.","marker":"[28]"},{"why":"Provides the Dyna planning loop that SimuDICE adapts, and the offline Dyna-Q baseline is SimuDICE with uniform sampling.","marker":"[33]"},{"why":"Supplies the tabular world model that stores average reward and the most frequently observed next state for each state-action pair.","marker":"[32]"},{"why":"Provides the dataset collection protocol of partially trained policies plus epsilon-greedy variants that is used in the experiments.","marker":"[39]"},{"why":"Sets the stabilization and evaluation protocol for off-policy Q-learning that the comparisons build on.","marker":"[8]"},{"why":"Defines the offline RL problem and the distribution mismatch that motivates the method.","marker":"[23]"}],"fun_headline_variants":["SimuDICE reweights world-model samples for efficient offline RL","Offline RL: reweighted world-model sampling needs less data","DICE-corrected world models cut offline RL data requirements","SimuDICE: fewer samples, same offline RL performance","Reweighted sampling improves offline RL sample efficiency"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything rests on the premise that the DualDICE stationary-distribution corrections, computed against the true environment and the current target policy, still point at the right experiences to reweight when the planner only samples from a simple tabular model that stores the most common next state and average reward for each pair.","fun_headline_variants_meta":{"raw":{"variants":["SimuDICE reweights world-model samples for efficient offline RL","Offline RL: reweighted world-model sampling needs less data","DICE-corrected world models cut offline RL data requirements","SimuDICE: fewer samples, same offline RL performance","Reweighted sampling improves offline RL sample efficiency"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000214,"raw_usage":{"total_tokens":1425,"prompt_tokens":948,"completion_tokens":477,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":564,"completion_tokens_details":{"reasoning_tokens":392}},"tokens_in":564,"tokens_out":477,"duration_ms":5119,"temperature":1.0,"reasoning_tokens":392,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T19:36:22.946727+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a discrete environment where the true transitions are multimodal, such that a tabular model's most-frequent-next-state approximation hides uncertainty, and compare the ordering of state-action pairs induced by $w^{\\pi/D}$ computed on the true model versus on the learned tabular model. If the rankings differ materially, SimuDICE's sampling guidance is based on a correction that does not match the model it is drawing from, and the reported sample-efficiency gains would not transfer.","supporting_citations":[{"cited_title":"In: Advances in Neural Information Processing Systems","cited_arxiv_id":null,"evidence_quote":"Supplies the DualDICE stationary distribution correction $w^{\\pi/D}$ that drives the reweighting of the world model's samples."},{"cited_title":"Proceedings of the Seventh International Conference on Machine Learning pp","cited_arxiv_id":null,"evidence_quote":"Supplies the tabular world model that stores average reward and the most frequently observed next state for each state-action pair."},{"cited_title":"In: Proceedings of the 36th International Conference on Ma- chine Learning","cited_arxiv_id":null,"evidence_quote":"Sets the stabilization and evaluation protocol for off-policy Q-learning that the comparisons build on."}],"review_version":1}