{"id":"8ebb670f-a0b6-4071-a271-fe33d791d2a3","arxiv_id":"2506.19643","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Training a diverse set of policies, relabeling their experience with the target reward, and selecting the highest-return buffer improves model-based offline RL on unknown tasks, supported by a Wasserstein-distance analysis.","lead":"Offline reinforcement learning usually inherits whatever data it is given, and bad data causes bad policies. This paper argues that collecting a diverse set of behaviors and then choosing the best single buffer for the target task lets an offline agent solve tasks it was never trained for.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Equation 8 rests on the unproven approximation D1≈D2; if it fails, model error under π* dominates and the behavioral-optimal Wasserstein distance is not the controlling quantity.","rationale":"I agree with the reader's identification of the weakest assumption. The paper's main theorem has two parts: the exact-looking bound in Eq. 6 with D1+D2, and the cleaned-up Eq. 8. Only Eq. 8 supports the paper's headline interpretation and the design of UDG. The step from Eq. 6 to Eq. 8 is not a corollary; it is a heuristic substitution of D2 for D1. Since D1 is a model-error term, it can be the dominant term in any offline-RL problem where the learned model is inaccurate under π*; this is precisely the regime MOPO was designed for. I also note the text itself concedes that D1>D2 is possible and that Eq. 12 is not a general minimizer of Eq. 11, so the paper's claim that diverse policies approximately minimize worst-case regret is not proven. These are internal-concession flags, not external disagreements. The empirical results are suggestive but do not measure D1 and D2; Table 2's comparison of top-1 vs top-2 vs all-mixed only tests the ranking predicted by D2 under the assumption, not the approximation itself. A direct computational test of Eq. 8 on a small MDP would settle the central claim. Thus I keep the reader's CONDITIONAL verdict: the contribution is plausible but the core bound is conditional on an unquantified approximation and the missing proof/appendices.","tokens_in":14080,"tokens_out":7039,"duration_ms":73256,"concrete_test":"Construct a small deterministic Lipschitz MDP (e.g., a 1D chain with transition s' = clamp(s+a)) and choose πβ and π* so that W1(ρπβ_T, ρπ*_T) is small but π* assigns non-negligible mass to a state-action region that the behavioral buffer covers only sparsely. With the Assumption 4.6 nearest-neighbor model trained on a finite buffer, estimate D1 and D2 from sampled trajectories and evaluate MOPO's return gap. If D1/D2 is not close to 1 or η_M(πhat) violates η_M(π*) − 2C D2, Eq. 8's approximation is falsified; repeat across buffer sizes and random seeds to show the failure is not a sampling artifact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central result used by the paper is the approximate bound η_M(πhat) ≥ η_M(π*) − 2C W1(ρπβ_T, ρπ*_T) (Eq. 8). This is not a theorem: it is obtained from the first line of Theorem 4.8 by replacing D1 := W1(ρπ*_T, ρπ*_\\hatT) with D2 := W1(ρπβ_T, ρπ*_T), justified only by the assertion that ρπ*_\\hatT will not be too distinct from the behavioral distribution. No quantitative bound is given, and the text explicitly says D1 > D2 is possible under non-smooth dynamics or multi-modal situations. This matters because D1 is the model-error term under the optimal policy's occupancy. In offline RL, π* can concentrate mass on state-action regions that are present in the behavioral buffer with low density or absent entirely; for a nearest-neighbor model (Assumption 4.6) D1 can then be large even when D2 is small, or vice versa. The purported support inclusion does not control D1: it concerns next states emitted by the memory, not the frequencies or action marginals of ρπ*_\\hatT. Without D1≈D2, the first line of Eq. 6 is dominated by D1, so the paper's headline interpretation — that the behavioral-optimal Wasserstein distance governs the gap — is not established. The surrogate objective (Eq. 12) is also acknowledged not to match the minimax regret objective (Eq. 11) in general, so the regret-optimality claim is likewise unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates how the distribution of the offline batch data affects the performance of model-based offline reinforcement learning. It builds on the MOPO performance lower bound and derives, under Lipschitz and nearest-neighbor model assumptions, a bound (Theorem 4.8) on the return of the MOPO-trained policy in terms of Wasserstein distances between the behavioral occupancy distribution and the optimal policy occupancy distribution. It then introduces an approximate bound (Eq. 8) that drops the model-error term by assuming D1≈D2, and uses this to formulate a worst-case regret objective (Eq. 11) and a diversity surrogate objective (Eq. 12) for task-agnostic data generation. The proposed framework, UDG, trains a set of diverse policies, collects buffers from their rollouts, relabels rewards once the task is revealed, selects the buffer with the highest average return, and trains MOPO on that buffer. Experiments on Ant-Angle and Cheetah-Jump compare UDG with supervised data, mixed buffers, and non-diverse baselines.","tokens_in":14545,"tokens_out":8227,"duration_ms":79916,"significance":"If the theoretical chain were rigorous, the paper would provide a useful and interpretable data-centric principle: in model-based offline RL, the gap to the optimal policy is governed by the Wasserstein distance between the behavioral and optimal occupancy distributions, so unsupervised diversity-based data generation is a principled choice for task-agnostic settings. The paper deserves credit for grounding the discussion in MOPO's bound, for transparently listing its own limitations (Lipschitz assumptions, non-parametric model, neural-model gap), and for including buffer-selection ablations that test the data-distance hypothesis. However, the headline result is currently an unquantified approximation, the relation between the minimax regret objective and the diversity surrogate is asserted rather than proved, and the experiments use a different model class than the theory. The contribution is therefore suggestive and potentially valuable, but the central theoretical claims are not yet established.","major_comments":[{"comment":"The step from the first line of Theorem 4.8 to Eq. (8) replaces D1 = W1(ρπ*_T, ρπ*_\\hatT) with D2 = W1(ρπβ_T, ρπ*_T) on the basis of the assertion that ρπ*_\\hatT 'will not be too distinct' from the behavioral distribution. This is not a theorem step: the support inclusion supp(ρπ*_\\hatT(s)) ⊆ supp(ρπβ_T(s)) only constrains the states emitted by the episodic memory, not the action marginals or the frequencies of ρπ*_\\hatT. D1 can therefore be large even when D2 is small, and since D1 is the model-error term under the optimal policy's occupancy, the claim that W1(ρπβ_T, ρπ*_T) controls the performance gap is not established without a quantitative bound on |D1−D2|. The authors should either prove such a bound under a stated additional assumption or present Eq. (8) explicitly as a heuristic and soften the corresponding claims in the abstract and interpretation.","section":"Section 4, Eq. (8) and text after Eq. (7)"},{"comment":"The surrogate diversity objective (Eq. 12) is admitted, in the text following Eq. (11), to have an optimal solution that does not match the minimax regret objective (Eq. 11) in general, and the paper says 'we assume the gap will be small' with an analogy to covering/packing and computational graphics. This is not a proof, and the third contribution is explicitly 'an objective of minimal worst-case regret'. Without a quantitative relation between the packing-type objective (Eq. 12) and the covering/regret objective (Eq. 11), the statement that UDG approximately minimizes worst-case regret is unsupported. The authors should either prove a bound under a stated condition on the policy class Π or clearly label Eq. (12) as a heuristic proxy and remove or qualify the regret-optimality claims.","section":"Section 4, Eq. (12) versus Eq. (11)"},{"comment":"The main theorem, Theorem 4.8, relies on Assumption 4.6 that the prediction model is a non-parametric nearest-neighbor model, but the experiments train MOPO with neural network dynamics models, as the Limitations section acknowledges. This mismatch means the experiments do not directly test Theorem 4.8. A claim that 'experimental results match the findings in theoretical analysis' needs either a nearest-neighbor variant of MOPO in the experiments or a separate argument that the bound extends to neural regressors, such as a Lipschitz or regression-error bound for the learned model.","section":"Section 5 and Section 4, Assumption 4.6"}],"minor_comments":[{"comment":"The statement that substituting π* with πβ yields ηM(πhat) ≥ ηM(πβ) does not follow from Eq. (6), since the bound still contains W1(ρπβ_T, ρπβ_\\hatT), and MOPO's own guarantee includes the model-error term 2γϵ_u(πβ).","section":"Section 4, after Theorem 4.8"},{"comment":"Definition 4.3 defines the radius of the minimal enclosing ball, not the diameter; this makes the quantity Bπβ in Eq. (6) ambiguous and should be renamed 'covering radius' or redefined.","section":"Definition 4.3"},{"comment":"The assumption uses three '≈' relations (\\hatρπβ_T ≈ ρπβ_T ≈ ρπβ_\\hatT) without quantifying the Wasserstein error; if these approximations affect the final bound, they should be stated as explicit assumptions with error terms.","section":"Assumption 4.7"},{"comment":"For Cheetah-Crawl, the diverse result (1348.19±274) overlaps heavily with the random result (1239.00±57), so the claim that diverse data performs better is not statistically supported by this table.","section":"Table 1"},{"comment":"There are several typos and minor wording issues, including 'connenction' (Section 4 heading), 'polices' (Figure 4 caption), 'an policy' (Section 5), and inconsistent use of 'Mujoco' vs. 'MuJoCo'.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper is potentially salvageable as an empirical and conceptual contribution if the D1≈D2 step and the surrogate-proxy step are honestly labeled as heuristics and the theoretical claims are weakened accordingly. If the authors intend to keep the full theoretical framing, they need a quantitative bound on D1 or an explicit additional assumption that controls it. The buffer-selection idea is a clear novelty relative to earlier unsupervised data generation work, and the transparency of the Limitations section is a positive sign."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper is best read as a method-level contribution with a nice theoretical motivation, not as a rigorous bound. What is new: it connects the divergence between the behavioral data distribution and the optimal policy's occupancy to the offline performance gap of model-based RL, using a Wasserstein distance. That reframing gives a principled reason why diverse unsupervised data should help for task-agnostic problems—at least one buffer should be close to any future optimal policy. The UDG pipeline (train diverse policies, relabel rewards, select the best buffer, train with MOPO) is simple and follows from the story.\n\nThe theoretical centerpiece, Equation 8, is not a theorem. The step D1 ≈ D2 is asserted with no quantitative control, and the stress-test note is correct: the episodic memory's support inclusion does not bound the model error under π*, because it does not control action marginals or frequencies. The authors admit D1 > D2 can happen, so the headline interpretation is only heuristic. The surrogate diversity objective (Eq. 12) is also conceded to be an unproven proxy for the minimax regret objective (Eq. 11). These are real soft spots, but they are not hidden—the text flags the approximation, and the limitations section is honest.\n\nThe experiments are the larger disappointment. On Ant-Angle, the baselines are just supervised and supervised-with-diversity; there is no comparison to existing exploratory-data methods like ExORL or to standard offline RL on the same data. Cheetah-Jump results appear as two numbers with standard deviations and no seed count. No code is released. For a paper whose contribution is primarily a framework, that is thin support.\n\nOn the positive side, the citation pattern is clean, the connection to WURL is prior published work by the same group, and the derivation from MOPO is transparent. No invented entities or fitted constants appear to prop up the theory.\n\nMy verdict: this deserves a serious referee, not a desk reject. The reframing is useful and timely, and even a heuristic relationship between data geometry and offline performance is worth airing. I would, however, send it back with a request to (1) either prove a version of the bound with mild coverage assumptions or clearly state the theorem with the approximation as a corollary, and (2) strengthen the experiments with external baselines and code release. I would cite the reframing in my own work, but not the quantitative bound.\n\nBring it to reading group if you want a good discussion about how much weight a theoretical motivation should carry when the headline inequality is approximate. Otherwise, a quick skim is enough.","headline":"A useful Wasserstein reframing of offline data quality, but the headline bound is an unproven approximation and the experiments are thin; worth refereeing nonetheless.","tokens_in":14974,"tokens_out":2640,"would_cite":true,"duration_ms":27797,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The performance of model-based offline RL is bounded by the Wasserstein distance between behavioral data and the optimal policy's occupancy, and diverse unsupervised data with buffer selection can approach optimality on unknown tasks.","keywords":["offline reinforcement learning","unsupervised reinforcement learning","data generation","Wasserstein distance","model-based offline RL","task-agnostic learning","policy diversity","buffer selection"],"falsifier":"Construct a Lipschitz MDP with a multi-modal transition (two very different next states both likely) and collect a behavioral buffer close to the optimal occupancy in $W_1$; if the measured gap $\\eta_M(\\pi^*) - \\eta_M(\\hat\\pi)$ repeatedly exceeds $2C\\,W_1(\\rho^{\\pi_\\beta}_T, \\rho^{\\pi^*}_T)$, then the $D_1 \\approx D_2$ supposition behind Equation 8 fails in that setting.","tokens_in":13912,"feed_emoji":"🎯","tokens_out":11855,"duration_ms":107800,"temperature":0.7,"pith_summary":"This paper tries to establish that, in model-based offline reinforcement learning, the important property of a batch dataset is how close its state-action occupancy is to the occupancy of the optimal policy, measured by the 1-Wasserstein distance. Under Lipschitz continuity and a nearest-neighbor learned transition model, the return gap between the learned policy and the optimal policy is bounded by a constant times that distance, so data generation becomes a distance-minimization problem. Because the task reward may be unknown when data are collected, the paper trains a family of policies with diversity rewards and later picks the buffer that scores best under the revealed reward. The resulting framework, UDG, outperforms data produced by a single supervised policy in the locomotion tasks tested.","feed_headline":"Offline RL's performance gap is a data-distance problem","feed_subtitle":"UDG trains diverse policies with no task reward, then picks the buffer that scores best on the revealed task, beating supervised data.","key_machinery":"The argument is carried by two objects. First, the telescoping performance-difference identity (Lemma 4.1), which expresses the return gap between true and learned dynamics as an expectation over the learned occupancy; combined with Lipschitz dynamics and value functions, the integrand is bounded by a model error that can be measured as a 1-Wasserstein distance. A non-parametric nearest-neighbor transition model (Assumption 4.6) keeps the support of model rollouts inside the behavioral buffer, and that is what lets the triangle inequality turn the bound into the distance $W_1(\\rho^{\\pi_\\beta}_T, \\rho^{\\pi^*}_T)$. Second, the surrogate diversity objective of Equation 12: maximize the minimum pairwise Wasserstein distance between the occupancy distributions of the candidate policies, which the paper argues approximates the minimax regret of Equation 11.","core_discovery":"On its own terms, the paper's central claim is that MOPO's model-error lower bound can be rewritten as a data-distribution bound: $\\eta_M(\\hat\\pi) \\ge \\eta_M(\\pi^*) - 2C\\,W_1(\\rho^{\\pi_\\beta}_T, \\rho^{\\pi^*}_T)$, with $C = 2c\\gamma L_r L_T$, whenever the unknown model error under the optimal policy is comparable to the behavioral-to-optimal distance. This is Theorem 4.8 together with the $D_1 \\approx D_2$ approximation behind Equation 8. The theorem also gives a mixture version (Equation 9), showing that blending many diverse buffers can widen the bound unless the mixture stays close to the optimal occupancy. The paper further claims that the unsupervised diversity objective $\\max \\min_{i\\neq j} W_1(\\rho^{\\pi_i}_T, \\rho^{\\pi_j}_T)$ approximately optimizes the worst-case regret $\\min_{\\pi_1,\\dots,\\pi_N}\\max_{\\pi^*}\\min_i L(\\pi_i,\\pi^*)$ over unknown tasks, and that choosing the highest-scoring buffer is therefore the right data-selection rule.","pith_inferences":["The bound suggests a cheap data-quality diagnostic: estimate $W_1$ between a candidate buffer and the occupancy of a reward-optimized policy, and pre-select buffers by that distance rather than by rollout evaluation, which would extend UDG to settings where the reward is only partially known.","The packing/covering relation between the surrogate diversity objective and the minimax regret (which the paper acknowledges is hard to analyze) implies that the number of policies $K$ needed for a given worst-case regret in continuous state spaces may scale with the metric entropy of policy-occupancy space; quantifying that scaling is a natural open problem.","Because the proof relies on a non-parametric nearest-neighbor model, the same buffer-selection recipe should be tested with neural dynamics models and model-free offline algorithms to see whether the Wasserstein distance remains the controlling quantity when the proof device is removed."],"forward_implications":["Data collection for offline RL should be judged by the occupancy distance to the intended task's optimal policy, not by raw state coverage; a single close buffer can beat a broad mixture.","When the task is unknown, unsupervised diversity is a rational strategy: a sufficiently diverse policy set makes it likely that at least one buffer is close to the optimal policy, and selecting that buffer by average return yields near-optimal offline behavior.","Mixing all generated buffers can hurt performance compared with selecting the best one, because the bound in Equation 9 depends on the mixture's distance as well as the nearest policy's distance; the Ant-Angle experiments confirm this.","Model-based offline RL with uncertainty penalties inherits its performance directly from data quality, so improving data collection and buffer selection is an alternative route to better offline RL that does not require changing the conservative algorithm."],"supporting_citations":[{"why":"Supplies the MOPO uncertainty-penalized model-based offline RL lower bound (Theorem 4.2) that the paper's proof starts from.","marker":"Yu et al. 2020c"},{"why":"Provides the model-based rollout paradigm and the rationale that model-based offline RL generalizes where the model is accurate.","marker":"Janner et al. 2019"},{"why":"Contributes the WURL diversity objective and the amortized primal-form Wasserstein estimation that UDG uses for its pseudo-reward.","marker":"He et al. 2022"},{"why":"Introduces unsupervised skill discovery through diversity rewards, the basis for the task-agnostic data-generation stage.","marker":"Eysenbach et al. 2019"},{"why":"Shows the equivalence between the minimal worst-case regret objective and maximal discriminability in a finite-state setting, which motivates the surrogate objective in Equation 12.","marker":"Eysenbach, Salakhutdinov, and Levine 2021"},{"why":"Defines integral probability metrics, which the paper uses to turn model error into a distance between transition distributions.","marker":"Müller 1997"},{"why":"Supplies the episodic-memory nearest-neighbor model that justifies the non-parametric transition assumption in the proof of Theorem 4.8.","marker":"Pritzel et al. 2017"}],"fun_headline_variants":["Offline RL gap traced to distribution distance","Unsupervised data diversity narrows offline RL gap","Why offline RL fails: it's the data distribution","Match optimal distribution, improve offline RL","UDG: unsupervised data generation beats supervised"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire chain collapses if the learned model's error on states visited by the optimal policy is not comparable to the Wasserstein distance from the behavioral data to the optimal policy, because the paper asserts this comparability is natural without offering a quantitative bound.","fun_headline_variants_meta":{"raw":{"variants":["Offline RL gap traced to distribution distance","Unsupervised data diversity narrows offline RL gap","Why offline RL fails: it's the data distribution","Match optimal distribution, improve offline RL","UDG: unsupervised data generation beats supervised"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000285,"raw_usage":{"total_tokens":1712,"prompt_tokens":1009,"completion_tokens":703,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":625,"completion_tokens_details":{"reasoning_tokens":634}},"tokens_in":625,"tokens_out":703,"duration_ms":7340,"temperature":1.0,"reasoning_tokens":634,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:30:30.913338+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a Lipschitz MDP with a multi-modal transition (two very different next states both likely) and collect a behavioral buffer close to the optimal occupancy in $W_1$; if the measured gap $\\eta_M(\\pi^*) - \\eta_M(\\hat\\pi)$ repeatedly exceeds $2C\\,W_1(\\rho^{\\pi_\\beta}_T, \\rho^{\\pi^*}_T)$, then the $D_1 \\approx D_2$ supposition behind Equation 8 fails in that setting.","supporting_citations":[{"cited_title":"The Information Geometry of Unsupervised Reinforcement Learning","cited_arxiv_id":"2110.02719","evidence_quote":"Shows the equivalence between the minimal worst-case regret objective and maximal discriminability in a finite-state setting, which motivates the surrogate objective in Equation 12."},{"cited_title":"P.; Vinyals, O.; Hassabis, D.; Wierstra, D.; and Blundell, C","cited_arxiv_id":null,"evidence_quote":"Supplies the episodic-memory nearest-neighbor model that justifies the non-parametric transition assumption in the proof of Theorem 4.8."}],"review_version":1}