REVIEW 6 major objections 5 minor 14 references
Reachability Weighted Offline Goal-conditioned Resampling
T0 review · 6 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read The paper claims that resampling offline goal-conditioned experience by a PU-learned reachability score, derived from goal-conditioned $Q$-values, consistently improves standard offline RL algorithms on six simulated robotic-manipulation…
desk verdict A useful empirical resampling heuristic that is effectively a softmax over Q-values; the benchmark gains are real, but the paper oversells the mechanism and needs a few revisions before it is fully convincing. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the reachability classifier $C_\theta$, a linear logistic regressor applied to the goal-conditioned value $q = Q_\psi(s,g,a)$, so that $f(s,g,a) = C_\theta(Q_\psi(s,g,a))$. It is trained with the non-negative PU loss, using a hindsight-relabeled positive set $D_P$ and a uniform-random unlabeled set $D_U$. Its output is converted into a sampling weight by an exponential transformation and normalization over sampled goals, $w(s,g,a) = \exp(C_\theta(Q_\psi(s,g,a))) \,/\, \frac{1}{N}\sum_{k} \exp(C_\theta(Q_\psi(s,a,g'_k)))$, so that reachable goals—near or far—receive comparable weight while unreachable ones are suppressed. The classifier is trained concurrently with the offline RL algorithm, avoiding a separate pretraining phase.
What would settle it
Take any offline goal-conditioned dataset with known ground-truth reachability for held-out state-goal pairs, train the RWS pipeline, and measure whether the assigned sampling probability is higher for reachable than for unreachable pairs; if the ordering is no better than chance, the mechanism is not doing the work. A cheaper check: run the same comparison after deliberately randomizing the $Q$-function's final layer to see whether the reported HandBlock-Z gain collapses.
Extended reading notes
Core claim
The central claim is that the main obstacle in offline goal-conditioned RL is not the absence of goals but the way goals are sampled: uniform relabeling creates many unreachable pairs and, through the policy-regularization term in offline RL objectives, pulls the policy toward suboptimal behavior. RWS replaces uniform goal sampling with a weighted sample whose priority is a reachability score. The score comes from a linear logistic classifier trained with non-negative PU learning on the scalar goal-conditioned $Q$-value; positive examples are constructed by hindsight relabeling within trajectories, and unlabeled examples by random goal draws. The paper's experiments on six robotic manipulation tasks show that this module improves several offline RL baselines, with the largest gain nearly 50% relative on HandBlock-Z, and that the learned weights are more interpretable than density- or advantage-based resampling in a 2D maze visualization. The authors also report that the benefit decreases when expert demonstrations make up only 10% of the dataset.
Load-bearing premise
The entire method rests on the offline-trained $Q$-function already ranking reachable goals above unreachable ones; if the $Q$-values are miscalibrated in poorly covered regions, the classifier and the sampling weights inherit that error, and the gains should shrink or reverse.
Editorial extensions
If this is right
- RWS is a drop-in resampling module: it can be combined with action-regularized offline RL algorithms without changing the base objective, and the combined method outperforms the base in all six tasks in the main table.
- Because the weight depends on reachability rather than raw $Q$-magnitude, RWS samples distant-but-reachable goals as often as nearby ones, preserving the goal-stitching behavior that random relabeling is meant to enable.
- The benefit scales with dataset quality: RWS consistently helps when expert trajectories make up 30–50% of the data, while at a 10% expert ratio the gains diminish, matching the paper's reported results.
- The learned reachability weights are more interpretable than density or advantage weights in the paper's 2D maze visualization: they concentrate mass on reachable regions and avoid unreachable ones.
- The method requires only a pre-existing or concurrently learned goal-conditioned $Q$-function, so it can be layered on top of standard offline GCRL pipelines without additional environment interaction.
Reading between the lines
- The paper leaves untested whether RWS transfers to non-vector goal spaces such as images or language; because the classifier consumes only the scalar $Q$-value, the same weighting scheme could apply to any goal representation that supports a learned $Q$-function. This is an extension beyond the paper's experiments.
- A direct but implicit consequence of using a linear logistic classifier on $Q$-values is that RWS is invariant to monotone rescaling of $Q$, so it should behave similarly across algorithms whose critics differ only in scale; the paper does not state this.
- The paper's own conclusion that RWS depends on reliable $Q$-value estimation suggests a natural test: substitute an ensemble of $Q$-functions or a conservative lower-confidence estimate as the classifier input to see whether the gains persist in low-coverage datasets. The authors do not run this variant.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Reachability Weighted Sampling (RWS), a plug-and-play resampling scheme for offline goal-conditioned reinforcement learning. RWS trains a positive-unlabeled (PU) logistic classifier whose input feature is the goal-conditioned Q-value, with positive examples generated by hindsight relabeling and unlabeled examples generated by uniform random goal sampling. The classifier output is transformed by an exponential function and normalized to obtain sampling weights over goals. The authors integrate RWS with TD3BC, ReBRAC, and MCQ and report improved scores on six simulated robotic manipulation tasks, most notably a near-50% improvement on HandBlock-Z, as well as experiments varying the expert-data ratio and comparisons with DW, GoFAR, and AW.
Significance. If the empirical gains are robust, RWS would be an attractively simple addition to offline GCRL pipelines. The evaluation is reasonably broad: six tasks, three offline RL backbones, three dataset-quality levels, 10 seeds, and comparisons with alternative weighting schemes, plus a qualitative 2D maze visualization. The paper also ships a clear algorithm description and openly states the method's dependence on reliable Q-value estimation. However, the central mechanism is questionable: because the PU classifier is a monotone transformation of Q, the sampling weights are a monotone function of Q, so the method cannot correct a wrong reachability ordering in Q, and the claim that RWS samples all reachable goals equally regardless of distance is not implemented by Eq. 6. The paper also lacks an ablation against direct Q-weighted (softmax) sampling, which would be needed to show that the PU step contributes beyond a temperature-scaled softmax. These issues, together with statistical overclaiming in Tables 1 and 2, currently leave the contribution in need of substantial revision.
major comments (6)
- [4.2, Eq. (6)] The stated motivation, 'to sample all potentially reachable state-goal-action pairs equally, regardless of the distance,' is not realized by the proposed weight. Since C_theta is a logistic function of Q and exp is strictly increasing, exp(C_theta(Q)) is strictly increasing in Q; for any finite-slope logistic, closer (higher-Q) goals receive strictly larger weights. Exact equality across reachable goals would require an infinite-slope step function, which PU training on noisy Q-values will not produce. This is not merely a wording issue: it changes the interpretation of the experimental gains, which may simply reflect softmax weighting over Q. The authors should either correct the claim or add a mechanism (e.g., thresholding or binning) that actually equalizes weights.
- [4.1, Eq. (5)] The positive set D_P contains only hindsight pairs from the same trajectory (i < h), while the unlabeled set D_U includes cross-trajectory goal pairs. PU learning with the non-negative risk estimator relies on the assumption that labeled positives are selected completely at random from the positive class. Here, the cross-trajectory reachable pairs (the goal-stitching cases the method most wants to identify) never appear as labeled positives, so the labeled-positive distribution is a biased subset of the positive class. The paper gives no argument that the PU assumption still holds for this construction, and this gap directly affects whether the classifier can learn the intended reachability concept.
- [4.3, Algorithm 1] The classifier is trained concurrently with the Q-function that provides its input feature, while the classifier's output in turn reweights the updates of that same Q-function. This is a non-stationary feedback loop: Q_psi changes as training proceeds, and the reachability labels implicit in C_theta are derived from a moving target. The paper mentions a two-stage alternative (freeze Q, then train classifier) but does not ablate it against the concurrent procedure. Without such an ablation or a stability analysis, it is unclear whether the reported improvements are due to the proposed mechanism or to side effects of this feedback.
- [5, first paragraph] The text promises a toy bandit experiment: 'We begin with a toy bandit experiment to illustrate the impact of prioritized sampling on behavior policies.' No toy bandit experiment appears in Section 5 or anywhere else in the manuscript. This is a missing promised analysis; the authors should either add it or remove the claim.
- [Tables 1 and 2] The claim that RWS 'significantly improves performance' is not supported by the reported statistics in several cells. For example, in Table 1, GC-TD3BC+RWS versus GC-TD3BC on FetchPick (31.63 +/- 3.24 versus 28.18 +/- 3.40) and on HandBlock-Parallel (24.39 +/- 6.23 versus 15.24 +/- 5.27) have differences well within the combined standard errors. The caption only says bold entries are best according to a t-test 'with respect to the highest mean,' not that each RWS-versus-baseline pairwise comparison is significant. The authors should report paired tests or confidence intervals and qualify the significance claims accordingly.
- [4.2, Eq. (6)] Because C_theta is a monotone transform of Q, the weight exp(C_theta(Q)) is, up to normalization, a softmax over Q with a learned slope and intercept. The paper does not compare RWS against direct Q-weighted sampling such as exp(Q / tau) or exp(Q) without the PU classifier. Such an ablation is necessary to establish that the PU step contributes beyond a temperature-scaled softmax; without it, the novelty of the reachability classifier is not demonstrated.
minor comments (5)
- [4.2, Eq. (6)] The first displayed fraction in Eq. (6) has C(Q(s, hat g, a)) in the numerator but exp(C_theta(Q(...))) in the denominator of the approximation; the notation should be made consistent, and the integral over the discrete dataset should be written as a sum.
- [5, Figure 7] The 2D maze visualization is qualitative only; a quantitative measure of weight quality (e.g., correlation with true reachability or downstream policy return) would strengthen the comparison.
- [6] The heading 'Conculusion' is a typo for 'Conclusion.' There are also several minor typos elsewhere, including 'classfier' in Section 4.3 and 'experimentalsetup' in Section 5.
- [3.3] The notation D_P is used both for the positive dataset in PU learning and for the hindsight relabeled positive set in Section 4.1; distinguish these to avoid confusion.
- [5.1] The baselines TD3BC, ReBRAC, MCQ, CRR, and IQL are not cited with their original references; the paper should provide proper citations and implementation details for these algorithms.
Circularity Check
Reachability classifier is a monotone transform of Q, so RWS reduces by construction to Q-value-weighted sampling; empirical comparisons remain independent.
-
self definitional
[Section 4.1, Eq. 5 and Section 4.2, Eq. 6]
"classifier f is defined as the composition of the goal-conditioned value function Q and a binary classifier, which is given by: f(s,g,a)=C(Q(s,g,a)). Given a pretrained goal-conditioned value function Qψ(s,g,a), we directly apply logistic regression on q=Qψ(s,g,a) to learn the reachability classifier. ... With a trained classifier Cθ (a simple linear logistic classifier), the classification score becomes proportional to the goal-conditioned Q-value, satisfying the stated requirement."
Because Cθ is a scalar logistic, its output is a monotone function of the scalar q=Qψ; the PU loss in Eq. 5 can only fit a slope and intercept and cannot reorder Q-values. Eq. 6 then makes the sampling weight w∝exp(Cθ(Q)), i.e., a softmax over Q. Since the paper itself defines Q as the expected number of steps to the goal (reachability), the 'reachability classifier' adds no information beyond Q: C(Q) is reachability by construction. Any error or miscalibration in Q is passed unchanged into the sampling priority, which the conclusion concedes ('RWS depends on reliable Q-value estimation'). The method's stated aim of sampling all reachable goals equally 'regardless of the distance' is also not implemented by a monotone exp transform, which keeps weighting closer (higher-Q) goals more.
full rationale
RWS is an empirical sampling heuristic, not a formal derivation, so most circularity patterns do not apply. There is a minor self-citation (Hong et al. 2023 includes author Pajarinen) but it is used only for background and the DW baseline and is not load-bearing. The experimental comparison against external baselines (TD3BC, MCQ, ReBRAC, GoFAR, etc.) on standard benchmarks is self-contained and independently testable. The one construction-level issue is that the reachability classifier is defined as C(Q) with C a linear logistic on the scalar Q; being monotone, it cannot change the Q-ordering, and the Eq. 6 sampling weight is a softmax over Q. Thus 'reachability-weighted sampling' is, by construction, Q-value-weighted sampling, and the PU-learning step is a reparameterization rather than an independent reachability signal. This does not invalidate the empirical results, but it means the central mechanism reduces to weighting by the same Q-function the algorithm is learning, so the paper's framing overstates the role of PU-learned reachability. Score 4 reflects this partial construction-level circularity/redundancy, not an allegation of fabricated results.
Assumptions & free parameters
free parameters (3)
- eta_p (positive class prior) =
0.5 (fixed by hand, no sensitivity analysis)
- Exponential temperature (implicitly 1.0 in Eq. 6) =
1.0 (implicit)
- Number of unlabeled goals sampled per batch (N in Algorithm 1) =
Not reported
assumptions (3)
- domain assumption The goal-conditioned Q-value Q(s,g,a) is a sufficient statistic for reachability, such that a linear classifier on Q can separate reachable from unreachable state-goal-action pairs.
- domain assumption PU learning assumptions hold: the unlabeled distribution is a mixture of positive and negative examples with known prior eta_p, and the positive examples are drawn from the true positive distribution.
- domain assumption The offline datasets used (Fetch and dexterous hand) are composed of a mixture of expert and random trajectories, as described in Section 5, and the standard evaluation protocol for offline GCRL applies.
Cite this review
Pith. "Pith review of Reachability Weighted Offline Goal-conditioned Resampling." pith.science (2026). https://pith.science/paper/6YVRTKVS
@misc{pith2026250602577,
author = {Pith},
title = {Pith review of: Reachability Weighted Offline Goal-conditioned Resampling},
year = {2026},
howpublished = {\url{https://pith.science/paper/6YVRTKVS}},
note = {Machine review of arXiv:2506.02577}
}
read the original abstract
Offline goal-conditioned reinforcement learning (RL) relies on fixed datasets where many potential goals share the same state and action spaces. However, these potential goals are not explicitly represented in the collected trajectories. To learn a generalizable goal-conditioned policy, it is common to sample goals and state-action pairs uniformly using dynamic programming methods such as Q-learning. Uniform sampling, however, requires an intractably large dataset to cover all possible combinations and creates many unreachable state-goal-action pairs that degrade policy performance. Our key insight is that sampling should favor transitions that enable goal achievement. To this end, we propose Reachability Weighted Sampling (RWS). RWS uses a reachability classifier trained via positive-unlabeled (PU) learning on goal-conditioned state-action values. The classifier maps these values to a reachability score, which is then used as a sampling priority. RWS is a plug-and-play module that integrates seamlessly with standard offline RL algorithms. Experiments on six complex simulated robotic manipulation tasks, including those with a robot arm and a dexterous hand, show that RWS significantly improves performance. In one notable case, performance on the HandBlock-Z task improved by nearly 50 percent relative to the baseline. These results indicate the effectiveness of reachability-weighted sampling.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[8]
Ofir Nachum, Yinlam Chow, Bo Dai, and Lihong Li. Dualdice: Behavior-agnostic estimation of discounted stationary distribution corrections.Advances in neural information processing systems, 32, 2019a. Ofir Nachum, Bo Dai, Ilya Kostrikov, Yinlam Chow, Lihong Li, and Dale Schuurmans. Algaedice: Policy gradient from arbitrary experience.arXiv preprint arXiv:1...
arXiv 1912
-
[9]
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning.arXiv preprint arXiv:1910.00177,
arXiv 1910
-
[12]
Mianchu Wang, Rui Yang, Xi Chen, Hao Sun, Meng Fang, and Giovanni Montana. Goplan: Goal-conditioned offline reinforcement learning by planning with learned models.arXiv preprint arXiv:2310.20025, 2023a. Tongzhou Wang, Antonio Torralba, Phillip Isola, and Amy Zhang. Optimal goal-reaching reinforcement learning via quasimetric learning. InInternational Conf...
-
[14]
Contrastive difference predictive coding
Chongyi Zheng, Ruslan Salakhutdinov, and Benjamin Eysenbach. Contrastive difference predictive coding. arXiv preprint arXiv:2310.20141,
-
[2008]
C-learning: Learning to achieve goals via recursive classification.arXiv preprint arXiv:2011.08909,
Benjamin Eysenbach, Ruslan Salakhutdinov, and Sergey Levine. C-learning: Learning to achieve goals via recursive classification.arXiv preprint arXiv:2011.08909,
arXiv 2011
-
[2014]
URLhttps://proceedings.neurips.cc/paper_files/paper/2014/file/ f032bc3f1eb547f716df87edb523b8f0-Paper.pdf. Charles Elkan and Keith Noto. Learning classifiers from only positive and unlabeled data. InProceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 213–220,
work page 2014
-
[2017]
11 Yevgen Chebotar, Karol Hausman, Yao Lu, Ted Xiao, Dmitry Kalashnikov, Jake Varley, Alex Irpan, Ben- jamin Eysenbach, Ryan Julian, Chelsea Finn, et al. Actionable models: Unsupervised offline reinforcement learning of robotic skills.arXiv preprint arXiv:2104.07749,
-
[2018]
Paria Rashidinejad, Hanlin Zhu, Kunhe Yang, Stuart Russell, and Jiantao Jiao. Optimal conservative offline rl with general function approximation via augmented lagrangian.arXiv preprint arXiv:2211.00716,
Show all 14 references
-
[2019]
Multi-goal reinforcement learning: Chal- lenging robotics environments and request for research.arXiv preprint arXiv:1802.09464,
Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Powell, Jonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, et al. Multi-goal reinforcement learning: Chal- lenging robotics environments and request for research.arXiv preprint arXiv:1802.09464,
-
[2020]
Rethinking goal-conditioned supervised learning and its connection to offline rl.arXiv preprint arXiv:2202.04478, 2022a
Rui Yang, Yiming Lu, Wenzhe Li, Hao Sun, Meng Fang, Yali Du, Xiu Li, Lei Han, and Chongjie Zhang. Rethinking goal-conditioned supervised learning and its connection to offline rl.arXiv preprint arXiv:2202.04478, 2022a. Rui Yang, Yiming Lu, Wenzhe Li, Hao Sun, Meng Fang, Yali D...
-
[2021]
Q-wsl: Optimizing goal-conditioned rl with weighted supervised learning via dynamic programming.arXiv preprint arXiv:2410.06648,
Xing Lei, Xuetao Zhang, Zifeng Zhuang, and Donglin Wang. Q-wsl: Optimizing goal-conditioned rl with weighted supervised learning via dynamic programming.arXiv preprint arXiv:2410.06648,
-
[2022]
Goal-conditioned reinforcement learning: Problems and solutions.arXiv preprint arXiv:2201.08299,
Minghuan Liu, Menghui Zhu, and Weinan Zhang. Goal-conditioned reinforcement learning: Problems and solutions.arXiv preprint arXiv:2201.08299,
-
[2023]
Goal-conditioned data augmentation for offline reinforcement learning.arXiv preprint arXiv:2412.20519,
Xingshuai Huang, Di Wu Member, and Benoit Boulet. Goal-conditioned data augmentation for offline reinforcement learning.arXiv preprint arXiv:2412.20519,
-
[2024]
Offline reinforcement learning: Tutorial, review, and perspectives on open problems.arXiv preprint arXiv:2005.01643,
12 Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems.arXiv preprint arXiv:2005.01643,
2005 arXiv
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.