Pith. sign in

REVIEW 3 major objections 6 minor 32 references

Good compatibility rankings do not make contrastive critics safe to maximize for action selection.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-31 01:22 UTC pith:7QRY2QYB

load-bearing objection Solid negative result: bilinear contrastive critics rank compatibility well but are unsafe to maximize under best-of-K; the toy, PointMaze, and objective controls carry the claim. the 3 major comments →

arxiv 2607.27422 v1 pith:7QRY2QYB submitted 2026-07-29 cs.LG

Good Rankers, Bad Objectives: Bilinear Contrastive Critics under Expressive Policy Search

Ayushman Singh , Siddharth Aphale This is my paper
classification cs.LG
keywords contrastive criticsgoal-conditioned RLbest-of-K selectionvalue calibrationbilinear scoresoffline RLexpressive policies
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Contrastive critics learn to score how well a state–action pair matches a goal. They work well as retrieval rankers, but practitioners now also use them as objectives: sample many candidate actions and pick the highest-scoring one. This paper shows that role is unsafe. Even when the critic ranks in-support actions correctly for compatibility, maximizing its score can prefer low-value actions—through embedding-norm inflation, bounded scores that still put bad actions on top, or simple misranking inside the data support. Bounding the score with cosine does not fix it. A Bellman-trained value scalar restores proper ordering. Realized harm depends on the task: it shows up when one action matters, and can vanish when a controller can self-correct later. The practical message is sharp: keep contrastive critics for ranking compatibility; use a value-calibrated scalar when you select actions.

Core claim

Accurate in-support compatibility ranking does not make a bilinear contrastive critic safe to maximize. Under best-of-K candidate search, raw, cosine, and hybrid contrastive scores incur large selection regret via norm drift, high-score false positives below the bound, or in-support misranking, while a Bellman-trained TD-Q scalar restores value ordering—including in parameter-matched controls.

What carries the argument

Candidate regret under best-of-K maximization (Proposition 2 and Corollary 1): even a perfect in-support ranker fails once any bad off-support or misranked set outscores the good set, whether scores are unbounded or cosine-capped; the polar decomposition of the bilinear score separates alignment from norm drift.

Load-bearing premise

That in-support return-to-go is a faithful enough progress signal to diagnose value decalibration on the navigation tasks, even where independent value judges disagree or single-step choices barely move return.

What would settle it

On fixed queries with shared candidate pools, show a cosine or hybrid contrastive readout that matches Bellman TD-Q on within-pool value ordering and near-oracle regret while still being trained only for binary compatibility—especially on PointMaze and the closed-form toy where one action clearly matters.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Do not use unnormalized or cosine contrastive scores as the objective for best-of-K, planning, or critic-guided generation.
  • Bounding or normalizing embeddings is not calibration; selection still needs a scalar ordered by value.
  • A two-head setup can keep contrastive retrieval while routing selection through a Bellman head.
  • Realized cost of misordering is task-dependent: expect harm when one step dominates, and possible nulls when later control can self-correct.
  • Contrastive critics remain appropriate as compatibility rankers on navigation and manipulation retrieval.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Any learned reranker or energy guide sitting on top of an expressive sampler inherits the same tail-exploitation risk unless its training signal enforces graded value order among positives.
  • Pairwise ranking or temporal-distance targets may close the gap without full Bellman recursion; the paper leaves that door open but untested.
  • Support-constrained actors (behavior cloning penalties) can mask the failure in deployment without repairing the scalar.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper argues that bilinear contrastive critics, though strong compatibility rankers, are unsafe objectives under candidate maximization (best-of-K). Theory shows off-support nonidentifiability and that raw bilinear scores can inflate via norms or finite bumps (Prop. 1); candidate max then yields regret even for perfect in-support rankers, and boundedness alone does not calibrate (Prop. 2, Cor. 1). Experiments separate mechanisms: a closed-form 2D toy attributes most raw regret to norm drift while cosine/hybrid still select off-support; on four OGBench navigation tasks, contrastive scores are weakly calibrated or inverted in the top decile and fail fixed-query value ordering, whereas Bellman TD-Q succeeds, including under parameter-matched and training/readout controls. Realized single-step costs appear on PointMaze and the toy; AntMaze/HumanoidMaze give powered nulls under self-correction. The conclusion is that contrastive critics remain useful retrievers, but action selection needs a value-calibrated scalar.

Significance. The ranking-versus-selection distinction is timely as contrastive critics are increasingly used for best-of-K, planning, and critic-guided generation with expressive policies. The contribution is not a new learner but a clear negative result with mechanism decomposition: existence theory, a support-controlled toy with exact Q*, dual-judge fixed-query audits, training/readout and 2×2 objective-versus-function-class controls, and honest task-dependent rollout costs. Strengths include falsifiable predictions (off-support rates, curse curves), explicit scoping of γd and reference-ensemble limits, and complementary near-perfect retrieval on navigation and manipulation. If the result holds, it should change how the community deploys contrastive scores under search and motivate value-calibrated or structured alternatives for the selection scalar.

major comments (3)
  1. [§7.2, Table 4, Abstract, §8] §7.2 and Table 4 carefully restrict AntMaze/HumanoidMaze to powered rollout nulls (reference Spearman 0.55/0.42 fail inclusion), yet the abstract and §8 still state that “action selection requires a value-calibrated scalar.” That prescription is load-bearing for the paper’s deployment message. Please tighten the claim to: (i) contrastive scores systematically misorder the searched tail (robust across toy, PointMaze, and score-semantics), and (ii) whether that misordering costs return is task-mediated (single-step effect vs self-correction). As written, readers may over-read the null tasks as evidence that calibration is always necessary for return, which the rollouts do not show.
  2. [§7.3, Table 5] §7.3 Table 5’s objective-versus-function-class 2×2 is decisive on PointMaze (Bellman raises τb by ~0.30–0.44; architecture match only ~0.06 for contrastive), but it is reported only there. The central attribution—“failure traces to the training objective”—is used globally. At minimum, state that the 2×2 is PointMaze-only and that higher-dim confirmation is left open; ideally add one seed-matched bilinear-Bellman or contrastive-MLP cell on AntMaze under the same fixed-query protocol so the objective story is not carried by a single low-dimensional task.
  3. [§5.1, §6.1, S4.1, S4.4] §5.1 and §6.1 correctly flag that TD-Q’s γd correlation partly reuses its Bellman target and treat it as a ceiling. The independent GCIQL seed (S4.4) and rollouts reduce circularity for selection. Residual risk remains for cross-query pooled-triple regret (Fig. S3, Tables S4–S6), where γd/DP disagreements and stitching gaps are large (e.g., PointMaze ρ=0.73, AntMaze ρ=0.57). Please keep those panels explicitly labeled as cross-query comparability diagnostics—not within-query selection harm—in the main text whenever they are cited in support of the “bad objective” claim, so the load-bearing evidence stays on the toy, PointMaze fixed-query, and controls.
minor comments (6)
  1. [Figure 2] Figure 2 bands are described as lying within the lines (per-seed std ≤0.02); consider a small vertical offset or inset Δγd bar so the flat/inverted cosine–hybrid tails remain visible in grayscale print.
  2. [Table 1, Eq. (S2)] Table 1 and Eq. (S2): hybrid’s auxiliary weight α is never given a default or sweep in the main text; a one-line value (or “α as in Zheng et al.”) would aid reproduction.
  3. [§4.1, Proposition 1] Proposition 1’s bump construction is clear in S1; a one-sentence pointer in §4.1 that the extra latent coordinate is continuous and support-vanishing would help readers who skip the supplement.
  4. [Footnote 1, S2] Footnote 1 / seed policy is exemplary; consider moving the one-sentence retrain note for HumanoidMaze TD-Q seed-43 into S2 only, to reduce main-text clutter.
  5. [§2, §5.1, §8] Related work cites RankQ (Choi & Xu, 2026) and quasimetric/temporal-distance critics as untested alternatives; a short clause that the claims cover only the four scalars in Table 1 (already in §5.1) could be repeated in the conclusion for skimmers.
  6. [Abstract] Minor typos: “cosinetrainingobjective” missing spaces in the abstract; “the cosinetrainingobjective” in the abstract body; ensure consistent “best-of-K” hyphenation.

Circularity Check

2 steps flagged

Minor acknowledged TD-Q/γd target reuse and same-family GCIQL reference notes; core ranking-vs-selection claim is independently secured by existence proofs, closed-form Q* toy, and multi-control selection audits.

specific steps
  1. self definitional [§6.1 Reading the TD-Q reference; Table 2]
    "TD-Q is the calibrated reference. Its Bellman fixed point on goal-directed data is γd, so evaluation against γd partly reuses its training target. We treat its strong ∆γd and τ as a sanity-check ceiling, not as independent evidence. The contrastive results do not share this overlap because binary compatibility training never encodes γd."

    TD-Q’s reported strong Kendall τ / ∆γd versus γd is partly guaranteed by training to the same progress proxy (Bellman fixed point on goal-directed data). That makes the positive-control ceiling partly self-definitional. The paper correctly scopes it as a sanity check; contrastive flat/inverted tails are not built into the evaluation target, so the failure claim is not forced by this overlap.

  2. self definitional [§S4.4 Table S5 dagger note; also Table S6]
    "†TD-Q belongs to the same GCIQL family as the reference, making its low regret partly circular. ... The TD-Q row is partly circular because the deployed td_q critic and the reference both use GCIQL. Its low regret is therefore not independent evidence; the contrastive rows provide the relevant comparison."

    Cross-query normalized regret for TD-Q against an independent-seed GCIQL DP oracle reuses the same critic family as the judge, so TD-Q’s near-oracle cells are partly circular by construction. The paper flags this with daggers and treats contrastive rows as the relevant comparison. This does not force the contrastive failure result, which remains high under both γd and the DP reference.

full rationale

The paper’s load-bearing chain does not reduce to its inputs by construction. Propositions 1–2 and Corollary 1 are existence/regret arguments from off-support nonidentifiability and candidate maximization; they do not fit parameters to the claimed failure. The controlled 2D toy uses closed-form Q*=−∥s+a−g∥ with explicit support disks and pool oracles (Table 3, Figs. 4/S1–S2). Contrastive failures on OGBench are measured by weak/inverted Kendall τ and top-decile γd calibration that binary NCE never targets, plus fixed-query τb/NRegret and paired rollouts (Tables 2, 4). Training/readout and 2×2 objective-vs-function-class controls attribute the gap to the contrastive objective rather than readout or capacity (Tables 5, S8). The only circular-adjacent elements are (i) TD-Q’s strong γd ordering partly reusing its Bellman fixed point—explicitly flagged as a sanity-check ceiling, not independent evidence—and (ii) some cross-query DP-oracle tables that compare against a same-family GCIQL reference (dagger notes in S4.4). Neither forces the central claim that good compatibility rankers are unsafe selection objectives. No self-citation uniqueness chain, fitted-input-as-prediction, or ansatz smuggling is present. Score 2 reflects those minor, paper-acknowledged overlaps without elevating them to load-bearing circularity.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 2 invented entities

The central claim rests on standard MDP/offline RL setup, the bilinear critic class, a contrastive population loss that only constrains scores on data support, candidate regret against a pool oracle, and evaluation proxies (γd, GCIQL ensembles, simulator rollouts). No new physical entities. Free choices are experimental (K, support radius, seeds, inclusion thresholds) rather than fitted universal constants that force the conclusion.

free parameters (5)
  • Candidate budget K and proposal distributions = K=64 primary OGBench; K=256 toy
    K (e.g. 64/256) and Gaussian/diffusion proposals define search pressure; results are shown across K but magnitudes depend on these design choices.
  • Support radius R in controlled toy = R=0.40 main
    Defines in- vs off-support partition; robustness across {0.25,0.40,0.55} is shown but R is experimenter-chosen.
  • Hybrid auxiliary weight α
    Scales TD residual added during hybrid representation learning (Q_hybrid = Q_TD + α f_cos); diagnostic only, but affects hybrid embeddings.
  • Reference-ensemble inclusion thresholds = Spearman≥0.7; resolvability≥50%
    Prespecified resolvability ≥50% and mean pairwise Spearman ≥0.7 decide which tasks enter reference-judged selection analysis.
  • Rollout minimum effect emin = ≈0.012
    Power criterion uses emin≈0.012 from realized PointMaze margin to declare well-powered nulls.
axioms (5)
  • domain assumption Contrastive population loss only evaluates triples on data support Ω_P, leaving f unconstrained off support (off-support nonidentifiability).
    Load-bearing for Proposition 1 and the bad-set construction in Proposition 2; standard for empirical risk on offline data (§4).
  • domain assumption Goal-conditioned MDP with offline dataset; candidate regret defined vs pool oracle on same K draws.
    Problem setup §3; regret is relative to the pool, not global optimality.
  • domain assumption In-support return-to-go γ^d is a valid navigation progress proxy for calibration/ordering analyses.
    Stated in §5.2/S2; invalid on manipulation play data; can undershoot under stitching.
  • standard math Bounded true returns Q(a)∈[Q_min,Q_max] and compact state-action spaces in finite-inflation construction.
    Used in §4.1 and Proposition 1 proof (bump function on separated closed sets).
  • domain assumption Binary contrastive positives need not be ordered by temporal distance among themselves.
    Remark in S1 explaining why compatibility ≠ graded value even in support.
invented entities (2)
  • Hybrid contrastive+auxiliary-TD critic used only as diagnostic readout no independent evidence
    purpose: Test whether TD representation shaping rescues bounded cosine selection without deploying the calibrated scalar.
    Not proposed as a method; still a paper-specific construct in the four-family sweep.
  • Finite off-support bilinear bump construction (extra latent coordinate) no independent evidence
    purpose: Show bounded embeddings can assign arbitrary finite margin off support without norm divergence.
    Existence proof device in Proposition 1 / S1; not claimed to be what SGD finds.

pith-pipeline@v1.2.0-daily-grok45 · 30610 in / 3675 out tokens · 72011 ms · 2026-07-31T01:22:18.328130+00:00 · methodology

0 comments
read the original abstract

Good action rankings do not make a contrastive critic safe to maximize. These critics increasingly act as value-like objectives for best-of-$K$ selection, planning, and critic-guided generation. Unbounded bilinear scores can let large embedding norms inflate off-support values, but cosine bounding does not remove the failure. A controlled support decomposition attributes most raw bilinear regret to norm drift. Cosine and hybrid critics nevertheless select off-support actions from most pools and incur comparable regret. Contrastive scores are weakly calibrated or inverted in the top score decile across four OGBench navigation tasks, and they fail to order fixed-query actions by value. Bellman-trained TD-Q succeeds, including in a parameter-matched function-class control. Realized costs depend on the task: simulator rollouts reveal single-step selection costs on PointMaze and the exact-$Q^*$ toy but well-powered nulls on AntMaze and HumanoidMaze, where the controller can self-correct. A training/readout decomposition traces the lost ordering to the cosine training objective; raw-trained embeddings retain weak ordering after inference-time normalization. Candidate maximization can therefore exploit false positives caused by norm drift, score saturation, or in-support misranking. Contrastive critics remain useful compatibility rankers on navigation and manipulation tasks, but action selection requires a value-calibrated scalar.

Figures

Figures reproduced from arXiv: 2607.27422 by Ayushman Singh, Siddharth Aphale.

Figure 1
Figure 1. Figure 1: Unified failure mechanism. Contrastive training separates compatible pairs but need not grade their value. Candidate maximization finds high-score false positives caused by in-support misranking, bounded scores at the cap, or raw norm drift (§4, Proposition 2). A Bellman-calibrated scalar restores the ordering in our experiments. The score can increase through closer alignment, a larger state–action embedd… view at source ↗
Figure 2
Figure 2. Figure 2: Score progress calibration (OGBench navigation). In-distribution (s, a, g) triples are binned by critic-score decile; each bin reports mean γ d . TD-Q is monotone, raw bilinear has a weak slope, and cosine and hybrid are flat or inverted in the high-score tail. Curves show seed means with ±std bands on all four navigation tasks. Per-seed std is ≤ 0.02, so the bands lie within the lines. pooled-triple tests… view at source ↗
Figure 3
Figure 3. Figure 3: Good rankers, bad objectives on navigation tasks. (a) Retrieval AUC (compatibility) versus Kendall τ against γ d (value ordering). Contrastive critics occupy the high-AUC, low-τ region; TD-Q is the calibrated reference. Markers show mean±std over 3 training seeds on all four tasks (Footnote 1). (b) Positive-pair score versus temporal distance (1 = nearest goal, 0 = random baseline), averaged over the three… view at source ↗
Figure 4
Figure 4. Figure 4: Qualitative score geometry in the controlled 2D environment. Columns show the four deployed score families (Section 3). Rows show the learned score surface, a contour with K=256 candidates and selected action a ⋆ K (⋆; orange support disk), and ∥∇af∥. Raw bilinear, cosine, and hybrid select outside the support disk in this representative run; TD-Q selects inside [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

32 extracted references · 16 linked inside Pith

  1. [1]

    Advances in Neural Information Processing Systems , year=

    Contrastive Learning as Goal-Conditioned Reinforcement Learning , author=. Advances in Neural Information Processing Systems , year=

  2. [2]

    Stabilizing Contrastive

    Zheng, Chongyi and Eysenbach, Benjamin and Walke, Homer and Yin, Patrick and Fang, Kuan and Salakhutdinov, Ruslan and Levine, Sergey , booktitle=. Stabilizing Contrastive. 2024 , url=

    2024

  3. [3]

    2025 , url=

    Park, Seohong and Frans, Kevin and Eysenbach, Benjamin and Levine, Sergey , booktitle=. 2025 , url=

    2025

  4. [4]

    Park, Seohong and Li, Qiyang and Levine, Sergey , journal=. Flow. 2025 , url=

    2025

  5. [5]

    Proceedings of the International Conference on Machine Learning , year=

    Learning Temporal Distances: Contrastive Successor Features Can Provide a Metric Structure for Decision-Making , author=. Proceedings of the International Conference on Machine Learning , year=

  6. [6]

    Conservative

    Kumar, Aviral and Zhou, Aurick and Tucker, George and Levine, Sergey , booktitle=. Conservative. 2020 , url=

    2020

  7. [7]

    2025 , url=

    Dong, Perry and Li, Qiyang and Sadigh, Dorsa and Finn, Chelsea , journal=. 2025 , url=

    2025

  8. [8]

    International Conference on Learning Representations , year=

    Offline Reinforcement Learning with Implicit Q-Learning , author=. International Conference on Learning Representations , year=

  9. [9]

    2023 , url=

    Nakamoto, Mitsuhiko and Zhai, Yuexiang and Singh, Anikait and Mark, Max Sobol and Ma, Yi and Finn, Chelsea and Kumar, Aviral and Levine, Sergey , booktitle=. 2023 , url=

    2023

  10. [10]

    2026 , url=

    Choi, Andrew and Xu, Wei , journal=. 2026 , url=

    2026

  11. [11]

    Proceedings of the International Conference on Machine Learning , year=

    Contrastive Energy Prediction for Exact Energy-Guided Diffusion Sampling in Offline Reinforcement Learning , author=. Proceedings of the International Conference on Machine Learning , year=

  12. [12]

    International Conference on Learning Representations , year=

    Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning , author=. International Conference on Learning Representations , year=

  13. [13]

    Diffusion-

    Mao, Liyuan and Xu, Haoran and Zhan, Xianyuan and Zhang, Weinan and Zhang, Amy , booktitle=. Diffusion-. 2024 , url=

    2024

  14. [14]

    Proceedings of the International Conference on Machine Learning , year=

    Off-Policy Deep Reinforcement Learning without Exploration , author=. Proceedings of the International Conference on Machine Learning , year=

  15. [15]

    Proceedings of Robotics: Science and Systems , year=

    Diffusion Policy: Visuomotor Policy Learning via Action Diffusion , author=. Proceedings of Robotics: Science and Systems , year=

  16. [16]

    Proceedings of the International Conference on Machine Learning , year=

    Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learning , author=. Proceedings of the International Conference on Machine Learning , year=

  17. [17]

    Diffusion policy: Visuomotor policy learning via action diffusion

    Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. In Proceedings of Robotics: Science and Systems, 2023. URL https://arxiv.org/abs/2303.04137

    Pith/arXiv arXiv 2023

  18. [18]

    RankQ : Offline-to-online reinforcement learning via self-supervised action ranking

    Andrew Choi and Wei Xu. RankQ : Offline-to-online reinforcement learning via self-supervised action ranking. arXiv preprint arXiv:2605.11151, 2026. URL https://arxiv.org/abs/2605.11151

    Pith/arXiv arXiv 2026

  19. [19]

    EXPO : Stable reinforcement learning with expressive policies

    Perry Dong, Qiyang Li, Dorsa Sadigh, and Chelsea Finn. EXPO : Stable reinforcement learning with expressive policies. arXiv preprint arXiv:2507.07986, 2025. URL https://arxiv.org/abs/2507.07986

    Pith/arXiv arXiv 2025

  20. [20]

    Contrastive learning as goal-conditioned reinforcement learning

    Benjamin Eysenbach, Tianjun Zhang, Ruslan Salakhutdinov, and Sergey Levine. Contrastive learning as goal-conditioned reinforcement learning. In Advances in Neural Information Processing Systems, 2022. URL https://arxiv.org/abs/2206.07568

    Pith/arXiv arXiv 2022

  21. [21]

    Off-policy deep reinforcement learning without exploration

    Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In Proceedings of the International Conference on Machine Learning, 2019. URL https://arxiv.org/abs/1812.02900

    Pith/arXiv arXiv 2019

  22. [22]

    Offline reinforcement learning with implicit q-learning

    Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. In International Conference on Learning Representations, 2022. URL https://arxiv.org/abs/2110.06169

    Pith/arXiv arXiv 2022

  23. [23]

    Conservative Q -learning for offline reinforcement learning

    Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative Q -learning for offline reinforcement learning. In Advances in Neural Information Processing Systems, 2020. URL https://arxiv.org/abs/2006.04779

    Pith/arXiv arXiv 2020

  24. [24]

    Contrastive energy prediction for exact energy-guided diffusion sampling in offline reinforcement learning

    Cheng Lu, Huayu Chen, Jianfei Chen, Hang Su, Chongxuan Li, and Jun Zhu. Contrastive energy prediction for exact energy-guided diffusion sampling in offline reinforcement learning. In Proceedings of the International Conference on Machine Learning, 2023. URL https://arxiv.org/abs/2304.12824

    Pith/arXiv arXiv 2023

  25. [25]

    Diffusion- DICE : In-sample diffusion guidance for offline reinforcement learning

    Liyuan Mao, Haoran Xu, Xianyuan Zhan, Weinan Zhang, and Amy Zhang. Diffusion- DICE : In-sample diffusion guidance for offline reinforcement learning. In Advances in Neural Information Processing Systems, 2024. URL https://arxiv.org/abs/2407.20109

    Pith/arXiv arXiv 2024

  26. [26]

    Learning temporal distances: Contrastive successor features can provide a metric structure for decision-making

    Vivek Myers, Chongyi Zheng, Anca Dragan, Sergey Levine, and Benjamin Eysenbach. Learning temporal distances: Contrastive successor features can provide a metric structure for decision-making. In Proceedings of the International Conference on Machine Learning, 2024. URL https://arxiv.org/abs/2406.17098

    Pith/arXiv arXiv 2024

  27. [27]

    Cal-QL : Calibrated offline RL pre-training for efficient online fine-tuning

    Mitsuhiko Nakamoto, Yuexiang Zhai, Anikait Singh, Max Sobol Mark, Yi Ma, Chelsea Finn, Aviral Kumar, and Sergey Levine. Cal-QL : Calibrated offline RL pre-training for efficient online fine-tuning. In Advances in Neural Information Processing Systems, 2023. URL https://arxiv.org/abs/2303.05479

    Pith/arXiv arXiv 2023

  28. [28]

    OGBench : Benchmarking offline goal-conditioned RL

    Seohong Park, Kevin Frans, Benjamin Eysenbach, and Sergey Levine. OGBench : Benchmarking offline goal-conditioned RL . In International Conference on Learning Representations, 2025 a . URL https://arxiv.org/abs/2410.20092

    Pith/arXiv arXiv 2025

  29. [29]

    Flow Q -learning

    Seohong Park, Qiyang Li, and Sergey Levine. Flow Q -learning. arXiv preprint arXiv:2502.02538, 2025 b . URL https://arxiv.org/abs/2502.02538

    Pith/arXiv arXiv 2025

  30. [30]

    Optimal goal-reaching reinforcement learning via quasimetric learning

    Tongzhou Wang, Antonio Torralba, Phillip Isola, and Amy Zhang. Optimal goal-reaching reinforcement learning via quasimetric learning. In Proceedings of the International Conference on Machine Learning, 2023 a . URL https://arxiv.org/abs/2304.01203

    Pith/arXiv arXiv 2023

  31. [32]

    Stabilizing contrastive RL : Techniques for robotic goal reaching from offline data

    Chongyi Zheng, Benjamin Eysenbach, Homer Walke, Patrick Yin, Kuan Fang, Ruslan Salakhutdinov, and Sergey Levine. Stabilizing contrastive RL : Techniques for robotic goal reaching from offline data. In International Conference on Learning Representations, 2024. URL https://arxiv.org/abs/2306.03346

    Pith/arXiv arXiv 2024

  32. [33]

    Diffusion policies as an expressive policy class for offline reinforcement learning

    Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. In International Conference on Learning Representations, 2023. URL https://arxiv.org/abs/2208.06193

    Pith/arXiv arXiv 2023