Pith. sign in

REVIEW 3 major objections 6 minor 35 references

CODS: Iterative Bellman-Residual Data Selection for Reusable Offline Reinforcement Learning

T0 review · 3 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Iterative Bellman-residual selection lets a frozen 10% subset of offline RL data retain 96.6% of the full eligible-pool performance, with reuse as the source of the compute savings.

desk verdict Careful empirical paper on iterative Bellman-residual data selection; the contribution is real and the evaluation is unusually disciplined, but the headline aggregate is conditional on two documented cell-level decisions that need a sensitivity analysis. read the letter →

arxiv 2608.07719 v1 pith:7L6TQY4V submitted 2026-08-07 cs.LG

Ibne Farabi Shihab , Sanjeda Akter , Abu Sa-Adat Mohamed Moon-Im Al Ahsan , Md Najmus Swaqeeb , Anuj Sharma This is my paper · ORCID
classification cs.LG
keywords offlinereinforcementlearningdatasubsetselectionBellmanresidualreusabledatasetD4RLbenchmarksiterativeprioritizedreplay
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CODS asks whether a fixed 10% subset of an offline reinforcement-learning transition pool can stand in for the full dataset when the subset is chosen by repeatedly refitting a critic and acquiring the transitions with the largest absolute Bellman residuals. The paper reports that it can, on its benchmark: across 20 valid D4RL task-algorithm cells the frozen subset retains 96.6% of the eligible-pool aggregate score, beats the six subset baselines in all 20 cells, and exceeds the two strongest comparators in 19. The authors frame the result as an empirical claim about a reusable selection procedure rather than a formal coreset guarantee, and they identify reuse across seeds and hyperparameters, rather than faster single-run convergence, as the source of the compute advantage. The practical payoff is that a single index artifact chosen once could replace repeated passes over a million-transition pool in offline RL sweeps.

What carries the argument

The central object is the algorithm-matched absolute Bellman residual $\delta_A(\tau)$, recomputed each round from a selection critic that follows the downstream learner's backup convention. A supporting theoretical proxy $I_{\mathrm{loc}}(\tau)=\frac12\delta_\theta(\tau)^2 g_\tau^\top I_\theta^{-1}g_\tau$ is proved to be within a constant factor of $|\delta_\theta(\tau)|$, but the constants are too loose to certify ranking. The load-bearing mechanism is the acquisition loop: fit the critic, score the unselected pool, add the top-$b_k$ transitions, refit, and repeat for $K$ rounds, then freeze the indices and train every downstream run from scratch. This converts a moving error signal into a fixed artifact whose one-time selection cost can be amortized over a sweep.

What would settle it

Give CODS the same five-round schedule and 100,000-update budget but replace the residual scores with random scores drawn once per round before selection; if the resulting 10% subsets retain close to 96.6% of pool performance, the Bellman-residual signal is not what carries the result. A cheaper check: verify on any D4RL cell that a CODS subset selected with a critic intentionally trained on corrupted rewards loses the reported margin over random selection, which the paper's 10% corruption row suggests it should.

Watch

Extended reading notes

Core claim

The paper's central claim is that iterative rescoring of an algorithm-matched Bellman residual produces a static, reusable data subset that preserves most offline RL performance at 10% of the original data. Starting from a 2% random burn-in, CODS spends five acquisition rounds fitting the downstream algorithm's critic on the current working set, scoring every unselected transition by the absolute TD residual for that algorithm (clipped target-policy noise for TD3+BC, unpenalized target for CQL, expectile value for IQL), and adding the highest-residual batch before freezing the indices. Under equal downstream updates, the subset's aggregate is 96.6% of the eligible 90% pool (mean gap -2.77 points), with a +4.72-point advantage over ReDOR and +6.84 over OPER; all six subset advantages survive Holm-adjusted hierarchical testing. The authors explicitly disclaim a policy-recovery, regret, or coreset guarantee, noting the local-score bound's worst-case factor is about $2\times10^3$ and only 0.02% of sampled pairs meet the sufficient separation condition.

Load-bearing premise

The load-bearing premise is that the absolute Bellman residual of the algorithm-matched critic reliably marks the transitions the downstream learner most needs; if residuals are dominated by noise, outliers, or critic misspecification, selection inherits those failures, as the paper's own corruption experiment shows.

Editorial extensions

If this is right

  • A 10% CODS subset can replace a 90% eligible pool for repeated compatible training: the paper's ledger breaks even at two epoch-scaled configurations and reaches a 5.45x total-time reduction at twenty equal-pass reuses.
  • Iterative rescoring, not just the residual rule, drives the gain: with total selector updates fixed, five acquisition rounds improve four representative cells by 11.23 points over one round and saturate by round ten.
  • Residual signals are algorithm-specific: matched selectors are best for all three downstream learners, and mismatched artifacts lose 1.4-3.9 retention points in the transfer study.
  • Whole-trace selection extends the principle to sequential reasoning: preserving prefix-closed traces retains 95.4% of pooled ALFWorld success and 96.5% of pooled GSM8K exact match, while fragmented step selection drops both.
  • The selector is not a universal accelerator: a single run is slower after selection cost is charged, and equal-update training gives only 1.04x total-time reduction at twenty reuses.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the corruption result (raw CODS falls from 58.0 to 27.8 at 10% reward corruption on CQL AntMaze, and clipping only recovers to 45.4) predicts that adding a residual-clipping or outlier-rejection step to acquisition would recover much of the clean-data margin; the paper stops at calling these diagnostic ablations.
  • Editorial inference: because the acquisition score depends only on the critic's backup convention, CODS should transfer to any offline algorithm that shares the same TD target; a cheap test is to retrain with a fourth algorithm that uses the same expectile or double-Q target and check the retention ratio.
  • Editorial inference: the equal-pass retention of 90.3% versus 93.8% under equal updates implies that part of the headline retention is reheating the same transitions more often; readers who care about wall-clock data efficiency should read the equal-hour condition (95.1% retention at 20 reuses) as the more honest number.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes CODS (Critic-guided Offline Data Selection), a method that iteratively fits an algorithm-matched critic on a growing burn-in subset, acquires the transitions with the largest absolute Bellman residuals, and freezes the resulting subset for downstream offline RL training. The central empirical claim, based on a 20-cell D4RL task–algorithm matrix, is that a 10% CODS subset retains 96.6% of the aggregate performance of the 90% eligible pool, exceeds ReDOR and OPER in 19/20 cells, and exceeds Random, ReD, Static-PER, and k-center in all 20 cells, with six predeclared Holm-adjusted hierarchical comparisons significant at p < 0.001. The paper also reports rescoring ablations, budget sweeps, selection-seed stability, corruption and duplication stress tests, cross-algorithm reuse, matched compute protocols, and a whole-trace language-agent extension on ALFWorld and GSM8K.

Significance. If the headline results hold, CODS is a simple and practical reusable data-selection method for offline RL, with a clearly stated scope (state-based D4RL benchmarks) and a clearly stated non-guarantee (no formal coreset or policy-recovery bound). The evaluation is unusually disciplined: dataset-blocked hierarchical bootstrap, Holm-corrected predeclared inference, fixed-selector-budget ablations, equal-pass and equal-hour controls, baseline fidelity audits, and raw-array/manifest artifacts that make the results machine-checkable. The paper's explicit negative results—the rejection of Remark 1 as a certification tool, the corruption failure, and the floor-level TD3+BC AntMaze analysis—are a genuine strength and increase confidence in the reported positive findings.

major comments (3)
  1. [Section 5.1, Table 1, Appendix J] The headline retention (96.6%) and dominance counts (19/20 and 20/20) are defined over the 20 'valid' cells, which result from two documented decisions: excluding the TD3+BC AntMaze cell as floor-level and replacing the IQL AntMaze value with the manifest-verified (0.9,10) setting. The manuscript never reports how the aggregate numbers would change if the excluded cell were included (with all baselines run) or if the unreconciled IQL value were used. Since the central claim is exactly an aggregate over this cell set, please add a sensitivity analysis: compute the retention ratio and the per-baseline positive-cell counts under (a) inclusion of the floor cell and (b) the unreconciled IQL row, or provide an upper bound on the resulting changes. Without this, the headline is contingent on two cell-level choices.
  2. [Section 5.1, Appendix L] The 'eligible-pool' baseline is trained on only 90% of the original data because 10% of trajectories are held out for an optional conformal diagnostic that is explicitly not used to make any claim. The retention and dominance results are therefore relative to a 90% pool, not the full dataset. Please report on at least two or three representative cells the Pool performance trained on the full 100% of original trajectories, and confirm that the 96.6% retention and baseline ordering are unchanged. If that is not feasible, state clearly in the main text that all claims are relative to the 90% eligible pool and that the effect of the holdout is unmeasured.
  3. [Section 4, Remark 1 and Appendix G] The paper's only theoretical statement, Remark 1, relates residual magnitude to a local information-gain proxy under strong conditioning assumptions, and the empirical audit shows a worst-case factor of about 2e3 with only 0.02% of pairs certified. The paper honestly concludes that this bound cannot explain the empirical gains. This leaves the acquisition mechanism—that high absolute Bellman residuals identify transitions the downstream learner needs—supported only by the aggregate benchmark results and the descriptive AntMaze composition audits. Given that the corruption study in the Limitations section shows raw residual selection can fail dramatically (CQL AntMaze drops from 58.0 to 27.8 at 10% reward corruption, below ReDOR), please either add a direct per-transition validation (e.g., correlation between residual scores and leave-one-out utility on a subset of cells) or explicitly state in the introduction that the method is a heuristic with no known conditions under which the 96.6% retention will hold.
minor comments (6)
  1. [Algorithm 1] The line 'Uniformly sampleb 0 indices' is missing a space and should read 'Uniformly sample b0 indices'.
  2. [Figure 1] The label 'Krounds' should be 'K rounds'.
  3. [Table 2] The column header 'Holmp' should be spaced as 'Holm p'.
  4. [Section 6] The cost equations 'Cpool(S) = 1.8S, C CODS(S) = 3.0 + 0.18S' would be clearer with consistent subscript formatting, e.g., C_pool(S) and C_CODS(S).
  5. [Appendix J] The phrase 'the historical 68.1 value isnotreused' is missing spaces and should read 'is not reused'.
  6. [References] The reference to 'V ovk et al.' should be 'Vovk et al.'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: CODS's empirical claims are self-contained benchmark measurements, with the theory explicitly non-certifying and no fitted parameters or self-citations carrying the result.

full rationale

The central claim is an empirical benchmark result: a frozen 10% subset selected by iterative Bellman residuals is compared, after training TD3+BC/CQL/IQL from scratch, against a 90% eligible-pool baseline and against external subset-selection baselines. No fitted constant is used to produce the headline retention or dominance numbers, and the acquisition critic is explicitly not reused for downstream training. The only theory, Remark 1, is audited and rejected as a ranking certificate: the paper states the worst-case factor is approximately 2e3 and only 0.02% of pairs satisfy the sufficient separation condition, so the empirical gains are not derived from that bound. The residual acquisition signal is a heuristic whose failures are documented in the corruption and duplication studies, and the Limitation section explicitly disclaims any formal coreset, regret, or policy-recovery guarantee. The paper's choices of the 20-cell set, the exclusion of floor-level TD3+BC AntMaze, and the reconciled IQL AntMaze row are transparent cell-selection decisions that affect the aggregate, but they are robustness/selective-reporting concerns rather than circular reductions of the claim to its own inputs. References are to external prior work by other authors; there are no load-bearing author self-citations. The derivation chain is therefore not circular: empirical measurements, independent baselines, and audited implementations carry the conclusions.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

The method's central premise, that absolute Bellman residuals are useful acquisition signals, is assumed rather than derived; the paper's only theoretical result (Remark 1) is explicitly too loose to certify rankings. All other inputs (budget, rounds, burn-in, update schedule) are hand-set hyperparameters that are ablated but not automatically fitted. No new entities are introduced.

free parameters (6)
  • Data budget B = 10% of original dataset N
    Chosen as the primary condition; budget sweep (Appendix O.4) shows diminishing returns beyond 10%, but 10% is not claimed as a universal optimum.
  • Burn-in size b0 = 2% of original dataset N
    Smallest tested burn-in near the plateau per Appendix O.7; counts against the total budget.
  • Acquisition rounds K = 5
    Fixed-critic-budget ablation (Sec 5.3, Appendix O.3) shows K=5 improves over K=1 by 11.23 points and K=10 gives no further gain; not universally optimal.
  • Per-round batch bk = 1.6% of N
    Derived from (B - b0)/K = (0.10 - 0.02)/5 so the rounds sum exactly to the 10% budget (Appendix B).
  • Selector update schedule U0, Uk = U0=25,000, Uk=15,000 (total 100,000)
    Hand-set fixed total across round ablations, documented in Appendix B; held constant to isolate rescoring.
  • Trace score weights and clipping percentile = weights (0.5, 0.5), top 20%, 95th percentile clip
    Hand-designed aggregation for language traces (Eq. 12); used only in the ALFWorld/GSM8K extension, not derived.
assumptions (6)
  • domain assumption Bellman residual magnitude is a useful proxy for acquisition value
    The entire method scores transitions by |delta|; Remark 1 only relates this to a local information-gain proxy under strong conditioning assumptions and explicitly does not certify rankings (Sec 4, Appendix G).
  • standard math Information matrix positive definite on the gradient span and bounded gradient norms
    Assumed in the proof of Remark 1 (Appendix A) to derive the cmin/cmax bounds.
  • domain assumption D4RL normalized scores are a valid measure of offline RL performance
    All headline comparisons use D4RL v2 locomotion and AntMaze scores (Sec 5.1).
  • domain assumption The 90% eligible pool is a valid baseline after holding out 10% of trajectories
    The held-out trajectories are excluded from both selection and training; retention is computed against the 90% pool, not the full original pool (Sec 5.1, Appendix L).
  • domain assumption Algorithm-matched residuals are the right scoring signal
    The paper argues matching the backup operator matters (Sec 3.1) but does not prove it; the transfer study (Appendix O.8) shows matched selectors outperform mismatched ones.
  • domain assumption Qwen2.5-7B value critic gives a meaningful credit signal for language traces
    Used for ALFWorld and GSM8K selection; only one model scale is tested (Appendix O.11).

how reviews work

0 comments
Cite this review

Pith. "Pith review of CODS: Iterative Bellman-Residual Data Selection for Reusable Offline Reinforcement Learning." pith.science (2026). https://pith.science/paper/7L6TQY4V

@misc{pith2026260807719,
  author       = {Pith},
  title        = {Pith review of: CODS: Iterative Bellman-Residual Data Selection for Reusable Offline Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7L6TQY4V}},
  note         = {Machine review of arXiv:2608.07719}
}
read the original abstract

Offline reinforcement learning repeatedly trains policies from a fixed transition pool, making redundant data costly across seeds and hyperparameters, while naive subsampling can remove rare transitions needed for long-horizon credit assignment. We introduce CODS, a critic-guided selector that alternates between fitting an algorithm-matched critic and acquiring high-residual transitions before freezing a reusable subset. Unlike prioritized replay, CODS produces a static artifact; unlike one-shot residual selection, it refreshes scores as the critic changes. At a 10\% budget, CODS retains 96.6\% of eligible-pool performance across 20 valid D4RL task--algorithm cells. It exceeds ReDOR and OPER on 19/20 cells and every other subset baseline on 20/20; all six subset advantages remain significant under predeclared hierarchical inference with Holm correction. Holding total selector updates fixed, five acquisition rounds improve four representative cells by 11.23 points over one round and saturate thereafter. Equal-pass and equal-hour evaluations clarify that reuse, rather than a single-run speedup, creates the compute advantage. Mechanism and corruption interventions expose both useful sparse-reward enrichment and sensitivity to outliers. Finally, a whole-trace extension retains 95.4\% of pooled ALFWorld success and 96.5\% of pooled GSM8K exact match. CODS is therefore a reusable selection procedure, not a formal coreset guarantee.

Figures

Figures reproduced from arXiv: 2608.07719 by the authors.

Figure 1
Figure 1. CODS separates iterative acquisition from [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Budget sweep for Walker2d-medium with TD3+BC. Points average five selection seeds and three downstream seeds; bands show marginal standard devi￾ations [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

35 extracted references · 14 canonical work pages

  1. [1]

    The Annals of Applied Statistics , volume=

    Learn then test: Calibrating predictive algorithms to achieve risk control , author=. The Annals of Applied Statistics , volume=. 2025 , publisher=

    2025

  2. [2]

    arXiv preprint arXiv:2208.02814 , year=

    Conformal risk control , author=. arXiv preprint arXiv:2208.02814 , year=

    arXiv

  3. [3]

    Advances in Neural Information Processing Systems , volume=

    Gone fishing: Neural active learning with fisher embeddings , author=. Advances in Neural Information Processing Systems , volume=

  4. [4]

    arXiv preprint arXiv:2004.07219 , year=

    D4rl: Datasets for deep data-driven reinforcement learning , author=. arXiv preprint arXiv:2004.07219 , year=

    arXiv 2004

  5. [5]

    2023 , eprint=

    Studying Large Language Model Generalization with Influence Functions , author=. 2023 , eprint=

    2023

  6. [6]

    arXiv preprint arXiv:2410.08020 , year=

    Efficiently learning at test-time: Active fine-tuning of llms , author=. arXiv preprint arXiv:2410.08020 , year=

    arXiv

  7. [7]

    International Conference on Machine Learning , year=

    Doubly robust off-policy value evaluation for reinforcement learning , author=. International Conference on Machine Learning , year=

    work page

  8. [8]

    International conference on machine learning , pages=

    Understanding black-box predictions via influence functions , author=. International conference on machine learning , pages=. 2017 , organization=

    2017

Show all 35 references
  1. [9]

    Advances in Neural Information Processing Systems , year=

    Learning active learning from data , author=. Advances in Neural Information Processing Systems , year=

  2. [10]

    arXiv preprint arXiv:2110.06169 , year=

    Offline reinforcement learning with implicit q-learning , author=. arXiv preprint arXiv:2110.06169 , year=

    arXiv

  3. [11]

    Advances in neural information processing systems , volume=

    Conservative q-learning for offline reinforcement learning , author=. Advances in neural information processing systems , volume=

  4. [12]

    Neural computation , volume=

    Information-based objective functions for active data selection , author=. Neural computation , volume=

  5. [13]

    International Conference on Machine Learning , pages=

    Coresets for data-efficient training of machine learning models , author=. International Conference on Machine Learning , pages=. 2020 , organization=

    2020

  6. [14]

    Nakamoto, Mitsuhiko and Zhai, Yuexiang and Singh, Anikait and Mark, Max Sobol and Ma, Yi and Finn, Chelsea and Kumar, Aviral and Levine, Sergey , booktitle=. Cal-

  7. [15]

    Mathematical programming , volume=

    An analysis of approximations for maximizing submodular set functions—I , author=. Mathematical programming , volume=. 1978 , publisher=

    1978

  8. [16]

    arXiv preprint arXiv:1511.05952 , year=

    Prioritized experience replay , author=. arXiv preprint arXiv:1511.05952 , year=

    arXiv

  9. [17]

    International Conference on Machine Learning , year=

    Gaussian process optimization in the bandit setting: No regret and experimental design , author=. International Conference on Machine Learning , year=

  10. [18]

    Advances in Neural Information Processing Systems , volume=

    Conformal off-policy prediction in contextual bandits , author=. Advances in Neural Information Processing Systems , volume=

  11. [19]

    International Conference on Machine Learning , year=

    Data-efficient off-policy policy evaluation for reinforcement learning , author=. International Conference on Machine Learning , year=

  12. [20]

    arXiv preprint arXiv:2402.04333 , year=

    Less: Selecting influential data for targeted instruction tuning , author=. arXiv preprint arXiv:2402.04333 , year=

    arXiv

  13. [21]

    2025 , eprint=

    Fewer May Be Better: Enhancing Offline Reinforcement Learning with Reduced Dataset , author=. 2025 , eprint=

    2025

  14. [22]

    arXiv preprint arXiv:2005.01643 , year=

    Offline reinforcement learning: Tutorial, review, and perspectives on open problems , author=. arXiv preprint arXiv:2005.01643 , year=

    2005 arXiv

  15. [23]

    2021 , eprint=

    RL Unplugged: A Suite of Benchmarks for Offline Reinforcement Learning , author=. 2021 , eprint=

    2021

  16. [24]

    Advances in neural information processing systems , volume=

    Deep learning on a data diet: Finding important examples early in training , author=. Advances in neural information processing systems , volume=

  17. [25]

    arXiv preprint arXiv:2210.09241 , year=

    Boosting offline reinforcement learning via data rebalancing , author=. arXiv preprint arXiv:2210.09241 , year=

    arXiv

  18. [26]

    arXiv preprint arXiv:1708.00489 , year=

    Active learning for convolutional neural networks: A core-set approach , author=. arXiv preprint arXiv:1708.00489 , year=

    arXiv

  19. [27]

    Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems , pages=

    A trajectory perspective on the role of data sampling techniques in offline reinforcement learning , author=. Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems , pages=

  20. [28]

    2025 , eprint=

    Decoupled Prioritized Resampling for Offline RL , author=. 2025 , eprint=

    2025

  21. [29]

    2005 , publisher=

    Algorithmic learning in a random world , author=. 2005 , publisher=

    2005

  22. [30]

    arXiv preprint arXiv:2107.07511 , year=

    A gentle introduction to conformal prediction and distribution-free uncertainty quantification , author=. arXiv preprint arXiv:2107.07511 , year=

    arXiv

  23. [31]

    Advances in Neural Information Processing Systems , volume =

    A Minimalist Approach to Offline Reinforcement Learning , author =. Advances in Neural Information Processing Systems , volume =. 2021 , url =

    2021

  24. [32]

    Offline Reinforcement Learning for

    Wang, Huaijie and Hao, Shibo and Dong, Hanze and Zhang, Shenao and Bao, Yilin and Yang, Ziran and Wu, Yi , booktitle =. Offline Reinforcement Learning for. 2025 , url =

    2025

  25. [33]

    International Conference on Learning Representations , year =

    Shridhar, Mohit and Yuan, Xingdi and C\^. International Conference on Learning Representations , year =

  26. [34]

    arXiv preprint arXiv:2110.14168 , year =

    Training Verifiers to Solve Math Word Problems , author =. arXiv preprint arXiv:2110.14168 , year =

    arXiv

  27. [35]

    ArXiv , year=

    Qwen2.5 Technical Report , author=. ArXiv , year=

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.