{"id":"5d816238-8eaa-467b-931a-559d2787f8dc","arxiv_id":"2608.07719","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"CODS iteratively picks high-Bellman-residual transitions into a frozen reusable subset, retaining 96.6% of eligible-pool D4RL performance at a 10% budget and beating one-shot and gradient-matching baselines.","lead":"The paper introduces CODS, a data selector for offline reinforcement learning that repeatedly rescans a fixed transition pool using a critic's Bellman residuals, then freezes a small subset for reuse across many training runs. At a 10% data budget it keeps about 96.6% of the eligible-pool performance across 20 D4RL task-algorithm combinations, which could cut the cost of large offline RL sweeps.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline retention and dominance counts depend on a 20-cell set that excludes the floor-level TD3+BC AntMaze cell and adopts a reconciled IQL AntMaze row; without a sensitivity analysis, the central claim is contingent on those two documented decisions.","rationale":"I read the paper in good faith. The arithmetic behind Table 3 checks: the six TD3+BC cells sum to 504.0 (Pool) and 486.9 (CODS), giving 96.6%; the seven CQL cells give 97.0%; the seven IQL cells give 96.1%; the overall 96.6% is internally consistent with Table 1. The fixed-budget round ablation is a reasonable design, and the paper is unusually explicit that Remark 1 is not a certification and that the corruption and duplication interventions expose real failure modes. The reader's weakest assumption, that Bellman residuals reliably identify useful transitions, is a genuine scope limit, but it does not by itself falsify the clean-benchmark claim; the paper's own language frames CODS as a reusable procedure, not a formal coreset. The more load-bearing issue is that the headline number and dominance counts are defined over a 20-cell set that depends on two documented but not yet stress-tested decisions: excluding the TD3+BC AntMaze floor cell and adopting the reconciled IQL AntMaze row after an internal inconsistency. Those decisions are legitimate if the manifest is correct, but the paper does not show the headline's sensitivity to them. That is testable by a straightforward recomputation, and conditional acceptance is the appropriate posture until the sensitivity analysis is reported. If the sensitivity analysis shows the counts and p-values are stable across variants, the central claim would be materially stronger.","tokens_in":23048,"tokens_out":7561,"duration_ms":74340,"concrete_test":"Recompute Table 2, Table 3, and the 96.6% retention figure under three variants: (a) include the TD3+BC AntMaze cell by running all subset baselines at that floor configuration, (b) replace the reconciled IQL AntMaze row with each of the two unreconciled historical values (39.5 under (0.7,3) and 68.1 from the aggregate table), and (c) drop IQL AntMaze entirely. For each variant, report the aggregate retention, the positive-cell count for each baseline, and the Holm-adjusted p-values from the hierarchical bootstrap. If the counts fall below 19/20 or 20/20, or if any subset comparison loses significance, the headline claim is not robust to the documented cell-selection decisions.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline claim is explicitly aggregate: 96.6% retention and the 19/20 and 20/20 dominance counts are defined over the 20 valid cells of Table 1. Those 20 cells are not untouched D4RL results. Appendix J documents that TD3+BC AntMaze is excluded because it is at the task floor, and that the IQL AntMaze row was changed to a manifest-verified (0.9,10) setting after an internal inconsistency was found between a sensitivity record and an aggregate table. The paper is transparent about both choices, but it never reports how the headline would shift if the floor cell were included (with all baselines actually run) or if the unreconciled IQL value were used. Because the central claim is exactly an aggregate over this particular cell set, the load-bearing assumption is that these two cell-level decisions are both correct and immaterial to the aggregate. This is compounded by the paper's own Remark 1, which certifies essentially none of the ranking (worst-case factor ~2e3, 0.02% certified pairs), and by the corruption study in the Limitations section, which shows raw CODS can fail when residuals are noisy. Together, these passages mean the empirical aggregate, and the cell-selection rules that define it, must carry the entire claim; the theoretical apparatus explicitly does not.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CODS (Critic-guided Offline Data Selection), a method that iteratively fits an algorithm-matched critic on a growing burn-in subset, acquires the transitions with the largest absolute Bellman residuals, and freezes the resulting subset for downstream offline RL training. The central empirical claim, based on a 20-cell D4RL task–algorithm matrix, is that a 10% CODS subset retains 96.6% of the aggregate performance of the 90% eligible pool, exceeds ReDOR and OPER in 19/20 cells, and exceeds Random, ReD, Static-PER, and k-center in all 20 cells, with six predeclared Holm-adjusted hierarchical comparisons significant at p < 0.001. The paper also reports rescoring ablations, budget sweeps, selection-seed stability, corruption and duplication stress tests, cross-algorithm reuse, matched compute protocols, and a whole-trace language-agent extension on ALFWorld and GSM8K.","tokens_in":23329,"tokens_out":11471,"duration_ms":108010,"significance":"If the headline results hold, CODS is a simple and practical reusable data-selection method for offline RL, with a clearly stated scope (state-based D4RL benchmarks) and a clearly stated non-guarantee (no formal coreset or policy-recovery bound). The evaluation is unusually disciplined: dataset-blocked hierarchical bootstrap, Holm-corrected predeclared inference, fixed-selector-budget ablations, equal-pass and equal-hour controls, baseline fidelity audits, and raw-array/manifest artifacts that make the results machine-checkable. The paper's explicit negative results—the rejection of Remark 1 as a certification tool, the corruption failure, and the floor-level TD3+BC AntMaze analysis—are a genuine strength and increase confidence in the reported positive findings.","major_comments":[{"comment":"The headline retention (96.6%) and dominance counts (19/20 and 20/20) are defined over the 20 'valid' cells, which result from two documented decisions: excluding the TD3+BC AntMaze cell as floor-level and replacing the IQL AntMaze value with the manifest-verified (0.9,10) setting. The manuscript never reports how the aggregate numbers would change if the excluded cell were included (with all baselines run) or if the unreconciled IQL value were used. Since the central claim is exactly an aggregate over this cell set, please add a sensitivity analysis: compute the retention ratio and the per-baseline positive-cell counts under (a) inclusion of the floor cell and (b) the unreconciled IQL row, or provide an upper bound on the resulting changes. Without this, the headline is contingent on two cell-level choices.","section":"Section 5.1, Table 1, Appendix J"},{"comment":"The 'eligible-pool' baseline is trained on only 90% of the original data because 10% of trajectories are held out for an optional conformal diagnostic that is explicitly not used to make any claim. The retention and dominance results are therefore relative to a 90% pool, not the full dataset. Please report on at least two or three representative cells the Pool performance trained on the full 100% of original trajectories, and confirm that the 96.6% retention and baseline ordering are unchanged. If that is not feasible, state clearly in the main text that all claims are relative to the 90% eligible pool and that the effect of the holdout is unmeasured.","section":"Section 5.1, Appendix L"},{"comment":"The paper's only theoretical statement, Remark 1, relates residual magnitude to a local information-gain proxy under strong conditioning assumptions, and the empirical audit shows a worst-case factor of about 2e3 with only 0.02% of pairs certified. The paper honestly concludes that this bound cannot explain the empirical gains. This leaves the acquisition mechanism—that high absolute Bellman residuals identify transitions the downstream learner needs—supported only by the aggregate benchmark results and the descriptive AntMaze composition audits. Given that the corruption study in the Limitations section shows raw residual selection can fail dramatically (CQL AntMaze drops from 58.0 to 27.8 at 10% reward corruption, below ReDOR), please either add a direct per-transition validation (e.g., correlation between residual scores and leave-one-out utility on a subset of cells) or explicitly state in the introduction that the method is a heuristic with no known conditions under which the 96.6% retention will hold.","section":"Section 4, Remark 1 and Appendix G"}],"minor_comments":[{"comment":"The line 'Uniformly sampleb 0 indices' is missing a space and should read 'Uniformly sample b0 indices'.","section":"Algorithm 1"},{"comment":"The label 'Krounds' should be 'K rounds'.","section":"Figure 1"},{"comment":"The column header 'Holmp' should be spaced as 'Holm p'.","section":"Table 2"},{"comment":"The cost equations 'Cpool(S) = 1.8S, C CODS(S) = 3.0 + 0.18S' would be clearer with consistent subscript formatting, e.g., C_pool(S) and C_CODS(S).","section":"Section 6"},{"comment":"The phrase 'the historical 68.1 value isnotreused' is missing spaces and should read 'is not reused'.","section":"Appendix J"},{"comment":"The reference to 'V ovk et al.' should be 'Vovk et al.'.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is a strong empirical contribution with an honest treatment of its limitations. The main risk is that the headline 96.6% retention and 20/20 dominance counts are defined over a cell set that excludes the TD3+BC AntMaze configuration and adopts a reconciled IQL AntMaze row. I would not reject on this basis, but I would require the sensitivity analysis before publication. The paper's narrow benchmark scope is acceptable for a cs.LG venue given the depth of the controls and the availability of raw arrays and manifests."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this one if you care about offline RL data curation or compute reuse. The core idea is simple and actually new: score the transition pool with an algorithm-matched Bellman residual, add the top batch, refit the critic, re-score, repeat, then freeze the selected indices for reuse. That sits cleanly between one-shot residual selection and dynamic prioritized replay. The paper's main technical contribution is showing that iterative rescoring matters under a fixed selector budget, and that a frozen 10% subset preserves most of the 90% eligible pool's aggregate D4RL performance.\n\nThe empirical discipline is the best part. They ran a dataset-blocked hierarchical bootstrap, predeclared six subset hypotheses with Holm correction, did a fixed-selector-budget round ablation, equal-pass and equal-hour controls, and a baseline fidelity audit. They also report explicit negative results, including the corruption study where raw CODS drops from 58.0 to 27.8 on CQL AntMaze at 10% reward corruption. And they are admirably honest about the theory: Remark 1 gives a local comparison bound, but they measure a worst-case factor around 2e3 and a certified pair-order rate of 0.02%, and say plainly it does not explain the empirical gains. That is the right way to run an empirical paper.\n\nThe soft spot is the aggregate claim itself. The 96.6% retention and the 19/20 and 20/20 dominance counts are defined over a 20-cell set that excludes the floor-level TD3+BC AntMaze cell and uses a reconciled IQL AntMaze row. Both decisions are documented and defensible, but there is no sensitivity analysis showing the headline would survive either inclusion of the floor cell with real baselines or use of the unreconciled IQL value. One table would settle it. Right now the central claim is conditional on two cell-level choices, and because the theoretical apparatus is explicitly non-certifying, those choices carry the whole argument. A second, smaller concern: the paper promises artifacts and manifests, but the arXiv version has no link. Again, common in anonymized submission, but 'auditable' is only as good as the URL.\n\nOverall: this is a solid method paper with a real but modest advance and an unusually honest evaluation. The residual-signal premise is heuristic and can fail under corruption; the authors admit that. I would send it to a serious referee, ask for the sensitivity analysis and the artifact link, and then likely publish. I would cite it if the artifacts actually appear.","headline":"Careful empirical paper on iterative Bellman-residual data selection; the contribution is real and the evaluation is unusually disciplined, but the headline aggregate is conditional on two documented cell-level decisions that need a sensitivity analysis.","tokens_in":23902,"tokens_out":3002,"would_cite":true,"duration_ms":28575,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Iterative Bellman-residual selection lets a frozen 10% subset of offline RL data retain 96.6% of the full eligible-pool performance, with reuse as the source of the compute savings.","keywords":["offline reinforcement learning","data subset selection","Bellman residual","reusable dataset","D4RL benchmarks","iterative selection","prioritized replay"],"falsifier":"Give CODS the same five-round schedule and 100,000-update budget but replace the residual scores with random scores drawn once per round before selection; if the resulting 10% subsets retain close to 96.6% of pool performance, the Bellman-residual signal is not what carries the result. A cheaper check: verify on any D4RL cell that a CODS subset selected with a critic intentionally trained on corrupted rewards loses the reported margin over random selection, which the paper's 10% corruption row suggests it should.","tokens_in":22863,"feed_emoji":"🤖","tokens_out":7289,"duration_ms":62141,"temperature":0.7,"pith_summary":"CODS asks whether a fixed 10% subset of an offline reinforcement-learning transition pool can stand in for the full dataset when the subset is chosen by repeatedly refitting a critic and acquiring the transitions with the largest absolute Bellman residuals. The paper reports that it can, on its benchmark: across 20 valid D4RL task-algorithm cells the frozen subset retains 96.6% of the eligible-pool aggregate score, beats the six subset baselines in all 20 cells, and exceeds the two strongest comparators in 19. The authors frame the result as an empirical claim about a reusable selection procedure rather than a formal coreset guarantee, and they identify reuse across seeds and hyperparameters, rather than faster single-run convergence, as the source of the compute advantage. The practical payoff is that a single index artifact chosen once could replace repeated passes over a million-transition pool in offline RL sweeps.","feed_headline":"Frozen 10% subset keeps 96.6% of offline-RL pool performance","feed_subtitle":"CODS rescored Bellman residuals to pick a reusable subset, beating six baselines on D4RL control tasks.","key_machinery":"The central object is the algorithm-matched absolute Bellman residual $\\delta_A(\\tau)$, recomputed each round from a selection critic that follows the downstream learner's backup convention. A supporting theoretical proxy $I_{\\mathrm{loc}}(\\tau)=\\frac12\\delta_\\theta(\\tau)^2 g_\\tau^\\top I_\\theta^{-1}g_\\tau$ is proved to be within a constant factor of $|\\delta_\\theta(\\tau)|$, but the constants are too loose to certify ranking. The load-bearing mechanism is the acquisition loop: fit the critic, score the unselected pool, add the top-$b_k$ transitions, refit, and repeat for $K$ rounds, then freeze the indices and train every downstream run from scratch. This converts a moving error signal into a fixed artifact whose one-time selection cost can be amortized over a sweep.","core_discovery":"The paper's central claim is that iterative rescoring of an algorithm-matched Bellman residual produces a static, reusable data subset that preserves most offline RL performance at 10% of the original data. Starting from a 2% random burn-in, CODS spends five acquisition rounds fitting the downstream algorithm's critic on the current working set, scoring every unselected transition by the absolute TD residual for that algorithm (clipped target-policy noise for TD3+BC, unpenalized target for CQL, expectile value for IQL), and adding the highest-residual batch before freezing the indices. Under equal downstream updates, the subset's aggregate is 96.6% of the eligible 90% pool (mean gap -2.77 points), with a +4.72-point advantage over ReDOR and +6.84 over OPER; all six subset advantages survive Holm-adjusted hierarchical testing. The authors explicitly disclaim a policy-recovery, regret, or coreset guarantee, noting the local-score bound's worst-case factor is about $2\\times10^3$ and only 0.02% of sampled pairs meet the sufficient separation condition.","pith_inferences":["Editorial inference: the corruption result (raw CODS falls from 58.0 to 27.8 at 10% reward corruption on CQL AntMaze, and clipping only recovers to 45.4) predicts that adding a residual-clipping or outlier-rejection step to acquisition would recover much of the clean-data margin; the paper stops at calling these diagnostic ablations.","Editorial inference: because the acquisition score depends only on the critic's backup convention, CODS should transfer to any offline algorithm that shares the same TD target; a cheap test is to retrain with a fourth algorithm that uses the same expectile or double-Q target and check the retention ratio.","Editorial inference: the equal-pass retention of 90.3% versus 93.8% under equal updates implies that part of the headline retention is reheating the same transitions more often; readers who care about wall-clock data efficiency should read the equal-hour condition (95.1% retention at 20 reuses) as the more honest number."],"forward_implications":["A 10% CODS subset can replace a 90% eligible pool for repeated compatible training: the paper's ledger breaks even at two epoch-scaled configurations and reaches a 5.45x total-time reduction at twenty equal-pass reuses.","Iterative rescoring, not just the residual rule, drives the gain: with total selector updates fixed, five acquisition rounds improve four representative cells by 11.23 points over one round and saturate by round ten.","Residual signals are algorithm-specific: matched selectors are best for all three downstream learners, and mismatched artifacts lose 1.4-3.9 retention points in the transfer study.","Whole-trace selection extends the principle to sequential reasoning: preserving prefix-closed traces retains 95.4% of pooled ALFWorld success and 96.5% of pooled GSM8K exact match, while fragmented step selection drops both.","The selector is not a universal accelerator: a single run is slower after selection cost is charged, and equal-update training gives only 1.04x total-time reduction at twenty reuses."],"supporting_citations":[{"why":"Supplies the D4RL v2 locomotion and AntMaze datasets that define the 20-cell evaluation matrix.","marker":"Fu et al., 2020"},{"why":"Defines TD3+BC, one of the three downstream algorithms whose critic target CODS matches during selection.","marker":"Fujimoto and Gu, 2021"},{"why":"Defines CQL; CODS uses its conservative critic for training but the unpenalized TD residual as the acquisition score.","marker":"Kumar et al., 2020"},{"why":"Defines IQL; CODS uses its learned expectile value to form the acquisition residual.","marker":"Kostrikov et al., 2021"},{"why":"ReDOR is the strongest gradient-matching subset baseline and the closest comparator, with CODS ahead on 19 of 20 cells.","marker":"Yang et al., 2025"},{"why":"OPER is the offline resampling baseline against which CODS is compared at equal cardinality.","marker":"Yue et al., 2025"},{"why":"ReD provides the trajectory-return baseline that CODS must beat in the subset comparisons.","marker":"Yue et al., 2022"},{"why":"Prioritized experience replay motivates the residual signal but is dynamic; CODS contrasts itself as a frozen artifact.","marker":"Schaul et al., 2015"},{"why":"OREO-style offline value learning supplies the language-model critic form for the whole-trace ALFWorld and GSM8K extension.","marker":"Wang et al., 2025"},{"why":"ALFWorld is one of the two sequence-level benchmarks used in the whole-trace extension.","marker":"Shridhar et al., 2021"}],"fun_headline_variants":["CODS preserves 96.6% of offline RL performance with 10% reusable data","Reusable 10% subset keeps 96.6% of offline RL performance","CODS beats six baselines with a frozen 10% subset","Iterative residual scoring makes a static 10% subset reusable","96.6% performance from 10% data via iterative residual selection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the absolute Bellman residual of the algorithm-matched critic reliably marks the transitions the downstream learner most needs; if residuals are dominated by noise, outliers, or critic misspecification, selection inherits those failures, as the paper's own corruption experiment shows.","fun_headline_variants_meta":{"raw":{"variants":["CODS preserves 96.6% of offline RL performance with 10% reusable data","Reusable 10% subset keeps 96.6% of offline RL performance","CODS beats six baselines with a frozen 10% subset","Iterative residual scoring makes a static 10% subset reusable","96.6% performance from 10% data via iterative residual selection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001365,"raw_usage":{"total_tokens":5582,"prompt_tokens":1040,"completion_tokens":4542,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":656,"completion_tokens_details":{"reasoning_tokens":4443}},"tokens_in":656,"tokens_out":4542,"duration_ms":27302,"temperature":1.0,"reasoning_tokens":4443,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T00:22:09.027147+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Give CODS the same five-round schedule and 100,000-update budget but replace the residual scores with random scores drawn once per round before selection; if the resulting 10% subsets retain close to 96.6% of pool performance, the Bellman-residual signal is not what carries the result. A cheaper check: verify on any D4RL cell that a CODS subset selected with a critic intentionally trained on corrupted rewards loses the reported margin over random selection, which the paper's 10% corruption row suggests it should.","supporting_citations":[{"cited_title":"Offline Reinforcement Learning for","cited_arxiv_id":null,"evidence_quote":"ReDOR is the strongest gradient-matching subset baseline and the closest comparator, with CODS ahead on 19 of 20 cells."},{"cited_title":"Offline Reinforcement Learning for","cited_arxiv_id":null,"evidence_quote":"OREO-style offline value learning supplies the language-model critic form for the whole-trace ALFWorld and GSM8K extension."}],"review_version":1}