{"id":"f8a3f74b-6fee-4f7b-baf1-2ef6577d09b6","arxiv_id":"2505.09114","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"CRDT improves Decision Transformers by generating counterfactual (low-probability) actions and their predicted outcomes, improving offline RL performance and enabling trajectory stitching.","lead":"A new training method, CRDT, feeds a decision transformer AI with 'what-if' experiences, including actions it did not take and the outcomes those actions might have produced. In tests on Atari and D4RL benchmarks, it generally outperforms the standard decision transformer, especially when data is limited or biased.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Outcome-model extrapolation is the load-bearing step: CRDT samples actions beyond the empirical support and filters on predicted returns, so reported gains may be an artifact of optimistic OOD predictions rather than valid counterfactual reasoning.","rationale":"The reader's weakest assumption already identifies the Outcome model's OOD reliability as the key vulnerability, and my analysis agrees: the counterfactual action selection mechanism (Eq. 8) deliberately targets the tail of the action distribution, and the filtering rule selects on predicted returns, which compounds any optimistic bias in O. This is load-bearing because D_crdt is the only new information CRDT injects into the underlying DT; if those trajectories are systematically wrong, the improved stitching and D4RL numbers would not be evidence of counterfactual reasoning. The proposed oracle test directly checks whether O's accepted counterfactual predictions match ground truth. I also considered an implementation-level ambiguity: Appendix D.1 describes exact-match MD5 hashing for n_enc, which in continuous state spaces would give n_enc=1 for most inputs under a literal reading, making Eq. 8 collapse to μ_t; however, if n_enc counts training epochs it becomes a global constant, so the method is not necessarily degenerate. I did not lead with that because the extrapolation-accuracy concern is more fundamental and remains regardless of how n_enc is resolved. Given the paper's conditional status, this concern does not move the verdict: the manuscript should be accepted only after the oracle validation or an equivalent check is performed.","tokens_in":21920,"tokens_out":7794,"duration_ms":85318,"concrete_test":"Run the released code on walker2d-medium-replay with the true MuJoCo environment as an oracle: for each accepted counterfactual transition in D_crdt, execute the counterfactual action from the factual state and record the true next state and return, then compare with O's predictions. If the mean absolute error on accepted counterfactual transitions is comparable to O's error on held-out factual transitions and the selected actions actually achieve the predicted higher returns, the concern is resolved. If accepted counterfactual predictions are systematically optimistic (true returns below predicted, or no better than the factual baseline), the buffer is biased and the central claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"CRDT's central claim depends on the counterfactual experiences in D_crdt being valid trajectories. In Section 3.2, Eq. 8 selects candidate actions from the extreme upper tail of T's predicted Gaussian: for j=0 the action is approximately μ_t + sqrt(2) σ_t sqrt(ln n_enc), the upper bound from Eq. 7 on the expected maximum of n_enc samples. These actions are deliberately low-probability and, in continuous state spaces, typically lie outside the empirical support of D_env. O is trained only on factual transitions via Eq. 6, so its predictions for such actions are extrapolations with no supervised signal. The filtering step does not repair this: Eq. 9's dropout variance measures internal disagreement among O's forward passes, not error against the environment, and the return filter (Algorithm 2, line 12: keep only if ĝ_{t+1} < g_{t+1}) actively retains actions for which O is most optimistic relative to the factual return. Thus D_crdt can be filled with self-consistent but invalid trajectories, and training DT on them (Eqs. 10-11) could inflate apparent performance on the toy stitching task and Maze2d without the agent learning valid counterfactual dynamics. The identifiability assumptions in Appendix B do not cover this regime: sequential overlap requires positive probability for every action, while Eqs. 7-8 explicitly target actions beyond the expected maximum of n_enc samples, whose empirical frequency in a finite dataset is near zero. If O's extrapolation is optimistically biased, the central claim that CRDT endows DT with stitching ability rests on invalid training data.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript introduces CRDT, a training-time augmentation of the Decision Transformer (DT). It first trains two transformer models: a Treatment model T that estimates the conditional action distribution given history, and an Outcome model O that predicts the next state and returns-to-go for a chosen action. It then generates counterfactual trajectories by substituting low-probability actions, filtering candidates with a dropout-based uncertainty threshold and a return-improvement criterion, and storing the successful trajectories in a buffer D_crdt. Finally, DT is trained on equal batches from the environment dataset D_env and D_crdt. Experiments report improved average scores on D4RL Locomotion and Ant, Maze2d, and Atari 1% DQN-replay datasets, together with a toy stitching environment and robustness checks under limited data and modified dynamics.","tokens_in":22261,"tokens_out":7053,"duration_ms":65194,"significance":"The empirical pattern is interesting and, if the generated trajectories are dynamically valid, would be a useful way to give DT stitching behavior without architectural changes. Strengths include released code, comparison with several sequential-modeling and model-free baselines, ablations of the filtering mechanisms, and explicit discussion of causal assumptions. However, the gains are modest and often within noise, and the core validity of the generated counterfactual experiences is not established; moreover, the paper's stated overlap assumption is violated by its own action-selection rule. The causal-identifiability framing therefore overstates what is currently shown.","major_comments":[{"comment":"The validity of D_crdt is the load-bearing step, and the manuscript does not establish it. O is trained only on factual transitions (Eq. 6), while Eq. 8 deliberately selects actions from the tail of T's predicted Gaussian; for continuous states these actions are generally outside the empirical support of D_env. The filtering in Eq. 9 uses dropout variance, which measures internal disagreement among O's forward passes, not error against the environment. Algorithm 2 (line 12) then keeps a candidate only if the predicted returns-to-go is lower than the factual one, i.e., it retains the most optimistic predictions. This can fill D_crdt with self-consistent but invalid trajectories and could explain the reported gains without the agent learning valid counterfactual dynamics. Please provide a direct diagnostic: for the chosen counterfactual actions, compare O's predicted next states and returns with the true environment outputs (or with a held-out factual subset), report prediction error as a function of the action's distance from the empirical action distribution, and compare CRDT against a variant that filters with ground-truth returns rather than predicted ones.","section":"Section 3.2, Eq. (8) and Algorithm 2"},{"comment":"The claimed identifiability is not achieved by the implemented procedure. Assumption A.2 (sequential overlap) requires every action to have positive probability under each observed history, while Eq. 8 explicitly targets actions above the expected maximum of n_enc samples; in a finite dataset these actions have empirical frequency near zero. Moreover, D.1 states that n_enc is obtained by exact-match MD5 hashing of the continuous input (h_t, s_{t+1}, g_{t+1}). In continuous state spaces exact matches are essentially unique, so n_enc is approximately 1 and sqrt(ln n_enc) is approximately 0, which makes Eq. 8 collapse to a_t = μ_t and removes the counterfactual search entirely. Please define n_enc precisely, and either weaken the identifiability claims or show that the assumptions actually cover the action-selection distribution used in the algorithm.","section":"Appendix B and Appendix D.1"},{"comment":"The claim that CRDT 'consistently enhances' DT is not supported with statistical evidence. No significance tests or paired comparisons are reported, and many per-task differences are smaller than one standard deviation; for example, Table 5 shows halfcheetah-med-rep with CRDT at 38.03±2.5 versus REINF at 38.3, and hopper-med with CRDT at 67.94±1.5 versus REINF at 75.2. On Atari, Table 2 shows a large regression on Pong (102.2 vs 140.2 human-normalized; raw scores in Table 6 are 15.8 vs 29.5), so the aggregate improvement is driven by other games. Please report paired bootstrap tests or significance tests across seeds and include per-task effect sizes with confidence intervals.","section":"Tables 1, 2 and Fig. 2"},{"comment":"The hyperparameter selection protocol is underspecified. Table 12 sets n_a, α, and n_e per dataset, and Appendix D.3 states that the uncertainty threshold α is obtained by tuning ς in steps of 0.5, but the manuscript does not say whether this tuning uses a validation split or the evaluation environments. If α, n_a, or n_e are selected using test performance, the comparisons are not model-selection-fair. Please state the selection criterion and, if possible, fix a single protocol across datasets or show that the main results are robust to these choices.","section":"Section 4 and Appendix F.12"}],"minor_comments":[{"comment":"The treatment-model loss uses parameter δ in the log-probability term, although T is parameterized by θ; the notation should be corrected.","section":"Section 3.1, Eq. (4)"},{"comment":"For β > 0, all sampled actions lie above μ_t because Φ^{-1}(0.08 − jβ) becomes more negative as j grows, so the claimed 'diverse range' of counterfactual actions is not actually diverse across both tails of the action distribution.","section":"Section 3.2, Eq. (8)"},{"comment":"T outputs a distribution or its parameters, not a sampled action; the paper should specify how â_t^(j) is obtained from T's output in the discrete and continuous cases.","section":"Algorithm 2, line 8"},{"comment":"The heading contains a typo: 'Stiching' should be 'Stitching'.","section":"Appendix A"},{"comment":"The statement that CRDT outperforms DT should be qualified given the Pong regression in Table 2 and the halfcheetah-med-rep and hopper-med rows in Table 5, where CRDT does not improve over the best baseline.","section":"Abstract and Section 4"},{"comment":"The text says CRDT improves on all datasets, but Table 5 shows halfcheetah-med-rep with CRDT at 38.03±2.5 versus REINF at 38.3; the exception should be acknowledged in the main text.","section":"Table 5"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely to be of interest to the offline RL and sequence modeling community, but the authors should be encouraged to position the method as an empirical data-augmentation technique rather than a formally identified counterfactual inference method. The lack of comparison with existing model-based counterfactual augmentation approaches, such as MoCoDA (cited as [25]), is a missed opportunity and should be addressed in a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"CRDT is a genuine and fairly implemented attempt to give Decision Transformers counterfactual stitching ability by training Treatment and Outcome models, sampling low-probability actions, filtering by predicted return and dropout uncertainty, and retraining the DT on mixed batches. The idea is new relative to MOCoDA and ACAMDA, and the toy environment result (0.4 to 0.9 success) plus the limited-data experiments are the strongest evidence. The paper is honest about its regressions (Pong) and gives ablations that show each filtering component helps.\n\nThe main soft spot is exactly what the stress-test note says: the Outcome model is asked to extrapolate to actions deliberately drawn from the tail of the action distribution, and the filtering step retains actions for which the model is optimistic about return. The dropout variance is an internal consistency check, not an error check. So D_crdt may be filled with self-consistent but invalid trajectories. The empirical gains on held-out environment return show this is not a pure curve-fitting artifact, but they don't establish that the counterfactual mechanism is doing what the paper says. The identifiability assumptions in Appendix B are not enough because the targeted actions are outside the support of the finite data.\n\nA second, more mundane issue is tuning. The per-environment alpha and n_a tables are extensive, and without significance tests (or at least error bars on the aggregate) the 701.38 vs 698.0 locomotion totals could be within noise. The stitching claim also relies heavily on a toy and Maze2d. None of this is fatal—the core idea is plausible and the implementation is careful—but it needs a skeptical referee.\n\nWho should read it: people working on offline RL, sequence modeling, or counterfactual data augmentation. It is a useful empirical contribution with a clear pipeline, not a breakthrough. I would send it to peer review: the question it asks (can counterfactual augmentation give DT stitching without architecture changes?) is worth a full review, and the authors have provided enough detail to be checked. The reviewer should ask for a direct diagnostic of whether predicted counterfactual outcomes match real environment outcomes, and maybe a sensitivity analysis over alpha and n_a that shows performance isn't a knife-edge.","headline":"A genuine counterfactual data augmentation pipeline for Decision Transformers with real but modest gains; the OOD optimism concern is legitimate but the paper still deserves peer review.","tokens_in":22869,"tokens_out":2200,"would_cite":false,"duration_ms":23378,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CRDT equips a Decision Transformer with counterfactual reasoning so it can stitch suboptimal offline trajectories into higher-return behavior, with no change to the DT architecture.","keywords":["offline reinforcement learning","Decision Transformer","counterfactual reasoning","potential outcome framework","trajectory stitching","data augmentation","sequence modeling","generalization"],"falsifier":"To test the mechanism directly: sample counterfactual actions with Eq. 8 on an environment where the true transition function is available, execute those actions in the environment to obtain ground-truth next states and returns, and compare O's predictions against them. If the accumulated prediction error on counterfactual actions is large while the performance gains persist, then accurate outcome prediction is not what carries the result. Conversely, a strong negative result would be: on a deterministic environment where O is trained to near-zero factual error, if CRDT still fails to improve over DT on the stitching task despite the pipeline in Algorithm 2 running to completion, the counterfactual selection and filtering loop itself, not the data scarcity, would be the bottleneck.","tokens_in":21697,"feed_emoji":"🔀","tokens_out":11365,"duration_ms":94741,"temperature":0.7,"pith_summary":"The paper claims that a Decision Transformer (DT), which learns to pick actions by imitating an offline dataset, can be made to reason beyond that data by training two auxiliary models that generate counterfactual experiences. A Treatment model learns the distribution of actions given history, state, and desired return, and an Outcome model predicts the next state and returns-to-go that each action would produce. CRDT samples actions from the low-probability tail of the action distribution, predicts where those actions would lead, keeps only the ones that promise higher returns under confident dynamics, and retrains DT on the original data plus these imagined trajectories. The paper reports that this raises success on a toy stitching task from about 40% for DT to about 90% for CRDT, lifts the total D4RL locomotion score from 677.0 to 701.38, and degrades far less than DT when training is limited to 10% of the dataset. The point of interest is that stitching and data efficiency are obtained by data generation alone, leaving the core transformer architecture untouched.","feed_headline":"Imagining other actions lifts Decision Transformers to ~90% success","feed_subtitle":"Adding counterfactual experiences lets offline RL stitch suboptimal data and hold up on 10% datasets","key_machinery":"The load-bearing machinery is a pair of auxiliary transformer models plus two admission filters. The Treatment model $T$ converts the DT's next-action prediction into a full conditional action distribution, which makes it possible to define 'an action the agent would not have taken' as a low-probability draw; the bound on the expectation of the maximum of Gaussian samples (Eq. 7) yields the sampling formula (Eq. 8) that produces the counterfactual action candidates. The Outcome model $O$, trained by Eq. 6 to predict $\\hat s_{t+1}$ and $\\hat g_{t+1}$, rolls those candidates forward. Two filters decide admission into the counterfactual buffer $D_{\\text{crdt}}$: a return filter requiring $\\hat g_{t+1} < g_{t+1}$ (lower returns-to-go means higher realized return), and an uncertainty filter $U_\\alpha$ that flags accumulated maximum variance across $m$ dropout-perturbed forward passes as unfamiliar when it exceeds a threshold $\\alpha$ calibrated on the static dataset. The DT agent $M$ is then trained on the combined loss $\\mathcal{L}_M = \\mathcal{L}_M^{\\text{env}} + \\mathcal{L}_M^{\\text{crdt}}$ over equal batches from both buffers. The paper grounds the construction in the potential-outcome framework and invokes consistency, sequential overlap, and sequential ignorability (Appendix B) to argue that counterfactual outcomes are identifiable from factual observations.","core_discovery":"The central claim is that counterfactual reasoning, built on the potential-outcome framework, gives a plain Decision Transformer abilities it is usually said to lack: stitching suboptimal trajectories, performing well with limited data, and coping with altered environment dynamics. The construction works in three steps. First, a Treatment model $T$ is trained to output the conditional action distribution rather than a single action—for continuous spaces a Gaussian with predicted mean $\\mu_t$ and variance $\\sigma_t^2$—and an Outcome model $O$ is trained by Eq. 6 to predict the next state $s_{t+1}$ and returns-to-go $g_{t+1}$. Second, candidate counterfactual actions are drawn from the low-probability tail using the Gaussian-max bound (Eq. 7), giving $\\hat a_t^{(j)} = \\mu_t - \\Phi^{-1}(0.08 - j\\beta)\\,\\sigma_t\\sqrt{\\ln n_{\\text{enc}}}$; $O$ rolls each candidate forward, and two filters admit only actions whose returns-to-go improve on the factual one and whose predicted states pass the dropout-variance uncertainty check $U_\\alpha$ (Eq. 9). Third, the surviving imagined trajectories are stored in a buffer $D_{\\text{crdt}}$ and the DT agent $M$ is trained on equal batches from $D_{\\text{env}}$ and $D_{\\text{crdt}}$ with combined loss. If the paper is right, the mechanism behind the reported gains is that this pipeline composes new high-return trajectories out of segments observed in the data—stitching by generation—without modifying the transformer architecture or its loss.","pith_inferences":["The counterfactual buffer behaves as a targeted, model-based data augmentation that concentrates synthetic data at the state where a better trajectory diverges from a worse one—exactly the intersections where offline stitching decisions are made.","Nothing in the pipeline is specific to Gaussian policies: any backbone with a tractable action distribution (a discretized continuous policy, a diffusion policy, or a categorical head) could be fed through the same selection-and-filter loop, which is a testable variant of the method.","The dropout-variance filter is a reusable out-of-distribution gate for offline model-based generation; applied to other augmentation schemes that generate rollouts (noise perturbations, dynamics-model sampling), it could filter their failure modes as well.","The framework is agnostic to the choice of dynamics model: replacing the Outcome transformer with any calibrated next-state predictor, while keeping the low-probability action sampler and the two filters, would isolate how much of the gain comes from the counterfactual selection logic versus the quality of the outcome model."],"forward_implications":["Data efficiency: with only 10% of the dataset, CRDT's score drops roughly 15%, while DT drops more than 21% (up to 40% in extreme cases); on Maze2d-umaze, DT loses more than 90% of its performance while CRDT loses about 25%.","Stitching: on the toy environment where good trajectories are outnumbered 10:1 to 50:1, DT reaches the goal only about 37-41% of the time, whereas CRDT succeeds in 83-92% of episodes.","Benchmark gains: the total Locomotion score improves from 677.0 (DT) to 701.38 (CRDT), with the largest single gain being 16.1% on walker2d-medium-replay; on Atari, CRDT beats its backbone in 3 of 4 games, up to 25% on Breakout.","Portability: wrapping REINF or EDT with the same counterfactual buffer still improves those backbones on the Locomotion tasks, suggesting the generated experiences transfer across sequence-modeling agents.","Robustness: in three of the four modified-dynamics environments (altered head and thigh parameters), CRDT outperforms both DT and REINF."],"supporting_citations":[{"why":"The Decision Transformer backbone whose next-action prediction the method replaces with a full action distribution; every reported gain is measured against it.","marker":"[6]"},{"why":"The potential-outcome framework that motivates counterfactual reasoning and supplies the consistency logic for treating observed transitions as factual outcomes.","marker":"[22, 28]"},{"why":"Extends potential outcomes to time-varying treatments, giving the sequential formulation of actions and outcomes used at each timestep.","marker":"[27]"},{"why":"The bound on the expectation of the maximum of Gaussian samples from which the counterfactual action sampling formula (Eq. 8) is derived.","marker":"[10]"},{"why":"Inspires the uncertainty-threshold calibration (mean plus sigma times a tuned factor over dataset samples) used to set alpha for the filtering mechanism.","marker":"[11]"},{"why":"Reinformer, the strongest sequence-modeling baseline, provides the Maze2d stitching results and random-dataset comparisons CRDT is measured against.","marker":"[48]"},{"why":"Elastic Decision Transformer, a baseline and an alternative backbone, and the source of the stitching-problem framing and Ant datasets.","marker":"[40]"},{"why":"Support the premise that DT improves when trained on higher-return data, which motivates keeping only counterfactuals with improved returns-to-go.","marker":"[4, 45]"},{"why":"The D4RL benchmark supplying the locomotion, ant, and maze2d datasets used in the main comparisons.","marker":"[8]"}],"fun_headline_variants":["Counterfactual what-ifs let Decision Transformers stitch suboptimal data","Imagined rare actions enable Decision Transformers to stitch","Counterfactual reasoning gives Decision Transformers stitching power","What-ifs for suboptimal data teach Decision Transformer to stitch","Rare-action imagination lets Decision Transformers stitch trajectories"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the Outcome model O, trained only on factual transitions, predicts next states and returns-to-go accurately for counterfactual actions drawn from the low-probability tail of the action distribution; if those out-of-distribution predictions are systematically biased, the counterfactual buffer injects distorted trajectories and the reported gains would not reproduce.","fun_headline_variants_meta":{"raw":{"variants":["Counterfactual what-ifs let Decision Transformers stitch suboptimal data","Imagined rare actions enable Decision Transformers to stitch","Counterfactual reasoning gives Decision Transformers stitching power","What-ifs for suboptimal data teach Decision Transformer to stitch","Rare-action imagination lets Decision Transformers stitch trajectories"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000393,"raw_usage":{"total_tokens":2120,"prompt_tokens":1053,"completion_tokens":1067,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":669,"completion_tokens_details":{"reasoning_tokens":984}},"tokens_in":669,"tokens_out":1067,"duration_ms":8024,"temperature":1.0,"reasoning_tokens":984,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:39:33.308231+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"To test the mechanism directly: sample counterfactual actions with Eq. 8 on an environment where the true transition function is available, execute those actions in the environment to obtain ground-truth next states and returns, and compare O's predictions against them. If the accumulated prediction error on counterfactual actions is large while the performance gains persist, then accurate outcome prediction is not what carries the result. Conversely, a strong negative result would be: on a deterministic environment where O is trained to near-zero factual error, if CRDT still fails to improve over DT on the stitching task despite the pipeline in Algorithm 2 running to completion, the counterfactual selection and filtering loop itself, not the data scarcity, would be the bottleneck.","supporting_citations":[{"cited_title":"Decision transformer: reinforcement learning via sequence mod- eling","cited_arxiv_id":null,"evidence_quote":"The Decision Transformer backbone whose next-action prediction the method replaces with a full action distribution; every reported gain is measured against it."},{"cited_title":"Estimation of the causal effects of time-varying exposures.Chap- man & Hall/CRC Handbooks of Modern Statistical Methods, pages 553–599, 2008","cited_arxiv_id":null,"evidence_quote":"Extends potential outcomes to time-varying treatments, giving the sequential formulation of actions and outcomes used at each timestep."},{"cited_title":"Bounds on the expectation of the maximum of samples from a gaussian.URL http://www","cited_arxiv_id":null,"evidence_quote":"The bound on the expectation of the maximum of Gaussian samples from which the counterfactual action sampling formula (Eq. 8) is derived."},{"cited_title":"Morel: model- based offline reinforcement learning","cited_arxiv_id":null,"evidence_quote":"Inspires the uncertainty-threshold calibration (mean plus sigma times a tuned factor over dataset samples) used to set alpha for the filtering mechanism."},{"cited_title":"Beyond the Known: Decision Making with Counterfactual Reasoning Decision Transformer","cited_arxiv_id":null,"evidence_quote":"Reinformer, the strongest sequence-modeling baseline, provides the Maze2d stitching results and random-dataset comparisons CRDT is measured against."},{"cited_title":"Elastic decision transformer","cited_arxiv_id":null,"evidence_quote":"Elastic Decision Transformer, a baseline and an alternative backbone, and the source of the stitching-problem framing and Ant datasets."}],"review_version":1}