Pith. sign in

REVIEW 4 major objections 6 minor 49 references

Beyond the Known: Decision Making with Counterfactual Reasoning Decision Transformer

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read CRDT equips a Decision Transformer with counterfactual reasoning so it can stitch suboptimal offline trajectories into higher-return behavior, with no change to the DT architecture.

desk verdict A genuine counterfactual data augmentation pipeline for Decision Transformers with real but modest gains; the OOD optimism concern is legitimate but the paper still deserves peer review. read the letter →

arxiv 2505.09114 v1 pith:OQXQPQXN submitted 2025-05-14 cs.AI cs.LG

Minh Hoang Nguyen , Linh Le Pham Van , Thommen George Karimpanal , Sunil Gupta , Hung Le This is my paper · ORCID
classification cs.AIcs.LG
keywords offlinereinforcementlearningDecisionTransformercounterfactualreasoningpotentialoutcomeframeworktrajectorystitchingdataaugmentationsequencemodelinggeneralization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a Decision Transformer (DT), which learns to pick actions by imitating an offline dataset, can be made to reason beyond that data by training two auxiliary models that generate counterfactual experiences. A Treatment model learns the distribution of actions given history, state, and desired return, and an Outcome model predicts the next state and returns-to-go that each action would produce. CRDT samples actions from the low-probability tail of the action distribution, predicts where those actions would lead, keeps only the ones that promise higher returns under confident dynamics, and retrains DT on the original data plus these imagined trajectories. The paper reports that this raises success on a toy stitching task from about 40% for DT to about 90% for CRDT, lifts the total D4RL locomotion score from 677.0 to 701.38, and degrades far less than DT when training is limited to 10% of the dataset. The point of interest is that stitching and data efficiency are obtained by data generation alone, leaving the core transformer architecture untouched.

What carries the argument

The load-bearing machinery is a pair of auxiliary transformer models plus two admission filters. The Treatment model $T$ converts the DT's next-action prediction into a full conditional action distribution, which makes it possible to define 'an action the agent would not have taken' as a low-probability draw; the bound on the expectation of the maximum of Gaussian samples (Eq. 7) yields the sampling formula (Eq. 8) that produces the counterfactual action candidates. The Outcome model $O$, trained by Eq. 6 to predict $\hat s_{t+1}$ and $\hat g_{t+1}$, rolls those candidates forward. Two filters decide admission into the counterfactual buffer $D_{\text{crdt}}$: a return filter requiring $\hat g_{t+1} < g_{t+1}$ (lower returns-to-go means higher realized return), and an uncertainty filter $U_\alpha$ that flags accumulated maximum variance across $m$ dropout-perturbed forward passes as unfamiliar when it exceeds a threshold $\alpha$ calibrated on the static dataset. The DT agent $M$ is then trained on the combined loss $\mathcal{L}_M = \mathcal{L}_M^{\text{env}} + \mathcal{L}_M^{\text{crdt}}$ over equal batches from both buffers. The paper grounds the construction in the potential-outcome framework and invokes consistency, sequential overlap, and sequential ignorability (Appendix B) to argue that counterfactual outcomes are identifiable from factual observations.

What would settle it

To test the mechanism directly: sample counterfactual actions with Eq. 8 on an environment where the true transition function is available, execute those actions in the environment to obtain ground-truth next states and returns, and compare O's predictions against them. If the accumulated prediction error on counterfactual actions is large while the performance gains persist, then accurate outcome prediction is not what carries the result. Conversely, a strong negative result would be: on a deterministic environment where O is trained to near-zero factual error, if CRDT still fails to improve over DT on the stitching task despite the pipeline in Algorithm 2 running to completion, the counterfactual selection and filtering loop itself, not the data scarcity, would be the bottleneck.

Watch

Extended reading notes

Core claim

The central claim is that counterfactual reasoning, built on the potential-outcome framework, gives a plain Decision Transformer abilities it is usually said to lack: stitching suboptimal trajectories, performing well with limited data, and coping with altered environment dynamics. The construction works in three steps. First, a Treatment model $T$ is trained to output the conditional action distribution rather than a single action—for continuous spaces a Gaussian with predicted mean $\mu_t$ and variance $\sigma_t^2$—and an Outcome model $O$ is trained by Eq. 6 to predict the next state $s_{t+1}$ and returns-to-go $g_{t+1}$. Second, candidate counterfactual actions are drawn from the low-probability tail using the Gaussian-max bound (Eq. 7), giving $\hat a_t^{(j)} = \mu_t - \Phi^{-1}(0.08 - j\beta)\,\sigma_t\sqrt{\ln n_{\text{enc}}}$; $O$ rolls each candidate forward, and two filters admit only actions whose returns-to-go improve on the factual one and whose predicted states pass the dropout-variance uncertainty check $U_\alpha$ (Eq. 9). Third, the surviving imagined trajectories are stored in a buffer $D_{\text{crdt}}$ and the DT agent $M$ is trained on equal batches from $D_{\text{env}}$ and $D_{\text{crdt}}$ with combined loss. If the paper is right, the mechanism behind the reported gains is that this pipeline composes new high-return trajectories out of segments observed in the data—stitching by generation—without modifying the transformer architecture or its loss.

Load-bearing premise

The load-bearing premise is that the Outcome model O, trained only on factual transitions, predicts next states and returns-to-go accurately for counterfactual actions drawn from the low-probability tail of the action distribution; if those out-of-distribution predictions are systematically biased, the counterfactual buffer injects distorted trajectories and the reported gains would not reproduce.

Editorial extensions

If this is right

  • Data efficiency: with only 10% of the dataset, CRDT's score drops roughly 15%, while DT drops more than 21% (up to 40% in extreme cases); on Maze2d-umaze, DT loses more than 90% of its performance while CRDT loses about 25%.
  • Stitching: on the toy environment where good trajectories are outnumbered 10:1 to 50:1, DT reaches the goal only about 37-41% of the time, whereas CRDT succeeds in 83-92% of episodes.
  • Benchmark gains: the total Locomotion score improves from 677.0 (DT) to 701.38 (CRDT), with the largest single gain being 16.1% on walker2d-medium-replay; on Atari, CRDT beats its backbone in 3 of 4 games, up to 25% on Breakout.
  • Portability: wrapping REINF or EDT with the same counterfactual buffer still improves those backbones on the Locomotion tasks, suggesting the generated experiences transfer across sequence-modeling agents.
  • Robustness: in three of the four modified-dynamics environments (altered head and thigh parameters), CRDT outperforms both DT and REINF.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The counterfactual buffer behaves as a targeted, model-based data augmentation that concentrates synthetic data at the state where a better trajectory diverges from a worse one—exactly the intersections where offline stitching decisions are made.
  • Nothing in the pipeline is specific to Gaussian policies: any backbone with a tractable action distribution (a discretized continuous policy, a diffusion policy, or a categorical head) could be fed through the same selection-and-filter loop, which is a testable variant of the method.
  • The dropout-variance filter is a reusable out-of-distribution gate for offline model-based generation; applied to other augmentation schemes that generate rollouts (noise perturbations, dynamics-model sampling), it could filter their failure modes as well.
  • The framework is agnostic to the choice of dynamics model: replacing the Outcome transformer with any calibrated next-state predictor, while keeping the low-probability action sampler and the two filters, would isolate how much of the gain comes from the counterfactual selection logic versus the quality of the outcome model.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The manuscript introduces CRDT, a training-time augmentation of the Decision Transformer (DT). It first trains two transformer models: a Treatment model T that estimates the conditional action distribution given history, and an Outcome model O that predicts the next state and returns-to-go for a chosen action. It then generates counterfactual trajectories by substituting low-probability actions, filtering candidates with a dropout-based uncertainty threshold and a return-improvement criterion, and storing the successful trajectories in a buffer D_crdt. Finally, DT is trained on equal batches from the environment dataset D_env and D_crdt. Experiments report improved average scores on D4RL Locomotion and Ant, Maze2d, and Atari 1% DQN-replay datasets, together with a toy stitching environment and robustness checks under limited data and modified dynamics.

Significance. The empirical pattern is interesting and, if the generated trajectories are dynamically valid, would be a useful way to give DT stitching behavior without architectural changes. Strengths include released code, comparison with several sequential-modeling and model-free baselines, ablations of the filtering mechanisms, and explicit discussion of causal assumptions. However, the gains are modest and often within noise, and the core validity of the generated counterfactual experiences is not established; moreover, the paper's stated overlap assumption is violated by its own action-selection rule. The causal-identifiability framing therefore overstates what is currently shown.

major comments (4)
  1. [Section 3.2, Eq. (8) and Algorithm 2] The validity of D_crdt is the load-bearing step, and the manuscript does not establish it. O is trained only on factual transitions (Eq. 6), while Eq. 8 deliberately selects actions from the tail of T's predicted Gaussian; for continuous states these actions are generally outside the empirical support of D_env. The filtering in Eq. 9 uses dropout variance, which measures internal disagreement among O's forward passes, not error against the environment. Algorithm 2 (line 12) then keeps a candidate only if the predicted returns-to-go is lower than the factual one, i.e., it retains the most optimistic predictions. This can fill D_crdt with self-consistent but invalid trajectories and could explain the reported gains without the agent learning valid counterfactual dynamics. Please provide a direct diagnostic: for the chosen counterfactual actions, compare O's predicted next states and returns with the true environment outputs (or with a held-out factual subset), report prediction error as a function of the action's distance from the empirical action distribution, and compare CRDT against a variant that filters with ground-truth returns rather than predicted ones.
  2. [Appendix B and Appendix D.1] The claimed identifiability is not achieved by the implemented procedure. Assumption A.2 (sequential overlap) requires every action to have positive probability under each observed history, while Eq. 8 explicitly targets actions above the expected maximum of n_enc samples; in a finite dataset these actions have empirical frequency near zero. Moreover, D.1 states that n_enc is obtained by exact-match MD5 hashing of the continuous input (h_t, s_{t+1}, g_{t+1}). In continuous state spaces exact matches are essentially unique, so n_enc is approximately 1 and sqrt(ln n_enc) is approximately 0, which makes Eq. 8 collapse to a_t = μ_t and removes the counterfactual search entirely. Please define n_enc precisely, and either weaken the identifiability claims or show that the assumptions actually cover the action-selection distribution used in the algorithm.
  3. [Tables 1, 2 and Fig. 2] The claim that CRDT 'consistently enhances' DT is not supported with statistical evidence. No significance tests or paired comparisons are reported, and many per-task differences are smaller than one standard deviation; for example, Table 5 shows halfcheetah-med-rep with CRDT at 38.03±2.5 versus REINF at 38.3, and hopper-med with CRDT at 67.94±1.5 versus REINF at 75.2. On Atari, Table 2 shows a large regression on Pong (102.2 vs 140.2 human-normalized; raw scores in Table 6 are 15.8 vs 29.5), so the aggregate improvement is driven by other games. Please report paired bootstrap tests or significance tests across seeds and include per-task effect sizes with confidence intervals.
  4. [Section 4 and Appendix F.12] The hyperparameter selection protocol is underspecified. Table 12 sets n_a, α, and n_e per dataset, and Appendix D.3 states that the uncertainty threshold α is obtained by tuning ς in steps of 0.5, but the manuscript does not say whether this tuning uses a validation split or the evaluation environments. If α, n_a, or n_e are selected using test performance, the comparisons are not model-selection-fair. Please state the selection criterion and, if possible, fix a single protocol across datasets or show that the main results are robust to these choices.
minor comments (6)
  1. [Section 3.1, Eq. (4)] The treatment-model loss uses parameter δ in the log-probability term, although T is parameterized by θ; the notation should be corrected.
  2. [Section 3.2, Eq. (8)] For β > 0, all sampled actions lie above μ_t because Φ^{-1}(0.08 − jβ) becomes more negative as j grows, so the claimed 'diverse range' of counterfactual actions is not actually diverse across both tails of the action distribution.
  3. [Algorithm 2, line 8] T outputs a distribution or its parameters, not a sampled action; the paper should specify how â_t^(j) is obtained from T's output in the discrete and continuous cases.
  4. [Appendix A] The heading contains a typo: 'Stiching' should be 'Stitching'.
  5. [Abstract and Section 4] The statement that CRDT outperforms DT should be qualified given the Pong regression in Table 2 and the halfcheetah-med-rep and hopper-med rows in Table 5, where CRDT does not improve over the best baseline.
  6. [Table 5] The text says CRDT improves on all datasets, but Table 5 shows halfcheetah-med-rep with CRDT at 38.03±2.5 versus REINF at 38.3; the exception should be acknowledged in the main text.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: CRDT's reported gains are measured on held-out environment returns, and no equation reduces to a fitted input or a load-bearing self-citation.

full rationale

The paper's counterfactual experiences are generated by Treatment and Outcome models trained on D_env, but the reported improvements are evaluated through independent environment rollouts, not through the models' own predicted returns. The filtering rule in Algorithm 2 uses the Outcome model's predicted returns-to-go to select training data; this is the intended data-augmentation mechanism rather than a re-labeling of the evaluation metric. No central equation is defined in terms of the target result: Eq. 8 is a Gaussian tail bound from an external source (Kamath), Eq. 9 is a dropout-variance uncertainty filter, and Eqs. 10-11 are standard DT losses. There are no load-bearing self-citations; all cited results are external benchmarks or external methods. The self-referential loop of generating D_crdt from models trained on D_env is not circular in the technical sense because the success metric is held-out environment return; it is better viewed as a validity risk if the Outcome model's out-of-distribution predictions are optimistically biased. The possible tension between the sequential-overlap assumption in Appendix B and the tail-sampling in Eq. 8 is a correctness and identifiability concern, not a by-construction reduction.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The method depends on several tuned hyperparameters (n_a, beta, alpha, n_e, gamma) and on causal identification assumptions stated informally in Appendix B. No new physical or conceptual entities are introduced.

free parameters (5)
  • n_a (number of counterfactual search actions) = 5 or 7, per environment
    Set per dataset (Table 12); Fig. F.3 shows performance varies with n_a, so the choice is effective tuning.
  • beta (step size in counterfactual action search) = 0.01
    Fixed for all environments (Appendix F.12); a hand-chosen constant, not tuned against the target.
  • alpha (uncertainty threshold) = 0.1 to 75, per environment
    Tuned per environment via ς in steps of 0.5 (Appendix D.3, Tables 12 and 14); this is a free parameter fitted to validation data.
  • n_e (counterfactual experience buffer size) = 500 to 4000, per environment
    Chosen per environment (Tables 12 and 14); for Atari set to 500 due to compute limits.
  • gamma (outlier action threshold, discrete spaces) = 0.25
    Tuned over [0.1, 0.3] with step 0.05 for all Atari games (Appendix F.12).
assumptions (5)
  • domain assumption Sequential ignorability: actions are independent of potential outcomes given observed history
    Appendix B, Assumption A.3. Required for counterfactual outcome identification; the paper relies on the history capturing all confounders.
  • domain assumption Sequential overlap: every action has nonzero probability under the behavior policy
    Appendix B, Assumption A.2. Needed so low-probability actions are in the support; may fail in narrow offline datasets.
  • domain assumption Consistency: observed next state and return are the true outcomes of the taken action
    Appendix B, Assumption A.1. Reasonable in MDP data collection.
  • domain assumption Continuous actions follow a Gaussian distribution conditional on history
    Section 3.1, Eq. 5. Used by the Treatment model and the counterfactual action sampler (Eq. 8); may be inaccurate for multi-modal action distributions.
  • domain assumption Dropout variance of the Outcome model measures prediction confidence
    Section 3.2, Eq. 9 and Appendix D.2. The entire filtering mechanism depends on this proxy being calibrated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Beyond the Known: Decision Making with Counterfactual Reasoning Decision Transformer." pith.science (2026). https://pith.science/paper/OQXQPQXN

@misc{pith2026250509114,
  author       = {Pith},
  title        = {Pith review of: Beyond the Known: Decision Making with Counterfactual Reasoning Decision Transformer},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OQXQPQXN}},
  note         = {Machine review of arXiv:2505.09114}
}
read the original abstract

Decision Transformers (DT) play a crucial role in modern reinforcement learning, leveraging offline datasets to achieve impressive results across various domains. However, DT requires high-quality, comprehensive data to perform optimally. In real-world applications, the lack of training data and the scarcity of optimal behaviours make training on offline datasets challenging, as suboptimal data can hinder performance. To address this, we propose the Counterfactual Reasoning Decision Transformer (CRDT), a novel framework inspired by counterfactual reasoning. CRDT enhances DT ability to reason beyond known data by generating and utilizing counterfactual experiences, enabling improved decision-making in unseen scenarios. Experiments across Atari and D4RL benchmarks, including scenarios with limited data and altered dynamics, demonstrate that CRDT outperforms conventional DT approaches. Additionally, reasoning counterfactually allows the DT agent to obtain stitching abilities, combining suboptimal trajectories, without architectural modifications. These results highlight the potential of counterfactual reasoning to enhance reinforcement learning agents' performance and generalization capabilities.

Figures

Figures reproduced from arXiv: 2505.09114 by the authors.

Figure 1
Figure 1. (a): A toy environment where the goal of the agent is to move from the green circle position to the red circle position given that data is biased toward moving from bottom-left to top-right (green trajectory/diagonal line) over top￾left to bottom-right (blue trajectory/ diagonal line). When using traditional DT, the agent will most likely follow the green trajectory and fail to reach the goal. (b): The empirical res… view at source ↗
Figure 2
Figure 2. Performance comparison on limited subset of [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Given two trajectories s a t−1 , st, sa t+1 , [PITH_FULL_IMAGE:figures/full_fig_p015_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

49 extracted references · 33 canonical work pages

  1. [1]

    An optimistic perspective on offline reinforcement learning

    Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. InProceedings of the 37th International Conference on Machine Learning, ICML’20. JMLR.org, 2020

    work page 2020

  2. [2]

    End-to-end balancing for causal con- tinuous treatment-effect estimation

    Taha Bahadori, Eric Tchetgen Tchetgen, and David Heckerman. End-to-end balancing for causal con- tinuous treatment-effect estimation. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepes- vari, Gang Niu, and Sivan Sabato, editors,Proceedings of the 39th International Conference on Machine Learning, volume 162 ofProceedings of Machine Learning Re...

    work page 2022

  3. [3]

    The arcade learning environment: An evaluation platform for general agents.Journal of Artificial Intelligence Research, 47:253–279, 2013

    Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents.Journal of Artificial Intelligence Research, 47:253–279, 2013

    2013

  4. [4]

    When should we prefer decision transformers for offline reinforcement learning? InThe Twelfth International Confer- ence on Learning Representations, 2024

    Prajjwal Bhargava, Rohan Chitnis, Alborz Geramifard, Shagun Sodhani, and Amy Zhang. When should we prefer decision transformers for offline reinforcement learning? InThe Twelfth International Confer- ence on Learning Representations, 2024

    work page 2024

  5. [5]

    Openai gym, 2016

    Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016

    2016

  6. [6]

    Decision transformer: reinforcement learning via sequence mod- eling

    Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: reinforcement learning via sequence mod- eling. InProceedings of the 35th International Conference on Neural Information Processing Systems, pages 15084–15097, 2021

    work page 2021

  7. [7]

    Estimating average causal effects from patient trajectories

    Dennis Frauen, Tobias Hatt, Valentyn Melnychuk, and Stefan Feuerriegel. Estimating average causal effects from patient trajectories. InProceedings of the Thirty-Seventh AAAI Conference on Artifi- cial Intelligence and Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence and Thirteenth Symposium on Educational Advances in Artificia...

    work page 2023

  8. [8]

    D4rl: Datasets for deep data-driven reinforcement learning.arXiv preprint arXiv:2004.07219, 2020

    Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning.arXiv preprint arXiv:2004.07219, 2020

    arXiv 2004

Show all 49 references
  1. [9]

    Cate meets ml: Conditional average treatment effect and machine learning.Digital Finance, 3(2):99–148, 2021

    Daniel Jacob. Cate meets ml: Conditional average treatment effect and machine learning.Digital Finance, 3(2):99–148, 2021

    2021

  2. [10]

    Bounds on the expectation of the maximum of samples from a gaussian.URL http://www

    Gautam Kamath. Bounds on the expectation of the maximum of samples from a gaussian.URL http://www. gautamkamath. com/writings/gaussian max. pdf, 10(20-30):31, 2015. 11

    2015

  3. [11]

    Morel: model- based offline reinforcement learning

    Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: model- based offline reinforcement learning. InProceedings of the 34th International Conference on Neural Information Processing Systems, pages 21810–21823, 2020

    2020

  4. [12]

    Counterfactually guided policy transfer in clinical settings

    Taylor W Killian, Marzyeh Ghassemi, and Shalmali Joshi. Counterfactually guided policy transfer in clinical settings. InConference on Health, Inference, and Learning, pages 5–31. PMLR, 2022

    2022

  5. [13]

    Decision convformer: Local filtering in metaformer is sufficient for decision making.arXiv preprint arXiv:2310.03022, 2023

    Jeonghye Kim, Suyoung Lee, Woojun Kim, and Youngchul Sung. Decision convformer: Local filtering in metaformer is sufficient for decision making.arXiv preprint arXiv:2310.03022, 2023

    2023 arXiv

  6. [14]

    Offline reinforcement learning with fisher divergence critic regularization

    Ilya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. In Marina Meila and Tong Zhang, editors,Proceedings of the 38th International Conference on Machine Learning, volume 139 ofProceedings of...

    2021

  7. [15]

    Offline reinforcement learning with implicit q- learning.arXiv preprint arXiv:2110.06169, 2021

    Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q- learning.arXiv preprint arXiv:2110.06169, 2021

    2021 arXiv

  8. [16]

    Conservative q-learning for offline reinforcement learning

    Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. InProceedings of the 34th International Conference on Neural Information Processing Systems, NIPS ’20, Red Hook, NY , USA, 2020. Curran Associates Inc

    2020

  9. [17]

    Offline reinforcement learning: Tutorial, review, and perspectives on open problems.arXiv preprint arXiv:2005.01643, 2020

    Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems.arXiv preprint arXiv:2005.01643, 2020

    2005 arXiv

  10. [18]

    G-net: a deep learning approach to g-computation for counterfactual outcome prediction under dynamic treatment regimes.arXiv preprint arXiv:2003.10551, 2020

    Rui Li, Zach Shahn, Jun Li, Mingyu Lu, Prithwish Chakraborty, Daby Sow, Mohamed Ghalwash, and Li-wei H Lehman. G-net: a deep learning approach to g-computation for counterfactual outcome prediction under dynamic treatment regimes.arXiv preprint arXiv:2003.10551, 2020

    2003 arXiv

  11. [19]

    Causal transformer for estimating coun- terfactual outcomes

    Valentyn Melnychuk, Dennis Frauen, and Stefan Feuerriegel. Causal transformer for estimating coun- terfactual outcomes. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors,Proceedings of the 39th International Conference on M...

    2022

  12. [20]

    Counterfactual credit assignment in model-free reinforcement learning.arXiv preprint arXiv:2011.09464, 2020

    Thomas Mesnard, Th ´eophane Weber, Fabio Viola, Shantanu Thakoor, Alaa Saade, Anna Harutyunyan, Will Dabney, Tom Stepleton, Nicolas Heess, Arthur Guez, et al. Counterfactual credit assignment in model-free reinforcement learning.arXiv preprint arXiv:2011.09464, 2020

    2011 arXiv

  13. [21]

    Human-level control through deep reinforcement learning.nature, 518(7540):529–533, 2015

    V olodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning.nature, 518(7540):529–533, 2015

    2015

  14. [22]

    On the application of probability theory to agricultural experiments

    Jerzy Neyman. On the application of probability theory to agricultural experiments. essay on princi- ples.Ann. Agricultural Sciences, pages 1–51, 1923

    1923

  15. [23]

    The book of why: The new science of cause and effect, 2018

    Judea Pearl and Dana Mackenzie. The book of why: The new science of cause and effect, 2018

    2018

  16. [24]

    Counterfactual data augmentation using locally fac- tored dynamics

    Silviu Pitis, Elliot Creager, and Animesh Garg. Counterfactual data augmentation using locally fac- tored dynamics. InProceedings of the 34th International Conference on Neural Information Processing Systems, NIPS ’20, Red Hook, NY , USA, 2020. Curran Associates Inc

    2020

  17. [25]

    Mocoda: model-based counterfac- tual data augmentation

    Silviu Pitis, Elliot Creager, Ajay Mandlekar, and Animesh Garg. Mocoda: model-based counterfac- tual data augmentation. InProceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, Red Hook, NY , USA, 2022. Curran Associates Inc. 12

    2022

  18. [26]

    Pomerleau

    Dean A. Pomerleau. Alvinn: an autonomous land vehicle in a neural network. InProceedings of the 2nd International Conference on Neural Information Processing Systems, NIPS’88, page 305–313, Cambridge, MA, USA, 1988. MIT Press

    1988

  19. [27]

    Estimation of the causal effects of time-varying exposures.Chap- man & Hall/CRC Handbooks of Modern Statistical Methods, pages 553–599, 2008

    James Robins and Miguel Hernan. Estimation of the causal effects of time-varying exposures.Chap- man & Hall/CRC Handbooks of Modern Statistical Methods, pages 553–599, 2008

    2008

  20. [28]

    Bayesian inference for causal effects: The role of randomization.The Annals of statistics, pages 34–58, 1978

    Donald B Rubin. Bayesian inference for causal effects: The role of randomization.The Annals of statistics, pages 34–58, 1978

    1978

  21. [29]

    Reinforcement learning upside down: Don’t predict rewards–just map them to actions.arXiv preprint arXiv:1912.02875, 2019

    Juergen Schmidhuber. Reinforcement learning upside down: Don’t predict rewards–just map them to actions.arXiv preprint arXiv:1912.02875, 2019

    1912 arXiv

  22. [30]

    Causal influence detection for improving efficiency in reinforcement learning

    Maximilian Seitzer, Bernhard Sch ¨olkopf, and Georg Martius. Causal influence detection for improving efficiency in reinforcement learning. InProceedings of the 35th International Conference on Neural Information Processing Systems, NIPS ’21, Red Hook, NY , USA, 2021. Curran A...

    2021

  23. [31]

    Johansson, and David Sontag

    Uri Shalit, Fredrik D. Johansson, and David Sontag. Estimating individual treatment effect: gen- eralization bounds and algorithms. In Doina Precup and Yee Whye Teh, editors,Proceedings of the 34th International Conference on Machine Learning, volume 70 ofProceedings of Machin...

    2017

  24. [32]

    Mastering the game of go without human knowledge.nature, 550(7676):354–359, 2017

    David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge.nature, 550(7676):354–359, 2017

    2017

  25. [33]

    Training agents using upside-down reinforcement learning.arXiv preprint arXiv:1912.02877, 2019

    Rupesh Kumar Srivastava, Pranav Shyam, Filipe Mutz, Wojciech Ja´skowski, and J¨urgen Schmidhuber. Training agents using upside-down reinforcement learning.arXiv preprint arXiv:1912.02877, 2019

    1912 arXiv

  26. [34]

    Offline imitation learning with variational counterfactual reasoning

    Zexu Sun, Bowei He, Jinxin Liu, Xu Chen, Chen Ma, and Shuai Zhang. Offline imitation learning with variational counterfactual reasoning. InProceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Red Hook, NY , USA, 2023. Curran Asso...

    2023

  27. [35]

    Acamda: improving data efficiency in reinforcement learning through guided counterfactual data aug- mentation

    Yuewen Sun, Erli Wang, Biwei Huang, Chaochao Lu, Lu Feng, Changyin Sun, and Kun Zhang. Acamda: improving data efficiency in reinforcement learning through guided counterfactual data aug- mentation. InProceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence a...

    2024

  28. [36]

    Sutton and Andrew G

    Richard S. Sutton and Andrew G. Barto.Reinforcement Learning: An Introduction. A Bradford Book, Cambridge, MA, USA, 2018

    2018

  29. [37]

    Learning robot in-hand ma- nipulation with tactile features

    Herke van Hoof, Tucker Hermans, Gerhard Neumann, and Jan Peters. Learning robot in-hand ma- nipulation with tactile features. In2015 IEEE-RAS 15th International Conference on Humanoid Robots (Humanoids), pages 121–127. IEEE, 2015

    2015

  30. [38]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. V on Luxburg, S. Bengio, H. Wal- lach, R. Fergus, S. Vishwanathan, and R. Garnett, editors,Advances in Neura...

    2017

  31. [39]

    Supervised reinforcement learning with recur- rent neural network for dynamic treatment recommendation

    Lu Wang, Wei Zhang, Xiaofeng He, and Hongyuan Zha. Supervised reinforcement learning with recur- rent neural network for dynamic treatment recommendation. InProceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pages 2447–2456, 2018

    2018

  32. [40]

    Elastic decision transformer

    Yueh-Hua Wu, Xiaolong Wang, and Masashi Hamaya. Elastic decision transformer. InProceedings of the 37th International Conference on Neural Information Processing Systems, pages 18532–18550, 2023

    2023

  33. [41]

    Bellman-consistent pessimism for offine reinforcement learning

    Tengyang Xie, Ching-An Cheng, Nan Jiang, Paul Mineiro, and Alekh Agarwal. Bellman-consistent pessimism for offine reinforcement learning. InProceedings of the 35th International Conference on Neural Information Processing Systems, NIPS ’21, Red Hook, NY , USA, 2021. Curran Ass...

    2021

  34. [42]

    Future-conditioned unsu- pervised pretraining for decision transformer

    Zhihui Xie, Zichuan Lin, Deheng Ye, Qiang Fu, Wei Yang, and Shuai Li. Future-conditioned unsu- pervised pretraining for decision transformer. InProceedings of the 40th International Conference on Machine Learning, ICML’23. JMLR.org, 2023

    2023

  35. [43]

    Q-learning decision transformer: Lever- aging dynamic programming for conditional sequence modelling in offline RL

    Taku Yamagata, Ahmed Khalil, and Raul Santos-Rodriguez. Q-learning decision transformer: Lever- aging dynamic programming for conditional sequence modelling in offline RL. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett...

    2023

  36. [44]

    Mopo: model-based offline policy optimization

    Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: model-based offline policy optimization. InProceedings of the 34th International Conference on Neural Information Processing Systems, pages 14129–14142, 2020

    2020

  37. [45]

    Offline trajectory generalization for offline reinforcement learning.arXiv preprint arXiv:2404.10393, 2024

    Ziqi Zhao, Zhaochun Ren, Liu Yang, Fajie Yuan, Pengjie Ren, Zhumin Chen, Xin Xin, et al. Offline trajectory generalization for offline reinforcement learning.arXiv preprint arXiv:2404.10393, 2024

    2024 arXiv

  38. [46]

    Online decision transformer

    Qinqing Zheng, Amy Zhang, and Aditya Grover. Online decision transformer. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors,Proceedings of the 39th International Conference on Machine Learning, volume 162 ofProceedings of M...

    2022

  39. [47]

    A boosting algorithm for estimating generalized propensity scores with continuous treatments.Journal of causal inference, 3(1):25–40, 2015

    Yeying Zhu, Donna L Coffman, and Debashis Ghosh. A boosting algorithm for estimating generalized propensity scores with continuous treatments.Journal of causal inference, 3(1):25–40, 2015

    2015

  40. [48]

    Beyond the Known: Decision Making with Counterfactual Reasoning Decision Transformer

    Zifeng Zhuang, Dengyun Peng, Jinxin Liu, Ziqi Zhang, and Donglin Wang. Reinformer: max-return sequence modeling for offline rl. InProceedings of the 41st International Conference on Machine Learn- ing, pages 62707–62722, 2024. 14 Supplementary Material for: “Beyond the Known: ...

    2024

  41. [49]

    Pearl-style231 counterfactual reasoning

    We can derive our formula to calculate each action:58 at =µ t−Φ−1 (0.08−j·β)σ t p ln(nenc). whereβis the step size andj= 0,1,...,n a indicates the index of thej-th action from the totaln a59 sampled counterfactual actions.Φ −1 is the quantile function of the standard normal di...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.