REVIEW 4 major objections 5 minor 3 cited by
A Survey on Explainable Deep Reinforcement Learning
T0 review · 4 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read This survey claims that explainable deep reinforcement learning is organized by four method families — feature-level, state-level, dataset-level, and model-level — and that these explanations can be measured, attacked with, defended with…
desk verdict A well-organized XRL survey whose four-category taxonomy is compromised by misattributed citations and an unaudited selection process. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The organizing machinery is the four-category taxonomy itself, which acts as the survey's lens. Feature-level methods highlight important parts of an observation through perturbation, gradients, or attention; state-level methods locate pivotal moments by comparing trajectories, learning masks, or extending the action space with a lazy action; dataset-level methods quantify per-example influence with influence functions, Shapley values, or masking; model-level methods change the policy model itself so it is interpretable, or extract rules from it. The taxonomy does the analytic work: it determines which papers are grouped together, which evaluation criteria apply, and which applications follow.
What would settle it
Take a large, independently assembled sample of XRL papers and have annotators assign each method to exactly one of the four proposed categories; if a substantial share cannot be assigned without stretching category definitions — for example, counterfactual explanations that change states rather than isolate features — the partition is not complete. A second check is to reproduce one representative method from each category and verify that the survey's description matches the actual algorithm and experimental setup.
Extended reading notes
Core claim
The paper's central claim is that existing approaches to explaining deep reinforcement learning can be broadly grouped into four categories: feature-level methods (saliency and attention maps), state-level methods (critical-step identification, whether through offline trajectory modeling or online interaction), dataset-level methods (influence functions, data Shapley values, and masking to find training examples that shaped the policy), and model-level methods (transparent architectures such as decision trees or rule extraction, plus sparse-autoencoder analyses). It further claims that these explanations can be measured along qualitative and quantitative axes, with perturbation-based fidelity — removing what an explanation calls important and checking performance loss — as the common yardstick. Finally, the survey claims that explanations are dual-use: they can be used to attack agents by targeting critical features, states, or data, to defend them by blinding or retraining at vulnerable moments, and to refine policies through human-in-the-loop correction or automated retraining from critical states.
Load-bearing premise
The taxonomy of four categories is assumed to be complete and non-overlapping, and the one-sentence summaries of the cited methods are assumed accurate.
Editorial extensions
If this is right
- If the taxonomy holds, practitioners can select an explanation method by the granularity they need: features for perception debugging, states for temporal debugging, data for data curation, and models for architectural transparency.
- A high-fidelity explanation, measured by performance drop under perturbation, is also a reliable attack blueprint; cited results show attacks succeed at just 10 percent of targeted time steps, so better interpretability can increase vulnerability.
- State-level explanations are the common engine for policy refinement, supporting both human feedback correction and automated retraining from critical states.
- Dataset-level explanations transfer to LLM alignment, where influence functions and Shapley values expose which human-preference data steering RLHF is most consequential and most poisonable.
- Explanation-guided defenses work locally: blinding observations at critical steps or retraining around backdoor triggers can restore agent performance in poisoned environments.
Reading between the lines
- Editorial inference: because the four levels are defined by what is explained — input, trajectory, data, or model — the survey's own proposed future direction of narrative or strategy-level explanations would likely form a fifth category rather than fitting inside the existing four.
- Editorial inference: if perturbation-based fidelity becomes the dominant metric, explanation methods validated only by user studies may appear weaker even when they are more practically useful, so a separate trust-based evaluation axis may be needed.
- Editorial inference: the same critical-state identification that supports automated retraining could be repurposed for curriculum design, using discovered bottlenecks as initial states to speed learning in sparse-reward or long-horizon tasks, a direction the survey suggests but does not develop.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This survey reviews explainable deep reinforcement learning (XRL). It proposes a four-category taxonomy—feature-level, state-level, dataset-level, and model-level explanations—presents qualitative and quantitative evaluation criteria, and surveys applications in adversarial attacks, defenses, and policy refinement. It also discusses the integration of RL with large language models, particularly RLHF. The survey's central claim is that the proposed taxonomy captures the existing XRL literature, supported by Figure 1 and the per-category discussions in Sections 3.1–3.4.
Significance. If the taxonomy and the individual method summaries are accurate, this survey would provide a useful structured entry point into XRL, especially its organization of evaluation metrics and the dual use of explanations for attacks and policy improvement. The explicit coverage of adversarial robustness and policy refinement, as well as the attention to LLM-related extensions, are strengths. However, the reliability of the central claim currently depends on citation accuracy and scope discipline; several cited works do not appear to be about deep RL explanation, and the absence of a stated search methodology prevents verification of completeness. The survey is therefore potentially valuable but not yet dependable as a guide to the field.
major comments (4)
- [Section 3.1, Figure 1] The description of Wang et al. [2016] as extending gradient-based saliency maps to deep RL by computing the Jacobian of output logits is a clear misattribution. The cited reference, Ziyu Wang, Tom Schaul, et al., ICML 2016, is the Dueling Network Architectures paper, which proposes dueling Q-networks and does not compute saliency maps or Jacobian-based explanations. Because this citation is used as the example of gradient-based feature-level explanation, the accuracy of other per-paper summaries in the taxonomy becomes questionable.
- [Section 3.3, Figure 1] The dataset-level category is not supported by the cited evidence. Li et al. [2024], Matelsky et al. [2024], and Ruis et al. [2024] are studies of influence functions for large language model fine-tuning, not for deep RL policies. The text acknowledges that these works concern LLMs, but the taxonomy in Figure 1 and the section heading present them as dataset-level XRL methods. Similarly, the Data Masking subcategory cites Dong et al. [2024] and Lin et al. [2024], both of which address LLM training data, with no established connection to DRL. The survey's claim to categorize DRL explanation methods is therefore not supported by these citations.
- [Section 3.4, Figure 1] The model-level category includes Demircan et al. [2024] as using sparse autoencoders within a policy network to explain LLM behavior. The cited paper studies sparse autoencoders in large language models, not in deep reinforcement learning policy networks. This is outside the stated scope of the survey's taxonomy and further weakens the claim that the four categories partition the XRL literature.
- [Section 3 (overall methodology)] The survey never states a systematic search methodology—no databases, search strings, inclusion/exclusion criteria, or time range—even though it claims to provide a comprehensive review. Without a reproducible selection process, the completeness of the taxonomy cannot be audited, and the reader cannot distinguish deliberate selection from accidental omission. This is a load-bearing omission for a survey whose central contribution is a categorization of the field.
minor comments (5)
- [Section 2.3] DPO and RAFT are described as 'RL approaches' for fine-tuning LLMs, but DPO is a direct preference optimization method that avoids a separate reward model and reinforcement learning loop; the characterization should be qualified to avoid misleading readers.
- [Figure 1] The figure lists Selvaraju et al. [2017] under gradient-based feature-level methods, but Grad-CAM is a general CNN visualization technique; the relevant DRL adaptation is Joo and Kim [2019]. Citing the original method is appropriate, but the taxonomy entry should make clear that the original is not a DRL-specific explanation method.
- [Section 5.2] The paragraph on SHINE states that the method 'provides theoretical guarantees of improved robustness' without specifying the nature of the guarantee or referencing a theorem; either state the guarantee precisely or soften the claim to 'empirically demonstrates improved robustness'.
- [Section 5.1] There is a redundant sentence: 'Recent work demonstrates that explanations... can be repurposed to compromise the agent’s performance' is followed immediately by a near-identical sentence beginning 'Recent studies have revealed...'. One of these should be removed.
- [References] Several author names contain formatting artifacts, e.g., 'Y uet al.', 'Y uanet al.', 'V an Waverenet al.', with spurious spaces inside author surnames. These should be corrected throughout the bibliography.
Circularity Check
No circularity: this is a descriptive survey whose four-part taxonomy is an organizational claim, not a derived result; the authors' self-citations are examples rather than load-bearing evidence, and the skeptic's citation concerns are audit issues, not circular loops.
full rationale
The paper is a survey, not a derivation chain. Its central claim, that existing XRL methods can be categorized into feature-level, state-level, dataset-level, and model-level explanation methods, is asserted as an organizational taxonomy in Section 3 and Figure 1; it is not obtained by fitting a parameter, solving an equation, or invoking a uniqueness theorem. The authors' own prior work (Cheng et al. 2023, 2024; Yu et al. 2023) appears as examples inside the taxonomy and in later application sections, but the taxonomy is not defined in terms of those papers, and the survey never uses them to prove any of its claims. These self-citations are therefore not load-bearing. No equation in Section 2 is repackaged as a prediction, no fitted quantity is renamed as an explanation result, and no uniqueness result is imported from the authors' earlier papers. The skeptic's documented citation problems (for example, the Wang et al. 2016 entry is Dueling DQN rather than a saliency-map method, and several LLM-focused influence-function papers are used as dataset-level XRL examples) are accuracy, scope, and selection-methodology concerns for a survey; they warrant correction but do not constitute circular reasoning. There is no step in which the survey's output is equivalent to its input by construction.
Assumptions & free parameters
assumptions (2)
- domain assumption The four-category taxonomy (feature-level, state-level, dataset-level, model-level) is complete and non-overlapping for the XRL literature.
- domain assumption The summaries of each cited paper accurately reflect that paper's contributions.
Cite this review
Pith. "Pith review of A Survey on Explainable Deep Reinforcement Learning." pith.science (2026). https://pith.science/paper/2RTV4HUG
@misc{pith2026250206869,
author = {Pith},
title = {Pith review of: A Survey on Explainable Deep Reinforcement Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/2RTV4HUG}},
note = {Machine review of arXiv:2502.06869}
}
read the original abstract
Deep Reinforcement Learning (DRL) has achieved remarkable success in sequential decision-making tasks across diverse domains, yet its reliance on black-box neural architectures hinders interpretability, trust, and deployment in high-stakes applications. Explainable Deep Reinforcement Learning (XRL) addresses these challenges by enhancing transparency through feature-level, state-level, dataset-level, and model-level explanation techniques. This survey provides a comprehensive review of XRL methods, evaluates their qualitative and quantitative assessment frameworks, and explores their role in policy refinement, adversarial robustness, and security. Additionally, we examine the integration of reinforcement learning with Large Language Models (LLMs), particularly through Reinforcement Learning from Human Feedback (RLHF), which optimizes AI alignment with human preferences. We conclude by highlighting open research challenges and future directions to advance the development of interpretable, reliable, and accountable DRL systems.
Figures
Forward citations
Cited by 3 Pith papers
-
Verification-Guided Falsification for Safe RL via Explainable Abstraction and Risk-Aware Exploration
A hybrid safe-RL framework combining explainable abstraction, formal model checking, and risk-guided search reports more diverse policy failures, but its headline PAC guarantee largely restates its own assumptions.
-
Interpret Policies in Deep Reinforcement Learning using SILVER with RL-Guided Labeling: A Model-level Approach to High-dimensional and Multi-action Environments
SILVER with RL-guided labeling: SHAP plus clustering plus policy-query labels plus decision trees or regression to interpret multi-action Atari policies.
-
Beyond Prediction: Reinforcement Learning as the Defining Leap in Healthcare AI
A survey of reinforcement learning in healthcare that frames RL as a paradigm shift from prediction to agentive clinical intelligence.
Reference graph
Works this paper leans on
-
[2024]
Sparse autoencoders reveal tem- poral difference learning in large language models
Can Demircan, Tankred Saanum, Akshay K Jagadish, Marcel Binz, and Eric Schulz. Sparse autoencoders reveal tem- poral difference learning in large language models. arXiv preprint arXiv:2410.01280,
-
[3]
Cdt: Cascading deci- sion trees for explainable reinforcement learning
Zihan Ding, Pablo Hernandez-Leal, Gavin Weiguang Ding, Changjian Li, and Ruitong Huang. Cdt: Cascading deci- sion trees for explainable reinforcement learning. arXiv preprint arXiv:2011.07553,
arXiv 2011
-
[7]
Empirical influence functions to understand the logic of fine-tuning
Jordan K Matelsky, Lyle Ungar, and Konrad P Kording. Em- pirical influence functions to understand the logic of fine- tuning. arXiv preprint arXiv:2406.00509 ,
-
[9]
Procedural knowledge in pretraining drives reasoning in large language models
Laura Ruis, Maximilian Mozes, Juhan Bae, Siddhartha Rao Kamalakara, Dwarak Talupuru, Acyr Locatelli, Robert Kirk, Tim Rockt¨ aschel, Edward Grefenstette, and Max Bartolo. Procedural knowledge in pretraining drives reasoning in large language models. arXiv preprint arXiv:2411.12580,
-
[10]
Proximal policy optimization al- gorithms
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Rad- ford, and Oleg Klimov. Proximal policy optimization al- gorithms. arXiv preprint arXiv:1707.06347 ,
-
[11]
Starcraft ii: A new challenge for re- inforcement learning
Oriol Vinyals, Timo Ewalds, Sergey Bartunov, Petko Georgiev, Alexander Sasha V ezhnevets, Michelle Y eo, Alireza Makhzani, Heinrich K¨ uttler, John Agapiou, Julian Schrittwieser, et al. Starcraft ii: A new challenge for re- inforcement learning. arXiv preprint arXiv:1708.04782 ,
-
[12]
Graying the black box: Understanding dqns
Tom Zahavy, Nir Ben-Zrihem, and Shie Mannor. Graying the black box: Understanding dqns. In Proc. of ICML, 2016
work page 2016
-
[2015]
Playing atari with deep reinforcement learning
V olodymyr Mnih. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 ,
Show all 12 references
-
[2017]
Do influence func- tions work on large language models? arXiv preprint arXiv:2409.19998,
Zhe Li, Wei Zhao, Yige Li, and Jun Sun. Do influence func- tions work on large language models? arXiv preprint arXiv:2409.19998,
-
[2020]
Training a helpful and harmless assistant with reinforcement learning from human feedback
Y untao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862 ,
-
[2022]
Reinforcement learning from imperfect corrective ac- tions and proxy rewards
Zhaohui Jiang, Xuening Feng, Paul Weng, Yifei Zhu, Y an Song, Tianze Zhou, Y ujing Hu, Tangjie Lv, and Changjie Fan. Reinforcement learning from imperfect corrective ac- tions and proxy rewards. arXiv preprint arXiv:2410.05782,
-
[2023]
Promptexp: Multi-granularity prompt explanation of large language models
Ximing Dong, Shaowei Wang, Dayi Lin, Gopi Krishnan Ra- jbahadur, Boquan Zhou, Shichao Liu, and Ahmed E Has- san. Promptexp: Multi-granularity prompt explanation of large language models. arXiv preprint arXiv:2410.13073 ,
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.