A Survey of Explainable Reinforcement Learning: Targets, Methods and Needs

L\'eo Sauli\`eres

REVIEW 3 major objections 4 minor 1 cited by

A Survey of Explainable Reinforcement Learning: Targets, Methods and Needs

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This survey of explainable reinforcement learning proposes a What/How taxonomy that classifies over 250 papers into policy-, sequence-, and action-level targets, and it reports that sequence-level explanation is sharply underrepresented…

desk verdict A useful qualitative map of XRL that undermines itself with arithmetic that doesn't add up. read the letter →

arxiv 2507.12599 v1 pith:ZFK7XN5J submitted 2025-07-16 cs.AI cs.LG

Léo Saulières This is my paper

classification cs.AIcs.LG

keywords explainablereinforcementlearningXRLtaxonomysequence-levelexplanationpolicy-levelaction-levelsurvey

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This survey proposes an organizing device for explainable reinforcement learning (XRL): every method is classified by the target it explains ('What') and by the way it delivers the explanation ('How'). Applied to more than 250 papers, the taxonomy yields the paper's central observation: 175 works explain the agent's policy, 89 explain a single action choice, and only 11 explain sequences of agent-environment interactions. The paper argues that sequence-level explanation is a genuinely underserved target, plausibly because most XRL methods inherit habits from classifier-focused XAI, which explains decisions locally or models globally. It also lists four needs for the field: comparable benchmarks, better metrics, more user studies, and dedicated user interfaces.

What carries the argument

The organizing device is the What/How taxonomy. 'What' names the explanation target (policy, sequence, action); 'How' names the explanation modality for each target, with categories such as interpretable policy construction, policy summarization, human-readable MDPs, visual analysis, counterfactual or important-element sequences, and, for actions, feature importance and expected outcomes. The taxonomy carries the argument because the headline counts are produced by sorting the surveyed works into those targets and modalities.

What would settle it

Recount the field with an explicit search protocol and inclusion criteria, and have independent annotators assign each paper to one of the three targets; if many papers land in more than one target or the sequence-level count grows substantially under the same definitions, the 175/89/11 imbalance is an artifact of the taxonomy rather than a fact about the field.

Watch

Extended reading notes

Core claim

The paper's central claim is that the XRL landscape can be organized by answering two questions in order. 'What does the method explain?' has three answers — the agent's policy, a sequence of interactions, or a single action; 'How is it explained?' then splits each target into concrete delivery modes such as interpretable policies, summaries, human-readable MDPs, visual analysis, counterfactual sequences, important elements, feature importance, and expected outcomes. Working through roughly 250 papers under this scheme, the survey counts 175 policy-level, 89 action-level, and only 11 sequence-level works, and reads that imbalance as low researcher interest in explaining sequences. The taxonomy is offered as a way for readers to find relevant work quickly, and the needs list — method comparison, metrics, user studies, interfaces — is the paper's agenda for maturing the field.

Load-bearing premise

The counts stand only if the roughly 250 surveyed papers fairly represent XRL research and if the three targets are distinct enough that every work fits into one of them; the survey states no systematic search protocol, and its own tables place some papers under both policy-level and action-level headings.

Editorial extensions

If this is right

If the taxonomy is correct and the counts are representative, sequence-level explanation is the most neglected target in XRL, so new work explaining whole trajectories would address a real gap.
Researchers looking for an existing method can use the What/How grid to locate a body of work by target and modality in one step.
The dominance of policy- and action-level work suggests XRL has inherited the local/global framing of classifier XAI, which may explain why trajectory-level questions are rare.
The needs list implies that progress in XRL depends less on new explanation algorithms than on standardized benchmarks, metrics, user studies, and interfaces.
Saliency-map methods dominate action-level feature importance, so methods that explain actions without requiring image states are comparatively scarce.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Re-annotating borderline works — for example summaries that mix sequences and policies — could shift the 11-paper sequence count, so the exact ratios are less stable than the qualitative gap itself.
The related-domains discussion points to a testable extension: importing algorithmic recourse into XRL would turn counterfactual states and sequences into actionable recommendations, a route the surveyed counterfactual methods do not yet take.
A benchmark that evaluates methods per target and modality, as the needs list calls for, would turn the taxonomy from a descriptive map into a comparative instrument; the survey points to its ingredients but does not build it.
The What/How grid could be applied to adjacent explainability domains, such as planning and model checking, to compare their coverage and expose similarly neglected targets.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Worth knowing: this is a broad, readable survey of explainable RL organized around a simple What/How taxonomy. The qualitative summaries of individual methods are consistent with the literature, and the needs list (compare methods, provide metrics, run user studies, build interfaces) is sensible. If you are entering XRL and want a map, this is a good place to start.

What's actually new: the taxonomy itself is a repackaging of earlier ones (Puiutta & Veith, Milani et al., Dazeley et al. are all cited), but the sequence-level target is a useful cell that gets little attention elsewhere, and the paper does document how thin that area is. Credit where earned: the survey covers a lot of ground, the section on related domains (planning, model checking, algorithmic recourse) is a helpful pointer, and the inclusion of interpretable-by-design methods alongside post-hoc ones is the right call.

The soft spot is real and it is in the numbers. The conclusion's headline counts--175 policy-level, 89 action-level, 11 sequence-level--do not follow from the tables. Table 1 says 'Surrogate Model (34)' but lists 35 entries; Table 6 says 'Model-agnostic Approach (15)' but lists 13 SHAP + 3 LIME = 16. Summing policy-level tables gives 176 or 177, not 175. Action-level gives 86 or 87, not 89. Only the sequence count (11) is internally consistent. Worse, these are not counts of distinct works: the same references (e.g., [378], [406], [309], [35], [17]) appear in both policy-level and action-level tables, and the paper gives no search protocol or inclusion criteria (Section 1 mentions only '12 states of the art'), so the denominator is unverifiable. The qualitative claim that sequence-level explanation is rare probably survives; 11 versus hundreds is a wide gap. But the specific numbers as printed are unsupported.

Minor: the taxonomy's categories are not shown to be disjoint, which matters for any claim about relative interest. This is a fixable problem, not a fatal one.

Who this is for: grad students and researchers new to XRL who want a map and a list of open problems. It does not resolve anything technically, but as a field-organizing survey it has value. It deserves a serious referee--the inconsistencies are correctable and the underlying survey is honest. I would send it to review with a requirement that the counts be fixed or removed, and the cross-listing made explicit.

Referee Report

3 major / 4 minor

Summary. The paper proposes a two-question taxonomy ('What?' and 'How?') for organizing explainable reinforcement learning (XRL) research, distinguishes policy-level, sequence-level, and action-level targets, reviews what it states are more than 250 papers, sketches related domains (planning, model checking, algorithmic recourse, causal RL), and lists needs for the field. The central quantitative claim is in Section 7: 175 policy-level works, 89 action-level works, and 11 sequence-level works, which the paper uses to support the conclusion that sequence-level explanation is understudied.

Significance. If its counts and taxonomies were reliable, the paper would make a useful contribution: an intuitive organizing device for a fragmented literature, a broad bibliography, and a concrete agenda of needs (comparison methods, metrics, user studies, interfaces). The qualitative observation that sequence-level explanations are rare is plausible and consistent with the works cited. However, the load-bearing quantitative evidence for that observation—the 175/89/11 figures—does not follow from the paper's own tables, and the survey methodology is not described. The taxonomy and the literature review are still valuable, but the headline claim needs substantial repair, and the number of works covered is not verifiable as written.

major comments (3)

[Section 7 and Tables 1–7] The counts quoted in the conclusion are not reproducible from the paper's tables. In Table 1, 'Surrogate Model (34)' contains subcategories that sum to 35 (17+9+5+2+2), so the table's actual policy-level total is 89 rather than 88; adding Tables 2 (29), 3 (49), and 4 (10) gives 176 or 177 policy-level works, not 175. In Table 6, 'Model-agnostic Approach (15)' lists 13 SHAP + 3 LIME = 16 entries, making the Feature Importance total 55 rather than 54; with Table 7's 32 works this yields 86 or 87 action-level works, not 89. Only the sequence-level count (Table 5 = 11) is internally consistent. Section 7 therefore asserts exact figures that the paper's own data contradict, and the conclusion's evidence for 'low interest' in sequence-level explanation needs to be recomputed and stated with the actual numbers.
[Section 1 and the survey methodology] The paper states in Section 1 that it is 'based on a total of 12 states of the art and complementary papers' but provides no search protocol, database list, year range, inclusion criteria, exclusion criteria, or screening procedure. This makes the denominator behind 'over 250 papers' and all of the target-level counts unverifiable. For a survey whose central quantitative conclusions depend on counting works, the absence of a methods description is a load-bearing gap; it should be fixed by adding a methodology subsection (or an appendix) that explains how the corpus was assembled and how works were assigned to categories.
[Tables 2 and 6 (and Tables 2 and 7)] The same references appear under multiple target categories without a stated decision rule: [378], [406], [309], and [35] appear in both Table 2 (Policy Summary SHAP) and Table 6 (Action-level SHAP); [17] appears in both Table 2 and Table 7; and [309] and [35] are also discussed at multiple levels. Because the taxonomy's three 'What' targets are presented as distinct categories, the 175/89/11 figures cannot be interpreted as counts of distinct works unless the authors define a primary-target assignment rule and apply it consistently. Without such a rule, the totals are sums of table entries, not counts of unique papers, and the conclusion should not treat them as comparable denominators.

minor comments (4)

[Table 3] The header 'Surrrogate Model' contains a typo and should read 'Surrogate Model'.
[Table 1] The stated count 'Surrogate Model (34)' should be corrected to 35 (or the subcategory entries adjusted); the same arithmetic correction is needed before the table totals can be used in the conclusion.
[Section 2.2.3] The paragraph introducing SHAP says it is used 'globally' in this section, but several listed works (e.g., [309], [35]) are later described in Section 4.1.2 as providing local explanations; the intended distinction between global and local use should be clarified in the text.
[Abstract] The abstract claims 'over 250 papers,' but the only supporting description in Section 1 mentions 12 prior surveys; please state or reference the actual number of distinct works reviewed, once the count is reconciled.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the survey's taxonomy and counts are self-contained descriptive claims, not predictions derived from fitted or self-cited inputs.

full rationale

This is a survey paper, not a derivation or prediction pipeline. There are no equations, no fitted parameters, no benchmark predictions, and no load-bearing self-citations. The central quantitative claim—that 175 works are policy-level, 89 action-level, and 11 sequence-level—is a direct summary of the author's own classification tables. That is the normal function of a taxonomy survey: define categories, assign papers, and report counts. The conclusion that sequence-level explanation is understudied is an interpretation of those counts, not a result that was independently derived and then shown to reduce to its inputs. Even if the counts are difficult to reproduce from the tables or the corpus selection is not fully systematic, those are correctness and rigor concerns, not circularity. The paper does not rename a known empirical result as a new one, does not import a uniqueness theorem from prior work by the same author, and does not fit any parameter and call it a prediction. Therefore the appropriate circularity score is 0.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

The paper contributes a survey and taxonomy; it introduces no free parameters and no invented entities. Its field-level claims rest on two unstated premises: the corpus is representative, and the categories are disjoint enough to count. The second premise is contradicted by the paper's own tables, where the same references appear under multiple levels.

assumptions (2)

domain assumption The surveyed set of roughly 250 papers is representative of the XRL literature.
The paper says it is based on 12 prior state-of-the-arts and complementary papers (Section 1), but gives no systematic search protocol, inclusion criteria, or inter-rater procedure, so its counts and trends assume representativeness.
domain assumption Papers can be assigned to exactly one of the three explanation targets (policy, sequence, action) for counting purposes.
The Conclusion's counts (175/89/11) require disjoint categories, yet [378, 406, 309, 35] are listed in both the policy-level SHAP table (Table 2) and the action-level SHAP table (Table 6), so the assumption of disjointness is not met.

how reviews work

0 comments

Cite this review

Pith. "Pith review of A Survey of Explainable Reinforcement Learning: Targets, Methods and Needs." pith.science (2026). https://pith.science/paper/ZFK7XN5J

@misc{pith2026250712599,
  author       = {Pith},
  title        = {Pith review of: A Survey of Explainable Reinforcement Learning: Targets, Methods and Needs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZFK7XN5J}},
  note         = {Machine review of arXiv:2507.12599}
}

read the original abstract

The success of recent Artificial Intelligence (AI) models has been accompanied by the opacity of their internal mechanisms, due notably to the use of deep neural networks. In order to understand these internal mechanisms and explain the output of these AI models, a set of methods have been proposed, grouped under the domain of eXplainable AI (XAI). This paper focuses on a sub-domain of XAI, called eXplainable Reinforcement Learning (XRL), which aims to explain the actions of an agent that has learned by reinforcement learning. We propose an intuitive taxonomy based on two questions "What" and "How". The first question focuses on the target that the method explains, while the second relates to the way the explanation is provided. We use this taxonomy to provide a state-of-the-art review of over 250 papers. In addition, we present a set of domains close to XRL, which we believe should get attention from the community. Finally, we identify some needs for the field of XRL.

Figures

Figures reproduced from arXiv: 2507.12599 by the authors.

**Figure 2.** Detailed taxonomy for policy-level methods. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Example of Abstract Policy Graph [346]. Each vertex is an abstract state with an associated action, and the edges between two vertices represent transitions (weighted by their probability of occurring). In the same vein, McCalmone et al. [244] propose CAPS, a method that constructs a directed graph composed of abstract states. The state space is abstracted using a decision tree-based clustering algorithm called CLT… view at source ↗

Figures from the paper (16 more)

**Figure 4.** Figure 4: A discrete decision tree for the CartPole envi [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]

**Figure 5.** Figure 5: Landmarks and their ordering obtained using TLdR [ [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]

**Figure 6.** Figure 6: SHAP for a set of states along an episode [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]

**Figure 7.** Figure 7: Explanation via ASAP [36] of the agent’s policy for the Mountain Car environment. The first line shows a representative state for each hyperstate with its associated action. The second line is divided into two parts. On the left, a policy graph models the policy based …

**Figure 8.** Figure 8: A reward function as a binary tree structure [PITH_FULL_IMAGE:figures/full_fig_p020_8.png]

**Figure 9.** Figure 9: PolicyExplainer visual interface [250]. The different tools are divided into three parts. Panels A-C provide a global view of the frequency of actions and rewards, the policy, the value function. Panels D-G provide a detailed analysis of particular states and their ass…

**Figure 10.** Figure 10: Semi-MDP built on top of a t-SNE map for the Atari 2600 Breakout environment [ [PITH_FULL_IMAGE:figures/full_fig_p025_10.png]

**Figure 11.** Figure 11: Detailed taxonomy for sequence-level meth [PITH_FULL_IMAGE:figures/full_fig_p025_11.png]

**Figure 12.** Figure 12: Possible explanations for the sequence of ac [PITH_FULL_IMAGE:figures/full_fig_p026_12.png]

**Figure 13.** Figure 13: An explanation of an episode from a game of Atari 2600 Pong [ [PITH_FULL_IMAGE:figures/full_fig_p027_13.png]

**Figure 14.** Figure 14: Detailed taxonomy for action-level methods. [PITH_FULL_IMAGE:figures/full_fig_p029_14.png]

**Figure 15.** Figure 15: Two saliency maps for a state-action pair from the Ms PacMan environment [ [PITH_FULL_IMAGE:figures/full_fig_p032_15.png]

**Figure 16.** Figure 16: Comparison of counterfactual states in the Atari 2600 Ms PacMan environment [ [PITH_FULL_IMAGE:figures/full_fig_p034_16.png]

**Figure 17.** Figure 17: Causal model for the LunarLander environ [PITH_FULL_IMAGE:figures/full_fig_p035_17.png]

**Figure 18.** Figure 18: RDX for a Human-Robot collaboration task [PITH_FULL_IMAGE:figures/full_fig_p037_18.png]

**Figure 19.** Figure 19: Explanation of each action in an episode in the Taxi environment [ [PITH_FULL_IMAGE:figures/full_fig_p038_19.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

A Differentiable Atari VCS:A Complex, Fully Known Ground Truth for Explainable AI
cs.AI 2026-06 conditional novelty 8.0 of 10

Differentiable reimplementations of the Atari VCS provide a complex, fully known ground-truth system for testing gradient-based explainable AI methods.

Reference graph

Works this paper leans on

300 extracted references · 58 canonical work pages · cited by 1 Pith paper

[35]

Daniel Beechey, Thomas M. S. Smith, and ¨Ozg¨ ur Sim- sek. Explaining reinforcement learning with shap- ley values. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, International Confer- ence on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA , volume 202 of Proceed- in...

2023
[17]

Explain- ing reinforcement learning agents through counter- factual action outcomes

Yotam Amitai, Yael Septon, and Ofra Amir. Explain- ing reinforcement learning agents through counter- factual action outcomes. In Michael J. Wooldridge, Jennifer G. Dy, and Sriraam Natarajan, editors,Thirty- Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence, IAAI 202...

2024
[1]

The regression Tsetlin machine: A Tsetlin machine for continuous output problems

Kuruge Darshana Abeyrathna, Ole-Christoffer Granmo, Lei Jiao, and Morten Goodwin. The regression Tsetlin machine: A Tsetlin machine for continuous output problems. In Paulo Moura Oliveira, Paulo Novais, and Lu ´ ıs Paulo Reis, editors,Progress in Artificial Intelligence, 19th EPIA Conference on Artificial Intelligence, EPIA 2019, Vila Real, Portugal, Sept...

2019
[2]

Explaining Conditions for Reinforcement Learning Behaviors from Real and Imagined Data

Aastha Acharya, Rebecca L. Russell, and Nisar R. Ahmed. Explaining conditions for reinforcement learn- ing behaviors from real and imagined data. CoRR, abs/2011.09004, 2020

work page Pith review arXiv 2011
[3]

Goodfellow, Moritz Hardt, and Been Kim

Julius Adebayo, Justin Gilmer, Michael Muelly, Ian J. Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicol` o Cesa-Bianchi, and Roman Garnett, editors, Advances in Neural Information Processing Systems 31: Annual 43 Conference on Neural Information Processing Sys...

2018
[4]

Symbolic relation networks for reinforcement learning

Dhaval Adjodah, Tim Klinger, and Joshua Joseph. Symbolic relation networks for reinforcement learning. In Proceedings of the Workshop on Relational Represen- tation Learning in Conference on Neural Information Processing Systems (NeurIPS), 2018

2018
[5]

A survey of statistical model checking

Gul Agha and Karl Palmskog. A survey of statistical model checking. ACM Trans. Model. Comput. Simul. , 28(1):6:1–6:39, 2018

2018
[6]

Unsupervised object-level deep reinforcement learning

William Agnew and Pedro Domingos. Unsupervised object-level deep reinforcement learning. In NeurIPS Workshop on Deep RL , 2018

2018

Show all 300 references

[7]

Towards reinforcement learning of human readable policies

Riad Akrour, Davide Tateo, and Jan Peters. Towards reinforcement learning of human readable policies. In The European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases: The 1st Workshop on Deep Continuous- Discrete Machine Learning, 2019

2019
[8]

Explicabilit´ e en apprentissage par renforcement: vers une taxinomie unifi´ ee

Maxime Alaarabiou, Nicolas Delestre, and Laurent Ver- couter. Explicabilit´ e en apprentissage par renforcement: vers une taxinomie unifi´ ee. JIAF-JFPDA, page 90, 2024

2024
[9]

Amal Alabdulkarim and Mark O. Riedl. Experien- tial explanations for reinforcement learning. CoRR, abs/2210.04723, 2022

2022 arXiv
[10]

REACT: revealing evolutionary action con- sequence trajectories for interpretable reinforcement learning

Philipp Altmann, C´ eline Davignon, Maximilian Zorn, Fabian Ritz, Claudia Linnhoff-Popien, and Thomas Gabor. REACT: revealing evolutionary action con- sequence trajectories for interpretable reinforcement learning. CoRR, abs/2404.03359, 2024

2024 arXiv
[11]

Axiomatic foundations of explainability

Leila Amgoud and Jonathan Ben-Naim. Axiomatic foundations of explainability. In Luc De Raedt, edi- tor, Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI 2022, Vi- enna, Austria, 23-29 July 2022 , pages 636–642. ij- cai.org, 2022

2022
[12]

HIGHLIGHTS: summa- rizing agent behavior to people

Dan Amir and Ofra Amir. HIGHLIGHTS: summa- rizing agent behavior to people. In Elisabeth Andr´ e, Sven Koenig, Mehdi Dastani, and Gita Sukthankar, ed- itors, Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, AA- MAS, pages 1168–1176....

2018
[13]

Agent strategy summarization

Ofra Amir, Finale Doshi-Velez, and David Sarne. Agent strategy summarization. In Elisabeth Andr´ e, Sven Koenig, Mehdi Dastani, and Gita Sukthankar, editors, Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS 2018, Stockholm, Sw...

2018
[14]

Sum- marizing agent strategies

Ofra Amir, Finale Doshi-Velez, and David Sarne. Sum- marizing agent strategies. Auton. Agents Multi Agent Syst., 33(5):628–644, 2019

2019
[15]

Weber, Prateek Goel, Owen Brooks, Archer Gandley, Brian Kitchell, and Aaron Zehm

Shideh Shams Amiri, Rosina O. Weber, Prateek Goel, Owen Brooks, Archer Gandley, Brian Kitchell, and Aaron Zehm. Data representing ground-truth explana- tions to evaluate XAI methods. CoRR, abs/2011.09892, 2020

2011 arXiv
[16]

”I don’t think so”: Summarizing policy disagreements for agent compar- ison

Yotam Amitai and Ofra Amir. ”I don’t think so”: Summarizing policy disagreements for agent compar- ison. In Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022, Thirty-Fourth Conference on Innovative Applications of Artificial Intelligence, IAAI 2022, The Twelve...

2022
[18]

Olson, Alan Fern, and Margaret Burnett

Andrew Anderson, Jonathan Dodge, Amrita Sadarangani, Zoe Juozapaitis, Evan Newman, Jed Irvine, Souti Chattopadhyay, Matthew L. Olson, Alan Fern, and Margaret Burnett. Mental models of mere mortals with explanations of reinforcement learning. ACM Trans. Interact. Intell. Syst. ...

2020
[19]

Mod- ular multitask reinforcement learning with policy sketches

Jacob Andreas, Dan Klein, and Sergey Levine. Mod- ular multitask reinforcement learning with policy sketches. In Doina Precup and Yee Whye Teh, ed- itors, Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Aus- tralia, 6-11 August 201...

2017
[20]

To- wards a more efficient computation of individual at- tribute and policy contribution for post-hoc explana- tion of cooperative multi-agent systems using Myerson values

Giorgio Angelotti and Natalia D ´ ıaz-Rodr ´ ıguez. To- wards a more efficient computation of individual at- tribute and policy contribution for post-hoc explana- tion of cooperative multi-agent systems using Myerson values. Knowledge-Based Systems, 260:110189, 2023

2023
[21]

Angelov and Dimitar P

Plamen P. Angelov and Dimitar P. Filev. An approach to online identification of Takagi-Sugeno fuzzy models. IEEE Trans. Syst. Man Cybern. Part B , 34(1):484–498, 2004

2004
[22]

Raghuram Mandyam Annasamy and Katia P. Sycara. Towards better interpretability in deep Q-networks. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AA...

2019
[23]

Arjona-Medina, Michael Gillhofer, Michael Widrich, Thomas Unterthiner, Johannes Brandstetter, and Sepp Hochreiter

Jose A. Arjona-Medina, Michael Gillhofer, Michael Widrich, Thomas Unterthiner, Johannes Brandstetter, and Sepp Hochreiter. RUDDER: return decomposition for delayed rewards. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d’Alch´ e-Buc, Emily B. Fox, and Roman...

2019
[24]

Building predictive models via feature syn- thesis

Ignacio Arnaldo, Una-May O’Reilly, and Kalyan Veera- machaneni. Building predictive models via feature syn- thesis. In Sara Silva and Anna Isabel Esparcia-Alc´ azar, editors, Proceedings of the Genetic and Evolution- ary Computation Conference, GECCO 2015, Madrid, Spain, July ...

2015
[25]

Optimal control of markov pro- cesses with incomplete state information i

Karl Johan ˚Astr¨ om. Optimal control of markov pro- cesses with incomplete state information i. Journal of mathematical analysis and applications , 10:174–205, 1965

1965
[26]

Akanksha Atrey, Kaleigh Clary, and David D. Jensen. Exploratory not explanatory: Counterfactual analysis of saliency maps for deep reinforcement learning. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenRev...

2020
[27]

Hanna, and Guni Sharon

James Ault, Josiah P. Hanna, and Guni Sharon. Learn- ing an interpretable traffic signal control policy. In Amal El Fallah Seghrouchni, Gita Sukthankar, Bo An, and Neil Yorke-Smith, editors, Proceedings of the 19th International Conference on Autonomous Agents and Multiagent S...

2020
[28]

It usu- ally works: The temporal logic of stochastic systems

Adnan Aziz, Vigyan Singhal, and Felice Balarin. It usu- ally works: The temporal logic of stochastic systems. In Pierre Wolper, editor, Computer Aided Verification, 7th International Conference, Li` ege, Belgium, July, 3-5, 1995, Proceedings, volume 939 of Lecture Notes in Com...

1995
[29]

On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation

Sebastian Bach, Alexander Binder, Gr´ egoire Montavon, Frederick Klauschen, Klaus-Robert M¨ uller, and Woj- ciech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one , 10(7):e0130140, 2015

2015
[30]

DRIVE: deep reinforced accident anticipation with visual explana- tion

Wentao Bao, Qi Yu, and Yu Kong. DRIVE: deep reinforced accident anticipation with visual explana- tion. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021 , pages 7599–7608. IEEE, 2021

2021
[31]

Model interpretability through the lens of computational complexity

Pablo Barcel´ o, Mika¨ el Monet, Jorge P´ erez, and Bernardo Subercaseaux. Model interpretability through the lens of computational complexity. In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Informa...

2020
[32]

Pablo V. A. Barros, Ana Tanevska, Francisco Cruz, and Alessandra Sciutti. Moody learners - explain- ing competitive behaviour of reinforcement learning agents. In Joint IEEE 10th International Conference on Development and Learning and Epigenetic Robotics, ICDL-EpiRob 2020, Va...

2020
[33]

Interpretable, verifiable, and robust re- inforcement learning via program synthesis

Osbert Bastani, Jeevana Priya Inala, and Armando Solar-Lezama. Interpretable, verifiable, and robust re- inforcement learning via program synthesis. In Andreas Holzinger, Randy Goebel, Ruth Fong, Taesup Moon, Klaus-Robert M¨ uller, and Wojciech Samek, editors, xxAI - Beyond Ex...

2020
[34]

Verifiable reinforcement learning via policy extraction

Osbert Bastani, Yewen Pu, and Armando Solar- Lezama. Verifiable reinforcement learning via policy extraction. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicol` o Cesa-Bianchi, and Roman Garnett, editors, Advances in Neural Infor- mation Processing Syst...

2018
[36]

ASAP: attention-based state space abstraction for policy sum- marization

Yanzhe Bekkemoen and Helge Langseth. ASAP: attention-based state space abstraction for policy sum- marization. In Berrin Yanikoglu and Wray L. Bun- tine, editors, Asian Conference on Machine Learning, ACML 2023, 11-14 November 2023, Istanbul, Turkey , volume 222 of Proceedings...

2023
[37]

Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling

Marc G. Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. J. Artif. Intell. Res. , 47:253–279, 2013

2013
[38]

Visu- alizing dynamics: from t-SNE to SEMI-MDPs

Nir Ben-Zrihem, Tom Zahavy, and Shie Mannor. Visu- alizing dynamics: from t-SNE to SEMI-MDPs. CoRR, abs/1606.07112, 2016

2016 arXiv
[39]

Approximate policy iteration: A survey and some new methods

Dimitri P Bertsekas. Approximate policy iteration: A survey and some new methods. Journal of Control Theory and Applications , 9(3):310–335, 2011

2011
[40]

Tripletree: A ver- satile interpretable representation of black box agents and their environments

Tom Bewley and Jonathan Lawry. Tripletree: A ver- satile interpretable representation of black box agents and their environments. In Thirty-Fifth AAAI Con- ference on Artificial Intelligence, AAAI 2021, Thirty- Third Conference on Innovative Applications of Arti- ficial Intell...

2021
[41]

Summarising and comparing agent dynamics with contrastive spatiotemporal abstraction

Tom Bewley, Jonathan Lawry, and Arthur Richards. Summarising and comparing agent dynamics with contrastive spatiotemporal abstraction. CoRR, abs/2201.07749, 2022

2022 arXiv
[42]

Interpretable preference-based reinforcement learning with tree- structured reward functions

Tom Bewley and Freddy L´ ecu´ e. Interpretable preference-based reinforcement learning with tree- structured reward functions. In Piotr Faliszewski, Vi- viana Mascardi, Catherine Pelachaud, and Matthew E. Taylor, editors, 21st International Conference on Au- tonomous Agents an...

2022
[43]

Aldo Faisal

Benjamin Beyret, Ali Shafti, and A. Aldo Faisal. Dot- to-dot: Explainable hierarchical reinforcement learning for robotic manipulation. In 2019 IEEE/RSJ Interna- tional Conference on Intelligent Robots and Systems, IROS 2019, Macau, SAR, China, November 3-8, 2019 , pages 5014–...

2019
[44]

Learning ”what-if” explanations for sequential decision-making

Ioana Bica, Daniel Jarrett, Alihan H¨ uy¨ uk, and Mihaela van der Schaar. Learning ”what-if” explanations for sequential decision-making. In 9th International Con- ference on Learning Representations, ICLR 2021, Vir- tual Event, Austria, May 3-7, 2021 . OpenReview.net, 2021

2021
[45]

Explain- able multi-agent reinforcement learning for temporal queries

Kayla Boggess, Sarit Kraus, and Lu Feng. Explain- able multi-agent reinforcement learning for temporal queries. In Proceedings of the Thirty-Second Inter- national Joint Conference on Artificial Intelligence, IJCAI 2023, 19th-25th August 2023, Macao, SAR, China, pages 55–63. i...

2023
[46]

How people explain their own and others’ behavior: a theory of lay causal explanations

Gisela B¨ ohm and Hans-R¨ udiger Pfister. How people explain their own and others’ behavior: a theory of lay causal explanations. Frontiers in psychology, 6:109763, 2015

2015
[47]

Evalu- ating the interpretability of the knowledge compilation map: Communicating logical statements effectively

Serena Booth, Christian Muise, and Julie Shah. Evalu- ating the interpretability of the knowledge compilation map: Communicating logical statements effectively. In Sarit Kraus, editor, Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelli- gence...

2019
[48]

Openai gym, 2016

Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Woj- ciech Zaremba. Openai gym, 2016

2016
[49]

From explanation to synthesis: Compo- sitional program induction for learning from demon- stration

Michael Burke, Svetlin Penkov, and Subramanian Ra- mamoorthy. From explanation to synthesis: Compo- sitional program induction for learning from demon- stration. In Antonio Bicchi, Hadas Kress-Gazit, and Seth Hutchinson, editors, Robotics: Science and Sys- tems XV, University ...

2019
[50]

Klassen, Richard Anthony Valenzano, and Sheila A

Alberto Camacho, Rodrigo Toro Icarte, Toryn Q. Klassen, Richard Anthony Valenzano, and Sheila A. McIlraith. LTL and beyond: Formal languages for reward function specification in reinforcement learning. In Sarit Kraus, editor, Proceedings of the Twenty- Eighth International Joi...

2019
[51]

Explainable ai for path following with model trees

Nicolas Blystad Carbone. Explainable ai for path following with model trees. Master’s thesis, NTNU, 2020. 46

2020
[52]

Michael Cashmore, Anna Collins, Benjamin Krarup, Senka Krivic, Daniele Magazzeni, and David E. Smith. Towards explainable AI planning as a service. CoRR, abs/1908.05059, 2019

1908 arXiv
[53]

The emerging landscape of ex- plainable automated planning & decision making

Tathagata Chakraborti, Sarath Sreedharan, and Sub- barao Kambhampati. The emerging landscape of ex- plainable automated planning & decision making. In Christian Bessiere, editor, Proceedings of the Twenty- Ninth International Joint Conference on Artificial In- telligence, IJCA...

2020
[54]

Plan explanations as model reconciliation: Moving beyond explanation as soliloquy

Tathagata Chakraborti, Sarath Sreedharan, Yu Zhang, and Subbarao Kambhampati. Plan explanations as model reconciliation: Moving beyond explanation as soliloquy. In Carles Sierra, editor, Proceedings of the Twenty-Sixth International Joint Conference on Arti- ficial Intelligenc...

2017
[55]

Coactive design of explain- able agent-based task planning and deep reinforcement learning for human-UA Vs teamwork

Wang Chang, WU Lizhen, YAN Chao, W ANG Zhichao, LONG Han, and YU Chao. Coactive design of explain- able agent-based task planning and deep reinforcement learning for human-UA Vs teamwork. Chinese Journal of Aeronautics, 33(11):2930–2945, 2020

2020
[56]

Interpretable end-to-end urban autonomous driving with latent deep reinforcement learning

Jianyu Chen, Shengbo Eben Li, and Masayoshi Tomizuka. Interpretable end-to-end urban autonomous driving with latent deep reinforcement learning. IEEE Trans. Intell. Transp. Syst. , 23(6):5068–5078, 2022

2022
[57]

Xgboost: A scal- able tree boosting system

Tianqi Chen and Carlos Guestrin. Xgboost: A scal- able tree boosting system. In Balaji Krishnapuram, Mohak Shah, Alexander J. Smola, Charu C. Aggarwal, Dou Shen, and Rajeev Rastogi, editors, Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and...

2016
[58]

Relax: Rein- forcement learning agent explainer for arbitrary pre- dictive models

Ziheng Chen, Fabrizio Silvestri, Jia Wang, He Zhu, Hongshik Ahn, and Gabriele Tolomei. Relax: Rein- forcement learning agent explainer for arbitrary pre- dictive models. In Mohammad Al Hasan and Li Xiong, editors, Proceedings of the 31st ACM International Conference on Informa...

2022
[59]

Stargan: Uni- fied generative adversarial networks for multi-domain image-to-image translation

Yunjey Choi, Min-Je Choi, Munyoung Kim, Jung-Woo Ha, Sunghun Kim, and Jaegul Choo. Stargan: Uni- fied generative adversarial networks for multi-domain image-to-image translation. In 2018 IEEE Confer- ence on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, U...

2018
[60]

Imitation learn- ing of car driving skills with decision trees and random forests

Pawel Cichosz and Lukasz Pawelczak. Imitation learn- ing of car driving skills with decision trees and random forests. Int. J. Appl. Math. Comput. Sci. , 24(3):579– 597, 2014

2014
[61]

Applying and verifying an explain- ability method based on policy graphs in the context of reinforcement learning

Antoni Climent, Dmitry Gnatyshak, and Sergio ´Alvarez-Napagao. Applying and verifying an explain- ability method based on policy graphs in the context of reinforcement learning. In Mateu Villaret, Teresa Alsinet, C` esar Fern´ andez, and A ¨ ıda Valls, editors,Ar- tificial Int...

2021
[62]

On integrating apprentice learn- ing and reinforcement learning

Jeffery Allen Clouse. On integrating apprentice learn- ing and reinforcement learning . University of Mas- sachusetts Amherst, 1996

1996
[63]

Quantifying generalization in reinforcement learning

Karl Cobbe, Oleg Klimov, Christopher Hesse, Taehoon Kim, and John Schulman. Quantifying generalization in reinforcement learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June...

2019
[64]

Symbolic learning for adaptive agents

Joshua Cole, JW Lloyd, and Kee Siong Ng. Symbolic learning for adaptive agents. In Proceedings of the An- nual Partner Conference, Smart Internet Technology Cooperative Research Centre, 2003

2003
[65]

Distilling deep reinforcement learning poli- cies in soft decision trees

Youri Coppens, Kyriakos Efthymiadis, Tom Lenaerts, Ann Now´ e, Tim Miller, Rosina Weber, and Daniele Magazzeni. Distilling deep reinforcement learning poli- cies in soft decision trees. In Proceedings of the IJCAI 2019 workshop on explainable artificial intelligence , pages 1–6, 2019

2019
[66]

Mood and personality in adulthood

Paul T Costa Jr and Robert R McCrae. Mood and personality in adulthood. In Handbook of emotion, adult development, and aging , pages 369–383. Elsevier, 1996

1996
[67]

Memory-based explainable reinforcement learning

Francisco Cruz, Richard Dazeley, and Peter Vamplew. Memory-based explainable reinforcement learning. In Jixue Liu and James Bailey, editors, AI 2019: Ad- vances in Artificial Intelligence - 32nd Australasian Joint Conference, Adelaide, SA, Australia, December 2-5, 2019, Procee...

2019
[68]

Explainable robotic systems: under- standing goal-driven actions in a reinforcement learning scenario

Francisco Cruz, Richard Dazeley, Peter Vamplew, and Ithan Moreira. Explainable robotic systems: under- standing goal-driven actions in a reinforcement learning scenario. Neural Comput. Appl. , 35(25):18113–18130, 2023. 47

2023
[69]

Evolu- tionary learning of interpretable decision trees

Leonardo Lucio Custode and Giovanni Iacca. Evolu- tionary learning of interpretable decision trees. IEEE Access, 11:6169–6184, 2023

2023
[70]

Interpreting a deep reinforcement learning model with conceptual embedding and perfor- mance analysis

Yinglong Dai, Haibin Ouyang, Hong Zheng, Han Long, and Xiaojun Duan. Interpreting a deep reinforcement learning model with conceptual embedding and perfor- mance analysis. Appl. Intell. , 53(6):6936–6952, 2023

2023
[71]

Enhanced oblique decision tree enabled policy extraction for deep rein- forcement learning in power system emergency control

Yuxin Dai, Qimei Chen, Jun Zhang, Xiaohui Wang, Yilin Chen, Tianlu Gao, Peidong Xu, Siyuan Chen, Siyang Liao, Huaiguang Jiang, et al. Enhanced oblique decision tree enabled policy extraction for deep rein- forcement learning in power system emergency control. Electric Power Sy...

2022
[72]

Danesh, Anurag Koul, Alan Fern, and Saeed Khorram

Mohamad H. Danesh, Anurag Koul, Alan Fern, and Saeed Khorram. Re-understanding finite-state repre- sentations of recurrent policy networks. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021...

2021
[73]

Learning sparse evidence- driven interpretation to understand deep reinforcement learning agents

Giang Dao, Wesley Houston Huff, and Minwoo Lee. Learning sparse evidence- driven interpretation to understand deep reinforcement learning agents. In IEEE Symposium Series on Computational Intelligence, SSCI 2021, Orlando, FL, USA, December 5-7, 2021 , pages 1–7. IEEE, 2021

2021
[74]

Deep reinforcement learning monitor for snapshot record- ing

Giang Dao, Indrajeet Mishra, and Minwoo Lee. Deep reinforcement learning monitor for snapshot record- ing. In M. Arif Wani, Mehmed M. Kantardzic, Moamar Sayed Mouchaweh, Jo˜ ao Gama, and Edwin Lughofer, editors, 17th IEEE International Confer- ence on Machine Learning and Appl...

2018
[75]

Fitted Q-learning for relational domains

Srijita Das, Sriraam Natarajan, Kaushik Roy, Ronald Parr, and Kristian Kersting. Fitted Q-learning for relational domains. CoRR, abs/2006.05595, 2020

2006 arXiv
[76]

d’Avila Garcez, Aimore Resende Riquetti Dutra, and Eduardo Alonso

Artur S. d’Avila Garcez, Aimore Resende Riquetti Dutra, and Eduardo Alonso. Towards symbolic re- inforcement learning with common sense. CoRR, abs/1804.08597, 2018

2018 arXiv
[77]

Feature-based interpretable reinforcement learning based on state- transition models

Omid Davoodi and Majid Komeili. Feature-based interpretable reinforcement learning based on state- transition models. In 2021 IEEE International Con- ference on Systems, Man, and Cybernetics, SMC 2021, Melbourne, Australia, October 17-20, 2021 , pages 301–

2021
[78]

Explainable reinforcement learning for broad-XAI: a conceptual framework and survey

Richard Dazeley, Peter Vamplew, and Francisco Cruz. Explainable reinforcement learning for broad-XAI: a conceptual framework and survey. Neural Comput. Appl., 35(23):16893–16916, 2023

2023
[79]

Leonardo Mendon¸ ca de Moura and Nikolaj S. Bjørner. Z3: an efficient SMT solver. In C. R. Ramakrishnan and Jakob Rehof, editors, Tools and Algorithms for the Construction and Analysis of Systems, 14th Interna- tional Conference, TACAS 2008, Held as Part of the Joint European ...

2008
[80]

Learning the structure of factored Markov decision processes in reinforcement learning problems

Thomas Degris, Olivier Sigaud, and Pierre-Henri Wuillemin. Learning the structure of factored Markov decision processes in reinforcement learning problems. In William W. Cohen and Andrew W. Moore, editors, Machine Learning, Proceedings of the Twenty-Third International Confere...

2006
[81]

Cracking open the black box: What observations can tell us about reinforcement learning agents

Arnaud Dethise, Marco Canini, and Srikanth Kandula. Cracking open the black box: What observations can tell us about reinforcement learning agents. In Pro- ceedings of the 2019 Workshop on Network Meets AI & ML, NetAI@SIGCOMM 2019, Beijing, China, August 23, 2019 , pages 29–36...

2019
[82]

Explicable reward de- sign for reinforcement learning agents

Rati Devidze, Goran Radanovic, Parameswaran Ka- malaruban, and Adish Singla. Explicable reward de- sign for reinforcement learning agents. In Marc’Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Ad- vances in Neural Info...

2021
[83]

Dhebar and Kalyanmoy Deb

Yashesh D. Dhebar and Kalyanmoy Deb. Interpretable rule discovery through bilevel optimization of split- rules of nonlinear decision trees for classification prob- lems. IEEE Trans. Cybern., 51(11):5573–5584, 2021

2021
[84]

Dhebar, Kalyanmoy Deb, Subramanya Nageshrao, Ling Zhu, and Dimitar P

Yashesh D. Dhebar, Kalyanmoy Deb, Subramanya Nageshrao, Ling Zhu, and Dimitar P. Filev. Interpretable-AI policies using evolutionary nonlin- ear decision trees for discrete action systems. CoRR, abs/2009.09521, 2020

2009 arXiv
[85]

Dijkstra

Edsger W. Dijkstra. A note on two problems in connex- ion with graphs. Numerische Mathematik , 1:269–271, 1959

1959
[86]

CDT: cascad- ing decision trees for explainable reinforcement learn- ing

Zihan Ding, Pablo Hernandez-Leal, Gavin Weiguang Ding, Changjian Li, and Ruitong Huang. CDT: cascad- ing decision trees for explainable reinforcement learn- ing. CoRR, abs/2011.07553, 2020. 48

2011 arXiv
[87]

Blies, Johannes Brandstetter, Jose A

Marius-Constantin Dinu, Markus Hofmarcher, Vi- hang Prakash Patil, Matthias Dorfer, Patrick M. Blies, Johannes Brandstetter, Jose A. Arjona-Medina, and Sepp Hochreiter. XAI and strategy extraction via re- ward redistribution. In Andreas Holzinger, Randy Goebel, Ruth Fong, Taes...

2020
[88]

Carlos Diuk, Andre Cohen, and Michael L. Littman. An object-oriented representation for efficient reinforce- ment learning. In William W. Cohen, Andrew McCal- lum, and Sam T. Roweis, editors, Machine Learning, Proceedings of the Twenty-Fifth International Confer- ence (ICML 20...

2008
[89]

no clear winner

Jonathan Dodge, Andrew Anderson, Roli Khanna, Jed Irvine, Rupika Dikkala, Kin-Ho Lam, Delyar Tabatabai, Anita Ruangrotsakun, Zeyad Shureih, Min- suk Kahng, et al. From “no clear winner” to an effective explainable artificial intelligence process: An empirical journey. Applied ...

2021
[90]

A natural language argumentation interface for explanation generation in Markov decision processes

Thomas Dodson, Nicholas Mattei, and Judy Gold- smith. A natural language argumentation interface for explanation generation in Markov decision processes. In Ronen I. Brafman, Fred S. Roberts, and Alexis Tsouki` as, editors,Algorithmic Decision Theory - Sec- ond International C...

2011
[91]

Explaining the behaviour of reinforcement learning agents in a multi-agent cooperative environ- ment using policy graphs

Marc Domenech i Vila, Dmitry Gnatyshak, Adrian Tormos, Victor Gimenez-Abalos, and Sergio Alvarez- Napagao. Explaining the behaviour of reinforcement learning agents in a multi-agent cooperative environ- ment using policy graphs. Electronics, 13(3):573, 2024

2024
[92]

Neural logic machines

Honghua Dong, Jiayuan Mao, Tian Lin, Chong Wang, Lihong Li, and Denny Zhou. Neural logic machines. CoRR, abs/1904.11694, 2019

1904 arXiv
[93]

Towards a rigorous science of interpretable machine learning, 2017

Finale Doshi-Velez and Been Kim. Towards a rigorous science of interpretable machine learning, 2017

2017
[94]

Tay- lor

Nathan Douglas, Dianna Yim, Bilal Kartal, Pablo Hernandez-Leal, Frank Maurer, and Matthew E. Tay- lor. Towers of saliency: A reinforcement learning visu- alization using immersive environments. In Bongshin Lee, Geehyuk Lee, Stacey D. Scott, Melanie Tory, and Jeonghyun Kim, edi...

2019
[95]

Learning digger using hierarchical reinforcement learning for concurrent goals

Kurt Driessens and Hendrik Blockeel. Learning digger using hierarchical reinforcement learning for concurrent goals. In Proceedings of the European Workshop on Reinforcement Learning, pages 11–12. CKI Utrecht University, 2001

2001
[96]

Ex- plainable artificial intelligence (XAI) for increasing user trust in deep reinforcement learning driven au- tonomous systems

Jeff Druce, Michael Harradon, and James Tittle. Ex- plainable artificial intelligence (XAI) for increasing user trust in deep reinforcement learning driven au- tonomous systems. CoRR, abs/2106.03775, 2021

2021 arXiv
[97]

Relational reinforcement learning

Saso Dzeroski, Luc De Raedt, and Kurt Driessens. Relational reinforcement learning. Mach. Learn. , 43(1/2):7–52, 2001

2001
[98]

A tale of two explanations: Enhancing human trust by explaining robot behavior

Mark Edmonds, Feng Gao, Hangxin Liu, Xu Xie, Siyuan Qi, Brandon Rothrock, Yixin Zhu, Ying Nian Wu, Hongjing Lu, and Song-Chun Zhu. A tale of two explanations: Enhancing human trust by explaining robot behavior. Sci. Robotics, 4(37), 2019

2019
[99]

Upol Ehsan, Brent Harrison, Larry Chan, and Mark O. Riedl. Rationalization: A neural machine transla- tion approach to generating natural language explana- tions. In Jason Furman, Gary E. Marchant, Huw Price, and Francesca Rossi, editors, Proceedings of the 2018 AAAI/ACM Confe...

2018
[100]

Upol Ehsan, Pradyumna Tambwekar, Larry Chan, Brent Harrison, and Mark O. Riedl. Automated ratio- nale generation: a technique for explainable AI and its effects on human perceptions. In Wai-Tat Fu, Shimei Pan, Oliver Brdiczka, Polo Chau, and Gaelle Calvary, editors, Proceeding...

2019
[101]

A new approach to plan-space explanation: Analyzing plan- property dependencies in oversubscription planning

Rebecca Eifler, Michael Cashmore, J¨ org Hoffmann, Daniele Magazzeni, and Marcel Steinmetz. A new approach to plan-space explanation: Analyzing plan- property dependencies in oversubscription planning. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020,...

2020
[102]

Iterative planning with plan-space explanations: A tool and user study

Rebecca Eifler and J¨ org Hoffmann. Iterative planning with plan-space explanations: A tool and user study. CoRR, abs/2011.09705, 2020. 49

2011 arXiv
[103]

Generating explanations based on Markov decision processes

Francisco Elizalde, Luis Enrique Sucar, Julieta Noguez, and Alberto Reyes. Generating explanations based on Markov decision processes. In Arturo Hern´ andez Aguirre, Ra´ ul Monroy Borja, and Carlos A. Reyes Garc ´ ıa, editors,MICAI 2009: Advances in Artificial Intelligence, 8t...

2009
[104]

Tree-based batch mode reinforcement learning

Damien Ernst, Pierre Geurts, and Louis Wehenkel. Tree-based batch mode reinforcement learning. J. Mach. Learn. Res., 6:503–556, 2005

2005
[105]

Explaining deep adaptive programs via reward decomposition

Martin Erwig, Alan Fern, Magesh Murali, and Anurag Koul. Explaining deep adaptive programs via reward decomposition. In IJCAI/ECAI workshop on explain- able artificial intelligence , 2018

2018
[106]

Learning explanatory rules from noisy data

Richard Evans and Edward Grefenstette. Learning explanatory rules from noisy data. J. Artif. Intell. Res., 61:1–64, 2018

2018
[107]

Search on the replay buffer: Bridging planning and reinforcement learning

Ben Eysenbach, Ruslan Salakhutdinov, and Sergey Levine. Search on the replay buffer: Bridging planning and reinforcement learning. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d’Alch´ e- Buc, Emily B. Fox, and Roman Garnett, editors, Ad- vances in Neural I...

2019
[108]

Ex- plaining online reinforcement learning decisions of self- adaptive systems

Felix Feit, Andreas Metzger, and Klaus Pohl. Ex- plaining online reinforcement learning decisions of self- adaptive systems. In Roberto Casadei, Elisabetta Di Nitto, Ilias Gerostathopoulos, Danilo Pianini, Ivana Dusparic, Timothy Wood, Phyllis R. Nelson, Evangelos Pournaras, N...

2022
[109]

Parkes, Jeffrey S

Mira Finkelstein, Lucy Liu, Yoav Kolumbus, David C. Parkes, Jeffrey S. Rosenshein, and Sarah Keren. Rein- forcement learning explainability via model transforms (student abstract). In Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022, Thirty-Fourth Conference ...

2022
[110]

Ex- plainable planning

Maria Fox, Derek Long, and Daniele Magazzeni. Ex- plainable planning. CoRR, abs/1709.10256, 2017

2017 arXiv
[111]

Combined reinforcement learn- ing via abstract representations

Vincent Fran¸ cois-Lavet, Yoshua Bengio, Doina Precup, and Joelle Pineau. Combined reinforcement learn- ing via abstract representations. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Ar- tificial Intelli...

2019
[112]

Nicholas Frosst and Geoffrey E. Hinton. Distilling a neural network into a soft decision tree. In Tarek R. Besold and Oliver Kutz, editors, Proceedings of the First International Workshop on Comprehensibility and Explanation in AI and ML 2017 co-located with 16th International...

2017
[113]

Plummer, and Kate Saenko

Julius Frost, Olivia Watkins, Eric Weiner, Pieter Abbeel, Trevor Darrell, Bryan A. Plummer, and Kate Saenko. Explaining reinforcement learning policies through counterfactual trajectories. CoRR, abs/2201.12462, 2022

2022 arXiv
[114]

Learning robust rewards with adversarial inverse reinforcement learning

Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse reinforcement learning. CoRR, abs/1710.11248, 2017

2017 arXiv
[115]

Application of instruction-based behavior explanation to a reinforcement learning agent with changing policy

Yosuke Fukuchi, Masahiko Osawa, Hiroshi Yamakawa, and Michita Imai. Application of instruction-based behavior explanation to a reinforcement learning agent with changing policy. In Derong Liu, Shengli Xie, Yuanqing Li, Dongbin Zhao, and El-Sayed M. El-Alfy, editors, Neural Inf...

2017
[116]

Autonomous self-explanation of be- havior for interactive reinforcement learning agents

Yosuke Fukuchi, Masahiko Osawa, Hiroshi Yamakawa, and Michita Imai. Autonomous self-explanation of be- havior for interactive reinforcement learning agents. In Britta Wrede, Yukie Nagai, Takanori Komatsu, Marc Hanheide, and Lorenzo Natale, editors, Proceedings of the 5th Inter...

2017
[117]

Induction and exploitation of subgoal automata for reinforcement learning

Daniel Furelos-Blanco, Mark Law, Anders Jonsson, Krysia Broda, and Alessandra Russo. Induction and exploitation of subgoal automata for reinforcement learning. J. Artif. Intell. Res. , 70:1031–1116, 2021

2021
[118]

Reccover: De- tecting causal confusion for explainable reinforcement 50 learning

Jasmina Gajcin and Ivana Dusparic. Reccover: De- tecting causal confusion for explainable reinforcement 50 learning. In Davide Calvaresi, Amro Najjar, Michael Winikoff, and Kary Fr¨ amling, editors,Explainable and Transparent AI and Multi-Agent Systems - 4th Interna- tional Wo...

2022
[119]

ACTER: diverse and actionable counterfactual sequences for explaining and diagnosing RL policies

Jasmina Gajcin and Ivana Dusparic. ACTER: diverse and actionable counterfactual sequences for explaining and diagnosing RL policies. CoRR, abs/2402.06503, 2024

2024 arXiv
[120]

RACCER: to- wards reachable and certain counterfactual explana- tions for reinforcement learning

Jasmina Gajcin and Ivana Dusparic. RACCER: to- wards reachable and certain counterfactual explana- tions for reinforcement learning. In Mehdi Dastani, Jaime Sim˜ ao Sichman, Natasha Alechina, and Virginia Dignum, editors, Proceedings of the 23rd International Conference on Aut...

2024
[121]

Redefining counterfactual explanations for reinforcement learn- ing: Overview, challenges and opportunities

Jasmina Gajcin and Ivana Dusparic. Redefining counterfactual explanations for reinforcement learn- ing: Overview, challenges and opportunities. ACM Comput. Surv. , 56(9):219:1–219:33, 2024

2024
[122]

Con- trastive explanations for comparing preferences of re- inforcement learning agents

Jasmina Gajcin, Rahul Nair, Tejaswini Pedapati, Radu Marinescu, Elizabeth Daly, and Ivana Dusparic. Con- trastive explanations for comparing preferences of re- inforcement learning agents. CoRR, abs/2112.09462, 2021

2021 arXiv
[123]

Maor Gaon and Ronen I. Brafman. Reinforcement learning with non-Markovian rewards. In The Thirty- Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applica- tions of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposiu...

2020
[124]

Towards deep symbolic reinforcement learning

Marta Garnelo, Kai Arulkumaran, and Murray Shana- han. Towards deep symbolic reinforcement learning. CoRR, abs/1609.05518, 2016

2016 arXiv
[125]

A survey on interpretable reinforcement learning

Claire Glanois, Paul Weng, Matthieu Zimmer, Dong Li, Tianpei Yang, Jianye Hao, and Wulong Liu. A survey on interpretable reinforcement learning. Mach. Learn., 113(8):5847–5890, 2024

2024
[126]

Coming up with good excuses: What to do when no plan can be found

Moritz G¨ obelbecker, Thomas Keller, Patrick Eyerich, Michael Brenner, and Bernhard Nebel. Coming up with good excuses: What to do when no plan can be found. In Ronen I. Brafman, Hector Geffner, J¨ org Hoffmann, and Henry A. Kautz, editors, Proceedings of the 20th Internationa...

2010
[127]

Un- supervised video object segmentation for deep rein- forcement learning

Vikash Goel, Jameson Weng, and Pascal Poupart. Un- supervised video object segmentation for deep rein- forcement learning. In Samy Bengio, Hanna M. Wal- lach, Hugo Larochelle, Kristen Grauman, Nicol` o Cesa- Bianchi, and Roman Garnett, editors, Advances in Neural Information P...

2018
[128]

Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C

Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial networks. CoRR, abs/1406.2661, 2014

2014 arXiv
[129]

Celi, Emma Brunskill, and Finale Doshi-Velez

Omer Gottesman, Joseph Futoma, Yao Liu, Sonali Parbhoo, Leo A. Celi, Emma Brunskill, and Finale Doshi-Velez. Interpretable off-policy evaluation in re- inforcement learning by highlighting influential tran- sitions. In Proceedings of the 37th International Con- ference on Mach...

2020
[130]

The Tsetlin machine - A game theoretic bandit driven approach to optimal pattern recognition with propositional logic

Ole-Christoffer Granmo. The Tsetlin machine - A game theoretic bandit driven approach to optimal pattern recognition with propositional logic. CoRR, abs/1804.01508, 2018

2018 arXiv
[131]

Visualizing and understanding Ataria- gents

Samuel Greydanus, Anurag Koul, Jonathan Dodge, and Alan Fern. Visualizing and understanding Ataria- gents. In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsm¨ assan, Stockholm, Sweden, Jul...

2018
[132]

Gros, David Groß, Stefan Gumhold, J¨ org Hoff- mann, Michaela Klauck, and Marcel Steinmetz

Timo P. Gros, David Groß, Stefan Gumhold, J¨ org Hoff- mann, Michaela Klauck, and Marcel Steinmetz. Trace- vis: Towards visualization for deep statistical model checking. In Tiziana Margaria and Bernhard Steffen, editors, Leveraging Applications of Formal Methods, Verification...

2020
[133]

Gros, Holger Hermanns, J¨ org Hoffmann, Michaela Klauck, and Marcel Steinmetz

Timo P. Gros, Holger Hermanns, J¨ org Hoffmann, Michaela Klauck, and Marcel Steinmetz. Deep sta- tistical model checking. In Alexey Gotsman and Ana Sokolova, editors, Formal Techniques for Distributed Objects, Components, and Systems - 40th IFIP WG 6.1 International Conference...

2020
[134]

π-light: Programmatic interpretable re- inforcement learning for resource-limited traffic signal control

Yin Gu, Kai Zhang, Qi Liu, Weibo Gao, Longfei Li, and Jun Zhou. π-light: Programmatic interpretable re- inforcement learning for resource-limited traffic signal control. In Michael J. Wooldridge, Jennifer G. Dy, and Sriraam Natarajan, editors, Thirty-Eighth AAAI Con- ference o...

2024
[135]

Generalizing plans to new environments in relational MDPs

Carlos Guestrin, Daphne Koller, Chris Gearhart, and Neal Kanodia. Generalizing plans to new environments in relational MDPs. In Georg Gottlob and Toby Walsh, editors, IJCAI-03, Proceedings of the Eighteenth In- ternational Joint Conference on Artificial Intelligence, Acapulco,...

2003
[136]

Ballard, Mary M

Sihang Guo, Ruohan Zhang, Bo Liu, Yifeng Zhu, Dana H. Ballard, Mary M. Hayhoe, and Peter Stone. Machine versus human attention in deep reinforce- ment learning tasks. In Marc’Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jen- nifer Wortman Vaughan, edit...

2021
[137]

Explainable deep reinforce- ment learning for aircraft separation assurance

Wei Guo and Peng Wei. Explainable deep reinforce- ment learning for aircraft separation assurance. In2022 IEEE/AIAA 41st Digital Avionics Systems Conference (DASC), pages 1–10. IEEE, 2022

2022
[138]

EDGE: explaining deep reinforcement learning policies

Wenbo Guo, Xian Wu, Usmann Khan, and Xinyu Xing. EDGE: explaining deep reinforcement learning policies. In Marc’Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, NeurIPS, pages 12222–12236, 2021

2021
[139]

Policy tree: Adaptive representation for policy gradient

Ujjwal Das Gupta, Erik Talvitie, and Michael Bowl- ing. Policy tree: Adaptive representation for policy gradient. In Blai Bonet and Sven Koenig, editors, Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, January 25-30, 2015, Austin, Texas, USA, pages ...

2015
[140]

Programming by example

Daniel Conrad Halbert. Programming by example . University of California, Berkeley, 1984

1984
[141]

Halpern and Judea Pearl

Joseph Y. Halpern and Judea Pearl. Causes and ex- planations: A structural-model approach - part II: explanations. In Bernhard Nebel, editor, Proceedings of the Seventeenth International Joint Conference on Artificial Intelligence, IJCAI 2001, Seattle, Washing- ton, USA, Augus...

2001
[142]

Deepsynth: Automata synthesis for auto- matic task segmentation in deep reinforcement learn- ing

Mohammadhosein Hasanbeig, Natasha Yogananda Jeppu, Alessandro Abate, Tom Melham, and Daniel Kroening. Deepsynth: Automata synthesis for auto- matic task segmentation in deep reinforcement learn- ing. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty...

2021
[143]

Escaping the big data paradigm with compact transformers

Ali Hassani, Steven Walton, Nikhil Shah, Abulikemu Abuduweili, Jiachen Li, and Humphrey Shi. Escaping the big data paradigm with compact transformers. CoRR, abs/2104.05704, 2021

2021 arXiv
[144]

Bradley Hayes and Julie A. Shah. Improving robot controller transparency through autonomous policy explanation. In Bilge Mutlu, Manfred Tscheligi, Astrid Weiss, and James E. Young, editors, Proceedings of the 2017 ACM/IEEE International Conference on Human- Robot Interaction, ...

2017
[145]

Deep explain- able relational reinforcement learning: A neuro- symbolic approach

Rishi Hazra and Luc De Raedt. Deep explain- able relational reinforcement learning: A neuro- symbolic approach. In Danai Koutra, Claudia Plant, Manuel Gomez Rodriguez, Elena Baralis, and Francesco Bonchi, editors, Machine Learning and Knowledge Discovery in Databases: Research...

2023
[146]

Explainable deep reinforcement learning for UA V autonomous path plan- ning

Lei He, Nabil Aouf, and Bifeng Song. Explainable deep reinforcement learning for UA V autonomous path plan- ning. Aerospace science and technology, 118:107052, 2021

2021
[147]

Dynamicsexplorer: Visual analytics for robot control tasks involving dynamics and LSTM-based control policies

Wenbin He, Teng-Yok Lee, Jeroen van Baar, Kent Wit- tenburg, and Han-Wei Shen. Dynamicsexplorer: Visual analytics for robot control tasks involving dynamics and LSTM-based control policies. In 2020 IEEE Pa- cific Visualization Symposium, PacificVis 2020, Tian- jin, China, June...

2020
[148]

Runkler, and Steffen Udluft

Daniel Hein, Alexander Hentschel, Thomas A. Runkler, and Steffen Udluft. Particle swarm optimization for generating interpretable fuzzy reinforcement learning policies. Eng. Appl. Artif. Intell. , 65:87–98, 2017. 52

2017
[149]

Runk- ler

Daniel Hein, Steffen Udluft, and Thomas A. Runk- ler. Generating interpretable reinforcement learning policies using genetic programming. In Manuel L´ opez- Ib´ a˜ nez, Anne Auger, and Thomas St¨ utzle, editors, Proceedings of the Genetic and Evolutionary Compu- tation Confere...

2019
[150]

Reinforcement learn- ing of causal variables using mediation analysis

Tue Herlau and Rasmus Larsen. Reinforcement learn- ing of causal variables using mediation analysis. In Thirty-Sixth AAAI Conference on Artificial Intelli- gence, AAAI 2022, Thirty-Fourth Conference on In- novative Applications of Artificial Intelligence, IAAI 2022, The Twelve...

2022
[151]

Causal Shapley values: Exploiting causal knowledge to explain individual predictions of complex models

Tom Heskes, Evi Sijben, Ioan Gabriel Bucur, and Tom Claassen. Causal Shapley values: Exploiting causal knowledge to explain individual predictions of complex models. In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Adva...

2020
[152]

Rainbow: Combining improvements in deep reinforcement learning

Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Hor- gan, Bilal Piot, Mohammad Gheshlaghi Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Sheila A. McIlraith and Kilian Q. Weinberger, edi...

2018
[153]

Explainability in deep rein- forcement learning

Alexandre Heuillet, Fabien Couthouis, and Na- talia D ´ ıaz Rodr ´ ıguez. Explainability in deep rein- forcement learning. Knowl. Based Syst. , 214:106685, 2021

2021
[154]

Collective explainable AI: ex- plaining cooperative strategies and agent contribution in multiagent reinforcement learning with Shapley val- ues

Alexandre Heuillet, Fabien Couthouis, and Na- talia D ´ ıaz Rodr ´ ıguez. Collective explainable AI: ex- plaining cooperative strategies and agent contribution in multiagent reinforcement learning with Shapley val- ues. IEEE Comput. Intell. Mag. , 17(1):59–71, 2022

2022
[155]

Explainability in deep reinforce- ment learning: A review into current methods and applications

Thomas Hickling, Abdelhafid Zenati, Nabil Aouf, and Phillippa Spencer. Explainability in deep reinforce- ment learning: A review into current methods and applications. ACM Comput. Surv. , 56(5):125:1–125:35, 2024

2024
[156]

Understanding RL vision

Jacob Hilton, Nick Cammarata, Shan Carter, Gabriel Goh, and Chris Olah. Understanding RL vision. Distill, 5(11):e29, 2020

2020
[157]

Long short- term memory

Sepp Hochreiter and J¨ urgen Schmidhuber. Long short- term memory. Neural Comput., 9(8):1735–1780, 1997

1997
[158]

Hoffman, Shane T

Robert R. Hoffman, Shane T. Mueller, Gary Klein, and Jordan Litman. Metrics for explainable AI: challenges and prospects. CoRR, abs/1812.04608, 2018

2018 arXiv
[159]

Explainable AI planning (XAIP): overview and the case of contrastive explanation (extended abstract)

J¨ org Hoffmann and Daniele Magazzeni. Explainable AI planning (XAIP): overview and the case of contrastive explanation (extended abstract). In Markus Kr¨ otzsch and Daria Stepanova, editors, Reasoning Web. Explain- able Artificial Intelligence - 15th International Summer Scho...

2019
[160]

Angelov, and Chengliang Yin

Jianfeng Huang, Plamen P. Angelov, and Chengliang Yin. Interpretable policies for reinforcement learn- ing by empirical fuzzy sets. Eng. Appl. Artif. Intell. , 91:103559, 2020

2020
[161]

Huang, Kush Bhatia, Pieter Abbeel, and Anca D

Sandy H. Huang, Kush Bhatia, Pieter Abbeel, and Anca D. Dragan. Establishing appropriate trust via critical states. In 2018 IEEE/RSJ International Con- ference on Intelligent Robots and Systems, IROS 2018, Madrid, Spain, October 1-5, 2018 , pages 3929–3936. IEEE, 2018

2018
[162]

Huang, David Held, Pieter Abbeel, and Anca D

Sandy H. Huang, David Held, Pieter Abbeel, and Anca D. Dragan. Enabling robots to communicate their objectives. Auton. Robots, 43(2):309–326, 2019

2019
[163]

Olson, and Elisabeth Andr´ e

Tobias Huber, Maximilian Demmler, Silvan Mertes, Matthew L. Olson, and Elisabeth Andr´ e. GANterfactual-RL: Understanding reinforcement learning agents’ strategies through visual counterfac- tual explanations. In Noa Agmon, Bo An, Alessandro Ricci, and William Yeoh, editors, P...

2023
[164]

Benchmarking perturbation-based saliency maps for explaining atari agents

Tobias Huber, Benedikt Limmer, and Elisabeth Andr´ e. Benchmarking perturbation-based saliency maps for explaining atari agents. Frontiers Artif. Intell., 5, 2022

2022
[165]

Enhancing explainability of deep reinforcement learn- ing through selective layer-wise relevance propagation

Tobias Huber, Dominik Schiller, and Elisabeth Andr´ e. Enhancing explainability of deep reinforcement learn- ing through selective layer-wise relevance propagation. In Christoph Benzm¨ uller and Heiner Stuckenschmidt, editors, KI 2019: Advances in Artificial Intelligence - 42n...

2019
[166]

Local and global explanations of agent behavior: Integrating strategy summaries with saliency maps

Tobias Huber, Katharina Weitz, Elisabeth Andr´ e, and Ofra Amir. Local and global explanations of agent behavior: Integrating strategy summaries with saliency maps. Artif. Intell. , 301:103571, 2021

2021
[167]

Explaining by imitating: Understanding decisions by interpretable policy learning

Alihan H¨ uy¨ uk, Daniel Jarrett, and Mihaela van der Schaar. Explaining by imitating: Understanding decisions by interpretable policy learning. CoRR, abs/2310.19831, 2023

2023 arXiv
[168]

Klassen, Richard An- thony Valenzano, and Sheila A

Rodrigo Toro Icarte, Toryn Q. Klassen, Richard An- thony Valenzano, and Sheila A. McIlraith. Using re- ward machines for high-level task specification and decomposition in reinforcement learning. In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th Internatio...

2018
[169]

Klassen, Richard Anthony Valenzano, Margarita P

Rodrigo Toro Icarte, Ethan Waldie, Toryn Q. Klassen, Richard Anthony Valenzano, Margarita P. Castro, and Sheila A. McIlraith. Learning reward machines for par- tially observable reinforcement learning. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d’Alch´ e...

2019
[170]

Synthesizing program- matic policies that inductively generalize

Jeevana Priya Inala, Osbert Bastani, Zenna Tavares, and Armando Solar-Lezama. Synthesizing program- matic policies that inductively generalize. In 8th In- ternational Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview.net, 2020

2020
[171]

Vi- sual explanation using attention mechanism in actor- critic-based deep reinforcement learning

Hidenori Itaya, Tsubasa Hirakawa, Takayoshi Ya- mashita, Hironobu Fujiyoshi, and Komei Sugiura. Vi- sual explanation using attention mechanism in actor- critic-based deep reinforcement learning. In Interna- tional Joint Conference on Neural Networks, IJCNN 2021, Shenzhen, Chin...

2021
[172]

Explainable reinforcement learning for human-robot collaboration

Alessandro Iucci, Alberto Hata, Ahmad Terra, Rafia Inam, and Iolanda Leite. Explainable reinforcement learning for human-robot collaboration. In 20th In- ternational Conference on Advanced Robotics, ICAR 2021, Ljubljana, Slovenia, December 6-10, 2021 , pages 927–934. IEEE, 2021

2021
[173]

Rahul Iyer, Yuezhang Li, Huao Li, Michael Lewis, Ramitha Sundar, and Katia P. Sycara. Transparency and explanation in deep reinforcement learning neural networks. In Jason Furman, Gary E. Marchant, Huw Price, and Francesca Rossi, editors, Proceedings of the 2018 AAAI/ACM Confe...

2018
[174]

A new, node-focused model for ge- netic programming

David Jackson. A new, node-focused model for ge- netic programming. In Alberto Moraglio, Sara Silva, Krzysztof Krawiec, Penousal Machado, and Carlos Cotta, editors, Genetic Programming - 15th European Conference, EuroGP 2012, M´ alaga, Spain, April 11-13,

2012
[175]

Jacobs, Michael I

Robert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton. Adaptive mixtures of local experts. Neural Comput., 3(1):79–87, 1991

1991
[176]

Lazy-MDPs: Towards interpretable reinforcement learning by learning when to act

Alexis Jacq, Johan Ferret, Olivier Pietquin, and Matthieu Geist. Lazy-MDPs: Towards interpretable reinforcement learning by learning when to act. CoRR, abs/2203.08542, 2022

2022 arXiv
[177]

Optimization of the Quine- McCluskey method for the minimization of the boolean expressions

Tarun Kumar Jain, Dharmender Singh Kushwaha, and Arun Kumar Misra. Optimization of the Quine- McCluskey method for the minimization of the boolean expressions. In Fourth International Conference on Autonomic and Autonomous Systems, ICAS 2008, 16- 21 March 2008, Gosier, Guadelo...

2008
[178]

Drlviz: Understanding decisions and memory in deep reinforcement learning

Theo Jaunet, Romain Vuillemot, and Christian Wolf. Drlviz: Understanding decisions and memory in deep reinforcement learning. Comput. Graph. Forum, 39(3):49–61, 2020

2020
[179]

Preprocessing reward functions for interpretability

Erik Jenner and Adam Gleave. Preprocessing reward functions for interpretability. CoRR, abs/2203.13553, 2022

2022 arXiv
[180]

Software model checking

Ranjit Jhala and Rupak Majumdar. Software model checking. ACM Comput. Surv. , 41(4):21:1–21:54, 2009

2009
[181]

Improved policy extraction via online Q-value distillation

Aman Jhunjhunwala, Jaeyoung Lee, Sean Sedwards, Vahdat Abdelzad, and Krzysztof Czarnecki. Improved policy extraction via online Q-value distillation. In 2020 International Joint Conference on Neural Net- works, IJCNN 2020, Glasgow, United Kingdom, July 19-24, 2020 , pages 1–8....

2020
[182]

Language as an abstraction for hierarchical deep reinforcement learning

Yiding Jiang, Shixiang Gu, Kevin Murphy, and Chelsea Finn. Language as an abstraction for hierarchical deep reinforcement learning. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d’Alch´ e-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Inf...

2019
[183]

Neural logic reinforce- ment learning

Zhengyao Jiang and Shan Luo. Neural logic reinforce- ment learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th Interna- tional Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA , vol- ume 97 of Proceedi...

2019
[184]

Creativity of AI: automatic symbolic option discovery for facilitating deep rein- forcement learning

Mu Jin, Zhihao Ma, Kebing Jin, Hankz Hankui Zhuo, Chen Chen, and Chao Yu. Creativity of AI: automatic symbolic option discovery for facilitating deep rein- forcement learning. In Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022, Thirty-Fourth Conference on In...

2022
[185]

Visualization of deep reinforcement learning using Grad-CAM: How AI plays Atarigames? In IEEE Conference on Games, CoG 2019, London, United Kingdom, August 20-23, 2019, pages 1–2

Ho-Taek Joo and Kyung-Joong Kim. Visualization of deep reinforcement learning using Grad-CAM: How AI plays Atarigames? In IEEE Conference on Games, CoG 2019, London, United Kingdom, August 20-23, 2019, pages 1–2. IEEE, 2019

2019
[186]

Deep reinforcement learning for safe local planning of a ground vehicle in unknown rough terrain

Shirel Josef and Amir Degani. Deep reinforcement learning for safe local planning of a ground vehicle in unknown rough terrain. IEEE Robotics Autom. Lett. , 5(4):6748–6755, 2020

2020
[187]

Explainable reinforcement learning via reward decomposition

Zoe Juozapaitis, Anurag Koul, Alan Fern, Martin Er- wig, and Finale Doshi-Velez. Explainable reinforcement learning via reward decomposition. In IJCAI/ECAI Workshop on explainable artificial intelligence , 2019

2019
[188]

Runkler, and Carl Henrik Ek

Markus Kaiser, Clemens Otte, Thomas A. Runkler, and Carl Henrik Ek. interpretable dynamics models for data-efficient reinforcement learning. In 27th European Symposium on Artificial Neural Networks, ESANN 2019, Bruges, Belgium, April 24-26, 2019 , 2019

2019
[189]

Explaining sympathetic actions of rational agents

Timotheus Kampik, Juan Carlos Nieves, and Helena Lindgren. Explaining sympathetic actions of rational agents. In Davide Calvaresi, Amro Najjar, Michael Schumacher, and Kary Fr¨ amling, editors,Explainable, Transparent Autonomous Agents and Multi-Agent Sys- tems - First Interna...

2019
[190]

M´ ely, Mohamed Eldawy, Miguel L´ azaro-Gredilla, Xinghua Lou, Nimrod Dorfman, Szymon Sidor, D

Ken Kansky, Tom Silver, David A. M´ ely, Mohamed Eldawy, Miguel L´ azaro-Gredilla, Xinghua Lou, Nimrod Dorfman, Szymon Sidor, D. Scott Phoenix, and Dileep George. Schema networks: Zero-shot transfer with a generative causal model of intuitive physics. In Doina Precup and Yee W...

2017
[191]

The mario AI benchmark and competitions

Sergey Karakovskiy and Julian Togelius. The mario AI benchmark and competitions. IEEE Trans. Comput. Intell. AI Games , 4(1):55–67, 2012

2012
[192]

A survey of algorith- mic recourse: definitions, formulations, solutions, and prospects

Amir-Hossein Karimi, Gilles Barthe, Bernhard Sch¨ olkopf, and Isabel Valera. A survey of algorith- mic recourse: definitions, formulations, solutions, and prospects. CoRR, abs/2010.04050, 2020

2010 arXiv
[193]

Algorithmic recourse: from counterfactual explanations to interventions

Amir-Hossein Karimi, Bernhard Sch¨ olkopf, and Isabel Valera. Algorithmic recourse: from counterfactual explanations to interventions. In Madeleine Clare Elish, William Isaac, and Richard S. Zemel, editors, F AccT ’21: 2021 ACM Conference on Fairness, Accountability, and Trans...

2021
[194]

Interpretable apprenticeship learning with temporal logic specifica- tions

Daniel Kasenberg and Matthias Scheutz. Interpretable apprenticeship learning with temporal logic specifica- tions. In 56th IEEE Annual Conference on Decision and Control, CDC 2017, Melbourne, Australia, Decem- ber 12-15, 2017 , pages 4914–4921. IEEE, 2017

2017
[195]

Barrett, Guy Katz, and Michael Schapira

Yafim Kazak, Clark W. Barrett, Guy Katz, and Michael Schapira. Verifying deep-RL-driven systems. In Proceedings of the 2019 Workshop on Network Meets AI & ML, NetAI@SIGCOMM 2019, Beijing, China, August 23, 2019 , pages 83–89. ACM, 2019

2019
[196]

Mar- leme: A multi-agent reinforcement learning model ex- traction library

Dmitry Kazhdan, Zohreh Shams, and Pietro Li` o. Mar- leme: A multi-agent reinforcement learning model ex- traction library. In 2020 International Joint Conference on Neural Networks, IJCNN 2020, Glasgow, United Kingdom, July 19-24, 2020 , pages 1–8. IEEE, 2020

2020
[197]

Particle swarm optimization

James Kennedy and Russell Eberhart. Particle swarm optimization. In Proceedings of International Con- ference on Neural Networks (ICNN’95), Perth, WA, Australia, November 27 - December 1, 1995 , pages 1942–1948. IEEE, 1995

1995
[198]

Omar Zia Khan, Pascal Poupart, and James P. Black. Minimal sufficient explanations for factored Markov decision processes. In Alfonso Gerevini, Adele E. Howe, Amedeo Cesta, and Ioannis Refanidis, editors, Proceed- ings of the 19th International Conference on Automated Planning...

2009
[199]

Bayesian inference of lin- ear temporal logic specifications for contrastive expla- nations

Joseph Kim, Christian Muise, Ankit Shah, Shubham Agarwal, and Julie Shah. Bayesian inference of lin- ear temporal logic specifications for contrastive expla- nations. In Sarit Kraus, editor, Proceedings of the Twenty-Eighth International Joint Conference on Arti- ficial Intell...

2019
[200]

Op- timal interpretability-performance trade-off of classi- fication trees with black-box reinforcement learning

Hector Kohler, Riad Akrour, and Philippe Preux. Op- timal interpretability-performance trade-off of classi- fication trees with black-box reinforcement learning. CoRR, abs/2304.05839, 2023

2023
[201]

Interpretable and editable programmatic tree policies for reinforcement learning

Hector Kohler, Quentin Delfosse, Riad Akrour, Kris- tian Kersting, and Philippe Preux. Interpretable and editable programmatic tree policies for reinforcement learning. CoRR, abs/2405.14956, 2024

2024 arXiv
[202]

Kurte, Yan Du, Kadir Amasyali, Robert W

Olivera Kotevska, Jeffrey Munk, Kuldeep R. Kurte, Yan Du, Kadir Amasyali, Robert W. Smith, and Helia Zandi. Methodology for interpretable reinforcement learning model for HV AC energy control. In Xintao Wu, Chris Jermaine, Li Xiong, Xiaohua Hu, Oliv- era Kotevska, Siyuan Lu, W...

2020
[203]

Learn- ing finite state representations of recurrent policy net- works

Anurag Koul, Alan Fern, and Sam Greydanus. Learn- ing finite state representations of recurrent policy net- works. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net, 2019

2019
[204]

Explainability in reinforcement learning: perspective and position

Agneza Krajna, Mario Brcic, Tomislav Lipic, and Juraj Doncevic. Explainability in reinforcement learning: perspective and position. CoRR, abs/2203.11547, 2022

2022 arXiv
[205]

Optimal control via reinforcement learning with sym- bolic policy approximation

Jiˇ r ´ ı Kubal ´ ık, Eduard Alibekov, and Robert Babuˇ ska. Optimal control via reinforcement learning with sym- bolic policy approximation. IF AC-PapersOnLine, 50(1):4162–4167, 2017

2017
[206]

Kulkarni, Karthik Narasimhan, Ardavan Saeedi, and Josh Tenenbaum

Tejas D. Kulkarni, Karthik Narasimhan, Ardavan Saeedi, and Josh Tenenbaum. Hierarchical deep rein- forcement learning: Integrating temporal abstraction and intrinsic motivation. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett, editors,...

2016
[207]

Exploring computational user models for agent policy summarization

Isaac Lage, Daphna Lifschitz, Finale Doshi-Velez, and Ofra Amir. Exploring computational user models for agent policy summarization. In Sarit Kraus, ed- itor, Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, ...

2019
[208]

Toward robust policy summariza- tion

Isaac Lage, Daphna Lifschitz, Finale Doshi-Velez, and Ofra Amir. Toward robust policy summariza- tion. In Edith Elkind, Manuela Veloso, Noa Agmon, and Matthew E. Taylor, editors, Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, AAMA...

2019
[209]

Petersen, Sookyung Kim, Cl´ audio P

Mikel Landajuela, Brenden K. Petersen, Sookyung Kim, Cl´ audio P. Santiago, Ruben Glatt, T. Nathan Mundhenk, Jacob F. Pettit, and Daniel M. Faissol. Discovering symbolic policies with deep reinforcement learning. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th...

2021
[210]

Michael T. Lash. HEX: human-in-the-loop explain- ability via deep reinforcement learning. CoRR, abs/2206.01343, 2022

2022 arXiv
[211]

Complementary reinforcement learn- ing towards explainable agents

Jung Hoon Lee. Complementary reinforcement learn- ing towards explainable agents. CoRR, abs/1901.00188, 2019

1901 arXiv
[212]

A synthesis of automated planning and reinforcement learning for efficient, robust decision-making

Matteo Leonetti, Luca Iocchi, and Peter Stone. A synthesis of automated planning and reinforcement learning for efficient, robust decision-making. Artif. Intell., 241:103–130, 2016

2016
[213]

State represen- tation learning for control: An overview

Timoth´ ee Lesort, Natalia D ´ ıaz Rodr ´ ıguez, Jean- Fran¸ cois Goudou, and David Filliat. State represen- tation learning for control: An overview. Neural Net- works, 108:379–392, 2018

2018
[214]

Explainable intelligence-driven de- fense mechanism against advanced persistent threats: A joint edge game and AI approach

Huiling Li, Jun Wu, Hansong Xu, Gaolei Li, and Mohsen Guizani. Explainable intelligence-driven de- fense mechanism against advanced persistent threats: A joint edge game and AI approach. IEEE Trans. Dependable Secur. Comput. , 19(2):757–775, 2022

2022
[215]

Explanations for human-on-the-loop: a proba- bilistic model checking approach

Nianyu Li, Sridhar Adepu, Eunsuk Kang, and David Garlan. Explanations for human-on-the-loop: a proba- bilistic model checking approach. In Shinichi Honiden, Elisabetta Di Nitto, and Radu Calinescu, editors, SEAMS ’20: IEEE/ACM 15th International Sympo- sium on Software Enginee...

2020
[216]

MRI reconstruction with in- terpretable pixel-wise operations using reinforcement learning

Wentian Li, Xidong Feng, Haotian An, Xiang Yao Ng, and Yu-Jin Zhang. MRI reconstruction with in- terpretable pixel-wise operations using reinforcement learning. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Application...

2020
[217]

Rein- forcement learning with temporal logic rewards

Xiao Li, Cristian Ioan Vasile, and Calin Belta. Rein- forcement learning with temporal logic rewards. In 2017 IEEE/RSJ International Conference on Intel- ligent Robots and Systems, IROS 2017, Vancouver, BC, Canada, September 24-28, 2017 , pages 3834–3839. IEEE, 2017

2017
[218]

Sycara, and Rahul Iyer

Yuezhang Li, Katia P. Sycara, and Rahul Iyer. Object- sensitive deep reinforcement learning. In Christoph Benzm¨ uller, Christine L. Lisetti, and Martin Theobald, editors, GCAI 2017, 3rd Global Conference on Arti- ficial Intelligence, Miami, FL, USA, 18-22 October 2017, volume...

2017
[219]

Roman Liessner, Jan Dohmen, and Marco A. Wiering. Explainable reinforcement learning for longitudinal control. In Ana Paula Rocha, Luc Steels, and H. Jaap van den Herik, editors, Proceedings of the 13th Interna- tional Conference on Agents and Artificial Intelligence, ICAART 2...

2021
[220]

What is answer set programming? In Dieter Fox and Carla P

Vladimir Lifschitz. What is answer set programming? In Dieter Fox and Carla P. Gomes, editors, Proceedings of the Twenty-Third AAAI Conference on Artificial Intelligence, AAAI 2008, Chicago, Illinois, USA, July 13-17, 2008 , pages 1594–1597. AAAI Press, 2008

2008
[221]

Combining reinforcement learning with rule- based controllers for transparent and general decision- making in autonomous driving

Amarildo Likmeta, Alberto Maria Metelli, Andrea Tirinzoni, Riccardo Giol, Marcello Restelli, and Danilo Romano. Combining reinforcement learning with rule- based controllers for transparent and general decision- making in autonomous driving. Robotics Auton. Syst., 131:103568, 2020

2020
[222]

Lillicrap, Jonathan J

Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In Yoshua Bengio and Yann LeCun, editors, 4th International Conference on Learning Representat...

2016
[223]

Wang, Eric Undersander, and Akshara Rai

Yixin Lin, Austin S. Wang, Eric Undersander, and Akshara Rai. Efficient and interpretable robot manip- ulation with graph neural networks. IEEE Robotics Autom. Lett., 7(2):2740–2747, 2022

2022
[224]

Con- trastive explanations for reinforcement learning via embedded self predictions

Zhengxian Lin, Kin-Ho Lam, and Alan Fern. Con- trastive explanations for reinforcement learning via embedded self predictions. In 9th International Confer- ence on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenReview.net, 2021

2021
[225]

Clustering via decision tree construction

Bing Liu, Yiyuan Xia, and Philip S Yu. Clustering via decision tree construction. Foundations and advances in data mining , pages 97–124, 2005

2005
[226]

Toward interpretable deep reinforcement learn- ing with linear model U-trees

Guiliang Liu, Oliver Schulte, Wang Zhu, and Qingcan Li. Toward interpretable deep reinforcement learn- ing with linear model U-trees. In Michele Berlingerio, Francesco Bonchi, Thomas G¨ artner, Neil Hurley, and Georgiana Ifrim, editors, Machine Learning and Knowl- edge Discove...

2018
[227]

Learning tree interpretation from object representation for deep reinforcement learn- ing

Guiliang Liu, Xiangyu Sun, Oliver Schulte, and Pascal Poupart. Learning tree interpretation from object representation for deep reinforcement learn- ing. In Marc’Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wort- man Vaughan, editors, Advances...

2021
[228]

Learning to identify critical states for reinforcement learning from videos

Haozhe Liu, Mingchen Zhuge, Bing Li, Yuhui Wang, Francesco Faccio, Bernard Ghanem, and J¨ urgen Schmidhuber. Learning to identify critical states for reinforcement learning from videos. In IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October ...

2023
[229]

Towards explainable reinforcement learning us- ing scoring mechanism augmented agents

Yang Liu, Xinzhi Wang, Yudong Chang, and Chao Jiang. Towards explainable reinforcement learning us- ing scoring mechanism augmented agents. In G´ erard Memmi, Baijian Yang, Linghe Kong, Tianwei Zhang, and Meikang Qiu, editors, Knowledge Science, Engi- neering and Management - ...

2022
[230]

Explaining robot actions

Meghann Lomas, Robert Chevalier, Ernest Vin- cent Cross II, Robert Christopher Garrett, John Hoare, and Michael Kopack. Explaining robot actions. In Holly A. Yanco, Aaron Steinfeld, Vanessa Evers, and Odest Chadwicke Jenkins, editors, International Con- ference on Human-Robot ...

2012
[231]

Explainable AI methods on a deep reinforce- ment learning agent for automatic docking

Jakob Løver, Vilde B Gjærum, and Anastasios M Lekkas. Explainable AI methods on a deep reinforce- ment learning agent for automatic docking. IF AC- PapersOnLine, 54(16):146–152, 2021

2021
[232]

Complex event processing in distributed systems

David C Luckham and Brian Frasca. Complex event processing in distributed systems. Computer Systems Laboratory Technical Report CSL-TR-98-754. Stanford University, Stanford, 28:16, 1998. 57

1998
[233]

Lundberg, Gabriel G

Scott M. Lundberg, Gabriel G. Erion, and Su-In Lee. Consistent individualized feature attribution for tree ensembles. CoRR, abs/1802.03888, 2018

2018 arXiv
[234]

Lundberg and Su-In Lee

Scott M. Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wal- lach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Pro- cessing Systems 30:...

2017
[235]

Lo- cal explanations for reinforcement learning

Ronny Luss, Amit Dhurandhar, and Miao Liu. Lo- cal explanations for reinforcement learning. In Brian Williams, Yiling Chen, and Jennifer Neville, editors, Thirty-Seventh AAAI Conference on Artificial Intel- ligence, AAAI 2023, Thirty-Fifth Conference on In- novative Applicatio...

2023
[236]

SDRL: interpretable and data-efficient deep reinforcement learning leveraging symbolic planning

Daoming Lyu, Fangkai Yang, Bo Liu, and Steven Gustafson. SDRL: interpretable and data-efficient deep reinforcement learning leveraging symbolic planning. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Arti...

2019
[237]

Learn- ing symbolic rules for interpretable deep reinforcement learning

Zhihao Ma, Yuzheng Zhuang, Paul Weng, Hankz Han- kui Zhuo, Dong Li, Wulong Liu, and Jianye Hao. Learn- ing symbolic rules for interpretable deep reinforcement learning. CoRR, abs/2103.08228, 2021

2021 arXiv
[238]

Distal explanations for explainable re- inforcement learning agents

Prashan Madumal, Tim Miller, Liz Sonenberg, and Frank Vetere. Distal explanations for explainable re- inforcement learning agents. CoRR, abs/2001.10284, 2020

2001 arXiv
[239]

Explainable reinforcement learning through a causal lens

Prashan Madumal, Tim Miller, Liz Sonenberg, and Frank Vetere. Explainable reinforcement learning through a causal lens. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial In- telligence Conference...

2020
[240]

Policy search in a space of simple closed-form formulas: Towards interpretability of rein- forcement learning

Francis Maes, Raphael Fonteneau, Louis Wehenkel, and Damien Ernst. Policy search in a space of simple closed-form formulas: Towards interpretability of rein- forcement learning. In Jean-Gabriel Ganascia, Philippe Lenca, and Jean-Marc Petit, editors, Discovery Sci- ence - 15th ...

2012
[241]

Zero-shot trans- fer with deictic object-oriented representation in rein- forcement learning

Ofir Marom and Benjamin Rosman. Zero-shot trans- fer with deictic object-oriented representation in rein- forcement learning. In Samy Bengio, Hanna M. Wal- lach, Hugo Larochelle, Kristen Grauman, Nicol` o Cesa- Bianchi, and Roman Garnett, editors, Advances in Neural Informatio...

2018
[242]

A self-organising network that grows when required

Stephen Marsland, Jonathan Shapiro, and Ulrich Nehmzow. A self-organising network that grows when required. Neural Networks, 15(8-9):1041–1058, 2002

2002
[243]

Relational reinforcement learning with guided demonstrations

David Mart ´ ınez Mart ´ ınez, Guillem Aleny` a, and Carme Torras. Relational reinforcement learning with guided demonstrations. Artif. Intell. , 247:295–312, 2017

2017
[244]

Alqahtani, and Dongwon Lee

Joe McCalmon, Thai Le, Sarra M. Alqahtani, and Dongwon Lee. CAPS: comprehensible abstract pol- icy summaries for explaining reinforcement learning agents. In Piotr Faliszewski, Viviana Mascardi, Cather- ine Pelachaud, and Matthew E. Taylor, editors, 21st International Conferen...

2022
[245]

Diet- terich, Rachel Houtman, Claire A

Sean McGregor, Hailey Buckingham, Thomas G. Diet- terich, Rachel Houtman, Claire A. Montgomery, and Ronald A. Metoyer. Interactive visualization for test- ing Markov decision processes: MDPVIS. J. Vis. Lang. Comput., 39:93–106, 2017

2017
[246]

Learning graph-based represen- tations for continuous reinforcement learning domains

Jan Hendrik Metzen. Learning graph-based represen- tations for continuous reinforcement learning domains. In Hendrik Blockeel, Kristian Kersting, Siegfried Ni- jssen, and Filip Zelezn´ y, editors, Machine Learning and Knowledge Discovery in Databases - European Conference, ECM...

2013
[247]

Explainable reinforcement learning: A survey and comparative review

Stephanie Milani, Nicholay Topin, Manuela Veloso, and Fei Fang. Explainable reinforcement learning: A survey and comparative review. ACM Comput. Surv. , 56(7):168:1–168:36, 2024

2024
[248]

Kamhoua, Evangelos E

Stephanie Milani, Zhicheng Zhang, Nicholay Topin, Zheyuan Ryan Shi, Charles A. Kamhoua, Evangelos E. 58 Papalexakis, and Fei Fang. MA VIPER: learning de- cision tree policies for interpretable multi-agent rein- forcement learning. In Massih-Reza Amini, St´ ephane Canu, Asja Fi...

2022
[249]

Miller and Peter Thomson

Julian F. Miller and Peter Thomson. Cartesian genetic programming. In Riccardo Poli, Wolfgang Banzhaf, William B. Langdon, Julian F. Miller, Peter Nordin, and Terence C. Fogarty, editors,Genetic Programming, European Conference, Edinburgh, Scotland, UK, April 15-16, 2000, Proc...

2000
[250]

Why? why not? when? visual explanations of agent behavior in reinforcement learning

Aditi Mishra, Utkarsh Soni, Jinbin Huang, and Chris Bryan. Why? why not? when? visual explanations of agent behavior in reinforcement learning. CoRR, abs/2104.02818, 2021

2021 arXiv
[251]

Vi- sual sparse bayesian reinforcement learning: A frame- work for interpreting what an agent has learned

Indrajeet Mishra, Giang Dao, and Minwoo Lee. Vi- sual sparse bayesian reinforcement learning: A frame- work for interpreting what an agent has learned. In IEEE Symposium Series on Computational Intelligence, SSCI 2018, Bangalore, India, November 18-21, 2018 , pages 1427–1434. ...

2018
[252]

Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu

Volodymyr Mnih, Adri` a Puigdom` enech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In Maria- Florina Balcan and Kilian Q. Weinberger, editors, Pro- ceedings of the...

2016
[253]

Rusu, Joel Veness, Marc G

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin A. Riedmiller, Andreas Fidje- land, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dhar- shan Kumaran, Daan Wierst...

2015
[254]

Gedanken-experiments on sequential machines

Edward F Moore. Gedanken-experiments on sequential machines. Automata studies, 34:129–153, 1956

1956
[255]

Venkatesh Babu

Konda Reddy Mopuri, Utsav Garg, and R. Venkatesh Babu. CNN fixations: An unraveling approach to visualize the discriminative image regions. IEEE Trans. Image Process., 28(5):2116–2125, 2019

2019
[256]

To- wards interpretable reinforcement learning using atten- tion augmented agents

Alexander Mott, Daniel Zoran, Mike Chrzanowski, Daan Wierstra, and Danilo Jimenez Rezende. To- wards interpretable reinforcement learning using atten- tion augmented agents. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d’Alch´ e-Buc, Emily B. Fox, and Roma...

2019
[257]

Murthy, Simon Kasif, and Steven Salzberg

Sreerama K. Murthy, Simon Kasif, and Steven Salzberg. A system for induction of oblique decision trees. J. Artif. Intell. Res. , 2:1–32, 1994

1994
[258]

Roger B. Myerson. Graphs and cooperation in games. Math. Oper. Res. , 2(3):225–229, 1977

1977
[259]

Subramanya Nageshrao, Bruno Costa, and Dimitar P. Filev. Interpretable approximation of a deep rein- forcement learning agent as a set of if-then rules. In M. Arif Wani, Taghi M. Khoshgoftaar, Dingding Wang, Huanjing Wang, and Naeem Seliya, editors, 18th IEEE International Con...

2019
[260]

Neerincx, Jasper van der Waa, Frank Kaptein, and Jurriaan van Diggelen

Mark A. Neerincx, Jasper van der Waa, Frank Kaptein, and Jurriaan van Diggelen. Using perceptual and cog- nitive explanations for enhanced human-agent team performance. In Don Harris, editor, Engineering Psy- chology and Cognitive Ergonomics - 15th International Conference, EP...

2018
[261]

Ng and Stuart Russell

Andrew Y. Ng and Stuart Russell. Algorithms for inverse reinforcement learning. In Pat Langley, edi- tor, Proceedings of the Seventeenth International Con- ference on Machine Learning (ICML 2000), Stanford University, Stanford, CA, USA, June 29 - July 2, 2000 , pages 663–670. ...

2000
[262]

Alkemy: A learning system based on an expressive knowledge representation formalism

Kee Siong Ng. Alkemy: A learning system based on an expressive knowledge representation formalism. submitted for publication , 2004

2004
[263]

Quinn, Thin Nguyen, and Truyen Tran

Tri Minh Nguyen, Thomas P. Quinn, Thin Nguyen, and Truyen Tran. Counterfactual explanation with multi- agent reinforcement learning for drug target prediction. CoRR, abs/2103.12983, 2021

2021 arXiv
[264]

Visualizing deep Q-learning to understanding behav- ior of swarm robotic system

Xiaotong Nie, Motoaki Hiraga, and Kazuhiro Ohkura. Visualizing deep Q-learning to understanding behav- ior of swarm robotic system. In Hiroshi Sato, Saori Iwanaga, and Akira Ishii, editors, Proceedings of the 23rd Asia Pacific Symposium on Intelligent and Evolu- tionary System...

2019
[265]

Nikolenko

Dmitry Nikulin, Anastasia Ianina, Vladimir Aliev, and Sergey I. Nikolenko. Free-lunch saliency via atten- tion in Atariagents. In 2019 IEEE/CVF International Conference on Computer Vision Workshops, ICCV Workshops 2019, Seoul, Korea (South), October 27-28, 2019, pages 4240–424...

2019
[266]

Michael Oberst and David A. Sontag. Counterfac- tual off-policy evaluation with Gumbel-Max structural causal models. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th Interna- tional Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long...

2019
[267]

Olson, Roli Khanna, Lawrence Neal, Fuxin Li, and Weng-Keen Wong

Matthew L. Olson, Roli Khanna, Lawrence Neal, Fuxin Li, and Weng-Keen Wong. Counterfactual state expla- nations for reinforcement learning agents via generative deep learning. Artif. Intell. , 295:103455, 2021

2021
[268]

Fuzzy centered explainable network for rein- forcement learning

Liang Ou, Yu-Cheng Chang, Yu-Kai Wang, and Chin- Teng Lin. Fuzzy centered explainable network for rein- forcement learning. IEEE Trans. Fuzzy Syst., 32(1):203– 213, 2024

2024
[269]

Paleja, Yaru Niu, Andrew Silva, Chace Ritchie, Sugju Choi, and Matthew C

Rohan R. Paleja, Yaru Niu, Andrew Silva, Chace Ritchie, Sugju Choi, and Matthew C. Gombolay. Learning interpretable, high-performing policies for autonomous driving. In Kris Hauser, Dylan A. Shell, and Shoudong Huang, editors, Robotics: Science and Systems XVIII, New York City...

2022
[270]

Canny, and Fisher Yu

Xinlei Pan, Xiangyu Chen, Qi-Zhi Cai, John F. Canny, and Fisher Yu. Semantic predictive control for ex- plainable and efficient policy learning. In International Conference on Robotics and Automation, ICRA 2019, Montreal, QC, Canada, May 20-24, 2019 , pages 3203–

2019
[271]

Bleu: a method for automatic evaluation of machine translation

Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, July 6-12, 2002, Philadelphia, PA, USA , pages 311–318. ACL, 2002

2002
[272]

Hierarchical reinforcement learn- ing: A comprehensive survey

Shubham Pateria, Budhitama Subagdja, Ah-Hwee Tan, and Chai Quek. Hierarchical reinforcement learn- ing: A comprehensive survey. ACM Comput. Surv. , 54(5):109:1–109:35, 2022

2022
[273]

Inductive logic pro- gramming via differentiable deep neural logic networks

Ali Payani and Faramarz Fekri. Inductive logic pro- gramming via differentiable deep neural logic networks. CoRR, abs/1906.03523, 2019

1906 arXiv
[274]

Incorporating rela- tional background knowledge into reinforcement learn- ing via differentiable inductive logic programming

Ali Payani and Faramarz Fekri. Incorporating rela- tional background knowledge into reinforcement learn- ing via differentiable inductive logic programming. CoRR, abs/2003.10386, 2020

2003 arXiv
[275]

The mirror agent model: A bayesian architecture for interpretable agent behavior

Michele Persiani and Thomas Hellstr¨ om. The mirror agent model: A bayesian architecture for interpretable agent behavior. In Davide Calvaresi, Amro Najjar, Michael Winikoff, and Kary Fr¨ amling, editors,Explain- able and Transparent AI and Multi-Agent Systems - 4th Internatio...

2022
[276]

RISE: randomized input sampling for explanation of black- box models

Vitali Petsiuk, Abir Das, and Kate Saenko. RISE: randomized input sampling for explanation of black- box models. In British Machine Vision Conference 2018, BMVC 2018, Newcastle, UK, September 3-6, 2018, page 151. BMV A Press, 2018

2018
[277]

Brittany Davis Pierson, Dustin Arendt, John Miller, and Matthew E. Taylor. Comparing explanations in RL. Neural Comput. Appl. , 36(1):505–516, 2024

2024
[278]

Strategic tasks for explainable reinforcement learning

Rey Pocius, Lawrence Neal, and Alan Fern. Strategic tasks for explainable reinforcement learning. In The Thirty-Third AAAI Conference on Artificial Intelli- gence, AAAI 2019, The Thirty-First Innovative Ap- plications of Artificial Intelligence Conference, IAAI 2019, The Ninth...

2019
[279]

Erika Puiutta and Eric M. S. P. Veith. Explainable re- inforcement learning: A survey. In Andreas Holzinger, Peter Kieseberg, A Min Tjoa, and Edgar R. Weippl, editors, Machine Learning and Knowledge Extraction - 4th IFIP TC 5, TC 12, WG 8.4, WG 8.9, WG 12.9 International Cross...

2020
[280]

Deshmukh, Balaji Krishna- murthy, and Sameer Singh

Nikaash Puri, Sukriti Verma, Piyush Gupta, Dhruv Kayastha, Shripad V. Deshmukh, Balaji Krishna- murthy, and Sameer Singh. Explain your move: Un- derstanding agent actions using specific and relevant feature attribution. In 8th International Conference on Learning Representatio...

2020
[281]

Ross Quinlan

J. Ross Quinlan. Induction of decision trees. Mach. Learn., 1(1):81–106, 1986

1986
[282]

S-RL toolbox: Environments, datasets and evalua- tion metrics for state representation learning

Antonin Raffin, Ashley Hill, Ren´ e Traor´ e, Timoth´ ee Lesort, Natalia D ´ ıaz Rodr ´ ıguez, and David Filliat. S-RL toolbox: Environments, datasets and evalua- tion metrics for state representation learning. CoRR, abs/1809.09369, 2018

2018 arXiv
[283]

Sindre Benjamin Remman and Anastasios M. Lekkas. Robotic lever manipulation using hindsight experience 60 replay and Shapley additive explanations. In 2021 Eu- ropean Control Conference, ECC 2021, Virtual Event / Delft, The Netherlands, June 29 - July 2, 2021 , pages 586–593. ...

2021
[284]

Sindre Benjamin Remman, Inga Str¨ umke, and Anasta- sios M. Lekkas. Causal versus marginal Shapley values for robotic lever manipulation controlled using deep reinforcement learning. In American Control Confer- ence, ACC 2022, Atlanta, GA, USA, June 8-10, 2022 , pages 2683–269...

2022
[285]

”Why should I trust you?”: Explaining the predictions of any classifier

Marco T´ ulio Ribeiro, Sameer Singh, and Carlos Guestrin. ”Why should I trust you?”: Explaining the predictions of any classifier. In Balaji Krishnapuram, Mohak Shah, Alexander J. Smola, Charu C. Aggarwal, Dou Shen, and Rajeev Rastogi, editors, Proceedings of the 22nd ACM SIGK...

2016
[286]

Finn Rietz, Sven Magg, Fredrik Heintz, Todor Stoy- anov, Stefan Wermter, and Johannes A. Stork. Hierar- chical goals contextualize local reward decomposition explanations. Neural Comput. Appl. , 35(23):16693– 16704, 2023

2023
[287]

Reinforcement learning with explainability for traffic signal control

Stefano Giovanni Rizzo, Giovanna Vantini, and Sanjay Chawla. Reinforcement learning with explainability for traffic signal control. In 2019 IEEE Intelligent Trans- portation Systems Conference, ITSC 2019, Auckland, New Zealand, October 27-30, 2019 , pages 3567–3572. IEEE, 2019

2019
[288]

U-net: Convolutional networks for biomedical image segmentation

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Nassir Navab, Joachim Hornegger, William M. Wells III, and Alejandro F. Frangi, edi- tors, Medical Image Computing and Computer-Assisted Intervention - MICCA...

2015
[289]

Better metrics for evaluating explain- able artificial intelligence

Avi Rosenfeld. Better metrics for evaluating explain- able artificial intelligence. In Frank Dignum, Alessio Lo- muscio, Ulle Endriss, and Ann Now´ e, editors,AAMAS ’21: 20th International Conference on Autonomous Agents and Multiagent Systems, Virtual Event, United Kingdom, M...

2021
[290]

Roth, Nicholay Topin, Pooyan Jamshidi, and Manuela Veloso

Aaron M. Roth, Nicholay Topin, Pooyan Jamshidi, and Manuela Veloso. Conservative Q-improvement: Reinforcement learning for an interpretable decision- tree policy. CoRR, abs/1907.01180, 2019

1907 arXiv
[291]

Explaining optimal trajecto- ries

C´ eline Rouveirol, Malik Kazi Aoual, Henry Soldano, and V´ eronique Ventos. Explaining optimal trajecto- ries. In Anna Fensel, Ana Ozaki, Dumitru Roman, and Ahmet Soylu, editors, Rules and Reasoning - 7th Inter- national Joint Conference, RuleML+RR 2023, Oslo, Norway, Septemb...

2023
[292]

Optimization of computer sim- ulation models with rare events

Reuven Y Rubinstein. Optimization of computer sim- ulation models with rare events. European Journal of Operational Research, 99(1):89–112, 1997

1997
[293]

Christian Rupprecht, Cyril Ibrahim, and Christopher J. Pal. Finding and visualizing weaknesses of deep rein- forcement learning agents. In 8th International Confer- ence on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview.net, 2020

2020
[294]

Conor Ryan, J. J. Collins, and Michael O’Neill. Gram- matical evolution: Evolving programs for an arbitrary language. In Wolfgang Banzhaf, Riccardo Poli, Marc Schoenauer, and Terence C. Fogarty, editors, Genetic Programming, First European Workshop, EuroGP’98, Paris, France, A...

1998
[295]

Towards global explainability of artificial intelligence agent tactics in close air combat

Emre Saldiran, Mehmet Hasanzade, Gokhan Inalhan, and Antonios Tsourdos. Towards global explainability of artificial intelligence agent tactics in close air combat. Aerospace, 11(6):415, 2024

2024
[296]

SAFE-RL: saliency-aware coun- terfactual explainer for deep reinforcement learning policies

Amir Samadi, Konstantinos Koufos, Kurt Debattista, and Mehrdad Dianati. SAFE-RL: saliency-aware coun- terfactual explainer for deep reinforcement learning policies. CoRR, abs/2404.18326, 2024

2024 arXiv
[297]

Model-agnostic and scalable counterfac- tual explanations via reinforcement learning

Robert-Florian Samoilescu, Arnaud Van Looveren, and Janis Klaise. Model-agnostic and scalable counterfac- tual explanations via reinforcement learning. CoRR, abs/2106.02597, 2021

2021 arXiv
[298]

Cooper, and Florence Dupin de Saint-Cyr

L´ eo Sauli` eres, Martin C. Cooper, and Florence Dupin de Saint-Cyr. Predicate-based explanation of a rein- forcement learning agent via action importance evalu- ation. In Rosa Meo and Fabrizio Silvestri, editors, Ma- chine Learning and Principles and Practice of Knowl- edge ...

2023
[1207]

International Foundation for Autonomous Agents and Multiagent Systems Richland, SC, USA / ACM, 2018

2018
[2012]

Springer, 2012

Proceedings, volume 7244 of Lecture Notes in Computer Science, pages 49–60. Springer, 2012

2012

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [35]

Daniel Beechey, Thomas M. S. Smith, and ¨Ozg¨ ur Sim- sek. Explaining reinforcement learning with shap- ley values. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, International Confer- ence on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA , volume 202 of Proceed- in...

2023

[2] [17]

Explain- ing reinforcement learning agents through counter- factual action outcomes

Yotam Amitai, Yael Septon, and Ofra Amir. Explain- ing reinforcement learning agents through counter- factual action outcomes. In Michael J. Wooldridge, Jennifer G. Dy, and Sriraam Natarajan, editors,Thirty- Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence, IAAI 202...

2024

[3] [1]

The regression Tsetlin machine: A Tsetlin machine for continuous output problems

Kuruge Darshana Abeyrathna, Ole-Christoffer Granmo, Lei Jiao, and Morten Goodwin. The regression Tsetlin machine: A Tsetlin machine for continuous output problems. In Paulo Moura Oliveira, Paulo Novais, and Lu ´ ıs Paulo Reis, editors,Progress in Artificial Intelligence, 19th EPIA Conference on Artificial Intelligence, EPIA 2019, Vila Real, Portugal, Sept...

2019

[4] [2]

Explaining Conditions for Reinforcement Learning Behaviors from Real and Imagined Data

Aastha Acharya, Rebecca L. Russell, and Nisar R. Ahmed. Explaining conditions for reinforcement learn- ing behaviors from real and imagined data. CoRR, abs/2011.09004, 2020

work page Pith review arXiv 2011

[5] [3]

Goodfellow, Moritz Hardt, and Been Kim

Julius Adebayo, Justin Gilmer, Michael Muelly, Ian J. Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicol` o Cesa-Bianchi, and Roman Garnett, editors, Advances in Neural Information Processing Systems 31: Annual 43 Conference on Neural Information Processing Sys...

2018

[6] [4]

Symbolic relation networks for reinforcement learning

Dhaval Adjodah, Tim Klinger, and Joshua Joseph. Symbolic relation networks for reinforcement learning. In Proceedings of the Workshop on Relational Represen- tation Learning in Conference on Neural Information Processing Systems (NeurIPS), 2018

2018

[7] [5]

A survey of statistical model checking

Gul Agha and Karl Palmskog. A survey of statistical model checking. ACM Trans. Model. Comput. Simul. , 28(1):6:1–6:39, 2018

2018

[8] [6]

Unsupervised object-level deep reinforcement learning

William Agnew and Pedro Domingos. Unsupervised object-level deep reinforcement learning. In NeurIPS Workshop on Deep RL , 2018

2018