REVIEW 4 major objections 5 minor 2 cited by
Towards Transparent AI: A Survey on Explainable Large Language Models
T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read A new survey organizes explainable-AI methods for large language models by the three transformer architectures — encoder-only, decoder-only, and encoder-decoder — and argues that matching methods to architecture is the right way to…
desk verdict A useful table of XAI methods, but the architecture-based taxonomy is a convenience label, not a method property; send out for review, don't cite for the taxonomy. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central organizing object is the three-way taxonomy of XAI methods based on transformer architecture: encoder-only, decoder-only, and encoder-decoder. Each category bundles the methods the paper argues fit that architecture: feature attribution, probing, and attention tools for encoder-only; feature attribution, in-context learning analysis, chain-of-thought prompting, and mechanistic interpretability for decoder-only; and feature attribution, classifier-based probing, attention, and self-explanation for encoder-decoder. The taxonomy is operationalized in a comparison table with dimensions such as method type, architecture, model, access (internal or external), scope (local or global), and integration stage, and this table is what carries the paper's claim of enabling meaningful cross-method comparison.
What would settle it
Check the methods in the survey's Table 1: if the same method class, for example SHAP-based attribution, appears in all three architecture sections without any architecture-specific modification, or if many of the surveyed methods are demonstrated on models from more than one architecture, then the architecture axis would no longer be a meaningful partition.
Extended reading notes
Core claim
The paper claims to be the first survey to categorize XAI methods for large language models according to the underlying transformer architecture. Under this taxonomy, encoder-only models are explained through their stable bidirectional contextual embeddings, using feature attribution, classifier-based probing, parameter-free probing, and attention visualization; decoder-only models require autoregression-aware methods, grouped into feature attribution, in-context-learning analysis, chain-of-thought prompting, and mechanistic interpretability; encoder-decoder models can be explained at both stages, with feature attribution, classifier-based probing, attention-based methods, and self-explanation. The survey further claims that this architecture-aware comparison, together with an analysis of evaluation mechanisms and practical applications, provides a comprehensive picture and identifies the main open challenges: missing ground-truth explanations, shortcut learning, interpretability efficiency, and ethics.
Load-bearing premise
The taxonomy assumes that a method's target transformer architecture — encoder-only, decoder-only, or encoder-decoder — is the right and complete way to group explainability techniques, so that every method can be assigned to one meaningful category.
Editorial extensions
If this is right
- A practitioner with an encoder-only model can narrow the XAI literature to feature attribution, probing, and attention-visualization tools designed for stable bidirectional embeddings.
- A practitioner with a decoder-only model should expect relevant methods to come from attribution, in-context learning analysis, chain-of-thought prompting, and mechanistic interpretability, all adapted to autoregressive generation.
- A practitioner with an encoder-decoder model can look for methods that explain both encoding and decoding, especially through cross-attention analysis and self-explanation.
- Choice of explanation method depends on model access: white-box methods such as attribution masking learning and ROME do not work on closed models, while black-box methods such as Progressive Inference and chain-of-thought prompting do.
- The lack of ground-truth explanations and benchmark datasets remains a barrier to evaluating faithfulness across all three architecture families.
Reading between the lines
- The architecture taxonomy could be combined with model access as a second axis: methods requiring internal weights, such as attribution masking learning, ROME, and ACDC, form one cluster, while API-only methods such as Progressive Inference and chain-of-thought prompting form another, sharpening the comparison table's Access column.
- If decoder-only models increasingly dominate deployment, the survey's structure predicts the field's attention will shift toward prompt-based and mechanistic interpretation methods, with probing methods receiving relatively less attention.
- A natural test of the taxonomy is to ask practitioners to pick an explanation method for a given model; if their choices do not separate by architecture, the organizing axis would need revision.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript is a survey of explainability methods for large language models (LLMs). It proposes a taxonomy based on transformer architecture—encoder-only, decoder-only, and encoder-decoder—and organizes a set of XAI methods into these three categories. It also discusses evaluation mechanisms, practical applications, available resources, and open challenges. The central contribution claimed is that this is the first comprehensive survey to provide an architecture-based standard taxonomy that enables meaningful comparison of XAI methods and their use cases.
Significance. If the central claim were well supported, the proposed taxonomy could be a useful organizing framework for practitioners selecting XAI tools. The survey compiles a diverse set of recent methods into a structured table and gives a balanced, critical discussion of many methods' limitations. However, the load-bearing taxonomy conflates the model on which a method was demonstrated with the method's architectural requirements, and no literature selection methodology is provided. These issues currently prevent the survey from supporting its advertised contribution, although the paper's breadth and structured presentation are genuine strengths that could be salvaged with a revised framing and a systematic methodology.
major comments (4)
- [Section 3, Table 1] The central taxonomy is not supported because the Architecture column records the model used in each study, not a property of the XAI method itself. LIME (Table 1, row [47]; Section 5.1.1) is a model-agnostic local surrogate; SHAP/TransSHAP (row [45]) is an additive attribution method independent of the underlying architecture; Progressive Inference (row [56]) is explicitly model-agnostic; CoT prompting (rows [15], [14]) can be applied to any generative or sequence-to-sequence model; and attention visualization tools such as BertViz and ExBERT (rows [33], [8]) are not confined to encoder-only models. The paper needs an explicit criterion distinguishing 'this method requires architectural property X' from 'this method was evaluated on model Y', and Table 1's architecture labels should be updated accordingly.
- [Section 5.2.4, Table 1] The ACDC entry is cited as [61] in both Table 1 and Section 5.2.4, but reference [61] is Gurnee et al., 'Language Models Represent Space and Time' (ICLR 2024), not the ACDC paper. No correct ACDC reference appears in the bibliography. Since this is a survey, mis-citing a core mechanistic-interpretability method materially reduces the reliability of the reference list; the authors should correct the citation and audit the remaining entries for similar mismatches.
- [Abstract, Section 4] The paper claims to be 'the first comprehensive survey' and to provide a 'systematic understanding', but Section 4 and Table 1 present 22 selected papers without any description of the literature search, inclusion/exclusion criteria, or comparison with existing surveys [22], [25]. Without a methodology, the comprehensiveness claim cannot be evaluated, and the reader cannot tell whether the architecture-based organization reflects the field or an arbitrary selection. Please add a methodology subsection and explicitly position the contribution against the cited surveys.
- [Section 5.2.1, Section 5.1.1] The paper is internally inconsistent about whether feature attribution is tied to architecture. Section 5.1.1 states that these techniques 'suit encoder-only models' pretrained with masked language modeling, but Section 5.2.1 applies feature attribution to decoder-only models and Table 1 lists AML and Progressive Inference under 'Decoder-only'. This confirms that the architecture labels function as convenience labels rather than method properties, which undercuts the Section 3 claim that the taxonomy enables 'meaningful comparisons'. The authors should revise the framing to describe architecture-specific applications and challenges rather than architecture-specific method categories.
minor comments (5)
- [References] The reference list has inconsistent formatting: [20] writes 'InEACL' instead of 'In EACL', [39] has a missing space in 'NeurIPS', and several entries give only an initial plus 'et al.' with incomplete author lists; please standardize the bibliography style.
- [Section 5.2.2] The description of Lampinen et al. [38] as using contrastive prompts with meaningless tokens does not clearly match the title 'Can language models learn from explanations in context?'; please verify that [38] is the correct source for those experiments.
- [Table 1] Because several methods in Table 1 (LIME, SHAP, Progressive Inference, CoT prompting) are model-agnostic, consider adding a separate column or flag indicating whether the method is architecture-specific or model-agnostic; this would make the table's information more accurate and usable.
- [Section 5.3.4] The author name 'Yordan et al.' in the text is inconsistent with the reference list entry [66], which gives only 'Yordan Y.'; please correct the spelling and complete the reference.
- [Throughout] Abbreviations such as AML, ICL, NLE, and CoT are used without a central notation table; adding a list of abbreviations would improve readability for a survey readership.
Circularity Check
No circularity found; the survey's claims are organizing and descriptive rather than derived from its own inputs.
full rationale
This paper is a literature survey; it does not derive predictions from data, fit parameters, or invoke its own prior results as evidence. The central claim, that XAI methods can be categorized by transformer architecture, is a proposed organizing taxonomy, not a conclusion derived from the surveyed methods in a way that reduces to its own inputs. Each cited method is attributed to an external paper, and the survey does not define the taxonomy in terms of its own conclusions or use a self-citation chain to force its structure. Concerns that the Architecture column records the tested model rather than an architectural requirement of the method are about the validity or informativeness of the taxonomy, not circularity. The paper also cites other surveys without comparing itself to them, but that is at most a completeness weakness, not a circular step. Under the stated criteria, no circular step can be exhibited with a quote showing equation or definitional equivalence, so the honest finding is no significant circularity.
Assumptions & free parameters
assumptions (2)
- domain assumption Transformer-based LLMs can be partitioned exhaustively into encoder-only, decoder-only, and encoder-decoder architectures for the purpose of organizing XAI methods.
- domain assumption The set of papers in Table 1 is representative and comprehensive enough to support the survey's conclusions.
Cite this review
Pith. "Pith review of Towards Transparent AI: A Survey on Explainable Large Language Models." pith.science (2026). https://pith.science/paper/UZF2D3H2
@misc{pith2026250621812,
author = {Pith},
title = {Pith review of: Towards Transparent AI: A Survey on Explainable Large Language Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/UZF2D3H2}},
note = {Machine review of arXiv:2506.21812}
}
read the original abstract
Large Language Models (LLMs) have played a pivotal role in advancing Artificial Intelligence (AI). However, despite their achievements, LLMs often struggle to explain their decision-making processes, making them a 'black box' and presenting a substantial challenge to explainability. This lack of transparency poses a significant obstacle to the adoption of LLMs in high-stakes domain applications, where interpretability is particularly essential. To overcome these limitations, researchers have developed various explainable artificial intelligence (XAI) methods that provide human-interpretable explanations for LLMs. However, a systematic understanding of these methods remains limited. To address this gap, this survey provides a comprehensive review of explainability techniques by categorizing XAI methods based on the underlying transformer architectures of LLMs: encoder-only, decoder-only, and encoder-decoder models. Then these techniques are examined in terms of their evaluation for assessing explainability, and the survey further explores how these explanations are leveraged in practical applications. Finally, it discusses available resources, ongoing research challenges, and future directions, aiming to guide continued efforts toward developing transparent and responsible LLMs.
Figures
Forward citations
Cited by 2 Pith papers
-
Evaluating the Impact of Explainable AI on Trust in AI-Assisted Code Review
Fuller XAI raises perceived trust in AI code review, while moderate explanations maximize agreement; no explanations minimize both, with no significant time effect.
-
Beyond Explainable AI (XAI): An Overdue Paradigm Shift and Post-XAI Research Directions
Current XAI methods for DNNs and LLMs rest on paradoxes and false assumptions that demand a paradigm shift to verification protocols, scientific foundations, context-aware design, and faithful model analysis rather th...
Reference graph
Works this paper leans on
- [22]
- [25]
- [61]
- [47]
-
[45]
Lundberg S. M. and et al. 2017. A Unified Approach to Interpreting Model Predictions. In NeurIPS
work page 2017
- [56]
- [15]
- [14]
- [33]
- [8]
Show all 71 references
-
[1]
and et al
Chaddad A. and et al. 2023. Survey of Explainable AI Techniques in Healthcare. Sensors 23 (01 2023), 634
2023
-
[2]
and et al
Katherine A. and et al. 2023. Cognitive network science reveals bias in gpt-3, gpt-3.5 turbo, and gpt-4 mirroring math anxiety in high-school students. Big Data and Cognitive Computing 7, 3 (2023), 124
2023
-
[3]
and et al
Lopes A. and et al. 2020. Lite Training Strategies for Portuguese- English and English-Portuguese Translation. arXiv:2008.08769 [cs.CL]
2020 arXiv
-
[4]
and et al
Nie A. and et al. 2019. DisSent: Learning Sentence Representations from Explicit Discourse Relations. In ACL. 4497–4510
2019
-
[5]
and et al
Vaswani A. and et al. 2017. Attention is all you need. In NeurIPS. 30
2017
-
[6]
Anthropic. 2025. Official Documentation for Anthropic APIs and Models. Accessed on June 16, 2025
2025
-
[7]
and et al
Hutchinson B. and et al. 2020. Accountability of AI under the law: The role of explanation. arXiv:1711.01134 [cs.AI]
2020 arXiv
-
[9]
and et al
Oren B. and et al. 2024. LLM Explainability via Attributive Masking Learning. In EMNLP. 9522–9537
2024
-
[10]
and et al
Tom B. and et al. 2020. Language models are few-shot learners.NeurIPS 33 (2020), 1877–1901
2020
-
[11]
and et al
Ng C. and et al. 2024. Educational personalized learning path planning with large language models. arXiv:2407.11773 [cs.AI]
2024 arXiv
-
[12]
and et al
Rudin C. and et al. 2019. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence 1, 5 (2019), 206–215
2019
-
[13]
and et al
Raffel C. and et al. 2023. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. arXiv:1910.10683 [cs.LG] Towards Transparent AI: A Survey on Explainable Large Language Models Conference’24, May 2024, Miami, FL, USA
2023 arXiv
-
[16]
and et al
Kokalj E. and et al. 2021. BERT meets shapley: Extending SHAP explanations to transformer-based classifiers. In EACL Hackashop. 16– 21
2021
-
[17]
and et al
Koto F. and et al. 2021. Discourse Probing of Pretrained Language Models. In NAACL-HLT. 3849–3864
2021
-
[18]
Rajani N. F. and et al. 2019. Explain Yourself! Leveraging Language Models for Commonsense Reasoning. In ACL. 4932–4942
2019
-
[19]
and et al
Brunner G. and et al. 2020. On Identifiability in Transformers. arXiv:1908.04211 [cs.CL]
2020 arXiv
-
[20]
and et al
Izacard G. and et al. 2021. Leveraging Passage Retrieval with Genera- tive Models for Open Domain Question Answering. InEACL. 874–880
2021
-
[21]
and et al
Chung H. and et al. 2022. Scaling Instruction-Finetuned Language Models. arXiv:2210.11416
2022 arXiv
-
[23]
and et al
Mohebbi H. and et al. 2021. Exploring the Role of BERT Token Repre- sentations to Explain Sentence Probing Results. In EMNLP. 792–806
2021
-
[24]
and et al
Touvron H. and et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv:2307.09288 [cs.AI]
2023 arXiv
-
[26]
and et al
Tenney I. and et al. 2019. What do you learn from context? Probing for sentence structure in contextualized word representations. In ICLR
2019
- [27]
-
[28]
and et al
Devlin J. and et al. 2018. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv:1810.04805 [cs.CL]
2018 arXiv
-
[29]
and et al
Enouen J. and et al. 2024. TextGenSHAP: Scalable Post-Hoc Explana- tions in Text Generation with Long Documents. InACL. 13984–14011
2024
-
[30]
and et al
Hewitt J. and et al. 2019. Designing and Interpreting Probes with Control Tasks. arXiv:1909.03368 [cs.CL]
2019 arXiv
-
[31]
and et al
Juraj J. and et al. 2021. Attention Is Indeed All You Need: Semantically Attention-Guided Decoding for Data-to-Text NLG. In INLG. 416–431
2021
-
[32]
and et al
Lee J. and et al. 2020. BioBERT: A Pre-trained Biomedical Language Representation Model for Biomedical Text Mining. Bioinformatics 36, 4 (2020), 1234–1240
2020
-
[34]
and et al
Wei J. and et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. NeurIPS 35 (2022), 24824–24837
2022
-
[35]
and et al
Wei J. and et al. 2022. Emergent abilities of large language models. arXiv:2206.07682 [cs.AI]
2022 arXiv
-
[36]
and et al
Yu J. and et al. 2022. INTERACTION: A Generative XAI Framework for Natural Language Inference Explanations. arXiv:2209.01061 [cs.CL]
2022 arXiv
-
[37]
and et al
Lee K. and et al. 2018. Higher-Order Coreference Resolution with Coarse-to-Fine Inference. In NAACL-HLT. 687–692
2018
-
[38]
and et al
Lampinen K. and et al. 2022. Can language models learn from expla- nations in context? arXiv:2204.02329 [cs.AI]
2022 arXiv
-
[39]
and et al
Meng K. and et al. 2022. Locating and editing factual associations in GPT. In NeurIPS. 17359–17372
2022
-
[40]
and et al
Solomon K. and et al. 1951. On Information and Sufficiency.The Annals of Mathematical Statistics 22, 1 (March 1951), 79–86
1951
-
[41]
and et al
Andrew L. and et al. 2023. Complementary Explanations: In-Context Learning via Composition of Natural Language and Computation Traces. In ACL
2023
-
[42]
and et al
He L. and et al. 2018. Jointly Predicting Predicates and Arguments in Neural Semantic Role Labeling. In ACL. 364–369
2018
-
[43]
and et al
Chen M. and et al. 2021. Evaluating large language models trained on code. arXiv:2107.03374 [cs.LG]
2021 arXiv
-
[44]
and et al
Lewis M. and et al. 2019. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Com- prehension. arXiv:1910.13461 [cs.CL]
2019 arXiv
-
[46]
Mitchell T. M. and et al. 1982. Generalization as search. Artificial Intelligence 18, 2 (1982), 203–226
1982
-
[48]
and et al
Nanda N. and et al. 2023. Progress measures for grokking via mecha- nistic interpretability. In ICLR
2023
-
[49]
and et al
Chris O. and et al. 2020. Zoom in: An introduction to circuits. Distill 5, 3 (2020), e00024–001
2020
-
[50]
and et al
Catherine O. and et al. 2022. In-context learning and induction heads. arXiv:2209.11895 [cs.CL]
2022 arXiv
-
[51]
and et al
Mavrepis P. and et al. 2024. XAI for All: Can Large Language Models Simplify Explainable AI? arXiv:2401.13110 [cs.AI]
2024 arXiv
-
[52]
and et al
Alec R. and et al. 2019. Language models are unsupervised multitask learners. OpenAI blog 1, 8 (2019), 9
2019
-
[53]
and et al
Bommasani R. and et al. 2021. On the opportunities and risks of foundation models. arXiv:2108.07258 [cs.AI]
2021 arXiv
-
[54]
and et al
Wang R. and et al. 2022. Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 Small. In ICLR
2022
-
[55]
and et al
Cao S. and et al. 2021. Attention Head Masking for Inference Time Content Selection in Abstractive Summarization. InNAACL-HLT. 5008– 5016
2021
-
[57]
and et al
Li S. and et al. 2023. Analyzing In-Context Learning with Saliency Methods: A Case Study on Contrastive Demonstrations. (2023). arXiv:2307.05052 [cs.CL]
2023 arXiv
-
[58]
and et al
Narang S. and et al. 2020. Wt5?! training text-to-text models to explain their predictions. arXiv:2004.14546 [cs.CL]
2020 arXiv
-
[59]
and et al
Kojima T. and et al. 2022. Large language models are zero-shot rea- soners. NeurIPS 35 (2022), 22199–22213
2022
-
[60]
Why should I trust you?
Ribeiro M. T. and et al. 2016. "Why should I trust you?": Explaining the predictions of any classifier. In KDD. 1135–1144
2016
-
[62]
and et al
Mann W. and et al. 1986. Assertions from Discourse Structure. In SC-NL Workshop
1986
-
[63]
and et al
Dang Y. and et al. 2024. Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Survey. arXiv:2412.02104
2024 arXiv
-
[64]
and et al
Liu Y. and et al. 2024. On the Reliability and Explainability of Language Models for Program Generation. ACM Trans. Softw. Eng. Methodol. 33, 5, Article 126 (2024), 26 pages
2024
-
[65]
and et al
Wang Y. and et al. 2023. CodeT5+: Open Code Large Language Models for Code Understanding and Generation. In EMNLP. 1069–1088
2023
-
[66]
and et al
Yordan Y. and et al. 2022. Few-Shot Out-of-Domain Transfer Learn- ing of Natural Language Explanations in a Label-Abundant Setup. In EMNLP. 3486–3501
2022
-
[67]
and et al
Zhou Y. and et al. 2021. DirectProbe: Studying Representations without Classifiers. In NAACL-HLT. 5070–5083
2021
-
[68]
and et al
Li Z. and et al. 2021. Implicit Representations of Meaning in Neural Language Models. In ACL-IJCNLP. 1813–1827
2021
-
[69]
and et al
Li Z. and et al. 2022. Automating Code Review Activities by Large-Scale Pre-training. arXiv:2203.09095 [cs.CL]
2022 arXiv
-
[70]
and et al
Wu Z. and et al. 2020. Perturbed Masking: Parameter-free Probing for Analyzing and Interpreting BERT. In ACL. 4166–4176
2020
-
[71]
and et al
Wu Z. and et al. 2021. On Explaining Your Explanations of BERT: An Empirical Study with Sequence Classification. arXiv:2101.00196
2021 arXiv
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.