Comparing Explanations is Not Enough, Explain the Change: New Standards are Needed to Explain Behavioral Shifts in Large Language Models

Fosca Giannotti; Francesco Giannini; Luca Pappalardo; Martino Ciaperoni; Marzio Di Vece; Roberto Pellungrini

arxiv: 2602.02304 · v2 · pith:P7GAEHENnew · submitted 2026-02-02 · 💻 cs.AI · cs.LG

Comparing Explanations is Not Enough, Explain the Change: New Standards are Needed to Explain Behavioral Shifts in Large Language Models

Martino Ciaperoni , Marzio Di Vece , Roberto Pellungrini , Luca Pappalardo , Fosca Giannotti , Francesco Giannini This is my paper

Pith reviewed 2026-05-21 13:33 UTC · model grok-4.3

classification 💻 cs.AI cs.LG

keywords explainable AIbehavioral shiftslarge language modelscomparative explanationsAI governancemodel auditinginterventionsXAI desiderata

0 comments

The pith

Explaining behavioral shifts in large language models requires treating the functional transition itself as the primary object of explanation.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Large language models exhibit behavioral shifts after interventions such as fine-tuning, scaling, or reinforcement learning with human feedback. Current explainability methods either analyze a single static model or compare independent explanations from separate checkpoints, which fails to capture how the intervention produces the observed change. This paper argues for Comparative XAI, or XAI_Δ, a new paradigm that explains the difference between a reference model and an updated model by focusing on the shift as the central target. It specifies desiderata that such explanations must meet, including comparability, validity, actionability, and monitoring, to support auditing. Preliminary experiments demonstrate the approach by generating transition reports usable for governance and regulatory documentation.

Core claim

The central claim is that current explainability methods are structurally ill-suited to explain behavioral shifts in large language models because they either treat models as static objects or merely compare independent explanations across different checkpoints. To address this, the paper introduces Comparative XAI (XAI_Δ) aimed at explaining the difference between two model checkpoints where a behavior has shifted, together with desiderata specifying what XAI_Δ explainers and explanations must satisfy, including comparability, validity, actionability, and monitoring, with the goal of grounding model auditing in explicit, measurable requirements.

What carries the argument

Comparative XAI (XAI_Δ), the paradigm that explains how and why an intervention transforms a reference model into an updated model with different behavior.

If this is right

Regulatory requirements for documenting causal chains of model modifications, such as those in the EU AI Act, can be met through explicit transition explanations.
Model auditing gains explicit, measurable requirements instead of relying on ad-hoc snapshot comparisons.
Interventions like fine-tuning or in-context learning become traceable in terms of their specific effects on behavior.
Transition reports generated by the new paradigm serve directly for governance and incident documentation.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Developers could use this approach to create standardized logs of behavioral changes across successive model releases in deployed systems.
The framework might help identify patterns in how certain interventions reliably produce or avoid specific behavioral shifts.
Extending the method to non-language AI systems could address similar explanation gaps when models are updated over time.

Load-bearing premise

The gap in explaining functional transitions between model instances creates significant governance risks because regulations require documenting causal chains for substantial system modifications.

What would settle it

A case study showing that applying existing static or comparative XAI techniques to pre- and post-intervention model checkpoints fully identifies and validates the causal mechanisms driving the behavioral shift without requiring any dedicated transition-focused method.

read the original abstract

Large-scale foundation models exhibit \emph{behavioral shifts} when subjected to interventions such as scaling, fine-tuning, reinforcement learning with human feedback, or in-context learning. Current explainability methods are structurally ill-suited to explain these shifts, because they either treat models as static objects, as traditional eXplainable AI (XAI) approaches do, or merely compare independent explanations across different checkpoints of a model. As a result, these approaches fail to explain the functional transition between two model instances in which a certain behavior has shifted following an intervention. This gap creates significant governance risks across jurisdictions including the EU AI Act, US state legislation, and Chinese AI regulations, which require documenting causal chains for substantial system modifications. This position paper argues that explaining behavioral shifts in large language models requires a principled approach that treats the shift itself as the primary object of explanation: namely, one that explains how and why an intervention transforms a reference model into an updated model with different behavior. To support this claim, we introduce \textit{Comparative} XAI (XAI$_\Delta$), a novel XAI paradigm aimed at explaining the difference between two model checkpoints where a behavior has shifted, together with a set of desiderata specifying what XAI$_\Delta$ explainers and explanations must satisfy, including comparability, validity, actionability, and monitoring, with the goal of grounding model auditing in explicit, measurable requirements. Finally, we provide preliminary evidence suggesting the need for XAI$_\Delta$ in practice through illustrative experiments, compiling the resulting findings into a transition report directly usable for governance and incident documentation.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

The paper flags a real gap in using static or comparative XAI for LLM behavioral shifts but does not test whether adapted existing methods could suffice, leaving the case for a new paradigm preliminary.

read the letter

Hi, the main thing to know is that this position paper argues current explainability tools cannot properly account for why an LLM's behavior changed after interventions like fine-tuning or scaling, and it pushes for treating the shift itself as the primary object to explain under a new XAI_Δ framing with four clear desiderata: comparability, validity, actionability, and monitoring. It ties the issue to practical governance needs such as documenting causal chains under the EU AI Act and similar rules, which is a useful connection. The illustrative experiments and the suggestion of compiling results into a transition report give a concrete sense of how this could support auditing in practice. That part lands reasonably well and gives the proposal some operational flavor. The soft spot is that the central claim of structural ill-suitedness for existing methods rests on conceptual distinction and examples rather than direct evidence. The manuscript does not run a systematic check on whether simple adaptations, such as subtracting attributions across checkpoints or paired counterfactual analysis on reference versus updated models, could isolate the drivers of the observed change. Without that, the argument for requiring a wholly new paradigm stays suggestive. The evidence overall is preliminary with no quantitative validation or error analysis, so the proposal functions more as a call for new standards than a demonstrated requirement. Citations to traditional XAI work look appropriate and there are no obvious logical gaps or invented math. This is aimed at XAI researchers and people working on AI compliance and auditing. Readers focused on model change documentation will find the desiderata and regulatory angle worth considering. It deserves peer review to see whether the community views the gap as fundamental or addressable through extensions of current tools.

Referee Report

3 major / 2 minor

Summary. The paper claims that existing XAI methods are structurally ill-suited to explain behavioral shifts in LLMs after interventions such as fine-tuning or RLHF, because they treat models as static or only compare independent explanations across checkpoints. It proposes a new paradigm called Comparative XAI (XAI_Δ) that treats the functional transition itself as the object of explanation, introduces desiderata including comparability, validity, actionability, and monitoring, and supports the position with illustrative experiments that produce a transition report for governance and auditing purposes.

Significance. If the central argument holds, the work could help shift XAI research and practice toward dynamic explanations of model changes, supporting more rigorous auditing and documentation of AI system modifications. The introduction of explicit desiderata and the concept of a transition report represent a constructive contribution to governance discussions, even if the supporting experiments remain preliminary.

major comments (3)

[Abstract and §1] Abstract and §1: The core claim that current methods 'fail to explain the functional transition' rests on a conceptual distinction but does not include a systematic argument or test ruling out adaptations of existing techniques (such as differencing attributions between paired checkpoints or intervention analysis on reference vs. updated models). Without addressing whether such adaptations could isolate the mechanisms driving the observed shift, the necessity of an entirely new XAI_Δ paradigm is not yet established.
[§4] §4 (Desiderata for XAI_Δ): The four desiderata are presented at a high level, but the manuscript does not provide operational definitions, measurable criteria, or even illustrative examples showing how 'validity' or 'actionability' would be assessed for explanations of behavioral shifts in models with billions of parameters.
[§5] §5 (Illustrative experiments): The experiments are described as preliminary and produce a transition report, yet the text provides no quantitative metrics, baseline comparisons against adapted standard XAI methods, or error analysis, leaving the practical advantage of XAI_Δ over existing approaches unquantified.

minor comments (2)

[Abstract] The abstract and introduction repeatedly use the term 'transition report' without defining its required structure, content, or how it would be generated from XAI_Δ outputs; a brief specification would improve clarity for readers interested in governance applications.
[Related Work] The paper would benefit from additional citations to recent work on model editing, continual learning, and attribution methods applied across training checkpoints to better situate the proposed paradigm relative to ongoing research.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the detailed and constructive feedback on our position paper. We address each major comment below, clarifying the scope and intent of the work while indicating revisions that will strengthen the manuscript.

read point-by-point responses

Referee: [Abstract and §1] Abstract and §1: The core claim that current methods 'fail to explain the functional transition' rests on a conceptual distinction but does not include a systematic argument or test ruling out adaptations of existing techniques (such as differencing attributions between paired checkpoints or intervention analysis on reference vs. updated models). Without addressing whether such adaptations could isolate the mechanisms driving the observed shift, the necessity of an entirely new XAI_Δ paradigm is not yet established.

Authors: The manuscript's argument in the abstract and §1 is primarily conceptual: existing XAI methods are architecturally oriented toward static models or independent post-hoc comparisons, which inherently do not model the intervention as a causal transformation between states. Adaptations such as attribution differencing or paired intervention analysis remain tethered to the assumptions of the original methods and do not produce an explanation of the transition itself. We will revise §1 to include an explicit subsection discussing common adaptations and why they remain insufficient for explaining functional shifts, thereby more directly addressing the necessity of the XAI_Δ paradigm. revision: partial
Referee: [§4] §4 (Desiderata for XAI_Δ): The four desiderata are presented at a high level, but the manuscript does not provide operational definitions, measurable criteria, or even illustrative examples showing how 'validity' or 'actionability' would be assessed for explanations of behavioral shifts in models with billions of parameters.

Authors: As a position paper, §4 introduces the desiderata at a conceptual level to define the requirements for the new paradigm. We agree that illustrative examples would improve clarity. We will add a subsection to §4 providing concrete examples of assessing validity (e.g., cross-validation against known intervention effects in smaller models) and actionability (e.g., linking explanations to specific modifiable components like data subsets or layers), while noting that full operationalization for billion-parameter models is a direction for subsequent research. revision: yes
Referee: [§5] §5 (Illustrative experiments): The experiments are described as preliminary and produce a transition report, yet the text provides no quantitative metrics, baseline comparisons against adapted standard XAI methods, or error analysis, leaving the practical advantage of XAI_Δ over existing approaches unquantified.

Authors: Section §5 explicitly frames the experiments as illustrative demonstrations of producing a transition report rather than a full empirical benchmark. We will revise the section to more clearly state this scope, expand the limitations discussion, and include qualitative comparisons to standard methods where feasible. However, adding rigorous quantitative baselines and error analysis would require new, extensive experiments that are outside the current scope and timeline of this position paper. revision: partial

Circularity Check

0 steps flagged

No significant circularity; conceptual position paper without derivations or reductions

full rationale

The manuscript is a position paper that advances a conceptual argument for new XAI standards to address behavioral shifts in LLMs. It defines XAI_Δ and lists desiderata (comparability, validity, actionability, monitoring) but contains no equations, fitted parameters, mathematical derivations, or self-citation chains that reduce any claim to its own inputs by construction. The central premise rests on analysis of limitations in static or comparative XAI approaches, supported by illustrative experiments and governance references, without renaming known results or smuggling ansatzes via prior work. The derivation chain is self-contained and does not exhibit any of the enumerated circularity patterns.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 1 invented entities

The central claim rests on the domain assumption that interventions produce behavioral shifts requiring new explanatory tools, and introduces the XAI_Δ paradigm as a novel construct without independent empirical grounding outside the paper.

axioms (1)

domain assumption Interventions such as scaling, fine-tuning, reinforcement learning with human feedback, or in-context learning produce behavioral shifts in large language models.
Presented as the foundational premise in the abstract without further justification or citation to specific empirical studies.

invented entities (1)

Comparative XAI (XAI_Δ) no independent evidence
purpose: A novel paradigm for explaining the functional transition between two model checkpoints where behavior has shifted.
Newly proposed explanatory framework without external validation or prior literature precedent cited.

pith-pipeline@v0.9.0 · 5853 in / 1290 out tokens · 37218 ms · 2026-05-21T13:33:03.027306+00:00 · methodology

discussion (0)

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

IndisputableMonolith/Foundation/AbsoluteFloorClosure.lean reality_from_one_distinction unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

We argue that to explain behavioral shifts in LLMs we need the development of a novel branch of AI: Comparative XAI (∆-XAI). A ∆-XAI method should be able to explain a behavioral shift, i.e., the difference between an LLM’s behavior before and after a given intervention.

What do these tags mean?

matches: The paper's claim is directly supported by a theorem in the formal canon.
supports: The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends: The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses: The paper appears to rely on the theorem as machinery.
contradicts: The paper's claim conflicts with a theorem or certificate in the canon.
unclear: Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.