Pith. sign in

REVIEW 3 major objections 5 minor 33 references

Rashomon in the Streets: Explanation Ambiguity in Scene Understanding

T0 review · 3 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Equally good driving-scene models give sharply different explanations for the same action, indicating that explanation ambiguity is intrinsic to the task.

desk verdict A useful empirical caution for driving XAI, but the 'inherent property' claim and the unresolved SHAP/integrated-gradients ambiguity need serious work before publication. read the letter →

arxiv 2509.03169 v1 pith:KHTDTZUJ submitted 2025-09-03 cs.LG cs.AI

Helge Spieker , J{o}rn Eirik Betten , Arnaud Gotlieb , Nadjib Lazaar , Nassim Belmecheri This is my paper · ORCID
classification cs.LGcs.AI
keywords RashomoneffectexplainableAIfeatureattributionsceneunderstandingautonomousdrivinggraphneuralnetworksgradientboostingqualitativegraphs
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether the Rashomon effect—many equally accurate models trained on the same data—undermines explainable AI for autonomous driving. It trains Rashomon sets of gradient-boosting models and graph neural networks on Qualitative Explainable Graph (QXG) scene representations derived from nuScenes and DriveLM, then measures how much the models' feature-attribution explanations agree. For the boosted models, agreement on full feature rankings is moderate (Kendall's W ≈ 0.32); for the GNNs it is near zero (W ≈ 0.07), even though all models in each set perform almost identically. The paper interprets this as evidence that explanation ambiguity is an inherent property of the action-explanation problem, and argues that trustworthy XAI should present a set of possible rationales rather than a single 'ground-truth' explanation.

What carries the argument

The central object is the Qualitative Explainable Graph (QXG), a symbolic spatio-temporal graph whose nodes are scene objects and whose edges carry qualitative relation labels from spatial and trajectory calculi. Around this, the paper builds a validation Rashomon set: models trained on the same data whose validation loss is within epsilon of the best model. Explanations are generated as feature-attribution rankings via SHAP, and agreement is measured by Fleiss' kappa (for top-k feature selection) and Kendall's W (for full-rank ordering). The comparison of explanations across a Rashomon set is the mechanism that carries the argument: near-identical performance with divergent attributions is

What would settle it

Re-run the same GNN and boosting models while replacing SHAP with another attribution method (e.g., integrated gradients, as the method section names) or repeatedly sampling SHAP background data for a single fixed model; if the near-zero Kendall's W disappears or matches the within-model attribution noise, the claim that ambiguity is inherent to the problem would be undermined.

Watch

Extended reading notes

Core claim

This paper claims that explanation ambiguity is not a modeling artifact but a property of the scene-understanding problem itself. To support this, it constructs validation Rashomon sets—sets of models whose validation loss is within a small epsilon (0.05) of the best model—for two model classes on a real-world driving dataset: 100 gradient-boosting decision trees solving a pair-based action explanation task, and 116 graph neural networks solving a graph-based version, of which 32 met the Rashomon criterion. Across 2,131 ego-vehicle action scenes, the models' SHAP feature attributions were compared using Fleiss' kappa on top-k features and Kendall's W on full rankings. The GNNs showed very lo

Load-bearing premise

The paper treats the divergence in SHAP rankings across equally well-performing models as evidence that the scene-explanation problem itself is ambiguous, rather than as noise from the SHAP attribution tool or from random variation in training.

Editorial extensions

If this is right

  • If explanation ambiguity is inherent, a single post-hoc explanation for a driving decision cannot be treated as the model's true reasoning in safety audits.
  • Consensus explanations computed across a Rashomon set could replace single-model attribution as a more stable basis for human review.
  • Explanation variance across a Rashomon set becomes a measurable form of uncertainty, useful for deciding when a prediction needs extra scrutiny.
  • Training with human relevance labels as an auxiliary loss (or other explanation-regularizing objectives) may reduce but cannot eliminate the multiplicity of rationales.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not establish that the near-zero GNN agreement comes from model multiplicity rather than from the instability of SHAP itself; a control using a second attribution method or repeated SHAP sampling on a single fixed model would separate these sources.
  • The large gap between boosted trees (W ≈ 0.32) and GNNs (W ≈ 0.07) suggests that model class and capacity shape explanation ambiguity independently of the task; a testable prediction is that simpler models on the same QXG features will always exhibit higher explanation concordance than expressive GNNs.
  • Since the action-explanation task is deliberately simplified to single-cause labels, the measured ambiguity is likely a lower bound; labeling multi-causal scenes would probably reveal even larger explanation divergence.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies the Rashomon effect for explanation generation in automated driving scene understanding. It builds Qualitative Explainable Graphs (QXGs) from nuScenes/DriveLM and trains two model families: pair-based LightGBM gradient boosting models and graph-based GNNs. For each family, multiple models are trained with different random seeds, and validation Rashomon sets are selected using an epsilon of 0.05. Feature attributions are computed for the annotated relevant object and ranked; agreement across models is quantified by Fleiss' kappa for top-k feature sets and Kendall's W for full rankings. The main empirical finding is low-to-moderate explanation agreement: Kendall's W is 0.32 for pair-based models and 0.07 for graph-based models. The paper interprets this as evidence that explanation ambiguity is an inherent property of the problem, not merely a modeling artifact.

Significance. If correct, the finding would be a useful caution for post-hoc XAI in safety-critical domains, extending earlier Rashomon-XAI work from controlled settings to a real driving dataset. The use of two structurally different model classes and a public driving dataset is a concrete step, and the authors provide a reproducibility link. However, the strength of the conclusion currently exceeds what the experimental protocol can establish: the attribution pipeline is under-specified, no no-model-change control is included, and only one dataset/split/model representation is examined. The empirical observation is valuable, but the 'inherent property' framing is not yet supported.

major comments (3)
  1. [Graph-based Action Explanation; Data Collection; Technical Setup] The attribution method is described inconsistently. The graph-based section states integrated gradients are 'used in the experiments', Data Collection says SHAP, and Technical Setup attributes all explanations to Captum. For LightGBM, Captum does not natively provide TreeSHAP, so it is unclear what quantity was ranked. More importantly, the low Kendall's W values could reflect instability of the attribution estimator rather than genuine model multiplicity. The paper needs a fixed-model control: for a single trained model, compute attributions multiple times (e.g., different baselines or background samples) and report the resulting W. Without this, the central claim that disagreement is due to the Rashomon set is not established.
  2. [Abstract; Conclusion] The claim that explanation ambiguity is 'an inherent property of the problem, not just a modeling artifact' goes beyond the evidence. The study uses one dataset (nuScenes subset), one train/validation split, one scene representation (QXG), two model families, one epsilon value, and one attribution pipeline. Such a design cannot establish an inherent property. The conclusion should be reframed to 'observed in this setting' or supported by additional experiments (e.g., different splits, model families, attribution methods, epsilon sensitivity). This is not a wording issue: the overclaim is load-bearing for the paper's stated contribution.
  3. [Experimental Setup, Model Selection; Eq. (2)] The operationalization of the Rashomon set is unclear. Definition 2 uses an additive epsilon on validation loss, but the text says 'at least 95% performance of the best model' and later mentions models are 'within 2% performance'. These are different selection criteria, and no per-model validation losses or accuracies are reported. Without showing the validation performance distribution, 'equally accurate' is not established, and a broad 5% band could include models with materially different behavior. Please report the validation performance of all selected models and clarify whether epsilon is additive in loss or relative to the best model's performance.
minor comments (5)
  1. [Metrics, Eq. (3)] The paper should clarify whether Fleiss' kappa is computed with fixed or free marginal probabilities, since the value depends on this choice. The cited Randolph reference suggests free-marginal kappa, but the text does not state it explicitly.
  2. [Metrics, Kendall's W] The formula W = 12S / (m^2(n^3 - n)) omits a tie correction. Feature attributions often contain ties, so please state whether ties were handled and, if so, how.
  3. [Figure 5] The legend labels 'Model' with 'Pair-based' and 'Graph-based' is clear, but the shaded areas are not defined in the caption. Please specify whether they are standard deviations or confidence intervals.
  4. [Background] Typo: 'it’s definition' should be 'its definition'. Also, the reference to 'M ¨uller' has inconsistent spacing in the bibliography.
  5. [Technical Setup] The footnote 'Online: https://doi.org/10.5281/zenodo.17045085' should describe exactly what is released (code, data, model weights) and the version used, so that the experiments are reproducible.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: explanation disagreement is empirically measured, not forced by construction or by self-citations.

full rationale

The paper's chain is empirical rather than derivational: it defines a validation Rashomon set by validation loss within an epsilon bound (Definition 2), trains many models from different seeds, selects those within 5% of the best validation performance, and then measures explanation agreement using external attribution libraries and standard rank statistics. The central observation—low Kendall's W (≈0.07 for GNNs, 0.32 for gradient boosting)—is not encoded in the Rashomon-set definition, which conditions only on validation loss, not on explanation similarity. The epsilon=0.05 cutoff is an experimental design choice and does not mathematically force the measured agreement values. The self-citations to Belmecheri et al. (QXG representation and GNN architecture) supply the input representation and model class, not the target conclusion; no uniqueness theorem, fitted parameter, or ansatz is imported to make the result true by definition. The paper does not rename a known result or fit a parameter and then call it a prediction. The main caveats—possible attribution-method noise, the SHAP vs. integrated-gradients inconsistency, and the absence of a no-model-change control—are threats to internal validity and interpretation, but they are not circularity: low agreement could have alternative causes, but that does not make the observed agreement equivalent to the paper's inputs by construction. Thus there is no significant circularity.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The study is empirical and observational. The main postulates are the sufficiency of the QXG representation, the faithfulness of SHAP explanations, the validity of validation-loss-based Rashomon set selection, and the simplification of action explanation to a single-cause problem. The only hand-chosen parameter is the Rashomon threshold epsilon=0.05.

free parameters (2)
  • epsilon (Rashomon threshold) = 0.05
    Hand-chosen threshold defining the Rashomon set as models within 95% of the best validation performance. The size of the Rashomon set, and thus the measured agreement, depends on this value.
  • number of trained models/seeds = 100 (pair-based), 116 (GNN)
    Arbitrary counts; different numbers of random seeds would change the statistical stability of the agreement metrics.
assumptions (4)
  • domain assumption Validation loss is an acceptable proxy for population loss in defining the Rashomon set
    Definition 2 replaces the population expectation with an empirical average on a validation set, a standard but nontrivial assumption that the selected models are truly equally performing on unseen data.
  • domain assumption The simplified action explanation problem (single causing object) captures the essence of scene understanding
    Section 'Problem Description' acknowledges actions can be multi-causal but assumes the single-cause label suffices for studying explanation ambiguity, which may limit the generalizability of the findings.
  • domain assumption SHAP/feature-attribution values faithfully reflect the models' decision rationale
    The entire agreement measurement rests on the validity of the chosen attribution method. The paper does not validate attribution stability or fidelity, and the attribution method is inconsistently described.
  • domain assumption The QXG representation contains enough scene information for action explanation
    The paper relies on QXG from prior work. If the representation omits action-relevant information, models may be forced to use spurious correlations, artificially inflating explanation disagreement.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Rashomon in the Streets: Explanation Ambiguity in Scene Understanding." pith.science (2026). https://pith.science/paper/KHTDTZUJ

@misc{pith2026250903169,
  author       = {Pith},
  title        = {Pith review of: Rashomon in the Streets: Explanation Ambiguity in Scene Understanding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KHTDTZUJ}},
  note         = {Machine review of arXiv:2509.03169}
}
read the original abstract

Explainable AI (XAI) is essential for validating and trusting models in safety-critical applications like autonomous driving. However, the reliability of XAI is challenged by the Rashomon effect, where multiple, equally accurate models can offer divergent explanations for the same prediction. This paper provides the first empirical quantification of this effect for the task of action prediction in real-world driving scenes. Using Qualitative Explainable Graphs (QXGs) as a symbolic scene representation, we train Rashomon sets of two distinct model classes: interpretable, pair-based gradient boosting models and complex, graph-based Graph Neural Networks (GNNs). Using feature attribution methods, we measure the agreement of explanations both within and between these classes. Our results reveal significant explanation disagreement. Our findings suggest that explanation ambiguity is an inherent property of the problem, not just a modeling artifact.

Figures

Figures reproduced from arXiv: 2509.03169 by the authors.

Figure 1
Figure 1. Illustration of the QXG built from a scene. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Extracted star graph centered around action [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Edge-list-based Action Explanation (adapted from [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Graph Neural Network architecture for action clas [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Agreement on top-k features among models. We observe a higher agreement for the Pair-based LightGBM than the [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

33 extracted references · 31 canonical work pages

  1. [1]

    Z.; He, H.; and et al

    Ansel, J.; Yang, E. Z.; He, H.; and et al. 2024. PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation. In ASPLOS

    work page 2024

  2. [2]

    Belmecheri, N.; Gotlieb, A.; Lazaar, N.; and Spieker, H. 2023. Acquiring Qualitative Explainable Graphs for Automated Driving Scene Interpretation. CoRR, abs/2308.12755

    work page Pith review arXiv 2023

  3. [3]

    Belmecheri, N.; Gotlieb, A.; Lazaar, N.; and Spieker, H. 2024. Toward Trustworthy Automated Driving through Qualitative Scene Understanding and Explanations. SAE International Journal of Connected and Automated Vehicles

    work page 2024

  4. [4]

    Belmecheri, N.; Gotlieb, A.; Lazaar, N.; and Spieker, H. 2025 a . Automatic Cause Determination in Road Scene Understanding Using Qualitative Reasoning and Four-Valued Logic. In Advancing Automated Driving in Highly Interactive Scenarios through Behavior Prediction, Trustworthy AI, and Remote Operations @ Intelligent Vehicles (IV)

    work page 2025

  5. [5]

    Belmecheri, N.; Gotlieb, A.; Lazaar, N.; and Spieker, H. 2025 b . Explainable Scene Understanding with Qualitative Representations and Graph Neural Networks. In Advancing Automated Driving in Highly Interactive Scenarios through Behavior Prediction, Trustworthy AI, and Remote Operations @ Intelligent Vehicles (IV)

    work page 2025

  6. [6]

    Breiman, L. 2001. Statistical Modeling : The Two Cultures (with comments and a rejoinder by the author). Statistical Science, 16(3): 199--231

    work page 2001

  7. [7]

    H.; Vora, S.; Liong, V

    Caesar, H.; Bankiti, V.; Lang, A. H.; Vora, S.; Liong, V. E.; Xu, Q.; Krishnan, A.; Pan, Y.; Baldan, G.; and Beijbom, O. 2020. nuScenes: A Multimodal Dataset for Autonomous Driving. In CVPR

    work page 2020

  8. [8]

    Dwivedi, R.; Dave, D.; Naik, H.; Singhal, S.; Omer, R.; Patel, P.; Qian, B.; Wen, Z.; Shah, T.; Morgan, G.; and Ranjan, R. 2023. Explainable AI ( XAI ): Core Ideas , Techniques , and Solutions . ACM Comput. Surv., 55(9): 194:1--194:33

    work page 2023

Show all 33 references
  1. [9]

    H.; Mossakowski, T.; Schneider, T.; van Delden, A.; van de Ven, J.; and Wolter, D

    Dylla, F.; Lee, J. H.; Mossakowski, T.; Schneider, T.; van Delden, A.; van de Ven, J.; and Wolter, D. 2017. A Survey of Qualitative Spatial and Temporal Calculi: Algebraic and Computational Properties. ACM Comput. Surv. , 50(1)

    2017

  2. [10]

    Fey, M.; and Lenssen, J. E. 2019. Fast Graph Representation Learning with PyTorch Geometric . In ICLR Workshop on Representation Learning on Graphs and Manifolds

    2019

  3. [11]

    Fisher, A.; Rudin, C.; and Dominici, F. 2019. All Models are Wrong , but Many are Useful : Learning a Variable ’s Importance by Studying an Entire Class of Prediction Models Simultaneously . JMLR , 20: 177

    2019

  4. [12]

    Fleiss, J. L. 1971. Measuring nominal scale agreement among many raters. Psychological bulletin, 76(5): 378

    1971

  5. [13]

    Friedman, J. H. 2001. Greedy function approximation: a gradient boosting machine. Annals of statistics, 1189--1232

    2001

  6. [14]

    Friedman, J. H. 2002. Stochastic gradient boosting. Computational statistics & data analysis, 38(4): 367--378

    2002

  7. [15]

    Geirhos, R.; Jacobsen, J.-H.; Michaelis, C.; Zemel, R.; Brendel, W.; Bethge, M.; and Wichmann, F. A. 2020. Shortcut learning in deep neural networks. Nature Machine Intelligence

    2020

  8. [16]

    Ke, G.; Meng, Q.; Finley, T.; Wang, T.; Chen, W.; Ma, W.; Ye, Q.; and Liu, T. 2017. LightGBM: A Highly Efficient Gradient Boosting Decision Tree. In NeurIPS , 3146--3154

    2017

  9. [17]

    Kendall, M. 1962. Rank Correlation Methods. Griffin

    1962

  10. [18]

    Kokhlikyan, N.; Miglani, V.; Martin, M.; Wang, E.; Alsallakh, B.; Reynolds, J.; Melnikov, A.; Kliushkina, N.; Araya, C.; Yan, S.; and Reblitz-Richardson, O. 2020. Captum: A unified and generic model interpretability library for PyTorch. arXiv:2009.07896

    2020 arXiv

  11. [19]

    F.; and G \'o mez, E

    Llorca, D. F.; and G \'o mez, E. 2021. Trustworthy Autonomous Vehicles: Assessment Criteria for Trustworthy AI in the Autonomous Driving Domain. Publications Office of the EU

    2021

  12. [20]

    M.; and Lee, S

    Lundberg, S. M.; and Lee, S. 2017. A Unified Approach to Interpreting Model Predictions. In NeurIPS

    2017

  13. [21]

    D.; Rezaei, M.; Kumar, N.; Hijji, M.; Bellavista, P.; and de Albuquerque, V

    Muhammad, K.; Hussain, T.; Ullah, H.; Ser, J. D.; Rezaei, M.; Kumar, N.; Hijji, M.; Bellavista, P.; and de Albuquerque, V. H. C. 2022. Vision-Based Semantic Segmentation in Scene Understanding for Autonomous Driving: Recent Achievements, Challenges, and Outlooks. IEEE Trans. I...

    2022

  14. [22]

    D.; and de Albuquerque, V

    Muhammad, K.; Ullah, A.; Lloret, J.; Ser, J. D.; and de Albuquerque, V. H. C. 2021. Deep Learning for Safe Autonomous Driving: Current Challenges and Future Directions. IEEE Trans. Intell. Transp. Syst. , 22(7): 4316--4336

    2021

  15. [23]

    M \" u ller, S.; Toborek, V.; Beckh, K.; Jakobs, M.; Bauckhage, C.; and Welke, P. 2023. An Empirical Evaluation of the Rashomon Effect in Explainable Machine Learning. In ECML PKDD

    2023

  16. [24]

    Nastjuk, I.; Herrenkind, B.; Marrone, M.; Brendel, A.; and Kolbe, L. 2020. What drives the acceptance of autonomous driving? An investigation of acceptance factors from an end-user's perspective. Technological Forecasting and Social Change, 161

    2020

  17. [25]

    Pedregosa, F.; Varoquaux, G.; Gramfort, A.; Michel, V.; Thirion, B.; Grisel, O.; Blondel, M.; Prettenhofer, P.; Weiss, R.; Dubourg, V.; VanderPlas, J.; Passos, A.; Cournapeau, D.; Brucher, M.; Perrot, M.; and Duchesnay, E. 2011. Scikit-learn: Machine Learning in Python. JMLR ,...

    2011

  18. [26]

    Randolph, J. J. 2005. Free-Marginal Multirater Kappa (multirater K [free]): An Alternative to Fleiss' Fixed-Marginal Multirater Kappa. Paper presented at the Joensuu Learning and Instruction Symposium (Joensuu, Finland)

    2005

  19. [27]

    Renz, J.; and Nebel, B. 2007. Qualitative Spatial Reasoning Using Constraint Calculi . In Handbook of Spatial Logics

    2007

  20. [28]

    Rudin, C.; Chen, C.; Chen, Z.; Huang, H.; Semenova, L.; and Zhong, C. 2022. Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges. Statistics Surveys, 16(none): 1--85

    2022

  21. [29]

    Semenova, L.; Rudin, C.; and Parr, R. 2022. On the Existence of Simpler Machine Learning Models . In 2022 ACM Conference on Fairness , Accountability , and Transparency

    2022

  22. [30]

    Sima, C.; Renz, K.; Chitta, K.; Chen, L.; Zhang, H.; Xie, C.; Luo, P.; Geiger, A.; and Li, H. 2023. DriveLM: Driving with Graph Visual Question Answering. CoRR, abs/2312.14150

    2023 arXiv

  23. [31]

    Sundararajan, M.; Taly, A.; and Yan, Q. 2017. Axiomatic Attribution for Deep Networks. In ICML

    2017

  24. [32]

    Yang, S.; Wang, W.; Liu, C.; and Deng, W. 2019. Scene Understanding in Deep Learning-Based End-to-End Controllers for Autonomous Vehicles. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 49(1): 53--63

    2019

  25. [33]

    I.; and Ravikumar, P

    Yeh, C.-K.; Hsieh, C.-Y.; Suggala, A.; Inouye, D. I.; and Ravikumar, P. K. 2019. On the ( In )fidelity and Sensitivity of Explanations . In NeurIPS

    2019

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.