REVIEW 3 major objections 5 minor 48 references
ExplainReduce: Generating global explanations from many local explanations
T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read A handful of local surrogate models, selected greedily to maximize coverage or minimize loss, can replace a large explanation set and match the closed-box model's fidelity, coverage, and stability on test data.
desk verdict Solid, useful aggregation method for local explanations; the 'five proxies suffice' claim rests on an unvalidated nearest-neighbor gating rule, but the work deserves serious refereeing. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the proxy set selection problem, formalized through a loss matrix $L_{ij} = \ell(g_i(x_j), \hat{y}_j)$ and a coverage threshold $\varepsilon$. The objective functions—coverage $C(S,\varepsilon)$, average loss $L(S)$, and balanced utility $U(S) = \lambda C(S,\varepsilon) + (1-\lambda)(L_{\text{base}}-L(S))/L_{\text{base}}$—are monotone submodular set functions. The greedy algorithm 'reduce' iteratively adds the explanation with the highest marginal gain, and its performance is bounded by the submodular maximization guarantee. For unseen test items, the paper assigns each item to the proxy of its nearest training item in Euclidean feature space.
What would settle it
Construct a closed-box model on a high-dimensional or discontinuous feature space—for example, a random forest with many axis-aligned thresholds—where neighboring points in Euclidean distance frequently land on opposite sides of decision boundaries. If ExplainReduce's five-proxy set then fails to reach the full explanation set's test fidelity, or if test fidelity drops sharply when the nearest-neighbor assignment is replaced with an oracle mapping from the true best local model, the claimed generalization would be contradicted.
Extended reading notes
Core claim
ExplainReduce's central claim is that a large set of local explanations can be reduced to a small 'proxy set' of simple models that serves as a generative global explanation. The reduction is formalized as three optimization problems: maximum coverage, minimum average loss, and a balanced joint utility combining both. Each objective is monotone submodular, so the standard greedy algorithm achieves at least a $(1-1/e)$ approximation to the optimal proxy set. Empirically, the balanced variant reaches or surpasses the full explanation set's fidelity, coverage, and stability with only about five proxies, and does so while being agnostic to the choice of closed-box model and local explanation method.
Load-bearing premise
The test-time evaluation assumes that a novel item can be mapped to the right proxy simply by taking the closest training item in Euclidean feature space, which only works if the closed-box model's local behavior changes smoothly between neighboring points.
Editorial extensions
If this is right
- Users of XAI methods can obtain a global view of a closed-box model by inspecting only a handful of simple models instead of reading hundreds of local explanations.
- ExplainReduce works across regression and classification, and with local explanation methods that produce generative surrogates, including LIME, SHAP, SLISEMAP, SmoothGrad, and LORE.
- The greedy algorithms find proxy sets in seconds, whereas the integer-programming baseline and GLocalX take hundreds to thousands of seconds on the same tasks.
- Proxy sets enable downstream uses such as clustering data by model behavior, outlier detection, and building an interpretable surrogate for the closed-box model.
- Test-set fidelity plateaus at small $k$, so a user can set $k$ around five and still get a faithful and interpretable global explanation.
Reading between the lines
- The paper's reliance on Euclidean nearest-neighbor assignment for test items implicitly assumes that the closed-box function is smooth enough that nearby feature-space points share the same best local explanation; on highly discontinuous or high-dimensional functions, the reported test fidelity may not transfer.
- The proxy-selection view could be adapted for concept-drift monitoring by tracking how the selected proxy set changes over time, since a shift in the most representative local models would signal a change in the closed-box function's behavior.
- For explanation methods that output feature attributions rather than generative surrogates, one could first fit a generative model to the attributions and then apply ExplainReduce, extending its scope beyond the methods tested in the paper.
- The balanced utility formulation could be tuned per application, e.g., prioritizing coverage in safety-critical settings or fidelity in high-stakes prediction, with the greedy guarantee preserved for any $\lambda \in [0,1]$.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes ExplainReduce, a model-agnostic procedure that reduces a large set of local surrogate explanations to a small 'proxy set' of simple models that can serve as a generative global explanation for a closed-box model. The reduction is formalized as three optimization problems (maximum coverage, minimum average loss, and a balanced joint utility), each approximated by greedy submodular maximization with standard (1-1/e)-type guarantees. The empirical study covers twelve datasets, several XAI methods (LIME, SHAP, SLISEMAP, SmoothGrad, LORE), and comparisons with submodular pick, GLocalX, and an IP-based aggregation method. The central claim is that as few as five proxies can faithfully emulate the closed-box model on test data.
Significance. If the empirical claim is fully established, ExplainReduce would be a practically useful, model-agnostic tool for compressing hundreds of local explanations into a compact global surrogate, with clear computational advantages over IP-based and rule-merging baselines. The theoretical component is standard but correct: the submodularity arguments for the coverage, loss-reduction, and balanced objectives are sound, and the experimental breadth is substantial. The main weakness is that the test-fidelity evaluation depends on an ad hoc Euclidean nearest-neighbor assignment for novel items, so the headline generalization claim is not supported as cleanly as the abstract states. Nevertheless, the optimization framework and the overall empirical picture are strong enough that the concerns are addressable within the manuscript's scope.
major comments (3)
- [Section 5, Figs. 5 and 8] The test-fidelity evaluation relies on a nearest-neighbor mapping that is not part of ExplainReduce. Section 5 states that 'each novel item is assigned to a proxy corresponding to the training item that is closest in the feature space using Euclidean distance,' but this gating rule is not learned, not derived from the loss matrix, and not compared against alternatives. Consequently, the reported test fidelity is a joint property of the proxy set and this Euclidean-NN heuristic, not of the proxy set alone. The abstract's claim that 'as few as five explanations can faithfully emulate the closed-box model' is therefore stronger than the evidence supports; the paper itself acknowledges in Section 6 that the method has no consideration of the spatial distribution of the data with respect to proxy assignment. I recommend that the authors report train and test fidelity separately, evaluate at least one alternative assignment rule (for example, direct loss-based assignment on a validation set, or a classifier trained on the training proxy assignments), and qualify the headline claim to refer to the proxy set together with the assignment rule used to deploy it.
- [Algorithm 2, Section 4.2] The mapping step in Algorithm 2 computes map[i] = arg min_{j in S_c} ell(g_j(x_i), y_i), using the original labels y_i, whereas the loss matrix L_ij = ell(g_i(x_j), hat y_j) and the fidelity metric are defined with respect to the closed-box predictions hat y_j. If the implementation follows the pseudocode, the training-time mapping is optimized against the wrong target; if the implementation uses hat y_i, the pseudocode should be corrected. This is a correctness ambiguity in a load-bearing part of the evaluation pipeline, and it should be clarified in the manuscript.
- [Figs. 5-8 and Appendix C] The main quantitative claims are presented as single runs without error bars or repeated-seed variation. For example, Figures 5 and 8 assert that k=5 proxies reach or surpass the fidelity of the full explanation set, but the reader cannot assess the variability of this conclusion across dataset splits or XAI initialization randomness. Since the paper makes a precise numerical claim ('as few as five'), reporting at least mean and standard deviation over several repetitions, or stating that each curve is a single fixed split, would materially strengthen the empirical support.
minor comments (5)
- [Abstract] The abstract contains formatting errors: 'includelime,shap, andslisemap' is missing spaces around the method names.
- [Figure 6 caption] The caption says 'different datasets are depicted as rows, with the XAI methods used to produce the initial local explanations shown as rows'; the repetition of 'rows' makes the intended layout unclear.
- [Appendix H, Lemma 1 proof] In the proof of supermodularity of the loss function, the line 'Since M_A >= M_B, then -M_A <= min(0, ell_v - M_B)' is not generally valid as written; the claimed inequality Delta(A,v) <= Delta(B,v) is true but requires a short case analysis. Please rephrase the proof step.
- [Section 5.2.1] The phrase 'we set the default number of subsamples to n=500' reuses the symbol n, which is already used for the size of the dataset D; use a distinct notation such as n_G or |G| for the number of local explanations.
- [Table 2] The column headers such as 'G. Maxc A. ratio' and 'G. Min Loss A. ratio' are difficult to parse; spell out 'greedy max coverage' and 'approximation ratio' or use a separate legend.
Circularity Check
No significant circularity: the proxy-set selection is a standard submodular optimization over a closed-box loss matrix, and the headline five-proxy fidelity claim is evaluated on held-out test data rather than being the training objective renamed.
full rationale
I walked the paper's derivation chain. The reduction task is formalized in Section 4.1 through a loss matrix L_ij = ell(g_i(x_j), yhat_j), and Problems 1-3 are max-coverage, min-loss, and balanced submodular maximization problems. The greedy algorithms are justified by the external Nemhauser et al. (1978) approximation theorem, not by a self-citation chain. The central claim that 'as few as five explanations can faithfully emulate the closed-box model' is supported by Section 5, which explicitly states that performance is measured primarily on test data: 'items not used to generate the initial set of local explanations.' The test-time assignment of novel items to proxies via Euclidean nearest neighbours is a heuristic, and the paper acknowledges its limitation in Section 6: 'there is no consideration of the spatial distribution of the data with respect to which proxy was associated with each item.' However, this is a limitation of the evaluation protocol, not a circular reduction: the proxy selection does not optimize the test fidelity, and the test metric is not identical to the training objective by construction. The self-citations, such as Björklund et al. (2023) for SLISEMAP and Seppäläinen et al. (2024) for the jet case study, are motivational or illustrative rather than load-bearing for the reduction result. I therefore find no step in which a predicted quantity is equivalent to a fitted input, and no definitional or self-citation-based circularity.
Assumptions & free parameters
free parameters (4)
- Proxy set size k =
5 (default)
- Loss threshold epsilon =
0.3 quantile of training loss
- Trade-off parameter lambda =
0.5
- Number of initial local explanations n =
500 (default)
assumptions (4)
- domain assumption The set of local explanations G contains at least one faithful approximation for each data item.
- domain assumption Local explanations can be used as generative models, i.e., produce predictions for previously unseen points.
- domain assumption The closed-box function is sufficiently smooth so that local explanations generalize across feature-space neighbors.
- standard math Submodularity and monotonicity of the coverage and loss-reduction objectives.
Cite this review
Pith. "Pith review of ExplainReduce: Generating global explanations from many local explanations." pith.science (2026). https://pith.science/paper/BBZ44A2V
@misc{pith2026250210311,
author = {Pith},
title = {Pith review of: ExplainReduce: Generating global explanations from many local explanations},
year = {2026},
howpublished = {\url{https://pith.science/paper/BBZ44A2V}},
note = {Machine review of arXiv:2502.10311}
}
read the original abstract
Most commonly used non-linear machine learning methods are closed-box models, uninterpretable to humans. The field of explainable artificial intelligence (XAI) aims to develop tools to examine the inner workings of these closed boxes. An often-used model-agnostic approach to XAI involves using simple models as local approximations to produce so-called local explanations; examples of this approach include LIME, SHAP, and SLISEMAP. This paper shows how a large set of local explanations can be reduced to a small "proxy set" of simple models, which can act as a generative global explanation. This reduction procedure, ExplainReduce, can be formulated as an optimisation problem and approximated efficiently using greedy heuristics. We show that, for many problems, as few as five explanations can faithfully emulate the closed-box model and that our reduction procedure is competitive with other model aggregation methods.
Reference graph
Works this paper leans on
-
[1]
botherref S Band , S. , Yarahmadi , A. , Hsu , C.-C. , Biyari , M. , Sookhak , M. , Ameri , R. , Dehzangi , I. , Chronopoulos , A.T. , Liang , H.-W. : Application of explainable artificial intelligence in medical health: A systematic review of interpretability methods (2023) 10.1016/j.imu.2023.101286 botherref
arXiv 2023
-
[2]
botherref Peng , G. , Cheng , Y. , Zhang , Y. , Shao , J. , Wang , H. , Shen , W. : Industrial big data-driven mechanical performance prediction for hot-rolling steel using lower upper bound estimation method (2022) 10.1016/j.jmsy.2022.08.014 botherref
-
[3]
barticle Seppäläinen , L. , Björklund , A. , Besel , V. , Puolamäki , K. : Using slisemap to interpret physical data . PLOS ONE ( 2024 ) 10.1371/journal.pone.0297714 barticle
-
[4]
botherref Guidotti , R. , Monreale , A. , Ruggieri , S. , Turini , F. , Giannotti , F. , Pedreschi , D. : A survey of methods for explaining black box models (2018) 10.1145/3236009 botherref
doi:10.1145/3236009 2018
-
[5]
botherref Björklund , A. , Mäkelä , J. , Puolamäki , K. : SLISEMAP : Supervised dimensionality reduction through local explanations (2023) 10.1007/s10994-022-06261-1 botherref
-
[6]
bchapter Ribeiro , M.T. , Singh , S. , Guestrin , C. : " Why Should I Trust You ?": Explaining the Predictions of Any Classifier . ( 2016 ). 10.1145/2939672.2939778 bchapter
arXiv 2016
-
[7]
bchapter Lundberg , S.M. , Lee , S.-I. : A Unified Approach to Interpreting Model Predictions . ( 2017 ). https://proceedings.neurips.cc/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf bchapter
work page 2017
-
[8]
bchapter Dombrowski , A.-K. , Alber , M. , Anders , C. , Ackermann , M. , Müller , K.-R. , Kessel , P. : Explanations can be manipulated and geometry is to blame . ( 2019 ). https://proceedings.neurips.cc/paper_files/paper/2019/file/bb836c01cdc9120a9c984c525e4b1a4a-Paper.pdf bchapter
work page 2019
Show all 48 references
-
[9]
, Runje , D
bchapter Shankaranarayana , S.M. , Runje , D. : ALIME : Autoencoder based approach for local interpretability . ( 2019 ). 10.1007/978-3-030-33607-3_49 bchapter
2019 doi
-
[10]
, Khan , N.M
botherref Zafar , M.R. , Khan , N.M. : DLIME: A Deterministic Local Interpretable Model-Agnostic Explanations Approach for Computer-Aided Diagnosis Systems (2019) botherref
2019
-
[11]
, Brčić , M
bchapter Došilović , F.K. , Brčić , M. , Hlupić , N. : Explainable artificial intelligence: A survey . ( 2018 ). 10.23919/MIPRO.2018.8400040 bchapter
2018
-
[12]
, Zaremba , W
botherref Szegedy , C. , Zaremba , W. , Sutskever , I. , Bruna , J. , Erhan , D. , Goodfellow , I. , Fergus , R. : Intriguing Properties of Neural Networks (2014). https://arxiv.org/abs/1312.6199 botherref
2014 arXiv
-
[13]
, Thorat , N
botherref Smilkov , D. , Thorat , N. , Kim , B. , Viégas , F. , Wattenberg , M. : SmoothGrad: removing noise by adding noise (2017). https://arxiv.org/abs/1706.03825 botherref
2017 arXiv
-
[14]
, Hilgard , A
botherref Slack , D. , Hilgard , A. , Singh , S. , Lakkaraju , H. : Reliable Post hoc Explanations: Modeling Uncertainty in Explainability (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/4e246a381baf2ce038b3b0f82c7d6fb4-Paper.pdf botherref
2021
-
[15]
, Renard , X
botherref Laugel , T. , Renard , X. , Lesot , M.-J. , Marsala , C. , Detyniecki , M. : Defining Locality for Surrogates in Post-hoc Interpretablity (2018). https://arxiv.org/abs/1806.07498 botherref
2018 arXiv
-
[16]
, Seppäläinen , L
bchapter Björklund , A. , Seppäläinen , L. , Puolamäki , K. : SLIPMAP : Fast and Robust Manifold Visualisation for Explainable AI . ( 2024 ). 10.1007/978-3-031-58553-1_18 bchapter
2024 doi
-
[17]
, Plaza , E
botherref Aamodt , A. , Plaza , E. : Case-based reasoning: Foundational issues, methodological variations, and system approaches (1994) botherref
1994
-
[18]
, Khanna , R
bchapter Kim , B. , Khanna , R. , Koyejo , O.O. : Examples are not enough, learn to criticize! Criticism for Interpretability . ( 2016 ). https://proceedings.neurips.cc/paper_files/paper/2016/file/5680522b8e2bb01943234bce7bf84534-Paper.pdf bchapter
2016
-
[19]
, Huang , W
bchapter Zhao , X. , Huang , W. , Huang , X. , Robu , V. , Flynn , D. : BayLIME : Bayesian local interpretable model-agnostic explanations . Proceedings of Machine Learning Research ( 2021 ). https://proceedings.mlr.press/v161/zhao21a.html bchapter
2021
-
[20]
, Haned , H
botherref Linden , I. , Haned , H. , Kanoulas , E. : Global Aggregations of Local Explanations for Black Box Models (2019). https://arxiv.org/abs/1907.03039 botherref
2019 arXiv
-
[21]
, Guidotti , R
botherref Setzu , M. , Guidotti , R. , Monreale , A. , Turini , F. , Pedreschi , D. , Giannotti , F. : GLocalX - from local to global explanations of black box AI models (2021) 10.1016/j.artint.2021.103457 botherref
2021
-
[22]
, Cummings , R
botherref Li , Q. , Cummings , R. , Mintz , Y. : Optimal Local Explainer Aggregation for Interpretable Prediction (2022) 10.1609/aaai.v36i11.21458 botherref
2022 doi
-
[23]
, Kean , A
botherref Mitchell , S. , Kean , A. , Mason , A. , O'Sullivan , M. , Phillips , A. , Peschiera , F. : PuLP : A Python Linear Programming API . https://github.com/coin-or/pulp botherref
-
[24]
, Wolsey , L.A
botherref Nemhauser , G.L. , Wolsey , L.A. , Fisher , M.L. : An analysis of approximations for maximizing submodular set functions— I (1978) 10.1007/BF01588971 botherref
1978 doi
-
[25]
: Submodular function minimization, (2005)
botherref McCormick , S.T. : Submodular function minimization, (2005). 10.1016/S0927-0507(05)12007-6 botherref
2005 doi
-
[26]
: An approximation guarantee of the greedy descent algorithm for minimizing a supermodular set function (2001) 10.1016/S0166-218X(00)00366-8 botherref
botherref Il'ev , V.P. : An approximation guarantee of the greedy descent algorithm for minimizing a supermodular set function (2001) 10.1016/S0166-218X(00)00366-8 botherref
2001 doi
-
[27]
, Koriche , F
bchapter Bounia , L. , Koriche , F. : Approximating probabilistic explanations via supermodular minimization . ( 2023 ). https://proceedings.mlr.press/v216/bounia23a.html bchapter
2023
-
[28]
, Tiittanen , H
botherref Oikarinen , E. , Tiittanen , H. , Henelius , A. , Puolamäki , K. : Detecting virtual concept drift of regressors without ground truth values (2021) 10.1007/s10618-021-00739-7 botherref
2021 doi
-
[29]
: Life Expectancy ( WHO )
botherref Rajarshi , K. : Life Expectancy ( WHO ). https://www.kaggle.com/datasets/kumarajarshi/life-expectancy-who botherref
-
[30]
, Verma , N
botherref Birla , N. , Verma , N. , Kushawa , N. : Vehicle Dataset. https://www.kaggle.com/datasets/nehalbirla/vehicle-dataset-from-cardekho botherref
-
[31]
10.24432/C5WC95 botherref
botherref Gas Turbine CO and NOx Emission Data Set . 10.24432/C5WC95 botherref
-
[32]
, Dral , P.O
botherref Ramakrishnan , R. , Dral , P.O. , Rupp , M. , Lilienfeld , O.A. : Quantum chemistry structures and properties of 134 kilo molecules (2014) 10.1038/sdata.2014.22 botherref
2014 doi
-
[33]
http://opendata.cern.ch/record/8882 botherref
botherref CMS Collaboration : Simulated Data QCD\_Pt-15to3000\_TuneZ2star\_Flat\_8TeV\_pythia6 in AODSIM Format for 2012 Collision Data. http://opendata.cern.ch/record/8882 botherref
2012
-
[34]
botherref Whiteson , D. : HIGGS . UCI Machine Learning Repository (2014). 10.24432/C5V312 botherref
2014 doi
-
[35]
https://cds.cern.ch/record/1599732 botherref
botherref CMS collaboration : Performance of Quark/Gluon Discrimination in 8 TeV Pp Data. https://cds.cern.ch/record/1599732 botherref
-
[36]
, Tian , Y.-S
botherref Moriwaki , H. , Tian , Y.-S. , Kawashita , N. , Takagi , T. : Mordred: A molecular descriptor calculator (2018) 10.1186/s13321-018-0258-y botherref
2018 doi
-
[37]
write newline
" write newline " cite write " FUNCTION editor.postfix editor num.names #1 > "( )" "( )" if FUNCTION editor.trans.postfix editor num.names #1 > "( )" "( )" if FUNCTION trans.postfix translator num.names #1 > "( )" "( )" if FUNCTION authors.editors.reflist.apa5 'field := 'dot :...
-
[38]
, " * write output.state after.block = add.period write newline
ENTRY address author booktitle chapter doi edition editor eid howpublished institution journal key keywords month note number organization pages publisher school series title type url volume year eprint archive archivePrefix primaryClass adsurl adsnote version label INTEGERS o...
-
[39]
write newline
" write newline "" before.all 'output.state := FUNCTION if.digit duplicate "0" = swap duplicate "1" = swap duplicate "2" = swap duplicate "3" = swap duplicate "4" = swap duplicate "5" = swap duplicate "6" = swap duplicate "7" = swap duplicate "8" = swap "9" = or or or or or or...
-
[40]
, " * write output.state after.block = add.period write newline
ENTRY address archive author booktitle chapter doi edition editor eid eprint howpublished institution journal key keywords month note number organization pages publisher school series title type url volume year archivePrefix primaryClass adsurl adsnote version label extra.labe...
-
[41]
write newline
" write newline "" before.all 'output.state := FUNCTION add.period duplicate empty 'skip "." * add.blank if FUNCTION if.digit duplicate "0" = swap duplicate "1" = swap duplicate "2" = swap duplicate "3" = swap duplicate "4" = swap duplicate "5" = swap duplicate "6" = swap dupl...
-
[42]
write newline
" write newline "" before.all 'output.state := FUNCTION output.doi doi empty skip "doi:" doi * "" * output if FUNCTION format.archive archivePrefix empty "" archivePrefix ":" * if FUNCTION format.primaryClass primaryClass empty "" " [" primaryClass * "] " * if FUNCTION format....
-
[43]
write newline
" write newline "" before.all 'output.state := FUNCTION string.to.integer 't := t text.length 'k := #1 'char.num := t char.num #1 substring 's := s is.num s "." = or char.num k = not and char.num #1 + 'char.num := while char.num #1 - 'char.num := t #1 char.num substring FUNCTI...
-
[44]
write newline
" write newline "" before.all 'output.state := FUNCTION string.to.integer 't := t text.length 'k := #1 'char.num := t char.num #1 substring 's := s is.num s "." = or char.num k = not and char.num #1 + 'char.num := while char.num #1 - 'char.num := t #1 char.num substring FUNCTI...
-
[45]
, " * write output.state after.block = add.period write newline
ENTRY address archive author booktitle chapter edition editor eprint howpublished institution journal key keywords month note number organization pages publisher school series title type url doi volume year archivePrefix primaryClass eid adsurl adsnote version label INTEGERS o...
-
[46]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
-
[47]
Available from:
ENTRY address assignee author booktitle chapter cartographer day edition editor howpublished institution inventor journal key keywords month note number organization pages part publisher school series title type volume word year eprint doi url lastchecked updated archive archi...
-
[48]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.