Pith. sign in

REVIEW 3 major objections 7 minor 31 references

Regional Tree Regularization for Interpretability in Black Box Models

T0 review · 3 major / 7 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Regional tree regularization trains a deep network to be simulable by a separate compact decision tree in each predefined region of the input space.

desk verdict A well-executed extension of tree regularization to region-specific explanations; the max-based penalty is a real improvement, but the evaluation needs tighter uncertainty reporting and a direct check of surrogate fidelity along the training trajectory. read the letter →

arxiv 1908.04494 v3 pith:MMK5KASW submitted 2019-08-13 cs.LG stat.ML

Mike Wu , Sonali Parbhoo , Michael Hughes , Ryan Kindle , Leo Celi , Maurizio Zazzi , Volker Roth , Finale Doshi-Velez This is my paper · ORCID
classification cs.LGstat.ML
keywords interpretabilitytreeregularizationdeepneuralnetworkssparsemaxaveragedecisionpathlengthregionalexplanationssimulabilityhealthcaremachinelearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that expecting one decision tree to explain an entire deep network is unreasonable, and that explanations should instead be regional: practitioners predefine parts of the input space where different logic applies, and the network is penalized so that each region is well approximated by its own compact decision tree. The central claim is that this regional penalty, combined with a sparsemax surrogate and careful stabilization of the tree-fitting subroutine, finds optima that are simultaneously more accurate than training separate trees per region and simpler than global tree regularization, without sacrificing predictive power. The payoff would be a plug-in regularizer for deep models in safety-critical settings such as critical care and HIV treatment, where clinicians want context-specific, human-simulable rules. If correct, it changes the interpretability trade-off from one global surrogate to a small set of context-dependent surrogates.

What carries the argument

The load-bearing object is the regional APL with a sparsemax penalty. APL (average decision path length) is the expected number of binary decisions a distilled CART tree traverses to produce a prediction; the paper replaces the non-differentiable tree fitting with a surrogate MLP that predicts each region's APL from the network parameters. The L0 regional penalty, max over regions of regional APL, prevents simple regions from being over-regularized, and sparsemax approximates that max by a Euclidean projection onto the simplex, giving a sparse, differentiable weighting that focuses on the most complex regions. Around this core, the three optimization innovations (Dirichlet-convex data augmentation for surrogate training, deterministic CART seeding, and reduced-error pruning) keep the surrogate accurate enough for the gradient to be meaningful.

What would settle it

Train a regional-tree-regularized network to completion, then recompute the true APL of every region at the final parameters and compare it to the surrogate's prediction; if the surrogate error at the visited parameters is large, and replacing the surrogate penalty with the true APL changes the final minima substantially, the central claim that the regularizer reliably drives simulability would fail. A stronger test: freeze the true APL as an oracle penalty and check whether the accuracy-simulability trade-off survives; if the oracle and surrogate-trained models diverge, the method's practical success depends on surrogate fidelity rather than on the regional objective itself.

Watch

Extended reading notes

Core claim

The discovery is a regularization objective, regional tree regularization, that pushes a deep network toward being simulable within each of several pre-specified regions of input space. The paper defines a region's complexity by the average decision path length (APL) of a decision tree distilled from the network on that region's data, and penalizes the maximum APL across regions (an L0-style penalty) rather than the sum, because summing over-regularizes simple regions. Because a hard max makes optimization slow and oscillatory, the method uses sparsemax to concentrate the penalty on the most complex regions while remaining differentiable and regularizing several regions at once. The paper further shows that stable training requires three optimizations: augmenting the surrogate's training data with convex combinations of past parameter vectors, fixing the randomness in the CART tree fitter, and pruning trees before measuring depth. It reports that this LSP regional penalty reaches the same minima as the L0 penalty roughly ten times faster.

Load-bearing premise

The method assumes the surrogate network that predicts each region's tree depth stays accurate for the parameter vectors the deep model actually visits during training, even though the paper reports that surrogate mean squared error can reach 0.298 at the maximum even after all stabilization innovations.

Editorial extensions

If this is right

  • Practitioners can define regions by clinical or domain categories and obtain a deep model that is simulable within each category, with region-specific distilled trees as explanations.
  • Regional tree regularization dominates global tree regularization on the accuracy-versus-simulability frontier across the UCI datasets and on the MIMIC and EUResist healthcare tasks.
  • LSP regional regularization matches L0 regional regularization in the minima it finds but converges in roughly one-tenth the epochs, removing the main computational objection to regional penalties.
  • The fidelity of a deep model to its regional trees can be tuned by the regularization strength, so users can choose how much of the network's nonlinear behavior the explanation must capture.
  • Experts reviewing the distilled trees found them compatible with their cognitive categories and usable for audit and hypothesis generation, for example noting that low-immunity HIV patients require trees with more drug-history splits.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same max-or-sparsemax structure should transfer to other simulable surrogates besides trees, such as rule lists or small linear models, whenever a practitioner can specify regions; the key quantity is a per-region complexity measure that is differentiable through a surrogate.
  • Overlapping regions, which the paper explicitly permits, could yield context-dependent explanations that share logic across contexts, an avenue the paper does not pursue experimentally.
  • Because the penalty is on the most complex region, the method implicitly balances explanation effort across regions; one could use the per-region APL gap as a diagnostic to tell modelers which region still needs more capacity or more data.
  • A testable extension would be to use the regional APL values as a model-selection signal: pick the regularization strength that yields the smallest maximum regional depth above a target fidelity, rather than sweeping all strengths.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper proposes regional tree regularization (LSP), a training regularizer for deep neural networks that encourages the model to be well approximated by separate decision trees on user-defined regions of input space. The regularizer penalizes a sparsemax-smoothed maximum of per-region average path lengths (APL), with the APL estimated by surrogate MLPs; three optimization innovations are introduced (Dirichlet-based data augmentation, deterministic CART, and reduced-error pruning). Experiments on a toy problem, four UCI datasets, MIMIC critical care, and EUResist HIV compare trade-offs between test APL and accuracy/F1/AUC, and report distilled region-specific trees with clinician feedback.

Significance. If the empirical claims hold, the method offers practitioners a practical regularizer for producing simulable, region-specific explanations of deep models, filling a useful middle ground between global tree regularization and local explanation methods. The paper has clear strengths: a clean formalization of a new objective, ablations isolating the roles of augmentation, determinism, and pruning, two healthcare case studies with expert validation, a public code repository, and explicit discussion of fidelity. The main risks are the reliance on surrogate APL estimates whose accuracy along the actual training trajectory is not established, and the evaluation metric's direct overlap with the training objective, both of which bear on the central empirical claim.

major comments (3)
  1. [Innovation: Three Keys to Reliable Optimization (Table 1, Fig. 3)] The surrogate APL estimator is the only channel through which the regularization term in Eq. (5) sees tree complexity, yet the paper never validates surrogate accuracy on the parameter vectors actually visited during training. Table 1 reports mean MSE as low as 0.015 with augmentation but max MSE of 0.298 in the augmented case and 0.371 in the deterministic case; these errors are not negligible relative to the 1-2 APL separations between neighboring points in Fig. 5. Fig. 3(c) shows that when augmentation is removed the method collapses to the unregularized solution, confirming that surrogate quality is material to the final optimum. The authors should report surrogate MSE and rank concordance on held-out parameter vectors from the training trajectory, including at the selected minima, or otherwise show that surrogate error does not bias the regularizer's gradient.
  2. [Experiments: Evaluation Metrics (Eq. 4, Eq. 5, Figs. 5-7)] The evaluation metric used for all baselines, Omega_regional-L1 defined in Eq. (4), is the sum of the same per-region APL values that the LSP training objective in Eq. (5) (through SparseMax) directly penalizes. Thus the trade-off curves in Figs. 5-7 partly measure how well each method optimizes the paper's own operational definition of interpretability; global tree regularization and L2 regularization are not optimizing this statistic, so the comparison is asymmetric. This is not a strict circularity, since APL is a legitimate definition of simulability, but the central claim that LSP yields simpler explanations without sacrificing predictive power would be stronger if it were confirmed with an independent complexity measure (e.g., number of leaves or depth of the distilled trees at matched accuracy) or with accuracy at fixed APL and explicit error bars.
  3. [UC Irvine Repository (Fig. 5, text, Table 3)] The claim that regional tree regularization achieves the highest test accuracy in all datasets is not supported by the evidence as presented. Fig. 5 shows curves made of single dots from three runs with no error bars or significance tests, and for Bank and Wine the differences between LSP and other methods are small at low APL. The same absence of variance reporting affects the fidelity values in Table 3. The authors should add seed-level error bars or statistical tests, or temper the claim to competitive or better and identify the APL ranges where the advantage is outside noise.
minor comments (7)
  1. [Fig. 3] The text says 'Only with all innovations (f)' but the figure has panels (a)-(e); the correct panel is (e).
  2. [Fig. 5 legend] The legend labels 'L Global Tree L Region Tree L Region Tree SP 01 L Region Tree2' are garbled; they should read 'L1 Regional Tree', 'L0 Regional Tree', and 'LSP Regional Tree'.
  3. [Table 1 caption] The caption phrase 'how this reduces noise' should read 'shows how this reduces noise'.
  4. [Regionally Faithful Explanations] The sentence 'we consider the problem in which we are given a collection of R regions that cover the entire input space' is followed by 'We do not require these regions to be disjoint nor tile the full space.' These statements are inconsistent; please clarify whether every input must belong to at least one region.
  5. [L0 Regional Tree Regularization] The sentence 'Eq. 5 is differentiable as the gradient through a max operator masks all indexes but one' is imprecise; max is subdifferentiable, and the later text correctly says SparseMax is differentiable a.e. Please align the wording.
  6. [Key 1. Data augmentation] The phrase 'did not compile a large enough dataset' is awkward; clarify that collecting only recent parameter vectors gave too few surrogate training examples.
  7. [Experiments: LSP vs. L0 convergence] The statement that LSP is ten-fold faster than L0 convergence is not backed by a quantitative comparison in the main text; please add epoch counts or convergence curves for the experiments.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the training objective and evaluation metric are distinct, the method is benchmarked against external baselines, and no prediction reduces to a fitted input.

full rationale

The paper's derivation is self-contained. The proposed regularizer (Eq. 5) penalizes max_r APL(X_r, f), with the LSP variant replacing the max by SparseMax over the same per-region APL values; these APL values themselves are estimated by a surrogate MLP because the true APL procedure (Alg. 1) is non-differentiable. The reported interpretability metric, however, is explicitly the sum-based quantity Ω_regional-L1, which the paper says is 'used only for evaluation; it does not appear in the objective' (Experiments, Evaluation Metrics). Since max and sum are not equal, optimizing the former does not by construction minimize the latter; if anything, the L1 regional baseline trains directly on the evaluation metric, so the comparison does not favor the proposed method by definition. The surrogate estimator is inherited from Wu et al. (2018), but this self-citation is not load-bearing in a circular way: the prior work is published, the surrogate's accuracy is separately measured in Table 1, and the reported APL values are computed from the true decision-tree procedure, not from the surrogate. The paper also validates against external benchmarks and clinician assessment, giving content beyond the optimized objective. No equation reduces to its own input, and no fitted parameter is renamed as a prediction.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The method depends on hand-tuned parameters (lambda, R, surrogate settings) and on several assumptions about the validity of APL and the surrogate estimator. No new physical or mathematical entities are introduced.

free parameters (4)
  • Regularization strength lambda = Selected from {0.0001, 0.0005, ..., 10.0}
    Swept over 14 values to produce the accuracy-simulability trade-off curves; the reported minima depend on this choice.
  • Number of regions R = 5 (UCI), 3 or 5 (critical care), 3 (HIV)
    Chosen by authors or domain experts without a formal selection criterion; the method's behavior depends directly on R.
  • Number of synthetic parameter vectors for surrogate augmentation = 100-1000
    The paper uses this range to train each surrogate APL network; no sensitivity analysis is reported.
  • Dirichlet concentration alpha for synthetic parameter mixing = alpha = 1
    Set to a uniform Dirichlet; the paper provides no justification for this specific value.
assumptions (4)
  • domain assumption APL of a CART tree is a valid measure of human simulability.
    The paper defines interpretability as simulability and uses tree depth as a proxy, without a human-subject validation of this link.
  • standard math SparseMax is differentiable almost everywhere and provides a usable approximation to max.
    This is a known property from Martins and Astudillo (2016), on which the LSP penalty depends.
  • domain assumption The surrogate MLP accurately predicts APL for parameter vectors encountered during training.
    The gradient of the regularizer is taken from the surrogate, yet the paper only measures surrogate MSE on a finite dataset and never verifies that surrogate error stays low throughout optimization.
  • ad hoc to paper Dirichlet-based data augmentation creates useful synthetic parameter vectors.
    The augmentation scheme is introduced without theoretical backing; only empirical evidence in Table 1 supports it.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Regional Tree Regularization for Interpretability in Black Box Models." pith.science (2026). https://pith.science/paper/MMK5KASW

@misc{pith2026190804494,
  author       = {Pith},
  title        = {Pith review of: Regional Tree Regularization for Interpretability in Black Box Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MMK5KASW}},
  note         = {Machine review of arXiv:1908.04494}
}
read the original abstract

The lack of interpretability remains a barrier to the adoption of deep neural networks. Recently, tree regularization has been proposed to encourage deep neural networks to resemble compact, axis-aligned decision trees without significant compromises in accuracy. However, it may be unreasonable to expect that a single tree can predict well across all possible inputs. In this work, we propose regional tree regularization, which encourages a deep model to be well-approximated by several separate decision trees specific to predefined regions of the input space. Practitioners can define regions based on domain knowledge of contexts where different decision-making logic is needed. Across many datasets, our approach delivers more accurate predictions than simply training separate decision trees for each region, while producing simpler explanations than other neural net regularization schemes without sacrificing predictive power. Two healthcare case studies in critical care and HIV demonstrate how experts can improve understanding of deep models via our approach.

Figures

Figures reproduced from arXiv: 1908.04494 by the authors.

Figure 2
Figure 2. Comparison of L1 and L0 penalties on per-region APL for blue-vs-green binary classification (each with two runs). Yellow and red patches represent regions. Empty re￾gions denote “trivial” decision functions. is essentially equivalent to global tree regularization (Wu et al., 2018) where the root decision node is constrained to split by region. We refer to this as L1 regional tree regularization. The trouble with thi… view at source ↗
Figure 3
Figure 3. Ablation analysis on a toy green-vs-blue classifi [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Synthetic data with a sparse training set (a) and a dense test set (b). Due to sparsity, the division of five rectangles is [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: Prediction quality vs. simulability tradeoff curves on four UCI datasets. Each dot represents the performance of one [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Tradeoff curves for Critical Care. Each subfigure compares APL and test AUC (higher is better) for a different [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Tradeoff curves on the HIV dataset. We split regions by the level of immunosuppression (abbreviated to immunity) at [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

31 extracted references · 22 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence 'output.state := if if FUNCTION not #0 #1 if FUNCTION and 'skip pop #0 if FUNCTIO...

  2. [2]

    Amir, D., and Amir, O. 2018. Highlights: Summarizing agent behavior to people. In Proc. of the 17th International conference on Autonomous Agents and Multi-Agent Systems (AAMAS)

    work page 2018

  3. [3]

    Bach, S.; Binder, A.; Montavon, G.; Klauschen, F.; M \"u ller, K.-R.; and Samek, W. 2015. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one 10(7):e0130140

    work page 2015

  4. [4]

    Binder, A.; Bach, S.; Montavon, G.; M \"u ller, K.-R.; and Samek, W. 2016. Layer-wise relevance propagation for deep neural network architectures. In Information Science and Applications (ICISA) 2016 . Springer. 913--922

    work page 2016

  5. [5]

    Che, Z.; Purushotham, S.; Khemani, R.; and Liu, Y. 2015. Distilling knowledge from deep networks with applications to healthcare domain. arXiv preprint arXiv:1512.03542

    work page Pith review arXiv 2015

  6. [6]

    H.; Asch, S

    Chen, J. H.; Asch, S. M.; et al. 2017. Machine learning and prediction in medicine-beyond the peak of inflated expectations. N Engl J Med 376(26):2507--2509

    work page 2017

  7. [7]

    Chuang, J., and Hsu, D. J. 2014. Human-centered interactive clustering for data analysis. In Conference on Neural Information Processing Systems (NIPS). Workshop on Human-Propelled Machine Learning

    work page 2014

  8. [8]

    Dheeru, D., and Karra Taniskidou, E. 2017. UCI machine learning repository

    work page 2017

Show all 31 references
  1. [9]

    Frosst, N., and Hinton, G. 2017. Distilling a neural network into a soft decision tree. arXiv preprint arXiv:1711.09784

    2017 arXiv

  2. [10]

    Guan, C.; Wang, X.; Zhang, Q.; Chen, R.; He, D.; and Xie, X. 2019. Towards a deep and unified understanding of deep neural models in nlp. In International Conference on Machine Learning , 2454--2463

    2019

  3. [11]

    C.; Wu, D.; Narayanaswamy, A.; Venugopalan, S.; Widner, K.; Madams, T.; Cuadros, J.; et al

    Gulshan, V.; Peng, L.; Coram, M.; Stumpe, M. C.; Wu, D.; Narayanaswamy, A.; Venugopalan, S.; Widner, K.; Madams, T.; Cuadros, J.; et al. 2016. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. Jama 316(...

    2016

  4. [12]

    E.; Pollard, T

    Johnson, A. E.; Pollard, T. J.; Shen, L.; Li-wei, H. L.; Feng, M.; Ghassemi, M.; Moody, B.; Szolovits, P.; Celi, L. A.; and Mark, R. G. 2016. Mimic-iii, a freely accessible critical care database. Scientific data 3:160035

    2016

  5. [13]

    Kim, B.; Rudin, C.; and Shah, J. A. 2014. The bayesian case model: A generative approach for case-based reasoning and prototype classification. In Advances in Neural Information Processing Systems , 1952--1960

    2014

  6. [14]

    Kim, B. 2015. Interactive and interpretable machine learning models for human machine collaboration . Ph.D. Dissertation, Massachusetts Institute of Technology

    2015

  7. [15]

    W., and Liang, P

    Koh, P. W., and Liang, P. 2017. Understanding black-box predictions via influence functions. arXiv preprint arXiv:1703.04730

    2017 arXiv

  8. [16]

    M.; Isbell, C

    Krening, S.; Harrison, B.; Feigh, K. M.; Isbell, C. L.; Riedl, M.; and Thomaz, A. 2017. Learning from explanations using sentiment and advice in rl. IEEE Transactions on Cognitive and Developmental Systems 9(1):44--55

    2017

  9. [17]

    Lipton, Z. C. 2016. The mythos of model interpretability. arXiv preprint arXiv:1606.03490

    2016 arXiv

  10. [18]

    Maaten, L. v. d., and Hinton, G. 2008. Visualizing data using t-sne. Journal of machine learning research 9(Nov):2579--2605

    2008

  11. [19]

    Martins, A., and Astudillo, R. 2016. From softmax to sparsemax: A sparse model of attention and multi-label classification. In International Conference on Machine Learning , 1614--1623

    2016

  12. [20]

    Miller, T. 2018. Explanation in artificial intelligence: Insights from the social sciences. Artificial Intelligence

    2018

  13. [21]

    A.; and Dudley, J

    Miotto, R.; Li, L.; Kidd, B. A.; and Dudley, J. T. 2016. Deep patient: an unsupervised representation to predict the future of patients from the electronic health records. Scientific reports 6:26094

    2016

  14. [22]

    Montavon, G.; Samek, W.; and M \"u ller, K.-R. 2018. Methods for interpreting and understanding deep neural networks. Digital Signal Processing 73:1--15

    2018

  15. [23]

    Mordvintsev, A.; Olah, C.; and Tyka, M. 2015. Inceptionism: Going deeper into neural networks. Google Research Blog. Retrieved June 20(14):5

    2015

  16. [24]

    Pedregosa, F.; Varoquaux, G.; Gramfort, A.; Michel, V.; Thirion, B.; Grisel, O.; Blondel, M.; Prettenhofer, P.; Weiss, R.; Dubourg, V.; et al. 2011. Scikit-learn: Machine learning in python. Journal of machine learning research 12(Oct):2825--2830

    2011

  17. [25]

    Quinlan, J. R. 1987. Simplifying decision trees. International journal of man-machine studies 27(3):221--234

    1987

  18. [26]

    T.; Singh, S.; and Guestrin, C

    Ribeiro, M. T.; Singh, S.; and Guestrin, C. 2016. Why should i trust you?: Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , 1135--1144. ACM

    2016

  19. [27]

    S.; Hughes, M

    Ross, A. S.; Hughes, M. C.; and Doshi-Velez, F. 2017. Right for the right reasons: Training differentiable models by constraining their explanations. arXiv preprint arXiv:1703.03717

    2017 arXiv

  20. [28]

    R.; Das, A.; Vedantam, R.; Cogswell, M.; Parikh, D.; and Batra, D

    Selvaraju, R. R.; Das, A.; Vedantam, R.; Cogswell, M.; Parikh, D.; and Batra, D. 2016. Grad-cam: Why did you say that? arXiv preprint arXiv:1611.07450

    2016 arXiv

  21. [29]

    T.; and Guestrin, C

    Singh, S.; Ribeiro, M. T.; and Guestrin, C. 2016. Programs as black-box explanations. arXiv preprint arXiv:1611.07579

    2016 arXiv

  22. [30]

    C.; Parbhoo, S.; Zazzi, M.; Roth, V.; and Doshi-Velez, F

    Wu, M.; Hughes, M. C.; Parbhoo, S.; Zazzi, M.; Roth, V.; and Doshi-Velez, F. 2018. Beyond sparsity: Tree regularization of deep models for interpretability. In Thirty-Second AAAI Conference on Artificial Intelligence

    2018

  23. [31]

    o nnerborg, A.; Struck, D.; Altmann, A.; Prosperi, M.; Rosen-Zvi, M.; Petroczi, A.; Peres, Y.; Sch \

    Zazzi, M.; Kaiser, R.; S \"o nnerborg, A.; Struck, D.; Altmann, A.; Prosperi, M.; Rosen-Zvi, M.; Petroczi, A.; Peres, Y.; Sch \"u lter, E.; et al. 2011. Prediction of response to antiretroviral therapy by human experts and by the euresist data-driven expert system (the eve stu...

    2011

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.