Optimizing for Interpretability in Deep Neural Networks with Tree Regularization

Finale Doshi-Velez; Michael C. Hughes; Mike Wu; Sonali Parbhoo; Volker Roth

REVIEW 4 major objections 5 minor 58 references

Optimizing for Interpretability in Deep Neural Networks with Tree Regularization

T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that training deep networks to be well approximated by compact decision trees makes them simulable by humans while retaining accuracy.

desk verdict A genuinely new training-time regularizer for simulability, with broad experiments, but the headline comparison is weakened by an asymmetry: network AUC is plotted against distilled-tree path length, and fidelity is only reported for tree-regularized models, never for L1/L2 baselines. read the letter →

arxiv 1908.05254 v1 pith:5L2CM45A submitted 2019-08-14 cs.LG stat.ML

Mike Wu , Sonali Parbhoo , Michael C. Hughes , Volker Roth , Finale Doshi-Velez This is my paper · ORCID

classification cs.LGstat.ML

keywords treeregularizationinterpretabilityhuman-simulabilityaveragepathlengthdecisionneuralnetworkregionalexplanationssparsemax

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a deep network can be trained to be interpretable, rather than interpreted afterward. Its proposal, tree regularization, adds a penalty to the training objective that rewards the network whenever its predictions can be reproduced by a compact axis-aligned decision tree, measured by the average number of decision nodes a human would have to step through. Because such trees can be followed by hand, a network that is well approximated by one becomes simulable by a clinician, regulator, or other user. Across synthetic problems, speech data, ICU records, and HIV treatment histories, the authors report that tree regularization reaches higher accuracy at low decision-tree complexity than L1 or L2 penalties, and that a regional version, fitting a separate compact tree for each expert-defined subpopulation, does even better.

What carries the argument

The central object is the average-path-length (APL) regularizer: fit an axis-aligned decision tree to the network's thresholded predictions on a reference set, then measure the average number of nodes traversed from root to leaf. Since tree fitting is not differentiable, the paper replaces APL with a small surrogate neural network that estimates APL from the target network's parameter vector; the surrogate is trained on parameter/APL pairs collected during optimization, using data augmentation, random restarts, and a sliding window of recent parameters to keep the estimate accurate. For regional tree regularization, one surrogate is maintained per human-defined region, and the sparsemax transform over regional APL estimates makes the max operator differentiable while penalizing only the most complex regions, so simple regions are not over-regularized.

What would settle it

Refit a decision tree to the tree-regularized network at every checkpoint, compute the true average path length, and compare it with the surrogate's estimate; if the two diverge as training progresses, or if the final network's average path length at matched accuracy is not below that of L1- or L2-regularized networks, the central claim fails.

Watch

Extended reading notes

Core claim

The central claim is that the usual accuracy-interpretability trade-off can be shifted by optimizing for a proxy of simulability during training. The paper's regularizer penalizes the average path length of a binary decision tree fitted to the target network's thresholded predictions; by making this penalty differentiable through a surrogate estimator of path length, gradient descent can find parameter minima whose decision boundaries are nearly tree-shaped. The authors assert that this produces networks that stay accurate while their distilled decision trees are short enough for a person to simulate, with fidelity between the network and its tree around 0.80–0.94 on real-world tasks. For settings where no single global tree can be both small and faithful, the paper extends the penalty region by region, using sparsemax to concentrate regularization on the most complex regions, and reports that this regional version finds better accuracy-versus-complexity trade-offs than any other regularizer tested.

Load-bearing premise

The method rests on the assumption that a small helper network can accurately and smoothly learn the mapping from the main network's parameters to the average path length of a decision tree fitted to that network; if that mapping is not learnable along the training trajectory, the regularization gradient is noise.

Editorial extensions

If this is right

Deep networks trained this way could be deployed in decision-critical settings where a human must audit each prediction, because the distilled tree gives the step-by-step logic and the network retains accuracy on cases the tree handles poorly.
If regional tree regularization works as claimed, expert knowledge about meaningful patient or input subgroups can be turned directly into a training constraint, yielding per-group explanations without requiring global simplicity.
Because the penalty is written as a regularizer on any differentiable model, the same procedure could make recurrent, convolutional, or hybrid sequence models simulable, not just plain feed-forward networks.
The reported accuracy gains over L1 and L2 at low average path length imply that norm-based penalties are not the right tool if the goal is human simulation; the tree regularizer targets the complexity measure that actually matters for step-through reasoning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial inference: average path length measures tree size, not actual human effort, so a fair test of the interpretability claim would be a user study where people simulate predictions from the distilled trees and compare error rates and response times against L1- or L2-regularized models at matched tree size.
Editorial inference: if the surrogate can learn the parameter-to-APL mapping, the same trick should apply to other discrete complexity measures, such as the number of distinct activation patterns or the description length of a rule list, suggesting a general recipe for procedure regularization beyond trees.
Editorial inference: the regional framework treats any user-defined partition as a place to inject prior knowledge; a natural extension would be to partition by protected attributes so the resulting model has simple, inspectable behavior within each demographic group.
Editorial inference: because fidelity is high but not perfect, the residual disagreement between network and tree defines a small set of hard cases; inspecting those cases could reveal where the tree explanation should not be trusted, which would make the method more usable in practice.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Two things you should know. First, the core idea is new and worth taking seriously: instead of distilling a trained network post hoc, they add a differentiable penalty that encourages the network to be well-approximated by a compact axis-aligned decision tree. The surrogate-network trick for estimating average path length is a real technical contribution, and the regional extension using sparsemax is a sensible way to keep all regions simple rather than letting one collapse. Second, the empirical story is broader and more careful than most interpretability papers: MIMIC, HIV, TIMIT, UCI benchmarks, fidelity numbers, and a stability analysis over random restarts. That is real work.

The soft spot is the central comparison. The trade-off plots put the deep network's held-out AUC on the y-axis and the distilled tree's APL on the x-axis. For tree-regularized models they report fidelity (0.80–0.94), so the tree is a reasonably faithful stand-in for the network. For L1 and L2 baselines they report neither fidelity nor the distilled tree's AUC. If an L1/L2 network's boundary is not axis-aligned, the tree fit to it can have low fidelity, and the plotted point then pairs a low APL with an AUC that a human simulating the tree would not actually get. That is not a minor quibble: the paper itself argues in the introduction that post-hoc distillation of unregularized networks is ill-posed for exactly this reason. So the headline claim—tree regularization gives models that are easier to simulate than L1/L2 without sacrificing predictive power—is not fully established by the reported comparisons. The fix is straightforward: report fidelity for the baselines' distilled trees, or plot the distilled tree's AUC instead of the network's.

There is also a mild circularity issue: APL is both the objective and the primary evaluation metric, so "tree regularization produces simpler models" is partly true by construction. The accuracy-at-matched-APL comparison is external and held-out, so the main claim survives, but confidence intervals or significance tests would strengthen it. The surrogate training involves several heuristics (augmentation, restarts, sliding windows), and those are free parameters that could be hard to transfer, though the paper is honest about them.

Bottom line: this deserves a serious referee. The idea is sound, the experiments are extensive, and the missing baseline fidelity is a fixable gap rather than a fatal flaw. I would send it out, and I would ask for that comparison before accepting.

Referee Report

4 major / 5 minor

Summary. The paper proposes tree regularization as a way to train deep networks whose decision boundaries are well approximated by compact, axis-aligned decision trees. The regularizer measures the average path length (APL) of a decision tree fit to the network's predictions, and a separately trained surrogate MLP is used to make the APL penalty differentiable. The method is demonstrated on synthetic tasks, UCI data, and medical time series (sepsis/MIMIC, HIV/EuResist, TIMIT), and is extended to regional tree regularization in which different, expert-defined regions of the input space are each matched to a compact tree, with sparsemax used to select the most complex region. The central empirical claim is that tree-regularized models achieve higher held-out AUC at low APL than models trained with L1 or L2 regularization.

Significance. If the central comparison is established, this is a valuable contribution: it moves interpretability from post-hoc explanation to direct optimization, introduces a non-trivial differentiable proxy for a discrete tree-complexity measure, and demonstrates the idea across several realistic tasks. The regional extension, with sparsemax over per-region APLs, is a reasonable and potentially useful generalization, and the paper reports fidelity of the distilled trees to the deep models, which is the right kind of evidence for simulability. The main empirical claim, however, is not fully supported as reported because the baseline comparison is asymmetric: the x-axis is the APL of a distilled tree, but the y-axis is the network's held-out AUC, and the fidelity of the distilled tree to L1/L2-regularized networks is never reported. The central result therefore needs additional evidence before the claimed 'easier to simulate without sacrificing predictive power' can be accepted.

major comments (4)

[Section 6.2, Figures 5c-d, 7a-g, 8a-c, 18-20, Tables 1 and 4] The headline comparison is asymmetric. In the trade-off plots, the x-axis is the APL of a decision tree fitted to the model (Algorithm 1 or Algorithm 2), while the y-axis is the held-out AUC of the deep network itself. Fidelity of the distilled tree to the deep model is reported only for tree-regularized models (Tables 1 and 4, values 0.80-0.94). For L1/L2 baselines, no fidelity or distilled-tree AUC is reported. Because a human simulating the model would simulate the tree rather than the network, the claim that tree regularization yields models that are easier to simulate than L1/L2 without sacrificing accuracy is not established: a low-APL tree fitted to an L1/L2 network could be unfaithful, in which case the plotted low APL would not be attainable by a human simulating the actual model. This is exactly the ill-posedness the paper itself attributes to post-hoc distillation in Section 1. Please report fidelity or held-out AUC of the distilled trees for L1/L2 baselines, or re-plot the comparison using the distilled tree's own AUC on the y-axis.
[Section 4, Eq. (10), Figure 2; Section 7.4] The validity of the entire optimization depends on the surrogate function \hat{\Omega}(\theta) providing accurate and smooth estimates of the true APL along the optimization trajectory. The paper shows that heuristics (data augmentation, random restarts, a sliding window) are needed even on a 2D toy problem, and Section 7.4 documents instability in the regional setting. However, no real-data diagnostic of surrogate accuracy is reported, such as MSE or correlation between \hat{\Omega}(\theta) and the true APL on held-out parameter vectors. If the surrogate's gradient is not a reliable descent direction, the regularization signal is noise and the empirical results may not reflect the proposed objective. Please include surrogate-versus-true-APL diagnostics for at least one real task, or an ablation showing that the downstream AUC/APL conclusions require an accurate surrogate.
[Section 6.2 and Figures 5, 7, 8, 18] No confidence intervals or significance tests are reported for the central trade-off curves. Figure 18 states that the plotted points are minima from 3 independent runs but does not show error bars or variance; Figures 5, 7, and 8 appear to show single runs. The abstract and Section 6.2 claim that tree regularization 'consistently' achieves higher accuracy in the low-APL regime. Given that the reported AUC differences are sometimes modest (0.03-0.15), the absence of repeated-run variability makes it difficult to judge whether the advantage is reliable. Please report the number of runs and the variance (e.g., confidence intervals or standard errors) for each curve, or state clearly which curves are single runs.
[Section 9.1 and Figures 18-20] The regional evaluation metric is subtly different from the global objective. For regional experiments, the reported test APL is always computed as \Omega^{regional}(\theta), even for globally regularized models, which are trained to minimize the global APL. The paper explains that this is to make models comparable, but this means the global tree models are evaluated on a metric that is not their training objective. Please make this mismatch explicit in the main text and discuss whether it could systematically disadvantage global tree regularization, since the global objective may allow one region to be complex as long as the overall APL is low, while the regional evaluation penalizes that complexity.

minor comments (5)

[Section 5, discussion of 2D Parabola] The text says 'Figure 4(b) shows each trained model as a single point in a 2D fitness space,' but Figure 4 contains decision boundaries; the fitness-space plot appears to be Figure 3(b). Please correct the cross-reference.
[Section 7.4, Figure 15] The body text refers to 'Table 15' ('Table 15 shows this to reduce noise'), while the cited entity appears to be the MSE table in Figure 15. Please unify the numbering and caption.
[Algorithm 1 and Equations (9)-(10)] Algorithm 1 is described as an APL cost function but returns only the mean path length; please define explicitly that the tree is trained on the thresholded predictions of the network and that the path length is computed on that tree, and state what happens when the tree's class differs from the network's probability output.
[Equation (8)] The HMM objective in Equation (8) is written as a maximum over a joint probability, but the earlier objectives (1) and (6) include a regularization term; it is unclear whether tree regularization is applied to the HMM component alone, the GRU component alone, or both in the GRU-HMM experiments. Please clarify the exact objective optimized for GRU-HMM.
[Section 7.2 and Equation (13)] The term 'L0 regional tree regularization' is used for the sparsemax-based method, while Equation (13) is a max over regions. Since sparsemax is a smooth approximation and not an exact L0 norm, the terminology could be misleading; please clarify the relationship or choose a different name.

Circularity Check

1 steps flagged · score 4.0 of 10

The 'tree-regularized models are simpler' result is true by construction because APL is both the regularizer and the evaluation metric; the accuracy comparisons at matched APL remain independent.

self definitional [Section 4, Eq. (9); Section 5, 'To evaluate model simulability']
"Thus, our regularizer is Ω(θ) ≜ APL({xn}N n=1,f (·;θ),h) ... We now come to our core contribution: we replace Ψ(θ) with a novel tree-regularizer, denoted Ω(θ), that encourages the model θ to be simulable. ... To evaluate model simulability, we use APL. Since Algorithm 1 can compute the APL for any fixed deep model given its parameters, we use it to measure decision boundary complexity under any regularization, including L1 or L2."

Equation 9 defines the regularizer as the average path length of a decision tree fitted to the target network, so the training objective directly minimizes APL. The paper's evaluation of simulability is the same APL quantity computed by Algorithm 1. Therefore, the headline observation that tree-regularized models are simpler or more simulable than L1/L2 models is a restatement of the objective rather than an independent empirical prediction. The independent content lies in the held-out AUC at matched APL and in fidelity/physician assessments; the APL-axis advantage is built in by construction.

full rationale

The central methodological claim, that minimizing the tree regularizer produces low-APL decision boundaries, is definitional because the regularizer itself is APL and the reported simulability metric is APL. This accounts for a score of 4 rather than 0. However, the paper's more substantive claims are not circular: the accuracy comparisons at matched APL use held-out AUC, an external and independent measure, and the fidelity numbers in Tables 1 and 4 confirm that the distilled trees are reasonably faithful stand-ins for the tree-regularized networks. The regional extension similarly compares all methods with a common regional APL evaluation metric while the training objective differs (L0 max vs. average), so those comparisons have independent content. I found no load-bearing self-citation chain, no uniqueness theorem imported from the authors, and no ansatz smuggled in via citation; citations to prior work by the authors are contextual. The skeptic's concern about missing fidelity for L1/L2 baselines is a serious evaluation-fairness issue but not a circularity, because it does not reduce a claimed result to its inputs by construction; it is a missing control.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new physical entities. Its surrogate network is an algorithmic component, not a postulated object. The load-bearing commitments are hyperparameters, the APL proxy, and the assumption that the surrogate can track the true APL during optimization.

free parameters (6)

Regularization strength lambda = varied per experiment, roughly 0.0001 to 15000
Chosen by grid search or selected from accuracy-complexity curves; central to every reported tradeoff.
Decision tree leaf parameter h = 1000 for real time-series tasks, 1 for UCI and toy tasks
Controls tree size and therefore the APL objective; hand-set per dataset.
Surrogate window size E = empirically chosen, not specified numerically
Section 4 states 'E is empirically chosen'; controls how much optimization history is used to train the APL surrogate.
Surrogate architecture and dataset size J = 1 hidden layer with 25 units; J=100 for applications
Small surrogate is a deliberate design choice; J and retraining frequency of every 25 or 50 steps are hand-set.
Region definitions = R=5 via k-means on UCI; R=3 SOFA and R=5 care unit on ICU; R=4 CDC immunity on HIV
Regions are either fit to data or provided by experts; all per-region APL results depend on this partition.
Choice of L0 regional norm via sparsemax = L0 used for final regional results
The norm over regional APLs is a modeling choice; the paper shows L1 regional regularization collapses important regions.

assumptions (5)

domain assumption APL of a fitted decision tree is a valid operationalization of human-simulability.
Section 4: 'We consider average path length a good proxy for simulability because human simulation requires stepping through every calculation.' No controlled human experiment verifies this mapping.
ad hoc to paper The surrogate network hat Omega(theta) can accurately predict APL from parameters along the optimization trajectory, so its gradient is a useful descent direction.
Equation 10 and Section 4 'Training the Surrogate Loss'; the paper shows this holds only with augmentation, restarts, and a sliding window.
domain assumption Deep models have many optima of similar accuracy, so adding an interpretability penalty can find a high-accuracy, low-complexity minimum.
Introduction, citing Goodfellow et al.; this motivates the entire approach and is not proven for the specific losses used.
domain assumption Sklearn CART with Gini impurity and deterministic seeds yields a stable, representative tree for APL computation.
Algorithms 1 and 2; Section 7.4 shows randomization in CART causes variance and must be controlled.
domain assumption For regional regularization, expert-defined or k-means partitions are available and fixed before training.
Section 7 and Section 9.2; the method assumes a priori regions and does not learn them jointly.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Optimizing for Interpretability in Deep Neural Networks with Tree Regularization." pith.science (2026). https://pith.science/paper/5L2CM45A

@misc{pith2026190805254,
  author       = {Pith},
  title        = {Pith review of: Optimizing for Interpretability in Deep Neural Networks with Tree Regularization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5L2CM45A}},
  note         = {Machine review of arXiv:1908.05254}
}

read the original abstract

Deep models have advanced prediction in many domains, but their lack of interpretability remains a key barrier to the adoption in many real world applications. There exists a large body of work aiming to help humans understand these black box functions to varying levels of granularity -- for example, through distillation, gradients, or adversarial examples. These methods however, all tackle interpretability as a separate process after training. In this work, we take a different approach and explicitly regularize deep models so that they are well-approximated by processes that humans can step-through in little time. Specifically, we train several families of deep neural networks to resemble compact, axis-aligned decision trees without significant compromises in accuracy. The resulting axis-aligned decision functions uniquely make tree regularized models easy for humans to interpret. Moreover, for situations in which a single, global tree is a poor estimator, we introduce a regional tree regularizer that encourages the deep model to resemble a compact, axis-aligned decision tree in predefined, human-interpretable contexts. Using intuitive toy examples as well as medical tasks for patients in critical care and with HIV, we demonstrate that this new family of tree regularizers yield models that are easier for humans to simulate than simpler L1 or L2 penalties without sacrificing predictive power.

Figures

Figures reproduced from arXiv: 1908.05254 by the authors.

**Figure 2.** (a) True average path lengths (yellow) and surrogate estimates Ωˆ (green) across many iterations of network parameter training iterations (on 2D Parabola). (b) Compares the effects of parameter augmentation and random restarts (retraining): The blue line shows the true APL of the decision tree at each epoch. All other lines show predicted APL using the surrogate MLP. By augmenting and restarting, we significantly im… view at source ↗

**Figure 3.** (a) 2D parabola dataset. The black line shows the true decision boundary; the gray lines define areas where noise is added. (b) A comparision of APL versus AUC for many regularizers. In the small average path length regime (0-5), tree-regularization produces models with higher AUC than L1 or L2. 5. Demonstration: A Tree-Regularized MLP and RNN We start by exploring two simple domains intended to build intuition for … view at source ↗

Figures from the paper (17 more)

**Figure 4.** Figure 4: Decision boundaries (black lines) have qualitatively different shapes for different regularization schemes, as regularization strength λ increases. We color each prediction as true positive (red), true negative (yellow), false negative (green), and false positive (blu…

**Figure 5.** Figure 5: Toy Signal-and-Noise HMM Task: (a)-(b) Decision trees trained to mimic predictions of GRU models at different regularization strengths λ; as expected, increasing λ decreases the size of the learned trees. Decision tree (b) suggests the model learns to predict positive…

**Figure 6.** Figure 6: Emission (5 states vs 7 features) and transition probabilities for the signal HMM [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]

**Figure 7.** Figure 7: SEPSIS task – Study of different regularizers for a GRU model with 100 states, trained to jointly predict 5 binary outcomes. Panels (a,c,e,g) show AUC vs. APL for 4 of the 5 outcomes; in all cases, tree regularization provides higher accuracy in the target regime of lo…

**Figure 8.** Figure 8: TIMIT and HIV tasks: – Study of different regularizers for a GRU model with 75 states. Panels (a)-(c) are tradeoff curves showing how predictive power and decision-tree complexity evolve with increasing strength of L1, L2 or tree regularization in both TIMIT (stop phon…

**Figure 9.** Figure 9: (a-c) Decision trees from 10 independent runs on the signal-and-noise HMM dataset with λ = 1000.0. Seven of the ten runs resulted in a tree of the same structure. The other three are similar, having additional subtrees but sharing the same splits and features. (d-f ) S…

**Figure 10.** Figure 10: Fitness curves for the GRU-HMM, showing prediction quality (AUC) vs. complex [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]

**Figure 11.** Figure 11: We show the differences between global (b), local (c), and regional (d) tree [PITH_FULL_IMAGE:figures/full_fig_p020_11.png]

**Figure 12.** Figure 12: An L1 penalty on per-region APLs can over-penalize, resulting in an entire region with far too simple predictions. Subplots (b) and (c) show results from two different initializations using the L1 norm, while (d) and (e) show the same using the L0 norm. to global tree…

**Figure 13.** Figure 13: Illustratoin of L0 regional tree regularization. Each round contains three trees representing regions. Light gray color indicates regions given 0 probability by sparsemax. Over the three rounds, different regions are given priority while other regions are given no wei…

**Figure 14.** Figure 14: (a-d) Decision trees using randomized training; (e-h) Decision trees using deterministic training. Note that randomized training leads to very different optima. it sufficient; we find that especially when there are multiple surrogates to be maintained, sophistication…

**Figure 15.** Figure 15: Comparison of the average and max mean squared error (MSE) between surrogate predictions and true average path lengths over 500 epochs. Non-deterministic training and lack of data introduces large errors. Data augmentation makes for a robust surrogate. Especially for …

**Figure 16.** Figure 16: (a) Ground truth decision boundary with 25 regions; green represents positive [PITH_FULL_IMAGE:figures/full_fig_p024_16.png]

**Figure 17.** Figure 17: Synthetic data with a sparse training set [PITH_FULL_IMAGE:figures/full_fig_p025_17.png]

**Figure 18.** Figure 18: (a-d) Comparison of regularizers (L2, global tree, regional tree) on four datasets [PITH_FULL_IMAGE:figures/full_fig_p026_18.png]

**Figure 19.** Figure 19: Comparison of regularization methods on the Critical Care dataset. Each output [PITH_FULL_IMAGE:figures/full_fig_p028_19.png]

**Figure 20.** Figure 20: Comparison of regularization methods on 15 output dimensions of the HIV dataset [PITH_FULL_IMAGE:figures/full_fig_p030_20.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

58 extracted references · 46 canonical work pages

[1]

A., Rybeck, G., Scheidegger, C., Smith, B., \ Venkatasubramanian, S

Adler, P., Falk, C., Friedler, S. A., Rybeck, G., Scheidegger, C., Smith, B., \ Venkatasubramanian, S. 2016 . Auditing black-box models for indirect influence \ In ICDM

work page 2016
[2]

\ \ Amir, O

Amir, D. \ \ Amir, O. 2018 . Highlights: Summarizing agent behavior to people \ In Proc. of the 17th International conference on Autonomous Agents and Multi-Agent Systems (AAMAS)

work page 2018
[3]

\ \ Kokkolaras, M

Audet, C. \ \ Kokkolaras, M. 2016 . Blackbox and derivative-free optimization: theory, algorithms and applications . Springer

work page 2016
[4]

Bach, S., Binder, A., Montavon, G., Klauschen, F., M \"u ller, K.-R., \ Samek, W. 2015 . On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation \ PloS one , 10\/ (7), e0130140

work page 2015
[5]

Bahdanau, D., Cho, K., \ Bengio, Y. 2014 . Neural machine translation by jointly learning to align and translate \ arXiv preprint arXiv:1409.0473

arXiv 2014
[6]

K., Rathod, V., Murphy, K

Balan, A. K., Rathod, V., Murphy, K. P., \ Welling, M. 2015 . Bayesian dark knowledge \ In NIPS

work page 2015
[7]

Binder, A., Bach, S., Montavon, G., M \"u ller, K.-R., \ Samek, W. 2016 . Layer-wise relevance propagation for deep neural network architectures \ In Information Science and Applications (ICISA) 2016 , \ 913--922. Springer

work page 2016
[8]

BuciluÇ, C., Caruana, R., \ Niculescu-Mizil, A. 2006 . Model compression \ In KDD

work page 2006

Show all 58 references

[9]

T., \ Liu, Y

Che, Z., Kale, D., Li, W., Bahadori, M. T., \ Liu, Y. 2015 . Deep computational phenotyping \ In KDD

2015
[10]

H., Asch, S

Chen, J. H., Asch, S. M., et al. 2017 . Machine learning and prediction in medicine-beyond the peak of inflated expectations \ N Engl J Med , 376\/ (26), 2507--2509

2017
[11]

Cho, K., Gulcehre, B. v. M. C., Bahdanau, D., Schwenk, F. B. H., \ Bengio, Y. 2014 . Learning phrase representations using RNN encoder--decoder for statistical machine translation \ In EMLNP

2014
[12]

T., Schuetz, A., Stewart, W

Choi, E., Bahadori, M. T., Schuetz, A., Stewart, W. F., \ Sun, J. 2016 . Doctor AI : Predicting clinical events via recurrent neural networks \ In Machine Learning for Healthcare Conference

2016
[13]

\ \ Shavlik, J

Craven, M. \ \ Shavlik, J. W. 1996 . Extracting tree-structured representations of trained networks \ In Advances in neural information processing systems , \ 24--30

1996
[14]

\ \ Karra Taniskidou, E

Dheeru, D. \ \ Karra Taniskidou, E. 2017 . UCI machine learning repository

2017
[15]

\ \ Le Cun, Y

Drucker, H. \ \ Le Cun, Y. 1992 . Improving generalization performance using double backpropagation \ IEEE Transactions on Neural Networks , 3\/ (6), 991--997

1992
[16]

Duchi, J., Shalev-Shwartz, S., Singer, Y., \ Chandra, T. 2008 . Efficient projections onto the l 1-ball for learning in high dimensions \ In Proceedings of the 25th international conference on Machine learning , \ 272--279. ACM

2008
[17]

Erhan, D., Bengio, Y., Courville, A., \ Vincent, P. 2009 . Visualizing higher-layer features of a deep network \ \ 1341, Department of Computer Science and Operations Research, University of Montreal

2009
[18]

\ \ Hinton, G

Frosst, N. \ \ Hinton, G. 2017 . Distilling a neural network into a soft decision tree \ arXiv preprint arXiv:1711.09784

2017 arXiv
[19]

Garofolo, J. S. \ et al. 1993 . TIMIT acoustic-phonetic continuous speech corpus \ Linguistic Data Consortium , 10\/ (5)

1993
[20]

C., Szolovits, P., \ Doshi-Velez, F

Ghassemi, M., Wu, M., Hughes, M. C., Szolovits, P., \ Doshi-Velez, F. 2017 . Predicting intervention onset in the icu with switching state space models \ AMIA Summits on Translational Science Proceedings , 2017 , 82

2017
[21]

Goodfellow, I., Bengio, Y., \ Courville, A. 2016 . Deep Learning . MIT Press

2016
[22]

C., Wu, D., Narayanaswamy, A., Venugopalan, S., Widner, K., Madams, T., Cuadros, J., et al

Gulshan, V., Peng, L., Coram, M., Stumpe, M. C., Wu, D., Narayanaswamy, A., Venugopalan, S., Widner, K., Madams, T., Cuadros, J., et al. 2016 . Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs \ Jama , ...

2016
[23]

Han, S., Pool, J., Tran, J., \ Dally, W. 2015 . Learning both weights and connections for efficient neural network \ In NIPS

2015
[24]

Hinton, G., Vinyals, O., \ Dean, J. 2015 . Distilling the knowledge in a neural network \ arXiv preprint arXiv:1503.02531

2015 arXiv
[25]

\ \ Schmidhuber, J

Hochreiter, S. \ \ Schmidhuber, J. 1997 . Long short-term memory \ Neural computation , 9\/ (8), 1735--1780

1997
[26]

Hu, Z., Ma, X., Liu, Z., Hovy, E., \ Xing, E. 2016 . Harnessing deep neural networks with logic rules \ In ACL

2016
[27]

E., Pollard, T

Johnson, A. E., Pollard, T. J., Shen, L., Lehman, L. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Celi, L. A., \ Mark, R. G. 2016 . MIMIC-III , a freely accessible critical care database \ Scientific Data , 3

2016
[28]

Kim, B., Rudin, C., \ Shah, J. A. 2014 . The bayesian case model: A generative approach for case-based reasoning and prototype classification \ In Advances in Neural Information Processing Systems , \ 1952--1960

2014
[29]

Kingma, D. P. \ \ Ba, J. 2014 . Adam: A method for stochastic optimization \ arXiv preprint arXiv:1412.6980

2014 arXiv
[30]

Koh, P. W. \ \ Liang, P. 2017 . Understanding black-box predictions via influence functions \ arXiv preprint arXiv:1703.04730

2017 arXiv
[31]

Kohavi, R. 1996 . Scaling up the accuracy of naive-bayes classifiers: a decision-tree hybrid. \ In KDD , 96, \ 202--207. Citeseer

1996
[32]

Krizhevsky, A., Sutskever, I., \ Hinton, G. E. 2012 . Imagenet classification with deep convolutional neural networks \ In Advances in neural information processing systems , \ 1097--1105

2012
[33]

H., \ Leskovec, J

Lakkaraju, H., Bach, S. H., \ Leskovec, J. 2016 . Interpretable decision sets: A joint framework for description and prediction \ In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , \ 1675--1684. ACM

2016
[34]

Lei, T., Barzilay, R., \ Jaakkola, T. 2016 . Rationalizing neural predictions \ arXiv preprint arXiv:1606.04155

2016 arXiv
[35]

Lipton, Z. C. 2016 . The mythos of model interpretability \ In ICML Workshop on Human Interpretability in Machine Learning

2016
[36]

\ \ Lee, S.-I

Lundberg, S. \ \ Lee, S.-I. 2016 . An unexpected unity among methods for interpreting model predictions \ arXiv preprint arXiv:1611.07478

2016 arXiv
[37]

Maaten, L. v. d. \ \ Hinton, G. 2008 . Visualizing data using t-sne \ Journal of machine learning research , 9\/ (Nov), 2579--2605

2008
[38]

\ \ Astudillo, R

Martins, A. \ \ Astudillo, R. 2016 . From softmax to sparsemax: A sparse model of attention and multi-label classification \ In International Conference on Machine Learning , \ 1614--1623

2016
[39]

Miller, T. 2018 . Explanation in artificial intelligence: Insights from the social sciences \ Artificial Intelligence

2018
[40]

A., \ Dudley, J

Miotto, R., Li, L., Kidd, B. A., \ Dudley, J. T. 2016 . Deep patient: an unsupervised representation to predict the future of patients from the electronic health records \ Scientific reports , 6 , 26094

2016
[41]

Montavon, G., Samek, W., \ M \"u ller, K.-R. 2018 . Methods for interpreting and understanding deep neural networks \ Digital Signal Processing , 73 , 1--15

2018
[42]

Mordvintsev, A., Olah, C., \ Tyka, M. 2015 . Inceptionism: Going deeper into neural networks \ Google Research Blog. Retrieved June , 20\/ (14), 5

2015
[43]

Moro, S., Cortez, P., \ Rita, P. 2014 . A data-driven approach to predict the success of bank telemarketing \ Decision Support Systems , 62 , 22--31

2014
[44]

Ochiai, T., Matsuda, S., Watanabe, H., \ Katagiri, S. 2017 . Automatic node selection for deep neural networks using group lasso regularization \ In ICASSP

2017
[45]

Organization, W. H. \ et al. 2005 . Interim who clinical staging of hvi/aids and hiv/aids case definitions for surveillance: African region \ , Geneva: World Health Organization

2005
[46]

Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., \ Duchesnay, E. 2011a . Scikit-learn: Machine learning in P ython \ Jour...
[47]

Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., et al. 2011b . Scikit-learn: Machine learning in python \ Journal of machine learning research , 12\/ (Oct), 2825--2830
[48]

Rastegari, M., Ordonez, V., Redmon, J., \ Farhadi, A. 2016 . XNOR - N et: Image N et classification using binary convolutional neural networks \ In ECCV

2016
[49]

T., Singh, S., \ Guestrin, C

Ribeiro, M. T., Singh, S., \ Guestrin, C. 2016 . Why should i trust you?: Explaining the predictions of any classifier \ In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , \ 1135--1144. ACM

2016
[50]

S., Hughes, M

Ross, A. S., Hughes, M. C., \ Doshi-Velez, F. 2017 . Right for the right reasons: Training differentiable models by constraining their explanations \ arXiv preprint arXiv:1703.03717

2017 arXiv
[51]

R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., \ Batra, D

Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., \ Batra, D. 2017 . Grad- CAM : Visual explanations from deep networks via gradient-based localization \ arXiv preprint arXiv:1610.02391v3

2017 arXiv
[52]

R., Das, A., Vedantam, R., Cogswell, M., Parikh, D., \ Batra, D

Selvaraju, R. R., Das, A., Vedantam, R., Cogswell, M., Parikh, D., \ Batra, D. 2016 . Grad-cam: Why did you say that? \ arXiv preprint arXiv:1611.07450

2016 arXiv
[53]

T., \ Guestrin, C

Singh, S., Ribeiro, M. T., \ Guestrin, C. 2016 . Programs as black-box explanations \ arXiv preprint arXiv:1611.07579

2016 arXiv
[54]

Sutskever, I., Vinyals, O., \ Le, Q. V. 2014 . Sequence to sequence learning with neural networks \ In NIPS

2014
[55]

Tang, W., Hua, G., \ Wang, L. 2017 . How to train a compact binary neural network with high accuracy? \ In AAAI

2017
[56]

Zazzi, M., Incardona, F., Rosen-Zvi, M., Prosperi, M., Lengauer, T., Altmann, A., Sonnerborg, A., Lavee, T., Schulter, E., \ Kaiser, R. 2012 . Predicting response to antiretroviral treatment by machine learning: the euresist project \ Intervirology , 55\/ (2), 123--127

2012
[57]

D., \ Jordan, M

Zhang, Y., Lee, J. D., \ Jordan, M. I. 2016 . l1-regularized neural networks are improperly learnable in polynomial time \ In ICML

2016
[58]

write newline

" write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence '...

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

A., Rybeck, G., Scheidegger, C., Smith, B., \ Venkatasubramanian, S

Adler, P., Falk, C., Friedler, S. A., Rybeck, G., Scheidegger, C., Smith, B., \ Venkatasubramanian, S. 2016 . Auditing black-box models for indirect influence \ In ICDM

work page 2016

[2] [2]

\ \ Amir, O

Amir, D. \ \ Amir, O. 2018 . Highlights: Summarizing agent behavior to people \ In Proc. of the 17th International conference on Autonomous Agents and Multi-Agent Systems (AAMAS)

work page 2018

[3] [3]

\ \ Kokkolaras, M

Audet, C. \ \ Kokkolaras, M. 2016 . Blackbox and derivative-free optimization: theory, algorithms and applications . Springer

work page 2016

[4] [4]

Bach, S., Binder, A., Montavon, G., Klauschen, F., M \"u ller, K.-R., \ Samek, W. 2015 . On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation \ PloS one , 10\/ (7), e0130140

work page 2015

[5] [5]

Bahdanau, D., Cho, K., \ Bengio, Y. 2014 . Neural machine translation by jointly learning to align and translate \ arXiv preprint arXiv:1409.0473

arXiv 2014

[6] [6]

K., Rathod, V., Murphy, K

Balan, A. K., Rathod, V., Murphy, K. P., \ Welling, M. 2015 . Bayesian dark knowledge \ In NIPS

work page 2015

[7] [7]

Binder, A., Bach, S., Montavon, G., M \"u ller, K.-R., \ Samek, W. 2016 . Layer-wise relevance propagation for deep neural network architectures \ In Information Science and Applications (ICISA) 2016 , \ 913--922. Springer

work page 2016

[8] [8]

BuciluÇ, C., Caruana, R., \ Niculescu-Mizil, A. 2006 . Model compression \ In KDD

work page 2006