Interactive Classification Metrics: A graphical application to build robust intuition for classification model evaluation

Davide Chicco; David H. Brown

REVIEW 2 major objections 4 minor 17 references

Interactive Classification Metrics: A graphical application to build robust intuition for classification model evaluation

T0 review · 2 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper presents Interactive Classification Metrics (ICM), a free interactive application that visualizes how binary classification evaluation metrics respond to changes in class distribution statistics and threshold.

desk verdict A modest, useful educational tool for binary classification metrics; the example needs a one-line clarification about unit-normalized MCC. read the letter →

arxiv 2412.17066 v1 pith:6JFXITA4 submitted 2024-12-22 cs.LG stat.ML

David H. Brown , Davide Chicco This is my paper · ORCID

classification cs.LGstat.ML

keywords machinelearningbinaryclassificationevaluationmetricsMatthewsCorrelationCoefficientROCcurveinteractivevisualizationmodelPythonapplication

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper presents a software application, Interactive Classification Metrics (ICM), that lets users explore binary classification evaluation metrics interactively. Users adjust sliders controlling the sample size, mean, standard deviation, and skew of the predicted scores for negative and positive classes, plus the classification threshold, and all plots and metrics update in real time. The tool is designed to make visible how metrics such as Accuracy, ROC AUC, PR AUC, F1-Score, and MCC behave differently under class imbalance and threshold shifts. The authors argue that this hands-on exploration helps practitioners and students choose appropriate metrics and interpret them carefully, without the overhead of data wrangling or model training.

What carries the argument

The central object is the interactive application itself, built with the bokeh server, where nine sliders control the sample size, mean, standard deviation, and skew of two distributions representing the predicted scores of the negative and positive classes, along with a classification threshold. The application computes and plots a comprehensive suite of metrics in response, including the MCC-F1 curve and a unit-normalized MCC. This direct manipulation mechanism lets users see how every derived metric changes in concert and how the relative size and shape of the two class distributions drive all downstream evaluation.

What would settle it

Run the installed package, set the sliders to a known configuration (for example, 100 negative and 500 positive examples with the threshold at the extreme), compute the confusion-matrix metrics directly from the distributions, and compare them with the app's displayed values; a correct implementation should show raw MCC equal to 0.0 for an all-positive predictor.

Watch

Extended reading notes

Core claim

The central claim is that a single interactive application can make the relationships among widely used binary classification metrics intuitive and transparent. ICM displays class distributions, ROC and PR curves, a confusion matrix, the MCC-F1 curve, and eleven numeric metrics, all driven by nine sliders. The paper's example shows a model that predicts everything as the positive class on an imbalanced dataset achieving over 80% Accuracy and PR AUC of 0.9, while MCC indicates chance performance; the interactive display makes the misleading nature of single-number summaries immediately visible. The authors present this as a pedagogical tool to build robust intuition for model evaluation.

Load-bearing premise

For the central claim to hold, every metric formula computed by the software must be correct, because the app's educational value depends entirely on the accuracy of the relationships it displays.

Editorial extensions

If this is right

Users can observe directly why Accuracy alone is misleading on imbalanced data, even when the threshold is set to predict the majority class almost always.
The app makes ROC AUC and PR AUC baselines explicit, so practitioners can see that a PR AUC of 0.9 may be near its floor when the positive class is large.
Comparing the unit-normalized MCC display with the confusion matrix helps users identify chance-level performance even when other metrics look strong.
The tool can be adopted as a lecture aid in machine learning courses to demonstrate metric tradeoffs without requiring students to write code.
Because the package is MIT-licensed and on PyPI, instructors and researchers can extend it with additional metrics or adapt it to their own pedagogical goals.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The same interaction pattern could be extended to multi-class evaluation, where confusion-matrix tradeoffs are more numerous and less commonly taught, though the paper does not implement this.
The app could serve as a parameterized figure generator for papers and lectures, letting authors produce static plots that document how a metric behaves across a chosen distribution grid.
Displaying raw MCC alongside the unit-normalized value, as a toggle, would make the tool's interpretation more transparent for advanced users.
The pedagogical effectiveness of the tool could be tested in a controlled study comparing metric-interpretation accuracy before and after use of the app.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

ICM is a modest, useful software artifact: a free, browser-based app that lets you slide two class distributions and a threshold and watch Accuracy, ROC AUC, PR AUC, MCC, and the MCC-F1 curve change together. That combination is new among the cited prior work, and it fills a real gap for teaching. The paper is honest about what it is: a pedagogical tool, not a new metric or a research claim.

What it does well: the underlying math is standard; the example in Section 3 makes the right point about Accuracy on imbalanced data; the code is public under MIT; and the screenshot and animation give a fair picture of the interaction. For a software paper, that is reasonable evidence of existence and function.

Where it gets soft: there is no test suite, no numerical verification, and no user study. For a tool whose whole purpose is to prevent misinterpretation, a few checks that the displayed values match the formulas would go a long way. The MCC = 0.5 in Section 3 is the one concrete red flag. The paper's metric list does say 'MCC (unit normalized),' so the 0.5 is consistent with normalizing raw MCC = 0 to the [0,1] range. But the example text just says 'MCC = 0.5' without flagging the normalization, which is exactly the kind of unlabeled transformation that would confuse the target audience. It is a labeling problem, not evidence that the app is broken. The stress-test worry that the whole metric suite might be wrong is not supported by anything in the paper; all the formulas are standard, and the single discrepancy has a plausible explanation.

The GitHub URL in the abstract is missing a character (there's a space in 'interactive classification metrics'), and there's no version number or commit hash. Minor, but worth fixing for reproducibility.

Who is this for? Instructors teaching binary classification, and practitioners who want to internalize why accuracy, ROC AUC, and PR AUC can mislead. Researchers won't learn new math, and the paper will not change research practice. That's fine; the contribution is pedagogical. A serious referee should take it: the artifact is real, the argument is clear, and a light revision — explain the normalized MCC, add a test snippet or a table of expected values, fix the URL, pin a release — would make it solid. I'd send it to review.

Referee Report

2 major / 4 minor

Summary. This paper presents Interactive Classification Metrics (ICM), an open-source Python application built with Bokeh that lets users interactively manipulate two class-conditional prediction distributions (mean, standard deviation, skew, and sample size) and a classification threshold, then observe how a suite of binary classification metrics change in concert. The displayed metrics include ROC and PR curves, their AUCs, a confusion matrix, the MCC-F1 curve, accuracy, recall, specificity, precision, NPV, F1, and a unit-normalized MCC. The authors argue that the tool fills a pedagogical gap by making metric tradeoffs and pitfalls (e.g., accuracy on imbalanced data, the low baseline of PR AUC) visible without requiring data wrangling or model training. A short example scenario demonstrates an overfit model on an imbalanced dataset where accuracy is high but MCC reveals chance performance.

Significance. If the metric computations are correct, ICM is a potentially valuable educational resource for machine learning courses and self-study. Its strengths include being freely available under the MIT license, providing an interactive rather than static visualization, covering a wider set of metrics than most existing educational tools, and directly addressing known caveats of ROC/PR curves and MCC. The paper is self-contained in its references to standard metric definitions and properly names its scholarly context. However, the scientific contribution rests almost entirely on whether the displayed metrics are computed correctly and labeled clearly; the manuscript provides no equations, no numerical verification of the displayed values, and no user study, and the Section 3 example contains an unexplained reporting of 'MCC = 0.5' for a case where the standard MCC is exactly zero. These gaps must be addressed before the central claim of a trustworthy and pedagogically effective application can be accepted.

major comments (2)

[Section 2, Section 3] The example scenario reports 'MCC = 0.5' for a model that predicts every instance as positive (TN=0, FN=0, TP=500, FP=100). The standard Matthews Correlation Coefficient for this confusion matrix is exactly 0, yet the text presents 0.5 as 'chance performance.' Section 2 lists 'MCC (unit normalized)' among the metrics, so the app likely rescales MCC, but neither the text nor the example explains this transformation. This is load-bearing because the pedagogical goal is to prevent misinterpretation of metrics; presenting a normalized value as if it were the standard MCC risks teaching the exact confusion the tool is meant to prevent. Please specify the normalization formula (e.g., (MCC+1)/2), state it in both the Section 3 narrative and the application interface, and report either raw MCC or clearly labeled normalized MCC throughout the example.
[Sections 2 and 3] The paper gives no equations, no numerical verification, and no test cases for any of the displayed metrics. For example, Section 3 states PR AUC = 0.9 and its baseline as 0.83 without any calculation, and the list of metrics in Section 2 has no accompanying formulas. Because the central claim is that the application computes and displays these metrics correctly, the manuscript should include a minimal verification section or point to a test suite in the repository that checks expected values on a simple confusion matrix or synthetic distribution. Without this, a reader cannot distinguish a correct implementation from a subtly incorrect one, and the 'robust intuition' the tool claims to build may be built on incorrect values.

minor comments (4)

[Abstract] The GitHub URL is broken: 'https://github.com/davhbrown/interactive classification metrics' contains a space and should be 'https://github.com/davhbrown/interactive_classification_metrics'.
[Ethical committee] 'Not applicabile' is a typo; it should be 'Not applicable.'
[Section 2] The sentence 'For example, their interpretation depends on class (im)balance and comparison to each other' is grammatically awkward; rephrasing would improve clarity.
[Section 4] The claim that no similar interactive visualization tool exists would be strengthened by citing specific examples (e.g., the listed static resources) and clarifying that the novelty is the combination of interactivity and coverage, rather than the absence of any related work.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: ICM is a visualization tool, not a derivation chain; metric definitions come from standard external references and no fitted quantity is presented as a prediction.

full rationale

This paper does not present a derivation, prediction loop, or fitted model. It introduces an interactive application that computes standard binary classification metrics from user-selected distribution parameters and a threshold. There is no equation in the paper by which one metric is defined in terms of another in a way that would make the displayed relationships true by construction; instead, metrics such as Accuracy, Recall, Precision, F1, ROC AUC, PR AUC, and MCC are standard quantities whose definitions are cited to external references [6, 7, 8, 9]. The central claim is that the software exists, is freely available under the MIT license, and lets users explore metric tradeoffs interactively; that claim is falsifiable by running the code, not by agreement with any fitted parameter. The paper contains self-citations co-authored by D. Chicco ([6], [8], [9]), but these are used as scholarly background for metric limitations and the MCC-F1 curve, not as load-bearing justification for any computational result in ICM; no uniqueness theorem or methodological premise is imported exclusively from the authors' prior work. The one notable anomaly, Section 3 reporting 'MCC = 0.5' for a model that predicts everything as positive (raw MCC = 0 with TN = 0, FN = 0, TP = 500, FP = 100), is a potential labeling, normalization, or implementation issue, but it is not circularity: an incorrect or ambiguous display does not make the application's claim reduce to its own inputs. Accordingly, the score is 0, with no circular steps identified.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

The paper introduces no free parameters fitted to data and no new entities. Its load-bearing assumptions are the adequacy of the simulated distributions for pedagogical purposes and the correctness of the metric implementations.

assumptions (2)

domain assumption Binary classification metrics are derived from the confusion matrix, which the app reconstructs from two class-conditional score distributions and a threshold.
The app models model outputs as skewed Gaussian distributions and computes the confusion matrix from their cumulative distribution functions at the threshold; this is a modeling convenience for visualization and is not asserted as a universal description of real model outputs.
standard math Standard metric definitions (ROC AUC, PR AUC, MCC, F1, etc.) are correct and correctly implemented.
The paper cites references for these metrics and does not re-derive them; the example in Section 3 suggests potential confusion about MCC normalization.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Interactive Classification Metrics: A graphical application to build robust intuition for classification model evaluation." pith.science (2026). https://pith.science/paper/6JFXITA4

@misc{pith2026241217066,
  author       = {Pith},
  title        = {Pith review of: Interactive Classification Metrics: A graphical application to build robust intuition for classification model evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6JFXITA4}},
  note         = {Machine review of arXiv:2412.17066}
}

read the original abstract

Machine learning continues to grow in popularity in academia, in industry, and is increasingly used in other fields. However, most of the common metrics used to evaluate even simple binary classification models have shortcomings that are neither immediately obvious nor consistently taught to practitioners. Here we present Interactive Classification Metrics (ICM), an application to visualize and explore the relationships between different evaluation metrics. The user changes the distribution statistics and explores corresponding changes across a suite of evaluation metrics. The interactive, graphical nature of this tool emphasizes the tradeoffs of each metric without the overhead of data wrangling and model training. The goals of this application are: (1) to aid practitioners in the ever-expanding machine learning field to choose the most appropriate evaluation metrics for their classification problem; (2) to promote careful attention to interpretation that is required even in the simplest scenarios like binary classification. Our application is publicly available for free under the MIT license as a Python package on PyPI at https://pypi.org/project/interactive-classification-metrics and on GitHub at https://github.com/davhbrown/interactive_classification_metrics.

Figures

Figures reproduced from arXiv: 2412.17066 by the authors.

**Figure 2.** The classic flaw of Accuracy on an imbalanced dataset. The negative class (black) has [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

17 extracted references · 10 canonical work pages

[1]

Occupational outlook handbook, data scientists,

Bureau of Labor Statistics, U.S. Department of Labor, “Occupational outlook handbook, data scientists,” 2024. [Online]. Available: https://www.bls.gov/ooh/math/data-scientists.htm

work page 2024
[2]

State of data science and machine learning,

Kaggle, “State of data science and machine learning,” 2022. [Online]. Available: https://www.kaggle.com/kaggle-survey-2022

work page 2022
[3]

A comparison of deep learning performance against health-care professionals in detecting diseases from medical imaging: a systematic review and meta-analysis,

X. Liu, L. Faes, A. U. Kale, S. K. Wagner, D. J. Fu, A. Bruynseels, T. Mahendiran, G. Moraes, M. Shamdas, C. Kern, J. R. Ledsam, M. K. Schmid, K. Balaskas, E. J. Topol, L. M. Bachmann, P. A. Keane, and A. K. Denniston, “A comparison of deep learning performance against health-care professionals in detecting diseases from medical imaging: a systematic revi...

work page doi:10.1016/s2589-7500(1 2019
[4]

Leakage and the reproducibility crisis in machine- learning-based science,

S. Kapoor and A. Narayanan, “Leakage and the reproducibility crisis in machine- learning-based science,” Patterns, vol. 4, no. 9, p. 100804, 2023. [Online]. Available: https://doi.org/10.1016/j.patter.2023.100804

arXiv 2023
[5]

Illusory generalizability of clinical prediction models,

A. M. Chekroud, M. Hawrilenko, H. Loho, J. Bondar, R. Gueorguieva, A. Hasan, J. Kambeitz, P. R. Corlett, N. Koutsouleris, H. M. Krumholz, J. H. Krystal, and M. Paulus, “Illusory generalizability of clinical prediction models,” Science, vol. 383, no. 6679, pp. 164–167, 2024. [Online]. Available: https://www.science.org/doi/abs/10.1126/science.adg8538

doi:10.1126/science.adg8538 2024
[6]

The Matthews correlation coefficient (MCC) should replace the ROC AUC as the standard metric for assessing binary classification,

D. Chicco and G. Jurman, “The Matthews correlation coefficient (MCC) should replace the ROC AUC as the standard metric for assessing binary classification,” BioData Mining, vol. 16, no. 4, pp. 1–23, 2023. [Online]. Available: https://doi.org/10.1186/s13040-023-00322-4

doi:10.1186/s13040-023-00322-4 2023
[7]

Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation,

D. M. W. Powers, “Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation,” 2020. [Online]. Available: https://arxiv.org/abs/2010.16061

arXiv 2020
[8]

The Matthews correlation coefficient (MCC) is more reliable than balanced accuracy, bookmaker informedness, and markedness in two-class confusion matrix evaluation,

D. Chicco, N. T¨ otsch, and G. Jurman, “The Matthews correlation coefficient (MCC) is more reliable than balanced accuracy, bookmaker informedness, and markedness in two-class confusion matrix evaluation,” BioData Mining, vol. 14, pp. 1–22, 2021. [Online]. Available: https://doi.org/10.1186/s13040-021-00244-z

doi:10.1186/s13040-021-00244-z 2021

Show all 17 references

[9]

The mcc-f1 curve: a performance evaluation technique for binary classification,

C. Cao, D. Chicco, and M. M. Hoffman, “The mcc-f1 curve: a performance evaluation technique for binary classification,” 2020. [Online]. Available: https://arxiv.org/abs/2006.11278

2020 arXiv
[10]

Table of confusion

Wikipedia, “Table of confusion.” [Online]. Available: https://en.wikipedia.org/wiki/ Confusion matrix#Table of confusion
[11]

ROC and AUC: A visual explanation of receiver operating characteristic curves and area under the curve,

J. Wilber, “ROC and AUC: A visual explanation of receiver operating characteristic curves and area under the curve,” June 2022. [Online]. Available: https://mlu-explain.github.io/roc-auc/

2022
[12]

Precision and recall: Accuracy is not enough,

——, “Precision and recall: Accuracy is not enough,” March 2022. [Online]. Available: https://mlu-explain.github.io/precision-recall/

2022
[13]

ROC curves,

Sharpr and Kakau, “ROC curves,” October 2015. [Online]. Available: https://upload.wikimedia. org/wikipedia/commons/4/4f/ROC curves.svg

2015
[14]

Understanding ROC curves,

R. Navaneethakrishnan, “Understanding ROC curves,” October 2014. [Online]. Available: http://navan.name/roc/

2014
[15]

Classification: Accuracy, recall, precision, and related metrics,

Google, “Classification: Accuracy, recall, precision, and related metrics,” November
[16]

Principles of effective data visualization,

S. R. Midway, “Principles of effective data visualization,” Patterns, vol. 1, no. 9, 2020. [Online]. Available: https://doi.org/10.1016/j.patter.2020.100141 6

2020
[2024]

Available: https://developers.google.com/machine-learning/crash-course/ classification/accuracy-precision-recall

[Online]. Available: https://developers.google.com/machine-learning/crash-course/ classification/accuracy-precision-recall

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

Occupational outlook handbook, data scientists,

Bureau of Labor Statistics, U.S. Department of Labor, “Occupational outlook handbook, data scientists,” 2024. [Online]. Available: https://www.bls.gov/ooh/math/data-scientists.htm

work page 2024

[2] [2]

State of data science and machine learning,

Kaggle, “State of data science and machine learning,” 2022. [Online]. Available: https://www.kaggle.com/kaggle-survey-2022

work page 2022

[3] [3]

A comparison of deep learning performance against health-care professionals in detecting diseases from medical imaging: a systematic review and meta-analysis,

X. Liu, L. Faes, A. U. Kale, S. K. Wagner, D. J. Fu, A. Bruynseels, T. Mahendiran, G. Moraes, M. Shamdas, C. Kern, J. R. Ledsam, M. K. Schmid, K. Balaskas, E. J. Topol, L. M. Bachmann, P. A. Keane, and A. K. Denniston, “A comparison of deep learning performance against health-care professionals in detecting diseases from medical imaging: a systematic revi...

work page doi:10.1016/s2589-7500(1 2019

[4] [4]

Leakage and the reproducibility crisis in machine- learning-based science,

S. Kapoor and A. Narayanan, “Leakage and the reproducibility crisis in machine- learning-based science,” Patterns, vol. 4, no. 9, p. 100804, 2023. [Online]. Available: https://doi.org/10.1016/j.patter.2023.100804

arXiv 2023

[5] [5]

Illusory generalizability of clinical prediction models,

A. M. Chekroud, M. Hawrilenko, H. Loho, J. Bondar, R. Gueorguieva, A. Hasan, J. Kambeitz, P. R. Corlett, N. Koutsouleris, H. M. Krumholz, J. H. Krystal, and M. Paulus, “Illusory generalizability of clinical prediction models,” Science, vol. 383, no. 6679, pp. 164–167, 2024. [Online]. Available: https://www.science.org/doi/abs/10.1126/science.adg8538

doi:10.1126/science.adg8538 2024

[6] [6]

The Matthews correlation coefficient (MCC) should replace the ROC AUC as the standard metric for assessing binary classification,

D. Chicco and G. Jurman, “The Matthews correlation coefficient (MCC) should replace the ROC AUC as the standard metric for assessing binary classification,” BioData Mining, vol. 16, no. 4, pp. 1–23, 2023. [Online]. Available: https://doi.org/10.1186/s13040-023-00322-4

doi:10.1186/s13040-023-00322-4 2023

[7] [7]

Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation,

D. M. W. Powers, “Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation,” 2020. [Online]. Available: https://arxiv.org/abs/2010.16061

arXiv 2020

[8] [8]

The Matthews correlation coefficient (MCC) is more reliable than balanced accuracy, bookmaker informedness, and markedness in two-class confusion matrix evaluation,

D. Chicco, N. T¨ otsch, and G. Jurman, “The Matthews correlation coefficient (MCC) is more reliable than balanced accuracy, bookmaker informedness, and markedness in two-class confusion matrix evaluation,” BioData Mining, vol. 14, pp. 1–22, 2021. [Online]. Available: https://doi.org/10.1186/s13040-021-00244-z

doi:10.1186/s13040-021-00244-z 2021