{"id":"3b2668ea-cc53-4156-9892-811eddacf255","arxiv_id":"2412.17066","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A free Python app lets users manipulate simulated class distributions and the threshold to see how classification metrics such as ROC AUC, MCC, and F1 change together.","lead":"Presents a free interactive web application that lets users adjust two simulated score distributions and a threshold, then watch how ROC, PR, MCC, F1, and other classification metrics change. It is a teaching tool for building intuition about why common evaluation metrics can mislead.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim hinges on metric computations being correct; Section 3's 'MCC = 0.5' for all-positive predictions is unexplained and may be a mislabeled normalization or a bug.","rationale":"The reader identified the same load-bearing assumption: the application's metric computations must be correct, and the Section 3 example about MCC = 0.5 is a warning sign. I agree that this is the central weak point. The paper's purpose is pedagogical, so an unexplained or mislabeled metric transformation is not a cosmetic issue; it directly undermines the claim that the tool builds correct intuition. The broken GitHub URL and lack of a versioned release are real availability concerns, but they are secondary to whether the metrics shown are trustworthy. Since the reader already issued a CONDITIONAL verdict and my concern is precisely the reason for that conditionality, the verdict should remain unchanged. A single reproducible numerical check on the all-positive-prediction scenario would settle whether the MCC display is a documented normalization or a genuine implementation error, and would establish a pattern for verifying the other metrics.","tokens_in":4435,"tokens_out":5187,"duration_ms":51589,"concrete_test":"Install the package from PyPI, reproduce the Section 3 configuration (negative class N=100, positive class N=500, threshold set below all predicted scores so every instance is predicted positive), and compare every displayed metric against direct confusion-matrix computations: Accuracy = 500/600, raw MCC = 0, PR AUC baseline = 500/600, and ROC AUC consistent with the two distributions. Also verify whether the displayed 'MCC (unit normalized)' equals (raw MCC + 1)/2; if it does, confirm that the UI and paper explicitly state this normalization, and if it does not, treat the metric computation as erroneous.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that the application exists, is freely available, and lets users interactively explore how evaluation metrics change as class distributions and thresholds vary. For that claim to be pedagogically valuable, every displayed metric must be computed correctly and labeled in a way that a learner can interpret. The paper provides no numerical validation, no test suite, and no equations for the metric calculations. Section 3 raises a concrete red flag: the scenario sets the threshold so low that the model predicts every instance as positive (TN = 0, FN = 0, TP = 500, FP = 100). The standard Matthews Correlation Coefficient for this confusion matrix is exactly 0, yet the text reports 'MCC = 0.5' as chance performance. The metric list mentions 'MCC (unit normalized)', but the example does not explain this transformation, leaving readers unable to tell whether the app is displaying raw MCC incorrectly, applying an undocumented normalization, or relying on a different definition. If the displayed 0.5 is a unit-normalized value, the app should state that explicitly and the Section 3 text should not present it as a standard MCC number. Without such clarification, the tool risks teaching the exact misinterpretation it aims to prevent. The same verification gap applies to ROC AUC, PR AUC, and the MCC-F1 curve, so one unexplained discrepancy is enough to cast doubt on the entire metric suite and on the central claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents Interactive Classification Metrics (ICM), an open-source Python application built with Bokeh that lets users interactively manipulate two class-conditional prediction distributions (mean, standard deviation, skew, and sample size) and a classification threshold, then observe how a suite of binary classification metrics change in concert. The displayed metrics include ROC and PR curves, their AUCs, a confusion matrix, the MCC-F1 curve, accuracy, recall, specificity, precision, NPV, F1, and a unit-normalized MCC. The authors argue that the tool fills a pedagogical gap by making metric tradeoffs and pitfalls (e.g., accuracy on imbalanced data, the low baseline of PR AUC) visible without requiring data wrangling or model training. A short example scenario demonstrates an overfit model on an imbalanced dataset where accuracy is high but MCC reveals chance performance.","tokens_in":4626,"tokens_out":2930,"duration_ms":28288,"significance":"If the metric computations are correct, ICM is a potentially valuable educational resource for machine learning courses and self-study. Its strengths include being freely available under the MIT license, providing an interactive rather than static visualization, covering a wider set of metrics than most existing educational tools, and directly addressing known caveats of ROC/PR curves and MCC. The paper is self-contained in its references to standard metric definitions and properly names its scholarly context. However, the scientific contribution rests almost entirely on whether the displayed metrics are computed correctly and labeled clearly; the manuscript provides no equations, no numerical verification of the displayed values, and no user study, and the Section 3 example contains an unexplained reporting of 'MCC = 0.5' for a case where the standard MCC is exactly zero. These gaps must be addressed before the central claim of a trustworthy and pedagogically effective application can be accepted.","major_comments":[{"comment":"The example scenario reports 'MCC = 0.5' for a model that predicts every instance as positive (TN=0, FN=0, TP=500, FP=100). The standard Matthews Correlation Coefficient for this confusion matrix is exactly 0, yet the text presents 0.5 as 'chance performance.' Section 2 lists 'MCC (unit normalized)' among the metrics, so the app likely rescales MCC, but neither the text nor the example explains this transformation. This is load-bearing because the pedagogical goal is to prevent misinterpretation of metrics; presenting a normalized value as if it were the standard MCC risks teaching the exact confusion the tool is meant to prevent. Please specify the normalization formula (e.g., (MCC+1)/2), state it in both the Section 3 narrative and the application interface, and report either raw MCC or clearly labeled normalized MCC throughout the example.","section":"Section 2, Section 3"},{"comment":"The paper gives no equations, no numerical verification, and no test cases for any of the displayed metrics. For example, Section 3 states PR AUC = 0.9 and its baseline as 0.83 without any calculation, and the list of metrics in Section 2 has no accompanying formulas. Because the central claim is that the application computes and displays these metrics correctly, the manuscript should include a minimal verification section or point to a test suite in the repository that checks expected values on a simple confusion matrix or synthetic distribution. Without this, a reader cannot distinguish a correct implementation from a subtly incorrect one, and the 'robust intuition' the tool claims to build may be built on incorrect values.","section":"Sections 2 and 3"}],"minor_comments":[{"comment":"The GitHub URL is broken: 'https://github.com/davhbrown/interactive classification metrics' contains a space and should be 'https://github.com/davhbrown/interactive_classification_metrics'.","section":"Abstract"},{"comment":"'Not applicabile' is a typo; it should be 'Not applicable.'","section":"Ethical committee"},{"comment":"The sentence 'For example, their interpretation depends on class (im)balance and comparison to each other' is grammatically awkward; rephrasing would improve clarity.","section":"Section 2"},{"comment":"The claim that no similar interactive visualization tool exists would be strengthened by citing specific examples (e.g., the listed static resources) and clarifying that the novelty is the combination of interactivity and coverage, rather than the absence of any related work.","section":"Section 4"}],"recommendation":"major_revision","confidential_remarks":"The MCC normalization issue in Section 3 is the main source of doubt: it could indicate a bug in the app's implementation or simply an unlabeled rescaling. A quick check of the repository's source code or test suite would settle this, and I recommend asking the authors to provide direct evidence of metric correctness as part of the revision. The paper's scope as a software/education note is appropriate for a journal like this, but the verification gap is real and should be addressed before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"ICM is a modest, useful software artifact: a free, browser-based app that lets you slide two class distributions and a threshold and watch Accuracy, ROC AUC, PR AUC, MCC, and the MCC-F1 curve change together. That combination is new among the cited prior work, and it fills a real gap for teaching. The paper is honest about what it is: a pedagogical tool, not a new metric or a research claim.\n\nWhat it does well: the underlying math is standard; the example in Section 3 makes the right point about Accuracy on imbalanced data; the code is public under MIT; and the screenshot and animation give a fair picture of the interaction. For a software paper, that is reasonable evidence of existence and function.\n\nWhere it gets soft: there is no test suite, no numerical verification, and no user study. For a tool whose whole purpose is to prevent misinterpretation, a few checks that the displayed values match the formulas would go a long way. The MCC = 0.5 in Section 3 is the one concrete red flag. The paper's metric list does say 'MCC (unit normalized),' so the 0.5 is consistent with normalizing raw MCC = 0 to the [0,1] range. But the example text just says 'MCC = 0.5' without flagging the normalization, which is exactly the kind of unlabeled transformation that would confuse the target audience. It is a labeling problem, not evidence that the app is broken. The stress-test worry that the whole metric suite might be wrong is not supported by anything in the paper; all the formulas are standard, and the single discrepancy has a plausible explanation.\n\nThe GitHub URL in the abstract is missing a character (there's a space in 'interactive classification metrics'), and there's no version number or commit hash. Minor, but worth fixing for reproducibility.\n\nWho is this for? Instructors teaching binary classification, and practitioners who want to internalize why accuracy, ROC AUC, and PR AUC can mislead. Researchers won't learn new math, and the paper will not change research practice. That's fine; the contribution is pedagogical. A serious referee should take it: the artifact is real, the argument is clear, and a light revision — explain the normalized MCC, add a test snippet or a table of expected values, fix the URL, pin a release — would make it solid. I'd send it to review.","headline":"A modest, useful educational tool for binary classification metrics; the example needs a one-line clarification about unit-normalized MCC.","tokens_in":5191,"tokens_out":3548,"would_cite":false,"duration_ms":30551,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper presents Interactive Classification Metrics (ICM), a free interactive application that visualizes how binary classification evaluation metrics respond to changes in class distribution statistics and threshold.","keywords":["machine learning","binary classification","evaluation metrics","Matthews Correlation Coefficient","ROC curve","interactive visualization","model evaluation","Python application"],"falsifier":"Run the installed package, set the sliders to a known configuration (for example, 100 negative and 500 positive examples with the threshold at the extreme), compute the confusion-matrix metrics directly from the distributions, and compare them with the app's displayed values; a correct implementation should show raw MCC equal to 0.0 for an all-positive predictor.","tokens_in":4173,"feed_emoji":"📊","tokens_out":7019,"duration_ms":53350,"temperature":0.7,"pith_summary":"The paper presents a software application, Interactive Classification Metrics (ICM), that lets users explore binary classification evaluation metrics interactively. Users adjust sliders controlling the sample size, mean, standard deviation, and skew of the predicted scores for negative and positive classes, plus the classification threshold, and all plots and metrics update in real time. The tool is designed to make visible how metrics such as Accuracy, ROC AUC, PR AUC, F1-Score, and MCC behave differently under class imbalance and threshold shifts. The authors argue that this hands-on exploration helps practitioners and students choose appropriate metrics and interpret them carefully, without the overhead of data wrangling or model training.","feed_headline":"A free app lets you watch classification metrics trade off live","feed_subtitle":"Adjust class imbalance, threshold, and skew to see why Accuracy alone misleads and PR AUC needs a baseline.","key_machinery":"The central object is the interactive application itself, built with the bokeh server, where nine sliders control the sample size, mean, standard deviation, and skew of two distributions representing the predicted scores of the negative and positive classes, along with a classification threshold. The application computes and plots a comprehensive suite of metrics in response, including the MCC-F1 curve and a unit-normalized MCC. This direct manipulation mechanism lets users see how every derived metric changes in concert and how the relative size and shape of the two class distributions drive all downstream evaluation.","core_discovery":"The central claim is that a single interactive application can make the relationships among widely used binary classification metrics intuitive and transparent. ICM displays class distributions, ROC and PR curves, a confusion matrix, the MCC-F1 curve, and eleven numeric metrics, all driven by nine sliders. The paper's example shows a model that predicts everything as the positive class on an imbalanced dataset achieving over 80% Accuracy and PR AUC of 0.9, while MCC indicates chance performance; the interactive display makes the misleading nature of single-number summaries immediately visible. The authors present this as a pedagogical tool to build robust intuition for model evaluation.","pith_inferences":["The same interaction pattern could be extended to multi-class evaluation, where confusion-matrix tradeoffs are more numerous and less commonly taught, though the paper does not implement this.","The app could serve as a parameterized figure generator for papers and lectures, letting authors produce static plots that document how a metric behaves across a chosen distribution grid.","Displaying raw MCC alongside the unit-normalized value, as a toggle, would make the tool's interpretation more transparent for advanced users.","The pedagogical effectiveness of the tool could be tested in a controlled study comparing metric-interpretation accuracy before and after use of the app."],"forward_implications":["Users can observe directly why Accuracy alone is misleading on imbalanced data, even when the threshold is set to predict the majority class almost always.","The app makes ROC AUC and PR AUC baselines explicit, so practitioners can see that a PR AUC of 0.9 may be near its floor when the positive class is large.","Comparing the unit-normalized MCC display with the confusion matrix helps users identify chance-level performance even when other metrics look strong.","The tool can be adopted as a lecture aid in machine learning courses to demonstrate metric tradeoffs without requiring students to write code.","Because the package is MIT-licensed and on PyPI, instructors and researchers can extend it with additional metrics or adapt it to their own pedagogical goals."],"supporting_citations":[{"why":"Supplies the MCC-versus-ROC-AUC comparison that the app visualizes and the argument for prioritizing MCC.","marker":"[6]"},{"why":"Provides the definitions and relationships among precision, recall, F-measure, ROC, informedness, markedness, and correlation used in the metric computations.","marker":"[7]"},{"why":"Supports the emphasis on MCC as a reliable metric in two-class evaluation, which the app illustrates.","marker":"[8]"},{"why":"Contributes the MCC-F1 curve module (py-mcc-f1) that the application incorporates.","marker":"[9]"},{"why":"Supplies the alternative naming conventions for metrics that the application references in its interface.","marker":"[10]"},{"why":"A static visual explanation of ROC and AUC that motivates the need for an interactive, more comprehensive tool.","marker":"[11]"},{"why":"A static visual explanation of precision and recall that motivates the need for an interactive, more comprehensive tool.","marker":"[12]"}],"fun_headline_variants":["Free app shows why 80% accuracy can mislead","Interactive tool reveals tradeoffs hidden in classification metrics","Slider-driven app clarifies why PR AUC needs a baseline","See how classification metrics mislead with just a few sliders"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"For the central claim to hold, every metric formula computed by the software must be correct, because the app's educational value depends entirely on the accuracy of the relationships it displays.","fun_headline_variants_meta":{"raw":{"variants":["Free app shows why 80% accuracy can mislead","Interactive tool reveals tradeoffs hidden in classification metrics","Slider-driven app clarifies why PR AUC needs a baseline","See how classification metrics mislead with just a few sliders"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00176,"raw_usage":{"total_tokens":6902,"prompt_tokens":859,"completion_tokens":6043,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":475,"completion_tokens_details":{"reasoning_tokens":5987}},"tokens_in":475,"tokens_out":6043,"duration_ms":39467,"temperature":1.0,"reasoning_tokens":5987,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T05:48:39.298476+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the installed package, set the sliders to a known configuration (for example, 100 negative and 500 positive examples with the threshold at the extreme), compute the confusion-matrix metrics directly from the distributions, and compare them with the app's displayed values; a correct implementation should show raw MCC equal to 0.0 for an all-positive predictor.","supporting_citations":[{"cited_title":"Table of confusion","cited_arxiv_id":null,"evidence_quote":"Supplies the alternative naming conventions for metrics that the application references in its interface."},{"cited_title":"ROC and AUC: A visual explanation of receiver operating characteristic curves and area under the curve,","cited_arxiv_id":null,"evidence_quote":"A static visual explanation of ROC and AUC that motivates the need for an interactive, more comprehensive tool."},{"cited_title":"Precision and recall: Accuracy is not enough,","cited_arxiv_id":null,"evidence_quote":"A static visual explanation of precision and recall that motivates the need for an interactive, more comprehensive tool."}],"review_version":1}