{"id":"0db55265-0a0c-4eeb-a04d-3d5326e4840b","arxiv_id":"2412.04377","paper_version":3,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"low","formal_verification":"none","parameter_count":2,"one_line_summary":"The paper defines six Tile-based visualization flavors (correlation, value, baseline, state-of-the-art, ranking, entity) and a four-profile guide for comparing binary classifiers, illustrated on 74 semantic segmentation models.","lead":"This paper presents a guided tour of the Tile, a two-dimensional map that organizes importance-weighted ranking scores for binary classifiers into one picture. It walks four user profiles through different ways to read the map, and illustrates them by ranking 74 semantic segmentation models.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Universality claim overreaches: the Tile family R_I in Eq. (3) does not contain all application preferences (e.g., balanced accuracy; the paper itself says mIoU is outside), so 'all possible preferences' is unsupported.","rationale":"The reader's verdict is CONDITIONAL; my read agrees. The math of the Tile flavors is coherent and the code is a practical asset. I do not see an internal error in Definitions 1-9. The load-bearing weakness is the scope of the central claim: Eq. (3) defines a two-parameter family of rational scores, and the paper neither proves nor tests that this family covers every preference a user might encode. The internal evidence is Scenario 4's admission that mIoU is not part of the Tile. The arbitrary label-grouping and posterior-threshold disclosure in Appendix A.4 is a separate, also valid concern about the demonstration of the 74-model rankings; it does not by itself threaten the Tile construction. My concern reinforces the reader's weakest_assumption, so the verdict should remain CONDITIONAL: the guide should be revised to either prove coverage or state the domain as the R_I family, and to temper the abstract's universality language.","tokens_in":33689,"tokens_out":14006,"duration_ms":149425,"concrete_test":"Using the 74 confusion matrices in Appendix A.7 and the code from the supplementary notebook, compute the rank ordering induced by binary balanced accuracy (or, equivalently, binary mIoU) and the ordering induced by R_I for a dense grid of (a,b) in [0,1]^2. Check whether any grid point has Spearman correlation 1.0 with the target ordering; more decisively, solve the feasibility problem: find (a,b) such that R_I(P_i)>R_I(P_j) for every consecutive pair (i,j) in the target ranking. If the constraint set is empty (or the maximum correlation is strictly below 1), the preference is not represented by any Tile point, and the 'all possible preferences' wording must be weakened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's central claim that the Tile 'captures the behavior of classifiers in a single visualization' while 'displaying all possible application-specific preferences' requires that every relevant binary-classifier preference be one of the ranking scores R_I in Eq. (3). This is a substantive restriction, not a normalization: R_I(P) is the ratio of one linear form in the four confusion probabilities, a*P(tp)+(1-a)*P(tn), over another, a*P(tp)+(1-a)*P(tn)+b*P(fn)+(1-b)*P(fp). Scores such as balanced accuracy, 0.5*(TPR+TNR), are sums of ratios and are not in this family; they can induce orderings that no single point (a,b) reproduces. The paper's own Scenario 4 concedes the point: it uses mIoU as 'a good criterion' but states 'this score is not part of the Tile,' resorting to the Correlation Tile instead. Thus the Tile does not itself accommodate every application-specific preference; it accommodates an infinite two-parameter subfamily. Since the guide's advertised completeness is load-bearing for the abstract and for Scenario 4's selection advice, the claim needs either a proof of coverage or an explicit domain restriction to the R_I family.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents a didactic guide to comparing binary classifiers using the Tile, a two-parameter family of ranking scores R_I defined in Eq. (3). It defines four user scenarios (theoretical analyst, method designer, benchmarker, application developer) and six flavors of the Tile (Correlation, Value, Baseline Value, State-of-the-Art Value, Ranking, and Entity), and illustrates them on 74 semantic segmentation models evaluated on BDD100K after collapsing labels into two classes. The paper claims that the Tile displays all possible application-specific preferences and effectively captures the behavior of classifiers in a single visualization.","tokens_in":33986,"tokens_out":6196,"duration_ms":62827,"significance":"If the paper is taken as a tutorial rather than a new theoretical contribution, it is a useful and well-executed guide: the definitions of the flavors are explicit, Eq. (3) correctly recovers TPR, TNR, PPV, NPV, accuracy, and F1 at the appropriate corners, and the supplementary material provides full numerical tables and code, making the 74-model illustration reproducible. The main value is pedagogical and organizational for practitioners who work within the R_I family. However, the advertised completeness ('all possible application-specific preferences') is not established, and the empirical illustration is an application of the authors' own framework rather than an external validation; these issues are fixable by restricting the claims.","major_comments":[{"comment":"The abstract's claim that the Tile displays 'all possible application-specific preferences' is unsupported by the definition of the family R_I. Eq. (3) restricts scores to ratios of one linear form in P({tp}), P({tn}) over a second linear form in all four events; scores such as balanced accuracy, 0.5(TPR+TNR), are sums of such ratios and are not members of the family, so their induced orderings need not be reproducible by any single point (a,b). The paper itself concedes this in Scenario 4 when it states that mIoU 'is not part of the Tile' and uses the Correlation Tile instead. The authors should either prove that every application-relevant preference (or every ordering) lies in the R_I family, or explicitly restrict the claims to scores of the form Eq. (3).","section":"Abstract; Section 3.1, Eq. (3); Section 3.2, Scenario 4"},{"comment":"The paper claims to 'demonstrate that the Tile effectively captures the behavior of classifiers' and to provide a full ranking of 74 state-of-the-art models, but the illustration cannot validate the framework: every flavor is constructed from the same R_I family it is meant to summarize, and no external benchmark, independent cost structure, or falsifiable prediction is used. Appendix A.4 further states that the results are specific to arbitrary choices and that changing the posterior threshold from pi+ to 0.5 'leads to a significantly different two-class classification problem, for which the ranking of the models is different.' At minimum, the claims about capturing behavior and ranking 74 models should be qualified as an illustration of the framework under explicitly stated conventions, not as a demonstration of the framework's validity.","section":"Abstract; Section 3.2, Scenario 4; Appendix A.4"}],"minor_comments":[{"comment":"The heading 'Acknowledments' should be spelled 'Acknowledgments'.","section":"Section 5 (Acknowledgments)"},{"comment":"The figure caption uses 'macro-IoU' while the text uses 'mean intersection over union (mIoU)'; please unify the terminology.","section":"Figure 4 and Section 3.1"},{"comment":"The hatched areas are explained in the text but never defined in the captions; adding a legend or a short caption sentence would improve readability.","section":"Figures 5-8"},{"comment":"The grid size parameter used to discretize the Tile is not reported; since all figures depend on this discretization, please state the default value and, ideally, a brief sensitivity check.","section":"Appendix A.2"},{"comment":"The codomain X(Omega,Sigma) is used in Definition 1 but defined only later in the supplementary symbol list; please define it before first use or rephrase the definition.","section":"Definition 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is heavily self-referential to the authors' prior work [36,37] and does not cite any independent evaluation of the Tile. The main contribution is pedagogical; if the journal expects novel theoretical results, the scope may be borderline. However, the presentation is clear and the code is provided, so with an honest scope statement and a restricted universality claim it could become a useful community resource."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read of arXiv:2412.04377.\n\nThe genuinely new thing is organizational: six named Tile flavors (Correlation, Value, Baseline, SoTA, Ranking, Entity) and a four-profile scenario guide. That is a real convenience. The math behind the Tile is clean, and the corner checks (TPR, TNR, PPV, NPV, accuracy, F1) are right. The supplementary tables reproduce the figures, and the code ships. The 74-model segmentation comparison is a serious exercise: they disclose the label grouping and posterior threshold choices in the appendix, and they note that thresholding at 0.5 rather than the prior changes the ranking substantially. Good transparency.\n\nThe soft spot is the abstract's claim that the Tile captures 'all possible application-specific preferences' and 'accommodates an infinite number of ranking scores.' The second clause is true; the first overreaches. The R_I family in Eq. (3) is the ratio of one linear form over another in the four confusion probabilities. Balanced accuracy, 0.5*(TPR+TNR), is a sum of ratios and does not sit in that family; it can induce orderings no single (a,b) point reproduces. The authors themselves concede this in Scenario 4, where they say mIoU 'is not part of the Tile' and fall back to the Correlation Tile. So the tile does not display all application-specific preferences; it displays an infinite two-parameter subfamily. That is still useful, but the claim needs either a proof of coverage or, more realistically, an explicit domain restriction.\n\nIs this load-bearing? For the guide's practical advice, mostly no: the Ranking Tile and Entity Tile are still fine conditional on the R_I family. For the abstract and the Scenario 4 selection story, yes, it is. A revision that softens the universality language and adds a one-line domain restriction would fix it.\n\nThe other soft spot is illustrative, not structural: the ranking of the 74 models depends on arbitrary binary grouping and the posterior threshold, and the authors admit these change results substantially. That weakens the demonstration as a ranking of 'state-of-the-art' models, but it does not touch the Tile mechanism.\n\nWho is this for? Practitioners and benchmarkers who want a single view across importance weights, and researchers interested in evaluation geometry. It deserves a serious referee; I'd accept it with the expectation of a revised abstract and a caveat in Scenario 4.","headline":"A genuinely useful guide to the Tile for two-class classifier evaluation, but the abstract overclaims universality for a two-parameter score family that does not contain balanced accuracy or mIoU.","tokens_in":34494,"tokens_out":1945,"would_cite":true,"duration_ms":19731,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62H30","68T10"],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that a single 2D map, the Tile, can show how a two-class classifier performs under every possible weighting of its four outcomes, replacing single-score comparisons with a complete picture of rankings.","keywords":["Tile visualization","two-class classification","ranking scores","classifier evaluation","semantic segmentation","importance weighting","benchmarking","performance ranking"],"falsifier":"Take a fixed set of classifiers and an application whose loss is a nonlinear function of the four outcome rates (for example, cost = TPR squared plus a fixed penalty per false alarm). If the application's ranking of those classifiers cannot be reproduced by $R_I$ at any point $(a,b)$ of the Tile, the claim that the Tile accommodates all application-specific preferences is refuted.","tokens_in":33507,"feed_emoji":"📊","tokens_out":9641,"duration_ms":92405,"temperature":0.7,"pith_summary":"The paper is a guide to a recently introduced visualization called the Tile, which arranges the infinite family of ranking scores for two-class classifiers on a unit square. Its central claim is that the Tile, read through six interpretative flavors, captures a classifier's behavior under every application-specific importance weighting in a single image. The guide organizes four user profiles—theoretical analyst, method designer, benchmarker, and application developer—and tells each which flavor to use and how to interpret it. It illustrates the approach by ranking 74 semantic segmentation models, showing that different models win in different regions of the map and that a single score is not enough to declare a winner.","feed_headline":"One 2D map holds every ranking score for binary classifiers","feed_subtitle":"A new guide shows how the Tile replaces one-score comparisons and ranks 74 segmentation models across all importance weights.","key_machinery":"The central object is the Tile, the mapping $(a,b) \\mapsto R_I$ defined on $[0,1]^2$, where $I(tn)=1-a$, $I(fp)=1-b$, $I(fn)=b$, $I(tp)=a$, and $R_I$ is the weighted ratio of correct outcomes to all four outcomes in Eq. (3). Each point of the Tile is one ranking score: the corners recover TPR, TNR, PPV, and NPV, the center is accuracy, and the right edge holds the $F_\\beta$ scores. The argument is carried by the fact that every score in this family is a function of the same four outcome probabilities, so value, rank, correlation, and entity identity can all be drawn on the same square without discarding information.","core_discovery":"For any two-class classifier, the paper places every importance-weighted ranking score $R_I$ at a point $(a,b)$ of the unit square, with $a$ controlling the importance of true positives relative to true negatives and $b$ controlling false negatives relative to false positives. The discovery is that this map can be read in six ways: as score value (Value Tile), as rank among a set of entities (Ranking Tile), as the identity of the entity at a fixed rank (Entity Tile), as correlation with a reference score (Correlation Tile), as the baseline envelope (Baseline Value Tile), and as the state-of-the-art envelope (State-of-the-Art Value Tile). On the 74-model illustration, the first rank belongs to four different models in different regions of the Tile, and the guide shows how an application developer can choose among them with known importance, with community practice encoded in a reference score, or with a minimax-rank rule when importance is unknown. Together these flavors make the Tile a single visualization that condenses the infinite family of scores into one picture.","pith_inferences":["The paper leaves implicit that choosing any scalar score is equivalent to choosing a point $(a,b)$ on the Tile; a natural next step is to define dominance regions where one entity beats another at every point, making 'which model is best' a set-valued answer.","The near-horizontal bands the paper observes in its illustration follow from an almost constant ratio $P({tp})/P({tn})$ across the 74 models; a testable prediction is that model families with heterogeneous priors will show fragmented tile regions, revealing models with genuinely different error trade-offs.","The discussion of multi-class extension suggests a concrete experiment: construct micro- and macro-averaged two-class Tiles for the same multi-class problem and compare their Entity Tiles with the full multi-class ranking, quantifying the information loss the paper warns about."],"forward_implications":["Benchmarking with a single score becomes a choice of a point on the Tile; a leaderboard that reports only one metric silently fixes an importance weighting.","For the 74 models analyzed, no model ranks first everywhere: the Entity Tile shows four different winners, so a fair challenge should report the winner as a region of the Tile rather than a single name.","When importance values are unknown, the minimax-rank rule (minimize the worst rank, then the average rank) gives a well-defined selection; in the illustration this chooses a model that never drops below rank 14 anywhere on the Tile.","The Correlation Tile offers a formal way to test whether a proposed new score carries information beyond a field-standard reference: compute its correlation with $R_I$ across the Tile and locate the low-correlation region."],"supporting_citations":[{"why":"Supplies the axiomatic definition of performance-based ranking and the importance-parameterized score family $R_I$ that the Tile organizes.","marker":"[37]"},{"why":"Introduces the Tile mapping and the mathematical properties of ranking scores on the unit square used throughout the guide.","marker":"[36]"},{"why":"Provides the collection of 74 semantic segmentation models used as the paper's illustration corpus.","marker":"[13]"},{"why":"Provides the driving test set on which the 74 models are evaluated as two-class classifiers.","marker":"[55]"},{"why":"Defines the bias notion used to read the direction of iso-value lines in the Value Tile.","marker":"[5]"}],"fun_headline_variants":["Six ways to read a single 2D map of classifier scores","Tile map condenses infinite classifier scores into one picture","One tile, six views: mastering binary classifier comparisons","How the Tile turns endless ranking scores into one useful map","A guide to reading classifier performance on a single 2D Tile"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central premise is that any practical way of scoring a two-class classifier can be expressed as a weighted ratio of the four possible prediction outcomes (true positive, true negative, false positive, false negative), with the two weights spanning the entire unit square.","fun_headline_variants_meta":{"raw":{"variants":["Six ways to read a single 2D map of classifier scores","Tile map condenses infinite classifier scores into one picture","One tile, six views: mastering binary classifier comparisons","How the Tile turns endless ranking scores into one useful map","A guide to reading classifier performance on a single 2D Tile"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000624,"raw_usage":{"total_tokens":2893,"prompt_tokens":956,"completion_tokens":1937,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":572,"completion_tokens_details":{"reasoning_tokens":1868}},"tokens_in":572,"tokens_out":1937,"duration_ms":12025,"temperature":1.0,"reasoning_tokens":1868,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T21:27:35.470672+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a fixed set of classifiers and an application whose loss is a nonlinear function of the four outcome rates (for example, cost = TPR squared plus a fixed penalty per false alarm). If the application's ranking of those classifiers cannot be reproduced by $R_I$ at any point $(a,b)$ of the Tile, the claim that the Tile accommodates all application-specific preferences is refuted.","supporting_citations":[{"cited_title":"Foundations of the theory of performance-based ranking","cited_arxiv_id":null,"evidence_quote":"Supplies the axiomatic definition of performance-based ranking and the importance-parameterized score family $R_I$ that the Tile organizes."},{"cited_title":"MMSegmentation: OpenMMLab semantic segmentation toolbox and benchmark","cited_arxiv_id":null,"evidence_quote":"Provides the collection of 74 semantic segmentation models used as the paper's illustration corpus."},{"cited_title":"BDD100K: A diverse driving dataset for hetero- geneous multitask learning","cited_arxiv_id":null,"evidence_quote":"Provides the driving test set on which the 74 models are evaluated as two-class classifiers."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the bias notion used to read the direction of iso-value lines in the Value Tile."}],"review_version":1}