Pith. sign in

REVIEW 5 major objections 6 minor 56 references

Set Visualizations for Comparing and Evaluating Machine Learning Models

T0 review · 5 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that grouping matched model predictions into sets, as realized in the SetMLVis tool, makes comparing object detection models more accurate and less mentally demanding than traditional visualization.

desk verdict SetMLVis and the set-based framing are worth a look, but the reported statistics are impossible and the study is confounded, so the empirical claim doesn't hold as written. read the letter →

arxiv 2502.14675 v1 pith:JJR2BGOU submitted 2025-01-30 cs.HC

Liudas Panavas , Tarik Crnovrsanin , Racquel Fygenson , Eamon Conway , Derek Millard , Norbou Buchler , Cody Dunne This is my paper · ORCID
classification cs.HC
keywords setvisualizationmachinelearningmodelcomparisonobjectdetectionvisualanalyticsUpcognitiveworkloadNASA-TLXevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Machine learning practitioners compare models by looking at aggregate metrics, then inspect individual images when metrics fail to explain differences. This paper claims that this order is backwards: predictions should first be matched across models and grouped into sets of agreement and disagreement, and only then evaluated against ground truth. It gives four criteria that make any machine learning output convertible to set data, demonstrates the conversion on clustering, regression, classification, and object detection, and builds this approach into SetMLVis, an interactive notebook-based tool. In a within-subjects study of eight participants, the set-based tool led to more accurate answers and lower self-reported cognitive workload than a conventional object-detection interface.

What carries the argument

The load-bearing mechanism is the set generation rule: a precise, task-specific criterion for deciding when two models agree on a prediction. For object detection the rule is an IOU threshold that turns a collection of bounding boxes into intersecting sets; for clustering it is cluster-label agreement, and for regression it is a proximity bound. Each set becomes a clickable bar in an UpSet-style matrix, connecting an aggregate overview to the underlying images and bounding boxes, which is what lets users move from asking which models are similar to asking why they differ without manually inspecting every instance.

What would settle it

Recompute the McNemar test from the per-participant accuracy data in the supplemental materials: the paper reports $\chi^2(1, N=35) = 0.0$ with $p = 0.004$, which is internally inconsistent, since a chi-square statistic of zero yields $p = 1$. If the raw data confirm this inconsistency, the claimed accuracy advantage is not established, and the central claim would need a new study with the corrected sample size to be tested.

Watch

Extended reading notes

Core claim

The central claim is that organizing model outputs into sets based on agreement between models changes the comparison task from a search through instance-level data into a query over meaningful subsets. The paper reorders the standard workflow: match predictions first, then score against ground truth. In object detection, matched bounding boxes become sets when their intersection-over-union exceeds a user-set threshold; clicking a set shows the instances where exactly those models agree or disagree. The paper's evaluation reports that this approach outperforms traditional visualization techniques in terms of task completion and reduces cognitive workload for users, with accuracy differences concentrated in the tasks that require the most comparison steps.

Load-bearing premise

The result that set visualizations improve task accuracy rests on the statistical analysis of just eight participants, a sample the paper itself says a corrected power analysis would need to be larger.

Editorial extensions

If this is right

  • If the central claim is right, model evaluation tools can adopt the reordered workflow: match predictions across models first, then score against ground truth, which gives practitioners a principled path from aggregate metrics to instance-level inspection.
  • The same four criteria apply across clustering, regression, classification, and object detection, so set-based comparison is not limited to one type of model.
  • New aggregate metrics, such as Jaccard similarity and Tversky containment, can complement average precision by capturing whether models actually make the same predictions rather than merely achieving similar scores.
  • Set-based comparison can support iterative model development, since visualizing agreements and disagreements across model versions or hyperparameter settings would make the effects of changes visible.
  • If set visualizations reduce cognitive workload, they are especially likely to help on comparison tasks that involve multiple reasoning steps, where users must combine several pieces of evidence.
  • The query system within SetMLVis, rather than the full set overview, carried much of the interaction in the study; a follow-up could test whether the set summary itself is necessary or whether a well-designed query interface alone suffices.
  • The set generation criteria, especially IOU thresholds, are user-chosen and can change the apparent agreement between models; exploring sensitivity to those thresholds would clarify when set visualizations help versus mislead.
  • Extending set comparisons to false negatives, which the paper lists as a limitation, would make the method more useful for practitioners who need a complete picture of model errors.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A properly powered replication is the natural next step: the paper itself notes that its initial power analysis was incorrect and more participants were needed, so the size of the reported accuracy advantage should be treated as preliminary.
  • The success of the query system suggests that the value of sets may come as much from turning comparison questions into discrete filters as from the aggregate UpSet overview; a follow-up could separate these two contributions.
  • Set-based agreement could be a lens for studying model diversity in ensembles, since the Tversky containment idea in the paper points toward directly measuring when one model's predictions add nothing beyond another's.
  • Because the set generation rules are user-defined, a practical extension would be guidance or defaults for choosing thresholds that match the intended comparison, since the paper shows these choices can change the conclusions.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. This paper proposes that machine learning practitioners compare models by transforming model outputs into set-type data and visualizing the resulting intersections with UpSet-style diagrams, rather than relying only on aggregate metrics or manual instance-level inspection. The authors state four criteria for when this transformation is possible (Section 3) and present four worked case studies (clustering, regression, image classification, object detection) in Sections 3.2-3.5. For object detection they contribute SetMLVis, an open-source Jupyter-widget tool that matches bounding boxes across models via IoU thresholds, evaluates matched sets against ground truth, and provides an UpSet-style overview, thumbnail and detail views, and an interactive query system (Section 4). The central empirical claim comes from a within-subjects study with 8 graduate students comparing SetMLVis against FiftyOne on five tasks: the abstract and Section 9 conclude that SetMLVis improves task accuracy and reduces cognitive workload. Section 5, however, states explicitly that the authors do not attribute observed differences directly to set visualization, and Section 7.3 reports that participants used the query system, not the set visualization, as their primary interaction.

Significance. The conceptual direction is timely and addresses a real gap in model evaluation tooling, and the paper grounds it appropriately in the Cabrera et al. sensemaking framework (Section 3.6). The case studies in Sections 3.2-3.5 are concrete and repeatable, and the paper ships substantial artifacts: open-source software (PyPI, GitHub), a preregistered protocol and data (OSF), and notebooks accompanying the case studies. If the empirical evaluation were sound, this would be a solid systems-and-evaluation contribution. The evaluation is the load-bearing part of the headline claim, however, and it fails on three independent grounds that the manuscript itself partly concedes: the reported test statistics in Section 6 are arithmetically impossible, Section 5.1 admits that the corrected power analysis required more than the 8 recruited participants, and Sections 5 and 7.3 concede that the design cannot isolate the set visualization from the query system. The failure is in the evaluation rather than in the core idea.

major comments (5)
  1. [Section 6 (Results)] The accuracy comparison is reported as 'p = 0.004, chi2(1, N = 35) = 0.0'. A chi-squared statistic of exactly 0 cannot produce p = 0.004, and the stated N = 35 does not match the 8 participants x 5 tasks = 40 paired observations described in Section 5.2, with no missing data reported. Additionally, the aggregated test treats the 40 participant-task observations as independent even though they are clustered within 8 participants. As reported, the headline accuracy result is internally inconsistent and cannot be verified from the manuscript.
  2. [Section 6 (Results)] The NASA-TLX workload comparison is reported as 'W = 1279.0, p < 0.001'. With n = 8 paired observations (one aggregated workload score per participant per system, per Section 5.3), the largest possible Wilcoxon signed-rank statistic is n(n+1)/2 = 36, so W = 1279 is arithmetically impossible. The cognitive-workload reduction claim is therefore not supported by the reported analysis.
  3. [Section 5.1 (Participants)] The authors state that their initial power analysis was incorrect and that the corrected power analysis indicated that more participants than the eight recruited were required. The study is thus underpowered by the authors' own criterion, and the statement that 'substantial differences' were still observed does not compensate, since the test statistics on which that observation rests are the invalid ones discussed above.
  4. [Sections 5 and 7.3 (Methodology and Discussion)] The study cannot isolate the effect of set visualization. Section 7.3 reports that nearly every participant used the query system (Fig. 10H) as the primary way of interacting with SetMLVis, and Section 5 explicitly states that the authors 'do not explicitly attribute any observed differences in performance directly to the use of set visualization.' Because the set visualization, the query system, and the Jupyter-widget environment are bundled in the SetMLVis condition, and FiftyOne lacks an equivalent query mechanism (Table 2), neither the abstract's claim that SetMLVis 'outperforms traditional visualization techniques' nor Section 9's claim that 'using sets improves accuracy ... and reduces cognitive workload' is supported by this design; supporting those claims requires new data from conditions that separate the query system from the set visualization.
  5. [Section 5.2 (Methodology)] Task correctness was adjudicated by the study administrators, who are also the primary and third authors, and no scoring rubric, blind scoring procedure, or inter-rater reliability measure is reported. Because the research question is precisely whether one system yields more correct answers, this adjudication procedure is a validity threat to the accuracy measure and should be addressed with a pre-specified rubric and independent scoring.
minor comments (6)
  1. [Section 3.1] The text contains a literal '(TODO LINK)' placeholder where the links to the supplemental notebooks should appear; the missing links should be supplied before any resubmission.
  2. [Front matter and bibliographic block] The ACM Reference Format block lists the venue as 'Woodstock '18' with a 2018 copyright date, and the Received/revised/accepted dates at the end include 'revised 12 March 2009'; these template leftovers are inconsistent with the 2025 arXiv submission and should be corrected.
  3. [Section 6 (Results)] The sentence 'SetMLVis consistently outperformed FiftyOne across all tasks' is contradicted by Table 4, where Task 5 is 100% correct for both systems; the text should say 'on Tasks 1-4', as the following paragraph itself makes clear.
  4. [Table 4] At n = 8, reporting only percentages (e.g., 0.0% vs. 37.5% for Task 1) hides the small counts (0/8 vs. 3/8); adding denominators would improve transparency.
  5. [References] The FiftyOne reference [30] is a GitHub link with no version identifier or access date, which is insufficient for reproducibility of the comparison condition.
  6. [Figure 3] The top panel reproduces a large table from Ding et al. (IEEE) with the publisher's copyright footer visible; the authors should confirm that this reproduction is permitted and properly annotated.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity; the paper's claims are empirical and not forced by definition, fit, or self-citation, though the reported statistics are invalid.

full rationale

The paper's central claim is empirical rather than derivational: it proposes a set-based formalization (Section 3), implements SetMLVis (Section 4), and evaluates it in a user study against FiftyOne (Section 5). No equation in the paper defines a predicted quantity in terms of fitted inputs, no parameter is fit and then renamed as a prediction, and no load-bearing premise is justified solely by the authors' own prior work. The UpSet-style visualization is adopted from Lex et al. [27] and the sensemaking framework from Cabrera et al. [9], both external citations. The headline comparison, 'using sets improves accuracy on model comparison tasks and reduces cognitive workload across all tasks' (Section 9), is a statistical claim about observed user performance, not a result that reduces to its own inputs by construction. The serious problems identified in the manuscript are validity and reporting problems, not circularity: Section 5.1 admits 'our initial (though incorrect) power analysis suggested that only five participants were needed'; Section 5 states 'we do not explicitly attribute any observed differences in performance directly to the use of set visualization'; Section 7.3 reports that 'nearly every participant used the query system' as the primary interaction, so the design cannot cleanly separate set visualization from the query system; and Section 6 reports a McNemar test with chi2(1, N = 35) = 0.0 and p = 0.004 and a Wilcoxon W = 1279.0 with n = 8, both arithmetically impossible. These defects undermine the evidential support for the conclusion, but they do not make the derivation circular. Accordingly, no circular step is exhibited, and the appropriate circularity score is 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

This paper is an empirical HCI study, so its central claim depends on the validity of the user-study design and statistical analysis rather than on fitted constants or derived equations. The main assumptions are the meaningfulness of IOU-based matching, the representativeness of the tasks, the validity of NASA-TLX, and the correctness of the statistical tests.

assumptions (4)
  • domain assumption Intersection over Union (IOU) with a chosen threshold is a valid set generation rule for matching object detections across models.
    Section 3.5 and Section 4.1 use IOU as the set generation rule; the paper assumes this captures meaningful agreement between model outputs without justifying the choice of threshold.
  • domain assumption The user study tasks are representative of real-world model comparison workflows and can be scored objectively.
    Section 5.2 Table 3 maps tasks to literature goals, but correctness is judged subjectively by the study administrators, and the tasks are simplified to six images with about 40 detections each.
  • domain assumption NASA-TLX provides a valid measure of cognitive workload for this interface comparison.
    Section 5.2 uses NASA-TLX as the workload instrument without questioning its applicability to short, interface-focused tasks in this within-subject setting.
  • domain assumption The McNemar and Wilcoxon signed-rank tests are correctly applied to the collected paired data.
    Section 5.3 describes these tests, but the reported statistics in Section 6 are internally inconsistent, so the correctness of their application is questionable.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Set Visualizations for Comparing and Evaluating Machine Learning Models." pith.science (2026). https://pith.science/paper/JJR2BGOU

@misc{pith2026250214675,
  author       = {Pith},
  title        = {Pith review of: Set Visualizations for Comparing and Evaluating Machine Learning Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JJR2BGOU}},
  note         = {Machine review of arXiv:2502.14675}
}
read the original abstract

Machine learning practitioners often need to compare multiple models to select the best one for their application. However, current methods of comparing models fall short because they rely on aggregate metrics that can be difficult to interpret or do not provide enough information to understand the differences between models. To better support the comparison of models, we propose set visualizations of model outputs to enable easier model-to-model comparison. We outline the requirements for using sets to compare machine learning models and demonstrate how this approach can be applied to various machine learning tasks. We also introduce SetMLVis, an interactive system that utilizes set visualizations to compare object detection models. Our evaluation shows that SetMLVis outperforms traditional visualization techniques in terms of task completion and reduces cognitive workload for users. Supplemental materials can be found at https://osf.io/afksu/?view_only=bb7f259426ad425f81d0518a38c597be.

Figures

Figures reproduced from arXiv: 2502.14675 by the authors.

Figure 1
Figure 1. When comparing object detection models, practitioners look at bounding box overlays on images (panel (a)). These visualizations [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. While all three models have the same average precision of 0.5, this metric hides key differences in their predictions. A closer [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 4
Figure 4. Illustration of set creation from model predictions. Top: The image is classified as a dog by both models, forming a set of model [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figures from the paper (8 more)
Figure 5
Figure 5. Figure 5: Example of using UpSet visualization to compare model predictions. (1) All three models make different predictions for the [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Transformation of data into sets based on cluster label agreement between K-Means and Birch algorithms. Clustering results [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: The figure shows how regression model predictions are converted into set-type data for house price estimation. In scenario [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]
Figure 9
Figure 9. Figure 9: Overview of the method used in SetMLVis for comparing object detection models. The process involves identifying overlapping [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]
Figure 10
Figure 10. Figure 10: This image demonstrates the SetMLVis interface using the Imagenette dataset [ [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]
Figure 11
Figure 11. Figure 11: The two interfaces compared in the study. FiftyOne is on the left and SetMLVis is on the right. [PITH_FULL_IMAGE:figures/full_fig_p018_11.png]
Figure 12
Figure 12. Figure 12: Comparison of NASA-TLX cognitive workload ratings between SetMLVis and FiftyOne across several dimensions: mental [PITH_FULL_IMAGE:figures/full_fig_p021_12.png]
Figure 13
Figure 13. Figure 13: Panel A shows two models with the same average precision (AP = 0.5) but different prediction overlaps, with Jaccard similarity [PITH_FULL_IMAGE:figures/full_fig_p025_13.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

56 extracted references · 16 canonical work pages

  1. [1]

    Shivam Agarwal and Fabian Beck. 2020. Set Streams: Visual exploration of dynamic overlapping sets. In Computer Graphics Forum, Vol. 39. Wiley Online Library, 383–391. https://doi.org/10.1111/cgf.13988

    work page doi:10.1111/cgf.13988 2020

  2. [2]

    Eric Alexander and Michael Gleicher. 2015. Task-driven comparison of topic models. IEEE transactions on visualization and computer graphics 22, 1 (2015), 320–329. https://doi.org/10.1109/TVCG.2015.2467618

    arXiv 2015

  3. [3]

    Basak Alper, Nathalie Riche, Gonzalo Ramos, and Mary Czerwinski. 2011. Design study of linesets, a novel set visualization technique. IEEE transactions on visualization and computer graphics 17, 12 (2011), 2259–2267. https://doi.org/10.1109/TVCG.2011.186

    doi:10.1109/tvcg.2011.186 2011

  4. [4]

    Bilal Alsallakh, Luana Micallef, Wolfgang Aigner, Helwig Hauser, Silvia Miksch, and Peter Rodgers. 2016. The state-of-the-art of set visualization. In Computer Graphics Forum, Vol. 35. Wiley Online Library, 234–260. https://doi.org/10.1111/cgf.12722

    doi:10.1111/cgf.12722 2016

  5. [5]

    Philippe Bardou, Jérôme Mariette, Frédéric Escudié, Christophe Djemiel, and Christophe Klopp. 2014. jvenn: an interactive Venn diagram viewer. BMC bioinformatics 15, 1 (2014), 1–7. https://doi.org/10.1186/1471-2105-15-293

    work page doi:10.1186/1471-2105-15-293 2014

  6. [6]

    Alex Bäuerle, Ángel Alexander Cabrera, Fred Hohman, Megan Maher, David Koski, Xavier Suau, Titus Barik, and Dominik Moritz. 2022. Symphony: Composing interactive interfaces for machine learning. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems . 1–14. https://doi.org/10.1145/3491102.3502102

    arXiv 2022

  7. [7]

    Ali Borji, Ming-Ming Cheng, Huaizu Jiang, and Jia Li. 2015. Salient object detection: A benchmark. IEEE transactions on image processing 24, 12 (2015), 5706–5722. https://doi.org/10.1109/TIP.2015.2487833

    work page arXiv 2015

  8. [8]

    Ángel Alexander Cabrera, Erica Fu, Donald Bertucci, Kenneth Holstein, Ameet Talwalkar, Jason I Hong, and Adam Perer. 2023. Zeno: An interactive framework for behavioral evaluation of machine learning. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems . 1–14. https://doi.org/10.1145/3544548.3581268

    arXiv 2023

Show all 56 references
  1. [9]

    Ángel Alexander Cabrera, Marco Tulio Ribeiro, Bongshin Lee, Robert Deline, Adam Perer, and Steven M Drucker. 2023. What did my AI learn? how data scientists make sense of model behavior. ACM Transactions on Computer-Human Interaction 30, 1 (2023), 1–27. https://doi.org/10.1145/3542921

    2023 doi

  2. [10]

    Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. 2020. End-to-end object detection with transformers. In European conference on computer vision . Springer, 213–229. https://doi.org/10.1007/978-3-030-58452-8_13

    2020 doi

  3. [11]

    Ángela Casado-García and Jónathan Heras. 2020. Ensemble methods for object detection. In ECAI 2020. IOS Press, 2688–2695. https://doi.org/10. 3233/FAIA200407

    2020

  4. [12]

    Jifeng Dai, Yi Li, Kaiming He, and Jian Sun. 2016. R-fcn: Object detection via region-based fully convolutional networks. Advances in neural information processing systems 29 (2016). https://doi.org/10.48550/arXiv.1605.06409

    2016 doi arXiv

  5. [13]

    Jesse Davis and Mark Goadrich. 2006. The relationship between Precision-Recall and ROC curves. In Proceedings of the 23rd international conference on Machine learning. 233–240. https://doi.org/10.1145/1143844.1143874 26 Set Visualizations for Comparing and Evaluating Machine L...

    2006

  6. [14]

    Thomas G Dietterich. 2000. Ensemble methods in machine learning. In International workshop on multiple classifier systems . Springer, 1–15. https://doi.org/10.1007/3-540-45014-9_1

    2000 doi

  7. [15]

    Jian Ding, Nan Xue, Gui-Song Xia, Xiang Bai, Wen Yang, Michael Ying Yang, Serge Belongie, Jiebo Luo, Mihai Datcu, Marcello Pelillo, et al. 2021. Object detection in aerial images: A large-scale benchmark and challenges. IEEE transactions on pattern analysis and machine intelli...

    2021

  8. [16]

    Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. 2010. The pascal visual object classes (voc) challenge. International journal of computer vision 88 (2010), 303–338. https://doi.org/10.1007/s11263-009-0275-4

    2010 doi

  9. [17]

    Jochen Görtler, Fred Hohman, Dominik Moritz, Kanit Wongsuphasawat, Donghao Ren, Rahul Nair, Marc Kirchner, and Kayur Patel. 2022. Neo: Generalizing confusion matrix visualization to hierarchical and multi-output labels. In Proceedings of the 2022 CHI Conference on Human Factor...

    2022

  10. [18]

    Sandra G Hart. 2006. NASA-task load index (NASA-TLX); 20 years later. In Proceedings of the human factors and ergonomics society annual meeting , Vol. 50. Sage publications Sage CA: Los Angeles, CA, 904–908. https://doi.org/10.1177/154193120605000909

    2006 doi

  11. [19]

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition . 770–778

    2016

  12. [20]

    Fred Hohman, Minsuk Kahng, Robert Pienta, and Duen Horng Chau. 2018. Visual analytics in deep learning: An interrogative survey for the next frontiers. IEEE transactions on visualization and computer graphics 25, 8 (2018), 2674–2693. https://doi.org/10.1109/TVCG.2018.2843369

    2018

  13. [21]

    Aspen Hopkins and Serena Booth. 2021. Machine learning practices outside big tech: How resource constraints challenge responsible development. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society . 134–145. https://doi.org/10.1145/3461702.3462527

    2021

  14. [22]

    Jeremy Howard and fast.ai. 2019. Imagenette: A smaller subset of 10 easily classified classes from Imagenet. https://github.com/fastai/imagenette. Accessed: 2024-05-31

    2019

  15. [23]

    Jonathan Huang, Vivek Rathod, Chen Sun, Menglong Zhu, Anoop Korattikara, Alireza Fathi, Ian Fischer, Zbigniew Wojna, Yang Song, Sergio Guadarrama, et al. 2017. Speed/accuracy trade-offs for modern convolutional object detectors. In Proceedings of the IEEE conference on compute...

    2017 doi

  16. [24]

    Paul Jaccard. 1901. Étude comparative de la distribution florale dans une portion des Alpes et des Jura. Bull Soc Vaudoise Sci Nat 37 (1901), 547–579

    1901

  17. [25]

    Minsuk Kahng, Dezhi Fang, and Duen Horng Chau. 2016. Visual exploration of machine learning results using data cube analysis. In Proceedings of the Workshop on Human-In-the-Loop Data Analytics . 1–6. https://doi.org/10.1145/2939502.2939503

    2016

  18. [26]

    Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems 25 (2012). https://doi.org/10.1145/3065386

    2012 doi

  19. [27]

    Alexander Lex, Nils Gehlenborg, Hendrik Strobelt, Romain Vuillemot, and Hanspeter Pfister. 2014. UpSet: visualization of intersecting sets. IEEE transactions on visualization and computer graphics 20, 12 (2014), 1983–1992. https://doi.org/10.1109/TVCG.2014.2346248

    2014

  20. [28]

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proce...

    2014

  21. [29]

    https://doi.org/10.1007/978-3-319-10602-1_48

    Springer, 740–755. https://doi.org/10.1007/978-3-319-10602-1_48

    doi

  22. [30]

    Quinn McNemar. 1947. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika 12, 2 (1947), 153–157

    1947

  23. [31]

    B. E. Moore and J. J. Corso. 2020. FiftyOne. GitHub. Note: https://github.com/voxel51/fiftyone (2020)

    2020

  24. [32]

    Steven Moore, Q Vera Liao, and Hariharan Subramonyam. 2023. fAIlureNotes: Supporting Designers in Understanding the Limits of AI Models for Computer Vision Tasks. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems . 1–19. https://doi.org/10.1145/35...

    2023 doi

  25. [33]

    Besmira Nushi, Ece Kamar, and Eric Horvitz. 2018. Towards accountable ai: Hybrid human-machine analyses for characterizing system failure. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing , Vol. 6. 126–135. https://doi.org/10.1609/hcomp.v6i1.13337

    2018 doi

  26. [34]

    Kemal Oksuz, Baris Can Cam, Emre Akbas, and Sinan Kalkan. 2018. Localization recall precision (LRP): A new performance metric for object detection. In Proceedings of the European conference on computer vision (ECCV) . 504–519. https://doi.org/10.48550/arXiv.1807.01696

    2018 doi arXiv

  27. [35]

    Rafael Padilla, Sergio L Netto, and Eduardo AB Da Silva. 2020. A survey on performance metrics for object-detection algorithms. In2020 international conference on systems, signals and image processing (IWSSIP) . IEEE, 237–242. https://doi.org/10.1109/IWSSIP48289.2020.9145130

    2020

  28. [36]

    Kayur Patel, Naomi Bancroft, Steven M Drucker, James Fogarty, Amy J Ko, and James Landay. 2010. Gestalt: integrated support for implementation and analysis in machine learning. In Proceedings of the 23nd annual ACM symposium on User interface software and technology . 37–46. h...

    2010

  29. [37]

    Kayur Patel, James Fogarty, James A Landay, and Beverly Harrison. 2008. Investigating statistical machine learning as a tool for software development. In Proceedings of the SIGCHI conference on human factors in computing systems . 667–676. https://doi.org/10.1145/1357054.1357160

    2008

  30. [38]

    Hanyang Peng and Shiqi Yu. 2021. A systematic IOU-related method: Beyond simplified regression for better localization. IEEE Transactions on Image Processing 30 (2021), 5032–5044. https://doi.org/10.1109/TIP.2021.3077144

    2021

  31. [39]

    Sebastian Raschka. 2018. Model evaluation, model selection, and algorithm selection in machine learning. arXiv preprint arXiv:1811.12808 (2018). https://doi.org/10.48550/arXiv.1811.12808

    2018 doi arXiv

  32. [40]

    Annika Reinke, Minu D Tizabi, Carole H Sudre, Matthias Eisenmann, Tim Rädsch, Michael Baumgartner, Laura Acion, Michela Antonelli, Tal Arbel, Spyridon Bakas, et al . 2021. Common limitations of image processing metrics: A picture story. arXiv preprint arXiv:2104.05642 (2021). ...

    2021 doi arXiv

  33. [41]

    Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems 28 (2015). https://doi.org/10.1109/TPAMI.2016.2577031

    2015

  34. [42]

    Dominik Sacha, Michael Sedlmair, Leishi Zhang, John Aldo Lee, Daniel Weiskopf, Stephen North, and Daniel Keim. 2016. Human-centered machine learning through interactive visualization. ESANN

    2016

  35. [43]

    Mark Sanderson and Justin Zobel. 2005. Information retrieval system evaluation: effort, sensitivity, and reliability. In Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval . 162–169. https://doi.org/10.1145/10...

    2005

  36. [44]

    Ben Shneiderman and Catherine Plaisant. 2010. Designing the user interface: strategies for effective human-computer interaction . Pearson Education India

    2010

  37. [45]

    Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014). https://doi.org/10.48550/arXiv.1409.1556

    2014 doi arXiv

  38. [46]

    Ashley Suh, Gabriel Appleby, Erik W Anderson, Luca Finelli, Remco Chang, and Dylan Cashman. 2022. Visualization Guidelines for Model Performance Communication Between Data Scientists and Subject Matter Experts. arXiv preprint arXiv:2205.05749 (2022). https://doi.org/10.48550/ ...

    2022 doi arXiv

  39. [47]

    Ashley Suh, Gabriel Appleby, Erik W Anderson, Luca Finelli, Remco Chang, and Dylan Cashman. 2023. Are Metrics Enough? Guidelines for Communicating and Visualizing Predictive Models to Subject Matter Experts. IEEE Transactions on Visualization and Computer Graphics (2023). http...

    2023

  40. [48]

    Dong Sun, Zezheng Feng, Yuanzhe Chen, Yong Wang, Jia Zeng, Mingxuan Yuan, Ting-Chuen Pong, and Huamin Qu. 2020. Dfseer: A visual analytics approach to facilitate model selection for demand forecasting. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Sys...

    2020

  41. [49]

    Mingxing Tan. 2019. Efficientnet: Rethinking model scaling for convolutional neural networks. arXiv preprint arXiv:1905.11946 (2019). https: //doi.org/10.48550/arXiv.1905.11946

    2019 doi arXiv

  42. [50]

    Amos Tversky. 1977. Features of similarity. Psychological review 84, 4 (1977), 327. https://doi.org/10.1037/0033-295X.84.4.327

    1977 doi

  43. [51]

    Carl Vondrick, Aditya Khosla, Tomasz Malisiewicz, and Antonio Torralba. 2013. Hoggles: Visualizing object detection features. In Proceedings of the IEEE International Conference on Computer Vision . 1–8. https://doi.org/10.1109/ICCV.2013.8

    2013 doi

  44. [52]

    Ž Vujović et al. 2021. Classification model evaluation metrics. International Journal of Advanced Computer Science and Applications 12, 6 (2021), 599–606

    2021

  45. [53]

    Junhong Wang, Yun Li, Zhaoyu Zhou, Chengshun Wang, Yijie Hou, Li Zhang, Xiangyang Xue, Michael Kamp, Xiaolong Zhang, and Siming Chen

  46. [54]

    Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. 2017. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition . 1492–1500. https://doi.org/10.1109/CVPR.2017.634 Receive...

    2017 doi

  47. [55]

    Frank Wilcoxon. 1992. Individual comparisons by ranking methods. In Breakthroughs in statistics: Methodology and distribution . Springer, 196–202. https://doi.org/10.1007/978-1-4612-4380-9_16

    1992 doi

  48. [2022]

    IEEE Transactions on Visualization and Computer Graphics (2022)

    When, where and how does it fail? A spatial-temporal visual analytics approach for interpretable object detection in autonomous driving. IEEE Transactions on Visualization and Computer Graphics (2022). https://doi.org/10.1109/TVCG.2022.3201101

    2022

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.