{"id":"6e040d30-5e0d-4583-a9b4-060bd03c92ac","arxiv_id":"2606.07503","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":5.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"DnD uses set operations on matched ground-truth detections to compare two object detectors more directly than separate mAP or TIDE scores.","lead":"The paper proposes Differences in Detection (DnD), a method that directly compares two object detection models by computing intersections, differences, and complements of the ground-truth objects each model detects. A smart generalist might read it to understand a practical tool for choosing between similar AI vision systems or focusing explainability efforts on meaningful examples.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"No significant objection identified","rationale":"The reader's weakest assumption concerns actionability relative to separate mAP/TIDE scores. The paper's strongest claim is narrower—only that the set view is more direct for identifying shared vs. individual matches—and this holds by construction once the matching step is fixed. Because the method introduces no new numerical claim or extrapolation, the actionability question is secondary and does not undermine the stated contribution.","tokens_in":1669,"tokens_out":327,"duration_ms":18647,"concrete_test":"Take the matching output (true-positive assignments) from any two detectors on a 100-image subset; recompute the three DnD sets (intersection, model-A-only, model-B-only, both-miss) and confirm they partition the ground-truth set without overlap or omission; if the partition fails for >5% of images the construction is incomplete.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that DnD yields a more direct comparison by computing the intersection of matched ground-truth labels recognized by both models, the corresponding difference sets, and the complement of labels missed by both, using the same matching algorithm. This is a direct set-algebraic reorganization of standard matching outputs (e.g., IoU-based assignment) and is explicitly positioned as complementary to mAP and TIDE rather than a replacement metric. No internal inconsistency, hidden assumption about boundedness or uniqueness of matches, or unsubstantiated formal step is visible in the claim; the extension to error-type confusion matrices follows naturally from existing TIDE-style categorization.","agreement_with_reader":"disagree"},"referee_report":{"model":"grok-4.3","summary":"The paper proposes Differences in Detection (DnD), a method to compare two object detection models by applying the same matching algorithm and then computing the intersection of ground-truth labels detected by both, the symmetric difference sets for each model, and the complement of labels missed by both. This set-algebraic view is positioned as complementary to mAP and TIDE, with natural extension to error-type confusion matrices, and is suggested as a way to select metric-relevant examples for explainability techniques such as ODAM.","tokens_in":1805,"tokens_out":513,"duration_ms":19912,"significance":"If the claimed intuitiveness and actionability hold, DnD would supply a lightweight, structured reorganization of existing matching outputs that could help practitioners isolate shared versus model-specific errors without replacing summary metrics. The approach is parameter-free and directly leverages standard IoU-based assignment, which is a strength, but its practical utility remains untested.","major_comments":[{"comment":"Abstract: the central claim that the resulting comparison 'is more direct and intuitive than a comparison of independent summary statistics' and 'reveals individual and shared mistakes' is asserted without any quantitative evaluation, user study, ablation, or concrete example showing that DnD changes conclusions drawn from mAP/TIDE; this is load-bearing for the paper's positioning as a useful complement.","section":"Abstract"},{"comment":"Method description (inferred from abstract and § on error types): while the set operations are a direct reorganization of standard matching outputs, the manuscript supplies no demonstration that the intersection/difference sets yield more actionable insights for guiding ODAM or other explainability methods than simply inspecting per-model TIDE error breakdowns.","section":"Method / Error-type analysis"}],"minor_comments":[{"comment":"The GitHub link is provided, supporting reproducibility; however, the manuscript does not indicate whether the released code includes the exact matching routine used in the reported comparisons.","section":null},{"comment":"Notation for the intersection, difference, and complement sets is described procedurally but not formalized with equations or pseudocode, which would improve clarity.","section":null}],"recommendation":"major_revision","confidential_remarks":"The manuscript reads as a short methods note whose primary contribution is a presentational reorganization rather than new algorithmic machinery or validated empirical gains; this may affect fit for journals that expect substantial experimental validation in computer vision submissions."},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the detailed and constructive report. The two major comments correctly identify that the manuscript's positioning relies on claims of intuitiveness and actionability that are not yet supported by concrete examples or demonstrations. We address each point below and will revise the manuscript to incorporate the requested evidence.","responses":[{"response":"We agree that the abstract's claims would be strengthened by at least one concrete example. The current manuscript presents DnD as a direct reorganization of standard matching outputs but does not show an instance where the intersection/symmetric-difference sets lead to different practitioner conclusions than separate mAP and TIDE scores. We will add such an example (e.g., two detectors with similar mAP but visibly different shared vs. model-specific errors) in a revised experiments or applications section and will adjust the abstract wording to reflect the addition.","revision_made":"yes","referee_comment":"[Abstract] Abstract: the central claim that the resulting comparison 'is more direct and intuitive than a comparison of independent summary statistics' and 'reveals individual and shared mistakes' is asserted without any quantitative evaluation, user study, ablation, or concrete example showing that DnD changes conclusions drawn from mAP/TIDE; this is load-bearing for the paper's positioning as a useful complement."},{"response":"We accept this observation. Although the set-algebraic view is parameter-free and directly reuses IoU matching, the manuscript contains no side-by-side illustration showing that the DnD subsets produce more targeted example selection for ODAM than TIDE error-type breakdowns alone. We will add a short illustrative case study in the revision that applies both approaches to the same pair of detectors and highlights any differences in the resulting explainability focus.","revision_made":"yes","referee_comment":"[Method / Error-type analysis] Method description (inferred from abstract and § on error types): while the set operations are a direct reorganization of standard matching outputs, the manuscript supplies no demonstration that the intersection/difference sets yield more actionable insights for guiding ODAM or other explainability methods than simply inspecting per-model TIDE error breakdowns."}],"tokens_in":1316,"tokens_out":458,"duration_ms":16680,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main point is that this paper defines Differences in Detection as the intersection of ground truths both models get right, the symmetric differences, and the complement of those both miss, all using the same matching step as mAP. It then folds error types into a confusion matrix and suggests feeding the resulting subsets into explainability methods like ODAM.\n\nWhat it does cleanly is make the comparison explicit and instance-level rather than forcing the reader to mentally align two separate mAP or TIDE reports. The procedural description is simple, the GitHub link is there, and the extension to error-type matrices follows naturally from TIDE-style categorization. That part is useful for anyone who already runs pairwise model checks.\n\nThe soft spots are proportionate. The abstract supplies no numbers, no ablation, and no side-by-side example showing that the sets actually change a decision that mAP plus TIDE would not. Because the method is just set algebra on already-computed matches, the claim of greater intuitiveness remains an assertion until someone measures it. Edge cases such as ambiguous IoU thresholds or multiple overlapping boxes are not discussed in the visible text.\n\nThis is for practitioners who routinely compare two detectors and want a structured way to pick examples for further inspection. It will not move the core metrics literature. A serious referee should see the full experiments and check whether the sets produce different conclusions in practice; the idea is coherent enough to merit that step.","headline":"DnD is a straightforward set-difference procedure on top of standard matching that gives a direct view of shared and unique detections between two models.","tokens_in":2266,"tokens_out":360,"would_cite":false,"duration_ms":17887,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"DnD compares two object detectors by taking set intersections and differences on their shared ground-truth matches.","keywords":["object detection","model comparison","explainability","mean average precision","error analysis","confusion matrix","ground truth matching","TIDE"],"falsifier":"A controlled user study in which practitioners using only DnD outputs select or debug models faster or more accurately than practitioners given only mAP and TIDE scores.","tokens_in":2578,"feed_emoji":"📊","tokens_out":618,"duration_ms":15877,"temperature":0.7,"pith_summary":"The paper introduces Differences in Detection (DnD) to compare two object detection models more directly than separate summary scores allow. Starting from the same matching algorithm used for standard metrics, it identifies the ground-truth instances both models detect correctly, the instances only one model detects, and the instances both models miss. This produces explicit shared and unique success and failure sets instead of independent mAP or TIDE numbers. When the same error typing is applied, the differences appear inside a confusion matrix that isolates where the models diverge in their mistakes. The resulting subsets can also steer explainability methods toward examples that actually affect the reported performance numbers.","feed_headline":"DnD compares detectors by shared and unique ground-truth matches","feed_subtitle":"Set differences on matched labels show individual and shared errors directly, guiding explainability to metric-relevant cases.","key_machinery":"Differences in Detection (DnD), which applies set intersection, difference, and complement operations to the ground-truth matches obtained from the same matching algorithm run on each model.","core_discovery":"DnD calculates the intersection of ground truth labels recognized by both models, followed by the corresponding difference sets and the complement set of ground truth labels missed by both models, providing a more direct and intuitive comparison than independent summary statistics.","pith_inferences":["The same set-construction logic could be applied to more than two models by computing multi-way intersections and symmetric differences.","The resulting subsets could serve as a natural training or validation split when the goal is to improve one model relative to another.","Because the method re-uses the existing matching step, it adds negligible computational cost while changing only the post-processing view of the results."],"forward_implications":["Reveals which detections are shared successes, unique to one model, or shared failures.","Allows detection errors to be analyzed in a standard confusion matrix once error types are attached.","Supplies structured subsets of examples that explainability methods can target because they affect the metrics.","Complements rather than replaces mAP and TIDE by adding instance-level overlap information."],"fun_headline_variants":["DnD splits ground truth matches into shared and model-unique sets","Intersection of detections highlights common and distinct model errors","DnD uses difference sets on matched labels for intuitive comparison","Error analysis via ground truth complement and difference sets in DnD"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"That performing set differences on the matched ground-truth instances produces insights that are meaningfully more actionable or less misleading than comparing separate mAP and TIDE scores.","fun_headline_variants_meta":{"raw":{"variants":["DnD splits ground truth matches into shared and model-unique sets","Intersection of detections highlights common and distinct model errors","DnD uses difference sets on matched labels for intuitive comparison","Error analysis via ground truth complement and difference sets in DnD"]},"model":"grok-4.3","cost_usd":0.004641,"raw_usage":{"total_tokens":2258,"prompt_tokens":588,"num_sources_used":0,"completion_tokens":65,"cost_in_usd_ticks":46412000,"prompt_tokens_details":{"text_tokens":588,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1605,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":588,"tokens_out":65,"duration_ms":14841,"temperature":1.0,"reasoning_tokens":1605,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-27T22:15:07.531883+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A controlled user study in which practitioners using only DnD outputs select or debug models faster or more accurately than practitioners given only mAP and TIDE scores.","supporting_citations":[],"review_version":1}