{"id":"ccc6e82f-527f-40cd-92cd-c3bcbdcd2bdd","arxiv_id":"2512.24592","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A generate-and-verify pipeline using LLMs and grounded VLMs discovers fine-grained error slices in detection/segmentation models, with a new FeSD benchmark reporting Precision@10 of 0.73 versus 0.31 for adapted baselines.","lead":"This paper presents SliceLens, a hypothesis-driven system that finds systematic failure patterns (\"error slices\") in object detection and segmentation models by generating visual-language hypotheses and verifying them with a vision-language model. It also introduces a new expert-annotated benchmark, FeSD, and reports large precision gains over prior slice-discovery methods, plus model-repair experiments.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"FeSD Precision@10 uses per-GT 'Best Predicted Slice' selection and the abstract's 0.63 baseline contradicts Table 2's 0.31, so the 0.42 SOTA gain is not yet established.","rationale":"The reader's weakest assumption is that the VLM cannot reliably answer grounded relational questions, so P(yes|q,r) from Eq. (1) fails to rank error regions. That is a legitimate methodological risk, but it is not the most load-bearing issue for the central empirical claim. If the VLM were unreliable, the FeSD results would indeed collapse; however, the paper's high Precision@10 on FeSD is itself indirect evidence that the VLM works, provided the benchmark and matching are trustworthy. The more fundamental problem is that the FeSD evaluation is not yet interpretable. Table 7's 'Best Predicted Slice' column is a red flag: averaging P@10 over 21 per-GT 'best' slices without specifying how 'best' is chosen can overstate performance, especially when the method generates 102 hypotheses. A fair comparison between methods requires the same selection rule for baselines, and the paper does not provide it. Additionally, the abstract and body disagree on the baseline precision (0.63 vs. 0.31/0.28), so even the magnitude of the claimed improvement is ambiguous. These issues are internal to the reported evidence and do not depend on assumptions about VLM capabilities. The reader's conditional verdict already asks for code, benchmark, matching protocol, and error bars; my concern strengthens that request but identifies the matching protocol and the abstract/body discrepancy as the single decisive point. A concrete recomputation with a fixed, non-oracle matching rule would settle whether the 0.42 gain is real or an artifact. Until then, the central SOTA claim is unverified, so I would mark the paper unverdict rather than conditionally acceptable.","tokens_in":19197,"tokens_out":8375,"duration_ms":91930,"concrete_test":"Release the FeSD annotations and the exact matching/reference script, then recompute Table 2 under a fixed protocol: for each method, apply the same semantic-relevance mapping from predicted slices to GT slices and use the top-ranked predicted slice per GT (or average over all predicted slices), never selecting a slice by its observed P@10. Also recompute the headline improvement against the abstract's 0.63 baseline rather than Table 2's 0.31. If, under non-oracle matching against the stronger baseline, SliceLens's advantage over HiBug* and FACTS* drops below 0.10, the claimed 0.42 gain is an artifact of the scoring protocol.","verdict_should_be":"UNVERDICTED","load_bearing_attack":"The central claim rests on an evaluation protocol that the paper does not specify. Table 7 reports, for each FeSD ground-truth slice, a 'Best Predicted Slice' and its Precision@10, and Table 2 averages these per-GT best values to 0.73. Equation (2) defines Precision@k for a single predicted slice, but the paper never states how the 'best' slice is selected: is it the hypothesis with the highest P@10 against that GT (an oracle/max-over-hypotheses choice), or the hypothesis picked by the LLM semantic-relevance evaluator from Section 5.3.1? If it is the former, the reported 0.73 is an upper bound over SliceLens's 102 hypotheses, not the precision a deployed discovery pipeline would achieve, and methods with more hypotheses are favored purely by selection. The comparison to HiBug* (0.31) and FACTS* (0.28) is valid only if the same best-slice selection is applied to the baselines, which is not documented. This is compounded by an internal contradiction: the abstract reports an improvement of 0.10 (0.73 vs. 0.63) on GESD, while Section 5.2 and Table 2 report 0.73 vs. 0.31/0.28, an improvement of 0.42. These are incompatible versions of the headline result. Because FeSD and the code are not released, no reader can determine which baseline is correct or whether the matching protocol is fair. The VLM-grounding worry identified by the reader is real, but it is downstream: even a perfect VLM would not rescue a SOTA claim that rests on an unspecified, possibly oracle-based scoring rule.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript (whose full text uses the name SliceLens, while the preceding abstract uses GH-ESD/GESD) proposes a hypothesis-driven error-slice discovery method for instance-level vision tasks. SliceLens generates failure hypotheses from an LLM in two ways — knowledge-driven, from task context, and data-driven, from VLM-generated grounded captions — and then verifies them by using a VLM to compute P(yes | q, r) for each error region, followed by a slope-based trend analysis that links slice confidence to model error. The paper also introduces FeSD, a new expert-annotated benchmark of fine-grained detection/segmentation error slices, and reports substantial Precision@10 gains over adapted FACTS and HiBug baselines, as well as model-repair experiments in both classification and object detection.","tokens_in":19645,"tokens_out":5174,"duration_ms":56185,"significance":"If the reported results are reliable, the paper would make a meaningful contribution: instance-level, relational error slices such as “bicycle obscured by a person” are not well captured by image-level tag or cluster methods, and the proposed generate-and-verify pipeline is well motivated. The paper is also commendable for attempting a new benchmark targeting detection and segmentation, for reporting detailed per-slice tables in the appendix, and for including repair experiments that test actionability. However, the central FeSD comparison is currently not reproducible because the benchmark is not released and, more importantly, the protocol for selecting the “Best Predicted Slice” in Table 7 is unspecified. Since the headline 0.42 gain rests on this protocol, the significance cannot be assessed until the evaluation is fully specified.","major_comments":[{"comment":"The FeSD Precision@10 comparison is not well-defined. Table 7 reports, for each ground-truth slice, a “Best Predicted Slice” and its P@10, and Table 2 averages these values to 0.73. The text never states how the “best” slice is chosen. If it is the hypothesis with maximum P@10 among SliceLens’s 102 hypotheses, then 0.73 is an oracle upper bound, not the precision of a deployed pipeline, and methods with more hypotheses are favored by selection alone. If it is chosen by the Gemini semantic-relevance evaluator of §5.3.1, then the reported metric is not the P@10 of Eq. (2) but a semantic-matching variant. The comparison to FACTS* and HiBug* is valid only if the same selection rule is applied to those baselines; this is not documented. Since FeSD is not released, a reader cannot verify the protocol. This issue is load-bearing for the central claim and must be fixed by specifying and applying","section":"§5.2, Table 7, Eq. (2)"},{"comment":"The headline result is internally inconsistent. The abstract states “improving Precision@10 by 0.10 (0.73 vs. 0.63) on the GESD benchmark,” while the full text, §5.2 and Table 2, report 0.73 vs. 0.31/0.28, an improvement of 0.42 above HiBug*. These are incompatible versions of the same experiment. A journal submission must have one unambiguous headline number; the discrepancy also makes it unclear whether the authors are comparing against a 0.63 baseline (perhaps an earlier version of their own benchmark) or the open-source baselines. This needs correction in the next version.","section":"Abstract and §5.2 / Table 2"},{"comment":"The semantic-relevance evaluation appears circular and is insufficiently validated. The evaluator that judges whether a discovered slice matches a ground-truth slice is Gemini-2.5-Pro, the same LLM family used to generate the hypotheses. The prompt permits loose matches (e.g., “bicycle partially occluded by a person” is listed as the best match for the GT “Bicycles seen from the front/back that being ridden/pushed by people”), and no human agreement is reported. If this evaluator is also used to select the “Best Predicted Slice” in Table 7, the reported P@10 is not model-error precision but the output of a semantic judge. Please report inter-annotator agreement with human experts and make the matching criterion explicit and identical for all compared methods.","section":"§5.3.1, Appendix A.3"},{"comment":"The pipeline relies on the assumption that a VLM can answer grounded yes/no questions about complex relational hypotheses from an error-region bounding box or point. This is not independently validated on the FeSD setting. The VLM ablation in Appendix B.1 is on NICO++ image classification, which uses global image-level decisions and simpler queries; it does not measure whether Qwen2.5-VL-7B can reliably judge descriptions like “bicycle obscured by a person” from a box. Since P(yes|q,r) is the ranking signal that feeds Eq. (1) and all downstream slice identification, the paper should report a small human-labeled validation set for FeSD-like grounded queries, or otherwise quantify VLM grounding accuracy.","section":"§3.2.1, Appendix B.1"}],"minor_comments":[{"comment":"The manuscript contains inconsistent naming: the top abstract uses GH-ESD and GESD, while the full text uses SliceLens and FeSD. The title also differs between the two versions. The authors should unify terminology and ensure the abstract matches the body.","section":"Throughout"},{"comment":"Overwriting all non-slice model predictions with ground truth makes the benchmark only contain the predefined error modes, potentially making slice discovery artificially clean. Please discuss why this cleaning does not inflate the reported precision relative to a real deployment setting with many unrelated errors.","section":"§4.1, Step 4"},{"comment":"The paper uses “slice confidence” for P(yes|q,r) but later slopes are computed between confidence and error rate. The exact ranking and thresholding procedure for retrieving the top k instances in a candidate slice should be stated in a single place; currently it is spread across §3.2.1, §3.2.2, and Eq. (2).","section":"Eq. (1), §3.2.2"},{"comment":"Typographical inconsistency: “GroupDRO + Slicelens” capitalizes different parts of the method name. Also, the caption could state more clearly what “Baseline model repair” means and how it differs from “No model repair.”","section":"Table 4"},{"comment":"The HiBug* baseline implementation details only list the attribute corpus; the actual tag-generation, tag-combination, and slice-selection procedure used to produce the 0.31 P@10 number is not described. Please provide the full protocol, ideally with code, so the comparison is reproducible.","section":"Appendix B.2"}],"recommendation":"major_revision","confidential_remarks":"The paper has a promising idea and extensive experiments, but the FeSD evaluation is currently not reproducible and the abstract/body discrepancy weakens confidence in the headline number. The authors should be asked to release the benchmark and code, precisely define the best-slice selection rule, and rerun the baselines under that rule. If the evaluation remains unspecified or the benchmark is not released, I would not be able to recommend acceptance. The circularity of the Gemini-based semantic match is also a concern, but it is addressable by reporting human agreement and applying the same judge to all methods."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short take: the method is worth a look, but the paper as written does not establish the SOTA claim on FeSD.\n\nWhat's new: SliceLens formulates slice discovery as LLM-generated hypotheses plus VLM-grounded verification, which is a genuine departure from cluster-then-explain and tag-then-slice. The FeSD benchmark, with 42 expert-annotated slices for detection and segmentation, is ambitious and potentially useful if released. The slope trend analysis is a nice idea - it avoids brittle error-rate thresholds, and the ablation shows it's more robust.\n\nThe soft spots are in the evaluation. First, the headline number is inconsistent: the abstract says 0.73 vs 0.63 (improvement 0.10) on GESD, but Section 5.2 and Table 2 say 0.73 vs 0.31 (improvement 0.42) on FeSD. Those are different claims, and the reader can't tell which is the intended result. That's a serious reporting error, not a nitpick.\n\nSecond, and more important, FeSD results in Table 7 are reported as 'Best Predicted Slice' for each ground-truth slice. The paper never specifies how that best slice is chosen. If the authors are selecting the predicted hypothesis with the highest Precision@10 using knowledge of the GT (an oracle over 102 hypotheses), then 0.73 is an upper bound, not the expected precision of a deployed discovery pipeline. The comparison to HiBug* (0.31) and FACTS* (0.28) is only fair if the identical selection rule is applied to those baselines; that's not documented. The stress-test note is right: this is load-bearing.\n\nThird, no error bars on FeSD, and no code or benchmark release, so external verification is impossible right now. The VLM grounding concern is real but downstream; even a perfect Qwen2.5-VL wouldn't fix the evaluation-protocol ambiguity.\n\nCredit where due: the classification results on NICO++ are modest and plausible (0.64-0.69 vs 0.60-0.67 for ViG-FACTS), the model-repair experiment for bicycle detection is a good demonstration of actionability, and the paper's discussion of prior art is fair. The fact that the same Gemini model is used to generate hypotheses and judge semantic relevance in the ablation is a circularity, but the main precision numbers don't depend on that judge - they depend on the slice matching protocol.\n\nBottom line: this is a paper for the error-slice-discovery community. It deserves a serious referee, but the referee should demand a reconciled headline, a precise description of the matching protocol, error bars, and a commitment to release FeSD and code. As it stands, I would not accept it; I'd condition acceptance on those revisions. I'd be happy to engage with the revised version.","headline":"A promising generate-and-verify framework for instance-level slice discovery, but the FeSD headline number is internally inconsistent and rests on an unspecified 'best slice' selection, so the SOTA claim needs scrutiny before being trusted.","tokens_in":20128,"tokens_out":3813,"would_cite":false,"duration_ms":38640,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SliceLens reformulates error slice discovery as grounded hypothesis generation and VLM verification, and claims it lifts detection Precision@10 from 0.31 to 0.73 on the new FeSD benchmark.","keywords":["error slice discovery","instance-level vision","object detection","instance segmentation","vision-language model","hypothesis generation","grounded verification","model repair"],"falsifier":"Take the FeSD detection slices and for each error region have human annotators label whether the region satisfies the corresponding relational hypothesis; then ask Qwen2.5-VL-7B the same box-prompted question. If human-VLM agreement is at or near chance on relational hypotheses (e.g., occlusion, part-whole), while agreement on simple attributes is high, then the slice-confidence ranking in Eq. (1) is not evidence about the hypothesis and the reported P@10 improvements would not transfer. Alternatively, run SliceLens with a VLM that always answers 'yes': if Precision@10 stays high, the groundin","tokens_in":19083,"feed_emoji":"🔍","tokens_out":6500,"duration_ms":59272,"temperature":0.7,"pith_summary":"The paper sets out to find the subsets of examples on which a vision model systematically fails — error slices — for instance-level tasks like detection and segmentation, not just image classification. It claims these failures are often relational and spatially local (e.g., 'bicycles partially occluded by a person'), so the right unit is the instance, not the image. To capture them, SliceLens generates hypotheses from an LLM (world knowledge plus data-driven attribute inference), retrieves candidate error regions by asking a VLM yes/no questions grounded by boxes or points, and verifies a slice by checking that error rate rises with slice confidence. On the paper's new FeSD benchmark, it reports Precision@10 of 0.73 for detection (vs. 0.31 for the adapted HiBug baseline) and 0.805 for segmentation. If correct, this turns slice discovery from clustering into a testable hypothesis-verification loop and makes discovered slices directly usable for model repair.","feed_headline":"SliceLens finds error slices at 0.73 precision, up from 0.31","feed_subtitle":"A generate-and-verify pipeline uses vision-language models to spot relational failure modes in detection and segmentation, then repairs them","key_machinery":"The load-bearing object is the grounded VLM verification head: the score P(yes|q,r) = softmax over the 'yes' and 'no' logits of a prompt that names the hypothesis q and localizes the error region r with a bounding box (detection) or point (segmentation). Around it sits the hypothesis generator (knowledge-driven LLM prompts plus a data-driven path: grounded attribute-aware captions → LLM attribute inference → LLM refinement into natural-language query phrases) and the slope-trend verifier, which replaces thresholding error rates with a sliding-window local linear fit between slice confidence and error rate, keeping slices whose maximum positive slope is large. Together they let SliceLens sear","core_discovery":"SliceLens claims that fine-grained instance-level error slices can be discovered by a generate-and-verify loop: an LLM proposes natural-language failure hypotheses; a VLM scores each error region's match to the hypothesis using box or point grounding and the softmax of 'yes'/'no' logits (Eq. 1); and a slope-trend analysis — not a fixed error-rate threshold — decides whether the retrieved slice is a genuine systematic error by requiring error likelihood to rise as slice confidence rises. On FeSD, a benchmark of 42 expert-annotated slices over COCO, KITTI, and public face imagery, the method reports average Precision@10 of 0.729 across 21 detection slices (21/21 matched) and 0.805 across 21 se","pith_inferences":["If VLM grounding is trustworthy, the same hypothesis-generate-verify loop could transfer to other fine-grained instance tasks (pose, tracking, video) and to open-vocabulary failure-mode search, since the query language is unrestricted natural language.","The slope-trend verifier is a domain-general criterion for 'slice-ness' that could sit on top of any retriever that produces a ranking — a clean way to test whether a ranking corresponds to a real error mode, independent of VLM scores.","The FeSD benchmark's 42 slices are only a sample of failure modes; high Precision@10 mostly reflects retrieval of known, annotated slices, and a full evaluation of discovery completeness would need a held-out 'unknown slice' protocol, which the paper leaves to future work."],"forward_implications":["Instance-level grounding is what unlocks detection/segmentation slices: image-level analysis dilutes failures (P@10 0.60 vs 0.80; slope 0.84 vs 1.32 on the partial-face slice).","Language-based hypotheses are more compact and semantically aligned than tag combinations: 102 hypotheses reach 0.90 recall / 0.74 precision, versus 38,639 tag combinations at 0.57/0.54.","The slope-trend test yields higher F1 (3.7% average improvement) and robustness to threshold choice than error-rate thresholding.","Discovered slices support targeted repair: instance-level GroupDRO on the 'bicycle occluded by person' slice improves bicycle mAP from 27.2 to 33.59 and mAR from 39.33 to 47.09.","On classification benchmarks the same pipeline matches or beats prior slice-discovery methods, reaching 1.0 Precision@10 on Waterbirds and CelebA and 0.64/0.66/0.69 on NICO++ across correlation strengths."],"fun_headline_variants":["GH-ESD finds instance-level error slices at 0.73 precision","Generate-and-verify loop uncovers relational failure modes","LLM-VLM pipeline discovers grounded error slices for vision","Beat baselines by 0.10 P@10 on detection with GH-ESD","Hypothesis-driven slice discovery for detection and segmentation"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The whole pipeline rests on VLM reliability for grounded relational questions: a 7B VLM, prompted with only a box or point, must correctly answer 'yes' or 'no' to complex queries such as 'bicycle obscured by a person'; the paper does not independently validate this grounding accuracy.","fun_headline_variants_meta":{"raw":{"variants":["GH-ESD finds instance-level error slices at 0.73 precision","Generate-and-verify loop uncovers relational failure modes","LLM-VLM pipeline discovers grounded error slices for vision","Beat baselines by 0.10 P@10 on detection with GH-ESD","Hypothesis-driven slice discovery for detection and segmentation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000295,"raw_usage":{"total_tokens":1574,"prompt_tokens":788,"completion_tokens":786,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":532,"completion_tokens_details":{"reasoning_tokens":697}},"tokens_in":532,"tokens_out":786,"duration_ms":8125,"temperature":1.0,"reasoning_tokens":697,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T13:17:22.348523+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the FeSD detection slices and for each error region have human annotators label whether the region satisfies the corresponding relational hypothesis; then ask Qwen2.5-VL-7B the same box-prompted question. If human-VLM agreement is at or near chance on relational hypotheses (e.g., occlusion, part-whole), while agreement on simple attributes is high, then the slice-confidence ranking in Eq. (1) is not evidence about the hypothesis and the reported P@10 improvements would not transfer. Alternatively, run SliceLens with a VLM that always answers 'yes': if Precision@10 stays high, the groundin","supporting_citations":[],"review_version":1}