{"id":"af5dae42-c6c0-4866-9f20-a78018d2b25a","arxiv_id":"2608.09122","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"VIGIL-8B, an LMM-based detector with synchronized multi-layer decoding, achieves state-of-the-art localization of synthetic and authentic image distortions in UGC photos.","lead":"VIGIL is a new image-quality model that finds and localizes distortions like blur and noise in everyday photos, trained on a large synthetic dataset and tested on real user content. It outperforms previous models on locating authentic distortions, suggesting a practical path for automated quality analysis and editing.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The S2A claim rests on an author-annotated 815-image test set with no inter-annotator agreement and no stated blinding to model outputs; if those labels are biased, the 0.12 mAP margin over Grounding-DINO is unsupported.","rationale":"I read the paper in good faith. The strongest claim is precisely the S2A result, and the load-bearing condition is that the authentic test set provides a valid, unbiased measure of distortion localization. The paper's own 'Validation / Test Set Construction' describes an iterative correction by five annotators but gives no agreement statistics, and no information about whether any annotator saw VIGIL outputs. The supplementary 'S2A Test Set' justification says the small size is due to annotation cost and strict filtering, but does not address independent validation. Since the central novelty is synthetic-to-authentic generalization, the test set is the linchpin. The rest of the paper is well structured: the multi-detector ablation (Tables 5, 7, 8) supports the architecture choices, the S2A-trick ablation (Table 6) shows a consistent gain, and the comparison to text-generation SFT (Table 4) is informative. The strongest claim is therefore conditionally supported, not unconditionally established. I also noted that the paper's mAP uses IoU thresholds starting at 0.05 (Supp. Metrics), which is non-standard and may inflate absolute numbers; however, this affects all models equally, so the relative claim is less threatened than the test-set reliability issue. An independent re-annotation study would settle whether the headline margin is real.","tokens_in":22261,"tokens_out":17852,"duration_ms":183831,"concrete_test":"Randomly sample 200 of the 815 test images. Have two independent annotators, not affiliated with the authors and blind to all model outputs, annotate distortions using the same 8-category protocol and the same 1-3 boxes/image rule. Measure inter-annotator agreement (box-level F1 at IoU>=0.5 and category-level Cohen's kappa). Then recompute VIGIL-8B and Grounding-DINO AP against each independent annotation set and against the original labels. If the VIGIL-minus-Grounding-DINO mAP margin drops below about 0.05, or if inter-annotator agreement is low, the S2A claim is not robust.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that VIGIL-8B, trained only on synthetic distortions, localizes authentic distortions in UGC images substantially better than strong baselines (Table 3: mAP 0.482 vs 0.362 for Grounding-DINO). The sole evidence is the AP computed on the 815-image authentic test set. That test set was constructed by the authors via an iterative annotation-correction procedure (Section 'Validation / Test Set Construction') with no inter-annotator agreement statistics, no independent re-annotation, and no statement that annotators were blind to the model's predictions. If the annotation protocol systematically favors distortions that resemble the synthetic training types, or if the authors' knowledge of VIGIL's outputs influenced which regions survived the final 'deletions-only' pass, then the ground truth is not an independent measure of authentic distortion localization. In that case the reported OOD margin could overstate the true S2A generalization, which is the paper's primary contribution.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes VIGIL, an LMM-based visual distortion detector trained entirely on a new synthetic dataset (VIGIL-140K, 145K images / 205K labeled regions). The model augments an InternVL-3-8B backbone with N area-query tokens and attaches independent classification/regression heads to M selected LLM decoder layers, treating each layer as a detector with its own Hungarian matching. At inference, the model keeps boxes that are classified as non-distortion and re-uses the second-highest class probability as a distortion candidate, followed by class-wise NMS and probability filtering. Experiments compare VIGIL against general LMMs (InternVL, Qwen3-VL, LLaVA-OV, GPT, Gemini) and object detectors (Faster R-CNN, DETR variants, YOLO-V11, Grounding-DINO) on two tasks: an in-domain synthetic distortion detection test set and an out-of-domain authentic distortion localization test set of 815 human-annotated COCO images. The paper reports state-of-the-art results on both, with the headline S2A claim being an mAP of 0.482 for VIGIL-8B versus 0.362 for the strongest baseline (Grounding-DINO) on the authentic test set.","tokens_in":22482,"tokens_out":6741,"duration_ms":68485,"significance":"If the S2A result is trustworthy, the paper makes a valuable empirical contribution: it shows that a model trained only on synthetic distortions can localize authentic distortions in user-generated content substantially better than current text-based LMMs and standard object detectors. The paper's strengths include a carefully constructed synthetic training set with quality filtering and distortion combination, a clean multi-detector design with ablations, a thorough comparison against both LMM and detection baselines, and useful supplementary analyses (parameter-freezing strategies, area-number generalization, threshold selection, and compatibility with text-based instruction tuning). The main risk is that the central S2A claim rests entirely on an author-constructed test set whose annotation reliability has not been demonstrated.","major_comments":[{"comment":"The central S2A claim (Table 3: mAP 0.482 vs. 0.362 for Grounding-DINO) is evaluated solely on the 815-image authentic test set. The described iterative annotation-correction procedure involves a leader annotator, four reviewers, and a final pass in which only deletions are allowed, but the paper provides no inter-annotator agreement statistics, no independent re-annotation, and no statement about whether annotators were blind to VIGIL's predictions. Because the final pass can only remove regions, the authors' prior knowledge of the model's likely outputs could systematically delete regions the model does not detect, inflating the reported AP. The authors should report pairwise agreement (e.g., IoU or kappa on a subset), conduct an independent re-annotation by outside annotators, and explicitly state whether model outputs were available during any annotation stage.","section":"Validation / Test Set Construction"},{"comment":"All AP/mAP values in Tables 2, 3, 4, 6, and 8 are reported as single numbers without variance, confidence intervals, or statistical significance tests. The headline margin of 0.120 mAP over Grounding-DINO in Table 3 could be within run-to-run noise or annotation noise. The authors should report means and standard deviations over at least three training runs (or resampled subsets of the test images with bootstrapping), and ideally a paired test for the VIGIL vs. baseline difference.","section":"Experiments Settings and Main Results"},{"comment":"The in-domain synthetic test set is generated with the same distortion pipeline (types, severity levels, region selection, and 1-3 regions per image) as the training set, so in-domain results largely measure the model's ability to fit that pipeline rather than generalization. More importantly, the authentic test set also uses the same 8-category taxonomy and the same 1-3 regions-per-image constraint as the synthetic training data, which may bias the S2A evaluation toward distortions that resemble the synthetic types. The paper should explicitly discuss this potential bias and, if possible, evaluate on an independently defined distortion taxonomy (e.g., the categories in ViDA-UGC or a broader UGC quality taxonomy) or report statistics showing that authentic annotations span the taxonomy naturally rather than being forced into it.","section":"Training Data Preparation / Validation / Test Set Construction"},{"comment":"The post-processing thresholds (TNMS=0.3, TProb=0.1 for authentic; 0.7, 0.8 for synthetic) are selected by a combinatorial search on a validation set of only 100 images, with step size 0.05 for both thresholds. With roughly 400 evaluated combinations, the selected thresholds may overfit this small validation set. The paper should report the robustness of the final AP to small perturbations of the selected thresholds (e.g., a sensitivity table or heatmap) and consider a larger validation set or cross-validated threshold selection.","section":"Additional Justification of Key Points (Supplementary)"}],"minor_comments":[{"comment":"There are multiple missing spaces in phrases such as \"thesynthetic-to-authentic (S2A)problem\" and \"theFG-BGindistinguishability inS2A\". The text should be copyedited throughout.","section":"Abstract and Introduction"},{"comment":"The section contains an unresolved reference to \"Sec.\" with no section number; this should be completed.","section":"Supplementary, Justification on the Use of In-domain Baselines"},{"comment":"Immediately after Figure 19, the supplementary text contains a long uninterpretable string of \"/uni...\" characters that appears to be a rendering artifact. This should be removed or replaced with the intended caption text.","section":"Supplementary, Authentic Distortion Test Set Statistics"},{"comment":"The column layout is dense and the meaning of the per-type \"mAP\" column (AP averaged over IoU thresholds 0.05-0.95 for that type) versus the overall mAP column is not explained in the caption. Adding a clear note would improve readability.","section":"Tables 2 and 3"},{"comment":"The table is captioned \"Area number generalization test\" but the version column is labeled \"Authentic-Supp\"; please clarify that the test set is the supplementary 100-image set with 4-5 distortion regions.","section":"Table 10"},{"comment":"The limitations paragraph is concise and appropriately acknowledges the absence of real-world training data, but it does not mention the annotation-reliability concern of the authentic test set; a sentence on this limitation would align the text with the actual evaluation risk.","section":"Limitations"}],"recommendation":"major_revision","confidential_remarks":"The paper is well within the scope of the journal and the technical approach is sound. The main risk is the authentic test set: without inter-annotator agreement or independent re-annotation, the S2A margin could be an artifact of the annotation protocol. I believe this is fixable with additional annotation analysis and a sensitivity study. I do not see a fundamental flaw in the method or the synthetic data construction."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth a careful read. It proposes VIGIL, an LMM-based detector for localized distortions, trained on a new synthetic dataset (VIGIL-140K) filtered from a million candidates. The two genuinely new pieces are the multi-detector setup—several LLM layers act as independent DETR-style heads with separate Hungarian matching—and the S2A trick of keeping the second-best class when a box is classified as non-distortion. Both are ablated cleanly, and the comparison against text-SFT is convincing. The model beat Grounding-DINO by a wide margin on the synthetic test, and on the authentic 815-image set it reports mAP 0.482 vs 0.362 for Grounding-DINO. If that holds, it is a real advance.\n\nThe soft spots are real but not necessarily fatal. The authentic test set was annotated by the authors; the iterative procedure with multiple annotators and a deletions-only final pass is plausible, but there is no inter-annotator agreement, no statement that annotators were blind to model outputs, and no error bars anywhere. The stress-test worry—that annotations could be biased toward synthetic-like distortions—is plausible, but the same test set is used for all baselines and the baselines were also trained on VIGIL-140K, so the comparison is still meaningful. Still, the absolute mAP on the authentic set is fragile without independent re-annotation or agreement statistics. The in-domain test shares the synthesis pipeline with training, the paper releases no code or data, and there is a garbled text passage after Figure 19 that looks like a rendering error.\n\nThis is a solid subfield contribution with a useful dataset and a method worth building on. The weaknesses are fixable: report annotator agreement, add error bars, release at least part of the data, and clean up the manuscript. I would send it to serious referees, with the authentic test-set construction as the central question. If the test set survives scrutiny, the S2A result is the kind of thing the community will want to cite.","headline":"VIGIL has a genuine new dataset and a clever LMM-based multi-detector design, but the headline synthetic-to-authentic result leans on an author-annotated 815-image test set that needs independent agreement data before the margin over baselines is fully trusted.","tokens_in":23045,"tokens_out":3369,"would_cite":true,"duration_ms":35240,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Trained only on 140K synthetic distortions, VIGIL-8B localizes authentic distortion regions in real UGC images with mAP 0.482.","keywords":["image quality assessment","distortion detection","large multimodal models","synthetic-to-authentic generalization","object detection","bounding box localization","user-generated content","multi-layer detectors"],"falsifier":"Re-annotate a random sample of the 815 test images with fresh annotators and compute VIGIL-8B's AP50 against majority-vote boxes: if AP50 drops to the level of the detection baselines, around 0.36, the reported 0.502 is an artifact of the original annotation protocol.","tokens_in":22045,"feed_emoji":"🔍","tokens_out":12147,"duration_ms":104363,"temperature":0.7,"pith_summary":"VIGIL makes the case that a large multimodal model is better used as a detection head stack than as a text generator for finding local image distortions. The authors build a 140K-image synthetic training set, attach classification and box-regression heads to multiple layers of the LLM decoder, and train with per-detector Hungarian matching; after post-processing, the resulting VIGIL-8B reports mAP 0.482 and AP50 0.502 on a human-annotated 815-image authentic-distortion test set, roughly 0.12 mAP above the strongest object-detection baseline. The paper argues that this closes much of the synthetic-to-authentic gap that has limited previous LMM-based IQA, and that the key enablers are multi-layer synchronous detection and recycling the second-best class of boxes the model calls background. If true, localized quality assessment can be scaled from synthetic data rather than labor-intensive real distortion annotation.","feed_headline":"Synthetic-only training beats object detectors on real image flaws","feed_subtitle":"VIGIL-8B reaches mAP 0.482 on authentic UGC distortions, ~0.12 over the strongest baseline.","key_machinery":"The load-bearing mechanism is the multi-detector LMM: N trainable area query tokens attend globally to image tokens while image tokens use causal attention, and each of M chosen LLM layers (10th, 15th, 20th, 25th, and 28th in the final model) carries its own linear head for 9-way classification (8 distortions plus non-distortion) and box regression. Each detector is trained with independent Hungarian matching against the set of ground-truth boxes, and the final loss is the average over detectors, with empty-class log-probabilities down-weighted by 0.05. For the synthetic-to-authentic gap, the paper reuses boxes classified as non-distortion by taking the second-highest class probability as a candidate distortion, then applies class-wise non-maximum suppression and probability filtering to produce final boxes. The training data is VIGIL-140K: more than 145K images filtered from a pool of over 1000K samples by six no-reference IQA scorers (threshold 85 on a 0-100 scale), with 1-3 irregular local regions injected per image across 8 distortion categories, 31 sub-types, and two severity levels.","core_discovery":"On its own terms, the paper's central claim is that an LMM with DETR-style detection heads attached to several LLM layers, trained only on synthetically distorted images, can localize authentic distortions in UGC images better than both general LMMs prompted to output bounding boxes and conventional object detectors trained on the same synthetic data. The reported evidence is that VIGIL-8B reaches mAP 0.482, AP50 0.502, and AP75 0.439 on the authentic test set, compared with 0.362/0.362/0.293 for Grounding-DINO and below 0.25 mAP for text-output LMMs. The paper also reports in-domain synthetic detection at mAP 0.736 and AP50 0.879, and ablations showing that discarding the non-distortion-class cue lowers authentic mAP from 0.482 to 0.407, while single-layer detection lowers it to 0.397.","pith_inferences":["The authors report no inter-annotator agreement statistics for the 815-image authentic test set; if fresh annotators disagree substantially on region boundaries and categories, the absolute AP numbers will shift, though the relative ranking may persist.","The supplementary Limitations section states that the dataset scale prevents verifying a data scaling law and that no large-scale authentic data is used in training, so the reported S2A gain is evidence for, not proof of, unbounded scaling of this approach.","A natural extension is to test VIGIL on distortions outside its eight categories, such as AI-generated artifacts or sensor noise; the multi-layer feature stacking might generalize further than the taxonomy suggests.","The background-cue trick is a form of pseudo-label mining; an adaptive rule that keeps the second-best class only when its probability exceeds a class-dependent threshold could reduce the false positives that the current global probability filter tolerates."],"forward_implications":["If the reported numbers hold, text-driven supervised fine-tuning for coordinate output is a bottleneck in LMM-based distortion localization, and regression-head detection should become the default for such tasks.","Synthetic distortion injection scaled to 140K images can substitute for costly authentic distortion annotation, at least for the eight categories tested.","The multi-layer detector design converges faster and predicts more regions than single-layer variants, which could carry over to other fine-grained perception tasks with scarce labels.","The S2A trick's margin (authentic mAP 0.482 versus 0.407 without it) suggests that even noisy background-class predictions carry useful cross-domain signal.","Joint training with Q-Instruct-200K preserves both distortion detection and text-based quality assessment abilities, so detection can be added to existing IQA-LMMs without sacrificing their general functionality."],"supporting_citations":[{"why":"Supplies the DETR set-prediction formulation (query tokens, Hungarian matching, box loss) that VIGIL reuses with multiple LLM layers as detectors.","marker":"Carion et al. 2020"},{"why":"Supplies the KADIS-700K distortion types and synthesis conventions from which VIGIL-140K's eight categories and 31 sub-types are derived.","marker":"Lin, Hosu, and Saupe 2019"},{"why":"Provides the COCO source images used in the training pool and the images used for the authentic test set.","marker":"Lin et al. 2014"},{"why":"Provides InternVL-3-8B, the base LMM whose vision encoder and LLM layers VIGIL repurposes.","marker":"Zhu et al. 2025"},{"why":"Grounding-DINO is the strongest object-detection baseline that VIGIL-8B claims to beat by about 0.12 mAP on authentic distortions.","marker":"Liu et al. 2024"},{"why":"Refine-IQA-S1 is the in-domain LMM baseline whose text-based coordinate output VIGIL contrasts with its detection-head design.","marker":"Jia et al. 2026a"},{"why":"Grounding-IQA represents the text-driven bounding-box SFT paradigm that VIGIL argues limits detection accuracy.","marker":"Chen et al. 2024c"},{"why":"KonIQ-10K is the pretraining database for the six no-reference IQA scorers used to filter the training pool.","marker":"Hosu et al. 2020"}],"fun_headline_variants":["VIGIL: synthetic-trained LMM beats object detectors on real image flaws","Trained on synthetic, detects real: VIGIL mAP .482","Multi-layer LMM detection surpasses object detectors on UGC flaws","LLM decoder layers as multiple detectors: VIGIL spots real distortions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes the 815 human-annotated test images, labeled by an iterative correction procedure with no reported agreement statistics, are a valid ground truth for real-world distortion location.","fun_headline_variants_meta":{"raw":{"variants":["VIGIL: synthetic-trained LMM beats object detectors on real image flaws","Trained on synthetic, detects real: VIGIL mAP .482","Multi-layer LMM detection surpasses object detectors on UGC flaws","LLM decoder layers as multiple detectors: VIGIL spots real distortions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00083,"raw_usage":{"total_tokens":3671,"prompt_tokens":1040,"completion_tokens":2631,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":656,"completion_tokens_details":{"reasoning_tokens":2551}},"tokens_in":656,"tokens_out":2631,"duration_ms":22370,"temperature":1.0,"reasoning_tokens":2551,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T23:05:27.558846+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate a random sample of the 815 test images with fresh annotators and compute VIGIL-8B's AP50 against majority-vote boxes: if AP50 drops to the level of the detection baselines, around 0.36, the reported 0.502 is an artifact of the original annotation protocol.","supporting_citations":[],"review_version":1}