{"id":"279951ff-7604-4825-9d43-94d88d211500","arxiv_id":"2411.10161","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"SEAGULL assesses region-of-interest quality with mask-guided vision-language instruction tuning, and releases two ROI-based IQA datasets.","lead":"This paper introduces SEAGULL, a vision-language model that scores the quality, importance, and distortion of a specific region of an image selected by a mask, rather than judging the whole image. It also releases two new ROI-level quality datasets and reports improvements over twelve existing image quality assessment models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 1 compares SEAGULL fine-tuned on SEAGULL-3k against baselines that are not fine-tuned on SEAGULL-3k, so the claimed superiority over existing models is not established.","rationale":"I read the paper in good faith as proposing a mask-conditioned VLM plus two datasets for ROI-level IQA. The strongest claim is that SEAGULL surpasses existing IQA models and VLMs, and Table 1 is the evidence for that claim. The reader's concern about TOPIQ pseudo-labels is legitimate for interpreting Table 3's pre-training benefit, but it is not the most load-bearing issue for the headline contribution: even if TOPIQ labels were perfect, Table 1 would still fail to establish superiority because baselines are not given the same fine-tuning. The final verdict should remain conditional: the framework and datasets may be useful, but the central comparison must be rerun under equal training conditions before the claim can be accepted.","tokens_in":16294,"tokens_out":7170,"duration_ms":72127,"concrete_test":"Re-run Table 1 after fine-tuning all baselines on the same SEAGULL-3k training split under identical LoRA settings (rank 128, alpha 256, same epochs and batch size) and the same ROI input format (masks where possible). If a baseline such as LLaVA-1.5-Q-Instruct or Osprey† then matches or exceeds SEAGULL on any sub-task, the claimed superiority is not supported. A complementary check is to evaluate SEAGULL immediately after SEAGULL-100w pre-training without SEAGULL-3k fine-tuning; if its zero-shot metrics drop toward the baseline level, the Table 1 gains are attributable to fine-tuning rather than the architecture. Report mean ± std over at least 3 seeds or test splits.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The main claim in Contribution 3, that SEAGULL 'remarkably surpasses existing advanced IQA models and VLMs in ROI quality analysis', rests on Table 1, but the comparison is not apples-to-apples. SEAGULL is pre-trained on SEAGULL-100w and then fine-tuned on the SEAGULL-3k training split (Sec. 5.1-A). The twelve baselines are only queried with ROIs and prompts (Sec. 5.2-A), with no indication that any baseline is fine-tuned on SEAGULL-3k; Osprey† is pre-trained on SEAGULL-100w but not fine-tuned on 3k. Because SEAGULL-3k images come from LIVEC, BID, SPAQ, and KonIQ (Sec. 3.2), and the test split is a random 20% of those same sources, SEAGULL is evaluated in-domain after task-specific training while baselines are not. The reported superiority could therefore reflect the fine-tuning procedure and dataset distribution rather than the proposed mask-based architecture. Small margins (e.g., 0.7452 vs 0.7176 SROCC for quality) with no error bars or significance tests make this confound hard to dismiss.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SEAGULL, a vision-language model for no-reference image quality assessment of user-specified regions of interest (ROIs) indicated by SAM masks. The method combines a ConvNeXt image encoder, a Mask-based Feature Extractor that pools multi-scale features inside the mask and fuses global and local view tokens, and a Vicuna LLM. The authors also introduce two datasets: SEAGULL-100w, containing roughly one million synthetic distorted images with about 33 million ROIs whose quality and importance labels are generated by TOPIQ, and SEAGULL-3k, containing 3,261 human-annotated ROIs from authentic images. After pre-training on SEAGULL-100w and fine-tuning on SEAGULL-3k, the model outputs ROI quality scores, importance scores, distortion types, and severity levels. The experimental section compares SEAGULL with twelve baselines on four sub-tasks and reports SROCC, PLCC, precision, recall, and F1, together with pre-training-scale ablations and component ablations.","tokens_in":16576,"tokens_out":10708,"duration_ms":95422,"significance":"If the results hold, the paper makes a useful contribution: it defines a new ROI-level IQA task, releases public datasets and code, and demonstrates a mask-guided VLM architecture that jointly predicts quality, importance, and distortion labels for arbitrary image regions. The final evaluation on the human-annotated SEAGULL-3k test set is a genuine strength and avoids the circularity that would arise if the test labels were derived from the same TOPIQ-generated pre-training labels. The main weaknesses are that the headline comparison in Table 1 does not put baselines on the same training footing, and the TOPIQ-generated pre-training labels are not validated against human judgments at ROI scale; these issues directly affect the strength of the central claim that SEAGULL remarkably surpasses existing models.","major_comments":[{"comment":"The central comparison in Table 1 is not apples-to-apples. SEAGULL is pre-trained on SEAGULL-100w and then fine-tuned on the SEAGULL-3k training split (Sec. 5.1-A), while the twelve baselines are only queried with ROIs and prompts (Sec. 5.2-A); no baseline is fine-tuned on SEAGULL-3k, and Osprey† receives only SEAGULL-100w pre-training. The reported margins, such as 0.7452 vs 0.7377 SROCC for Quality Score and 59.08% vs 56.25% F1 for Distortion Types, are therefore compatible with the explanation that task-specific fine-tuning on the benchmark distribution, rather than the proposed mask-based architecture, is responsible for the gains. Since Contribution 3 and the abstract claim that SEAGULL remarkably surpasses existing models rest on this table, the authors should fine-tune the strongest baselines on SEAGULL-3k under the same protocol or clearly present the comparison as zero-shot/frozen-model evaluation and temper the claim accordingly.","section":"Table 1; Secs. 5.1-A, 5.2-A"},{"comment":"The paper's own numbers contradict the headline claim on one of the four sub-tasks. On Importance Score, Osprey† achieves SROCC 0.8811 and PLCC 0.8756, while SEAGULL achieves 0.8603 and 0.8468, so SEAGULL is not the best model on this sub-task. The text calls this a slight advantage for Osprey, but the 0.0208 SROCC gap is larger than SEAGULL's 0.0075 SROCC margin over the second-best Quality Score model. The authors should either revise the contribution statement to say that SEAGULL outperforms baselines on most sub-tasks, or provide an analysis of why the proposed model underperforms on Importance Score.","section":"Table 1; Sec. 5.2-A; Contribution 3"},{"comment":"The automatically generated labels in SEAGULL-100w are not validated against human perception at the ROI scale. ROI Quality Scores are TOPIQ scores on mask-cropped patches, and ROI Importance Scores are defined as the TOPIQ difference between the reference image and the image with the distorted ROI pasted back in; this proxy conflates the visibility of a distortion with its semantic importance to the overall image. Because Table 3 measures the pre-training benefit using models trained on these labels and then fine-tuned on human labels, the improvement could in part reflect learning TOPIQ's regional biases rather than a general ROI-quality perception ability. The paper should provide a human-agreement analysis, for example the correlation between TOPIQ-derived scores and SEAGULL-3k human labels on a held-out subset, or an ablation that replaces TOPIQ labels with human labels.","section":"Sec. 3.1.3 A-B; Table 3"},{"comment":"No uncertainty quantification is reported for any of the headline metrics. The margins over the second-best model are small, and the Sample-Average metrics aggregate many ROIs from the same image, so the effective sample size is unclear. The authors should report confidence intervals obtained by bootstrapping at the image level, or significance tests, and state how many distinct test images contribute to the results in Table 1.","section":"Sec. 5.2-A; Table 1"},{"comment":"The evaluation of Distortion Severity Degrees is underspecified for authentic images. The annotation protocol in Sec. 3.2 assigns a severity rating per distortion type and notes that authentic images often have multiple distortions, but the severity sub-task is evaluated as a single 30-class label, with a prediction considered correct only when both the severity degree and the distortion type align. The paper should specify how multiple per-type severity ratings are aggregated into the single class, how Precision, Recall, and F1 are computed for this multi-label setting, and what the chance level is; without these details, the severity results in Table 1 cannot be interpreted.","section":"Sec. 3.2; Sec. 5.2-A; Table 1"}],"minor_comments":[{"comment":"The acronym for the mask-based feature extractor is written as 'MEF' once in the ablation paragraph ('remove the local and global tokens from the MEF'), but the model component is called MFE elsewhere; please standardize the notation.","section":"Sec. 5.3-B"},{"comment":"In the sentence 'The ROIs and text prompts are used to qurey the models for ROI quality analysis', 'qurey' should be 'query'.","section":"Sec. 5.2-A"},{"comment":"In 'we employe the ConvNeXt-Large model', 'employe' should be 'employ'.","section":"Sec. 4.1"},{"comment":"In the visualization discussion, 'it's reliability' should be 'its reliability'.","section":"Sec. 5.3-C"},{"comment":"The paper does not describe how the twenty parameter values for each ISP distortion type are spaced, which matters for the reliability and reproducibility of the synthetic labels. In addition, the bin boundaries in Eq. (1) are stated as M*i/5 < st <= M*(i+1)/5, which assigns a score exactly equal to M*i/5 to the lower bin while the upper boundary is inclusive; please clarify the intended discretization.","section":"Sec. 3.1.1; Eq. (1)"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a solid systems-and-datasets contribution with public data and code, and the use of a human-annotated final test set means the central result is not circular by construction. The stress-test concern about Table 1 is valid: the comparison mixes fine-tuned and zero-shot baselines, and the paper should either equalize training or soften the claim. The TOPIQ-label validation issue is also real and fixable. I recommend major revision rather than rejection; both load-bearing issues can be addressed within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take. SEAGULL is a real attempt at a neglected problem: fine-grained IQA for mask-specified ROIs, with three output dimensions (quality, importance, distortion). The architecture—SAM masks plus a VLM with mask-pooled global and local tokens—is sensible, and the two datasets are a concrete contribution, especially the 3k human-annotated one. The paper is worth reading for anyone working on region-level quality assessment.\n\nWhat's good: the ablation studies are the most informative part. Removing the global token collapses importance prediction (SROCC from 0.86 to 0.25), and the pre-training scale curve shows a clear benefit up to 50% of SEAGULL-100w. These are honest, controlled comparisons within the model family.\n\nThe soft spot is Table 1. SEAGULL is pre-trained on 100w and then fine-tuned on the SEAGULL-3k training split. The twelve baselines are only queried zero-shot—no fine-tuning on the 3k training split, no mention of it. Osprey† is pre-trained on 100w but still not fine-tuned on 3k. So the reported margins (e.g., quality SROCC 0.7452 vs 0.7176) conflate the architecture with the fine-tuning procedure. This isn't a fatal flaw in the model, but it means the 'remarkably surpasses' claim in the contributions is not established. The fix is straightforward: fine-tune a few baselines on the same 3k split, or explicitly label Table 1 as zero-shot comparison.\n\nA second concern: the 100w pre-training labels are TOPIQ-based with no validation at ROI scale. That's a real risk, but the 3k test set is human-annotated, so the final evaluation is independent; the leftover worry is that some of the pre-training gain in Table 3 could reflect learning TOPIQ's biases rather than general perceptual ability. A quick human-rating agreement check on a sample of ROIs would settle it.\n\nRecommendation: worth a serious referee. The datasets and task are useful, and the core ideas are solid. The authors need to address the baseline fairness and add a small validation of the pseudo-labels. I'd accept for review.","headline":"A useful ROI-IQA system and datasets, but the headline superiority claim rests on an unfair comparison (SEAGULL fine-tuned on the test distribution's training split, baselines not).","tokens_in":17089,"tokens_out":3562,"would_cite":true,"duration_ms":33234,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SEAGULL claims that a mask-guided vision-language model can return quality, importance, and distortion labels for any image region after instruction tuning on 33 million masked ROIs.","keywords":["image quality assessment","region of interest","vision-language model","instruction tuning","mask-based feature extraction","no-reference image quality","distortion analysis","Segment Anything Model"],"falsifier":"Take a random sample of, say, 300 ROIs from SEAGULL-100w, collect human ratings on the same five-level quality and importance scales used in SEAGULL-3k, and compute SROCC and PLCC between mean human scores and the TOPIQ-derived pseudo-labels. If agreement is low or varies strongly by distortion type, the pretraining signal is not a trustworthy stand-in for human ROI perception, and the pretraining gains in Table 3 could be an artifact of matching TOPIQ rather than matching people.","tokens_in":16139,"feed_emoji":"🎯","tokens_out":10125,"duration_ms":89994,"temperature":0.7,"pith_summary":"This paper tries to establish that image quality assessment can be moved from whole images to arbitrary regions of interest without a reference image: given a mask produced by a segmentation model, a vision-language model can return a quality score, an importance score, and a distortion type with severity for the masked region. To make that work, the paper introduces a mask-based feature extractor that feeds both global context tokens and local region tokens into a language model, and it builds two ROI-labeling datasets: a roughly one-million-image synthetic set with 33 million automatically labeled ROIs for pretraining, and a 3,261-ROI authentic set annotated by 24 people for fine-tuning. On its test split, SEAGULL reports higher correlation and F1 scores than twelve prior IQA and VLM baselines on most ROI sub-tasks. Region-level quality labels matter because they give actionable guidance for targeted enhancement, compression, and video optimization, which whole-image scores cannot provide.","feed_headline":"Mask-guided model scores image regions and names their distortions","feed_subtitle":"SEAGULL adds region-level quality, importance, and distortion labels after pretraining on 33 million masked ROIs.","key_machinery":"The load-bearing component is the Mask-based Feature Extractor (MFE). It mask-pools the four scales of ConvNeXt features inside a binary ROI mask to form basic tokens, runs self-attention over them, then uses them as queries in cross-attention over the full-image tokens to build a global view token, while two CNNs on the mask-cropped patch produce a local view token. These two tokens are inserted as special placeholders next to the image tokens in the Vicuna LLM, so the model sees both the region's own pixels and its context. The other essential mechanism is the dataset construction: SEAGULL-100w generates distortions from RAW images through an ISP pipeline and derives ROI quality and importance labels from TOPIQ scores, while SEAGULL-3k provides human labels; both datasets use Analysis Instruction-Response and Judgment Instruction-Response pairs to tune the language model.","core_discovery":"On the paper's own terms, SEAGULL is a no-reference ROI quality assessor: SAM converts a click or box into a precise mask, and the model fuses that mask with the full image to produce three outputs at once — ROI quality score, ROI importance score, and distortion type with severity. The key training recipe is two-stage: pretrain the projector, mask-based feature extractor, and LLM on SEAGULL-100w, where TOPIQ, a full-reference quality model, provides the pseudo-labels for quality and importance, then fine-tune on SEAGULL-3k, where human annotators provide the labels. The reported results are that SEAGULL outperforms existing vision-based and VLM-based IQA models on ROI quality, importance, and distortion identification (quality SROCC 0.7452, importance PLCC 0.8468, average distortion-type F1 46.93%), with the single exception that Osprey, a mask-based VLM that was also pretrained on SEAGULL-100w, keeps a small edge on importance-score prediction. The paper interprets these gains as evidence that mask-based region specification plus global and local token extraction is what ROI-level quality analysis needs.","pith_inferences":["If the auto-generated labels in SEAGULL-100w are systematically biased, then the pretraining gains shown in Table 3 may partly be learning TOPIQ's region-level biases; validating a sample of those pseudo-labels against human ratings would separate genuine perceptual learning from model imitation.","The importance score's operational definition — how much a full-reference metric drops when the distorted region is pasted back into the reference — makes importance depend on distortion visibility, which may understate the importance of clean but semantically central regions; human importance judgments could diverge from this metric exactly in high-quality ROIs.","The paper's own scaling analysis shows the synthetic-to-authentic gap flattening quality-score gains between 50% and 100% of SEAGULL-100w, so the large synthetic set contributes most to importance and distortion labels; this suggests future work should focus on authentic ROI data for quality-score gains.","The same mask-plus-global-and-local-token architecture could be adapted to video ROI quality by replacing SAM masks with tracked object masks over time, and to generated-image quality by querying edited regions."],"forward_implications":["If SEAGULL's results hold, ROI-level IQA no longer requires cropping or bounding boxes: a precise mask gives a model the region's exact boundary, and the same model can output score, importance, and distortion labels in one pass.","Pretraining on a large synthetic set with model-generated ROI labels transfers to authentic images: even 25% of SEAGULL-100w raises all sub-task metrics over no pretraining, and the full set raises importance SROCC from 0.7512 to 0.8603.","Global context tokens are what make importance prediction possible: removing them drops importance SROCC from 0.8603 to 0.2475, so a region's impact on overall image quality can only be judged when the model sees the whole image.","Because the model reports distortion type and severity, not just a number, it can be used not only to rank regions but to tell users what is wrong with a region and how badly.","The released datasets give the field a common training and evaluation ground for ROI-based IQA, which previously lacked fine-grained region labels."],"supporting_citations":[{"why":"Generates the mask-based ROIs that specify which image region SEAGULL should assess.","marker":"[30]"},{"why":"Provides the full-reference scores used as ROI quality pseudo-labels and the difference scores used as importance pseudo-labels in SEAGULL-100w.","marker":"[5]"},{"why":"Detects objects in reference images and supplies boxes that seed object-level SAM masks.","marker":"[72]"},{"why":"Supplies the discrete-level scoring scheme and instruction-tuning format that SEAGULL adapts for quality and importance prediction.","marker":"[56]"},{"why":"Provides VLM-based IQA baselines and instruction data used for comparison in the experiments.","marker":"[57]"},{"why":"Supplies RAW images that are passed through an ISP to generate roughly one million synthetic distortion images.","marker":"[12]"},{"why":"One of four authentic distortion datasets whose images contribute ROIs to the human-annotated SEAGULL-3k set.","marker":"[16]"},{"why":"Another authentic distortion dataset contributing images to SEAGULL-3k.","marker":"[10]"},{"why":"Another authentic distortion dataset contributing images to SEAGULL-3k.","marker":"[13]"},{"why":"Another authentic distortion dataset contributing images to SEAGULL-3k.","marker":"[20]"}],"fun_headline_variants":["ROI quality, importance, and distortion labels from one masked model","SEAGULL turns SAM masks into region-level quality verdicts","Vision-language IQA: score any region, name its distortion","Mask-based IQA model outputs score, importance, and distortion type","Two-stage training gives SEAGULL fine-grained ROI assessment"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The pre-training labels for region quality and importance come from an automatic scoring model rather than from people, and the paper does not show that those automatic region-level labels agree with human judgments of a region's quality and importance.","fun_headline_variants_meta":{"raw":{"variants":["ROI quality, importance, and distortion labels from one masked model","SEAGULL turns SAM masks into region-level quality verdicts","Vision-language IQA: score any region, name its distortion","Mask-based IQA model outputs score, importance, and distortion type","Two-stage training gives SEAGULL fine-grained ROI assessment"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000222,"raw_usage":{"total_tokens":1518,"prompt_tokens":1075,"completion_tokens":443,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":691,"completion_tokens_details":{"reasoning_tokens":356}},"tokens_in":691,"tokens_out":443,"duration_ms":4841,"temperature":1.0,"reasoning_tokens":356,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T19:54:10.764494+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of, say, 300 ROIs from SEAGULL-100w, collect human ratings on the same five-level quality and importance scales used in SEAGULL-3k, and compute SROCC and PLCC between mean human scores and the TOPIQ-derived pseudo-labels. If agreement is low or varies strongly by distortion type, the pretraining signal is not a trustworthy stand-in for human ROI perception, and the pretraining gains in Table 3 could be an artifact of matching TOPIQ rather than matching people.","supporting_citations":[{"cited_title":"Berg, Wan-Yen Lo, Piotr Doll ´ar, and Ross Girshick","cited_arxiv_id":null,"evidence_quote":"Generates the mask-based ROIs that specify which image region SEAGULL should assess."},{"cited_title":"Topiq: A top-down approach from semantics to distortions for image quality assessment","cited_arxiv_id":null,"evidence_quote":"Provides the full-reference scores used as ROI quality pseudo-labels and the difference scores used as importance pseudo-labels in SEAGULL-100w."},{"cited_title":"Segment everything everywhere all at once","cited_arxiv_id":null,"evidence_quote":"Detects objects in reference images and supplies boxes that seed object-level SAM masks."},{"cited_title":"Q-instruct: Improving low-level visual abilities for multi-modality foundation models","cited_arxiv_id":null,"evidence_quote":"Provides VLM-based IQA baselines and instruction data used for comparison in the experiments."},{"cited_title":"Raise: a raw images dataset for digital image forensics","cited_arxiv_id":null,"evidence_quote":"Supplies RAW images that are passed through an ISP to generate roughly one million synthetic distortion images."},{"cited_title":"No-reference blur assessment of digital pictures based on multifeature classifiers","cited_arxiv_id":null,"evidence_quote":"Another authentic distortion dataset contributing images to SEAGULL-3k."},{"cited_title":"Koniq-10k: An ecologically valid database for deep learning of blind image quality assessment","cited_arxiv_id":null,"evidence_quote":"Another authentic distortion dataset contributing images to SEAGULL-3k."}],"review_version":1}