{"id":"d6309d84-431c-4831-8be8-558e36cdac41","arxiv_id":"2412.14672","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"FiVL augments vision-language instruction data with GPT-4o-extracted key expressions and segmentation masks, trains LLaVA with a vision-modeling loss that predicts vocabulary tokens for image patches, and measures visual reliance via accuracy drop when key regions are masked.","lead":"This paper introduces a pipeline that finds the words in a question-answer pair that need the image, segments those words in the picture, and uses that data to train and test vision-language models. The trained model answers better on many benchmarks, and the masking-based score is proposed as a way to measure whether a model is really looking at the image.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The training improvement claim is not yet established because the comparison is against the released LLaVA-1.5-7b checkpoint, with no same-recipe control fine-tuned on un-augmented data; Table 6 even lists SGD, unlike standard LLaVA training, so the gain may not come from the VM loss.","rationale":"The reader's weakest assumption focuses on GroundedSAM mask accuracy. That is a legitimate concern, and the reported numbers (58% human-relevant, 46% Seg1) support it. However, the more load-bearing gap for the headline training claim is the absence of a same-recipe control: the paper compares its FiVL-trained model to the released LLaVA-1.5-7b checkpoint, not to a model fine-tuned on the original 665K data with the same optimizer, schedule, and data ordering but without the VM loss. The Table 6 entry listing SGD as the optimizer deepens this concern, because the standard LLaVA-1.5 fine-tuning recipe uses AdamW; if the FiVL run used a different optimizer than the released baseline, then the comparison conflates optimizer choice with the proposed loss. This does not invalidate the dataset and evaluation contributions, which are still plausible and partially supported by the random-mask control in Table 2 and the ablations in Appendix F. It does mean the central 'better performance than the baseline' claim is not yet established. The correct verdict remains CONDITIONAL, but the condition is stronger than the reader stated: add a no-VM control and resolve the optimizer discrepancy before accepting the training result. I therefore leave the reader's CONDITIONAL verdict unchanged, while flagging the missing control as the decisive test.","tokens_in":17127,"tokens_out":6184,"duration_ms":53726,"concrete_test":"Fine-tune LLaVA-1.5-7b from the same projector/Vicuna init on un-augmented LLaVA-1.5-mix-665K with the exact Table 6 recipe (including SGD if that is what was used) and lambda=0, and also with lambda=0.1 on FiVL-Instruct; run each with at least 3 seeds and report the Fig. 3 benchmarks. If the lambda=0 control matches or exceeds the released checkpoint, or the lambda=0.1 model's margin over the control is within noise, the training improvement claim fails. Also report per-benchmark error bars and the exact optimizer name.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the FiVL augmented dataset plus Vision Modeling loss produces an LVLM that outperforms LLaVA-1.5-7b. The evidence in Section 5.1 and Figure 3, however, compares the model to the released LLaVA-1.5-7b checkpoint, not to a control model trained in the same pipeline. Section 5.1 says they 'fine-tuned an LVLM, specifically LLaVA-1.5-7b, referred as to the baseline, on FiVL-Instruct,' which is ambiguous, and Appendix F compares against 'the baseline LLaVA-v1.5-7B model,' indicating the stock checkpoint. Without a no-VM control (lambda=0) trained on the original LLaVA-1.5-mix-665K with the same epochs, learning rate, data order, and optimizer, every benchmark gain in Fig. 3 is confounded by recipe differences. Table 6 lists Optimizer 'SGD,' whereas the standard LLaVA-1.5 fine-tuning recipe uses AdamW; if the FiVL model was actually trained with SGD, the comparison to a released AdamW-trained checkpoint is not isolating the VM loss at all. This is anterior to the mask-quality concern: even if every segmentation mask were perfect, the current experimental design cannot attribute the observed improvement to the proposed training task. The mask accuracy issue (58% human-relevant, 46% Seg1) further weakens the interpretation, but the missing control is the load-bearing gap.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FiVL, a framework that uses GPT-4o to extract key expressions from question-answer pairs and GroundedSAM to produce segmentation masks, yielding FiVL-Instruct (an augmentation of LLaVA-1.5-mix-665K) and FiVL-POPE, FiVL-VQAv2, and FiVL-GQA evaluation sets. The authors fine-tune LLaVA-1.5-7b with a combined language-modeling and vision-modeling loss (Eq. 1) and report performance improvements over a baseline on several benchmarks. They also introduce a Visual Reliance Score (Eq. 2) based on masking key expressions, and they use Spearman correlations between segmentation masks and attention to identify vision-language aligned attention heads. The paper further claims that the trained model can produce approximate segmentation maps from vision logits.","tokens_in":17452,"tokens_out":4852,"duration_ms":37009,"significance":"If the training improvement were established, the FiVL framework would be a useful, low-cost way to create grounded instruction data for LVLMs and to evaluate how much a model relies on visual evidence. The paper includes human and automated quality assessments of the generated data, an open-source release, a simple perturbation-based evaluation metric applicable to any dataset, and a complementary explainability analysis. These are valuable additions to the literature. However, the central training claim is not yet supported because the experiments compare against a released checkpoint rather than a same-recipe control model; several smaller methodological gaps further weaken the interpretation.","major_comments":[{"comment":"The central claim that the Vision Modeling loss improves performance over the baseline is not supported by the current experimental design. The model is compared against the released LLaVA-1.5-7B checkpoint, not against a model fine-tuned on the original un-augmented LLaVA-1.5-mix-665K under identical conditions. This matters because Table 6 lists the optimizer as SGD while the standard LLaVA-1.5 recipe uses AdamW; without a lambda=0 control trained with the same data order, epochs, learning rate, and optimizer, the gains in Figure 3 could be due to recipe differences rather than to the proposed loss. I request a same-recipe control on the original data, ideally with multiple seeds and error bars, before the improvement claim can be accepted.","section":"Section 5.1 / Figure 3 / Table 6 / Appendix F"},{"comment":"The training labels for the vision-modeling loss come from segmentation masks whose quality is low: human evaluation found only 58% of masks relevant to their key expression, and the Seg1 judge found 46%. The paper states that masks are filtered by size (Figure 6) and that only nouns are used, but it never specifies the size threshold, the fraction of data removed, or the annotation/judge agreement after filtering. Since every image patch in a retained mask is assigned a vocabulary token as its visual label in Eq. (1), systematic mask errors become training-label noise. Please report the filtering criterion, the post-filter mask relevance, and ideally perform a sensitivity analysis of the training result to mask quality.","section":"Section 4.1 / Section 4.2 / Section 5.1"},{"comment":"The claim that Visual Reliance Score correlates with overall model performance is not quantified. Table 3 reports VRS for seven models, and the text compares leaderboard ranks qualitatively, but no correlation coefficient, confidence interval, or statistical test is provided, and the external leaderboard source is not described in enough detail to reproduce the rank mapping. Please provide a quantified correlation with uncertainty, or soften the claim to a qualitative observation.","section":"Section 5.2 / Table 3"},{"comment":"The segmentation-capability evaluation is partly circular: Grounded-SAM is used as ground truth for IoU, but Grounded-SAM is also the system that produced the training masks. A model trained to mimic Grounded-SAM masks will score higher by construction. In addition, the baseline and FiVL model produce very different numbers of max-v tokens, which makes the per-token IoU comparison difficult to interpret. Please evaluate with human-annotated masks or an independent segmenter, and align the comparison across a common set of tokens or use a detection-style metric.","section":"Appendix G / Table 7"}],"minor_comments":[{"comment":"The text says 'we evaluated five models' but Table 3 lists seven models (Qwen2-VL, LLaVA-7B, LLaVA-13B, GPT4o, BLIP-2, Pixtral, Phi3). Please correct the wording.","section":"Section 3.3 / Table 3"},{"comment":"Random perturbation sometimes yields negative Visual Reliance Scores (e.g., LLaVA-13B on VQA-v2: -0.05), which means the model accuracy improves after perturbation rather than drops. Please clarify the interpretation of negative values or use a signed metric.","section":"Eq. (2) / Table 2"},{"comment":"The text says 'We maintained the original training setup' but Table 6 reports Optimizer SGD, while the standard LLaVA-1.5 fine-tuning setup uses AdamW. Please clarify whether the released baseline was retrained with SGD or whether this is a deliberate deviation, and discuss the impact on the comparison.","section":"Table 6"},{"comment":"There are several typos and small errors, including 'Additionnaly', 'lenght', 'refer as to the baseline', and the reference to 'Appendix 13' in Section 3.1, which should be 'Figure 13'.","section":"Throughout"},{"comment":"The format of Table 5 is not aligned: the first row lists three numbers without column headers, making it hard to read. Please align the table or add explicit column labels.","section":"Appendix A / Table 5"}],"recommendation":"major_revision","confidential_remarks":"The paper has a useful framework and a potentially interesting training objective, but the experimental comparison is not yet convincing. The missing same-recipe control (especially with the SGD/AdamW discrepancy) is the main blocking issue; it is fixable within the scope of the work. I would also encourage the editor to ask the authors to report the mask filtering details and a quantified VRS correlation, as these are needed to support the paper's broader claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: FiVL has a genuinely new training objective and a reusable dataset pipeline, but the paper's headline claim—that the VM loss improves benchmark performance—is not actually supported by the experiments as reported. The comparison is against the released LLaVA-1.5-7b checkpoint, not a control fine-tuned on un-augmented LLaVA-1.5-mix-665K under the same recipe. Table 6 lists SGD as the optimizer, while the standard LLaVA recipe uses AdamW. So every point in Figure 3 is confounded by optimizer choice, data ordering, and the augmentation itself. That's a load-bearing gap, not a nitpick.\n\nWhat's genuinely new: the vision-modeling cross-entropy loss on visual logits is a different way to inject grounding, and the pipeline of GPT-4o key-expression extraction + GroundedSAM masks for both training and evaluation is not in the prior work they cite. The VRS perturbation metric is a reasonable idea, and the random-mask control in Table 2 is a good sanity check. The attention-head explainability section is speculative but not egregious.\n\nThe soft spots in proportion: mask quality is weak—58% human relevance and 46% GPT-4o Seg1. The paper's own Figure 6 shows size filtering helps, but that filter is then applied selectively and not fully specified. The evaluation datasets are filtered to 40–95% of the originals, and the \"strong correlation\" between VRS and leaderboard rank is asserted without a correlation statistic or error bars. No multiple seeds anywhere. These are addressable.\n\nThe paper's own Limitation section acknowledges the GroundedSAM issue, which is honest. The authors clearly know the field and cite relevant work. The central training claim needs a same-recipe control with λ=0 on original data, at least one more seed, and a statement of optimizer that matches the code.\n\nVerdict: this deserves a serious referee. The framework is worth engaging, but the experimental design needs revision before the training claim can stand. I'd send it to review with a request for the control experiment and cleaned-up training details. I wouldn't cite the benchmark gains in the meantime, though I'd keep an eye on the dataset release.","headline":"Useful dataset-and-evaluation framework with a novel VM loss, but the main training claim is confounded by the lack of a same-recipe control and an SGD/AdamW discrepancy.","tokens_in":18012,"tokens_out":2511,"would_cite":false,"duration_ms":19565,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adding a vision-modeling loss to LLaVA-1.5 training improves most benchmarks and gives the model a built-in segmentation capability.","keywords":["vision-language alignment","visual grounding","segmentation masks","hallucination","visual reliance score","instruction tuning","vision-modeling loss","attention explainability"],"falsifier":"Train the identical LLaVA-1.5-7b setup with random same-size masks in place of the genuine key-expression masks; if the reported benchmark gains persist, then the content of the masks is not what drives the improvement. Alternatively, train only on the 58% of masks that human annotators judged relevant and check whether gains grow, shrink, or vanish.","tokens_in":16945,"feed_emoji":"👁️","tokens_out":8347,"duration_ms":55632,"temperature":0.7,"pith_summary":"The paper introduces FiVL, a pipeline that turns ordinary image-text instruction data into visually grounded data by extracting key expressions from question-answer text, localizing those expressions with segmentation masks, and then training the model so that every image patch inside a mask predicts the word naming it. The central claim is that this vision-modeling loss, added to the usual language-modeling loss, makes a fine-tuned LLaVA-1.5-7b outperform its baseline on a majority of ten downstream benchmarks while also giving the model a built-in ability to segment images. The same masks power a Visual Reliance Score that measures how much a model depends on the image by comparing accuracy on original versus masked images, and the masks let the authors rank attention heads by how strongly they align vision and language. If the claim is right, hallucinations in large vision-language models can be reduced simply by teaching image patches their own names, and evaluation can directly measure whether a model is actually looking at the image.","feed_headline":"A vision-modeling loss beats the LLaVA-1.5 baseline on most benchmarks","feed_subtitle":"Auto-generated masks train image patches to name themselves, yielding better grounding, free segmentation, and a visual-reliance score.","key_machinery":"The load-bearing object is the vision-modeling (VM) loss in equation (1), $L = \\lambda \\cdot CE_{VM} + (1-\\lambda) \\cdot CE_{LM}$ with $\\lambda = 0.1$, applied during LLaVA's fine-tuning stage. For each image patch inside a selected segmentation mask, the visual logit—the last linear layer's output over the text vocabulary—is trained to predict the noun key expression assigned to that patch; patches outside masks are ignored, and the LM loss still trains the textual logits. This turns image tokens into vocabulary predictions, explicitly teaching the model what each image region is called, while the same segmentation masks serve evaluation (masked-image accuracy drop equals the Visual Reliance Score) and explainability (Spearman correlation between mask and vision-to-language attention identifies heads (10,6) and (14,11)).","core_discovery":"On the paper's own terms, the central discovery is that pixel-level grounding can be generated automatically and then injected into an LVLM through a simple auxiliary loss. Specifically, GPT-4o reads only the question-answer text and names the key expressions, GroundedSAM turns those expressions into segmentation masks, and each image patch inside a mask is trained to predict the corresponding key expression's text token from the model's vocabulary. The resulting model outperforms LLaVA-1.5-7b on a majority of the ten benchmarks tested, produces argmax token maps that serve as coarse segmentation maps (about 10 tokens per image versus about 73 for the baseline, with roughly three times higher IoU against GroundedSAM), and enables a Visual Reliance Score that compares models and benchmarks by the accuracy drop when key regions are masked.","pith_inferences":["The same VM-loss recipe should transfer to other LVLMs whose last-layer logits are accessible, but the paper only demonstrates it on LLaVA-1.5-7b, so that transfer is an extrapolation.","Because only 58% of masks passed human relevance, the training signal is partly noisy; a mask-quality filter or a relevance-weighted loss could plausibly raise the reported gains further, and the paper's own size-based selection is an early version of that idea.","Visual Reliance Score could be repurposed as a data-quality filter: samples whose key-expression masking barely changes accuracy are likely answerable from language priors, so they could be downweighted or removed from instruction data.","The cross-model correlation between VRS and leaderboard rank suggests that visual reliance is a proxy for general model capability, but verifying that requires comparing models with identical architecture and different training data."],"forward_implications":["The FiVL-trained model outperforms LLaVA-1.5-7b on most of the ten benchmarks reported: OK-VQA, MME, POPE, ScienceQA, MMBench, LLaVA-Bench-COCO, LLaVA-in-the-wild, Text-VQA, VizWiz, and GQA.","The vision logits can double as segmentation maps: FiVL produces about 10 max-v tokens per image versus about 73 for the baseline, and IoU against GroundedSAM roughly triples at every threshold tested.","The Visual Reliance Score distinguishes benchmarks: FiVL-VQAv2 demands the most image reliance, and the average VRS across benchmarks tracks overall model ranking on a broad leaderboard.","The Spearman-correlation head summary isolates attention heads (10,6) and (14,11) as the strongest vision-language aligners, giving a concrete target for studying visual hallucinations."],"supporting_citations":[{"why":"Supplies the LLaVA-1.5-mix-665K instruction-tuning dataset that FiVL augments with key expressions and masks.","marker":"(Liu et al., 2024b)"},{"why":"Provides the GroundedSAM pipeline that turns key expressions into bounding boxes and segmentation masks.","marker":"(Ren et al., 2024)"},{"why":"Provides the Segment Anything model used for precise mask creation inside the GroundedSAM pipeline.","marker":"(Kirillov et al., 2023)"},{"why":"Defines the LLaVA-1.5-7b baseline model that FiVL fine-tunes and compares against, including several benchmark settings.","marker":"(Liu et al., 2023)"},{"why":"Supplies GPT-4o, which extracts key expressions from text and later serves as an automated judge of keyword and mask quality.","marker":"(OpenAI, 2024)"},{"why":"Supplies the POPE benchmark that FiVL converts into FiVL-POPE to test visual reliance.","marker":"(Li et al., 2023b)"},{"why":"Supplies VQAv2, which FiVL converts into FiVL-VQAv2, the evaluation set showing the highest visual-reliance demand.","marker":"(Goyal et al., 2017)"},{"why":"Supplies GQA, which FiVL converts into FiVL-GQA for compositional visual-reasoning evaluation.","marker":"(Hudson and Manning, 2019)"},{"why":"Provides the Qwen2-VL-7B model that FiVL evaluates with its Visual Reliance Score to show cross-model generalization.","marker":"(Wang et al., 2024)"},{"why":"Supplies the head-summary and Spearman-correlation technique used to rank attention heads for vision-language alignment.","marker":"(Aflalo et al., 2022)"}],"fun_headline_variants":["Auto-masked token loss boosts LVLM grounding and cuts hallucinations","New training task makes LVLMs actually use the image, not just text priors","FiVL: Pixel-level grounding via auto-generated masks outperforms LLaVA-1.5","Visual Reliance Score reveals when models truly need the image"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the automatically generated masks, derived from phrases extracted from text alone, are accurate enough to serve as training targets and evaluation perturbations, even though human judges found only 58% of masks relevant to their phrase.","fun_headline_variants_meta":{"raw":{"variants":["Auto-masked token loss boosts LVLM grounding and cuts hallucinations","New training task makes LVLMs actually use the image, not just text priors","FiVL: Pixel-level grounding via auto-generated masks outperforms LLaVA-1.5","Visual Reliance Score reveals when models truly need the image"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000451,"raw_usage":{"total_tokens":2270,"prompt_tokens":943,"completion_tokens":1327,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":559,"completion_tokens_details":{"reasoning_tokens":1245}},"tokens_in":559,"tokens_out":1327,"duration_ms":9531,"temperature":1.0,"reasoning_tokens":1245,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:01:08.663855+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the identical LLaVA-1.5-7b setup with random same-size masks in place of the genuine key-expression masks; if the reported benchmark gains persist, then the content of the masks is not what drives the improvement. Alternatively, train only on the 58% of masks that human annotators judged relevant and check whether gains grow, shrink, or vanish.","supporting_citations":[],"review_version":1}