{"id":"cf751991-b56c-436a-9fdf-a4227207cdfe","arxiv_id":"2501.04568","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"SVP uses self-captioning with grounding feedback and a scoring filter to adapt vision-language models with about 1,000 images, improving captioning, referring, object recall, and hallucination control.","lead":"This paper introduces SVP, a method that improves vision-language models by using an object-detection model as feedback to generate and select better training captions from a small set of images. It reports gains in captioning, referring, object recall, and hallucination reduction across several LLaVA variants, without using curated image-text pairs.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SVP's central mechanism assumes GroundingDINO's detections are a reliable training target; the paper's own grounded-caption pipeline shows that any object the detector fails to output is absent from the training target, so the detector's omissions and mislabels can be learned by the adapted VLM.","rationale":"The reader's weakest assumption is that the pretrained grounding model supplies accurate and complete object and spatial information, and this is indeed the most load-bearing condition for the central claim. The entire SVP pipeline—sampling, scoring, and selection—is driven by the detector's output, and the adaptation step trains the model to reproduce detector-conditioned captions. If the detector systematically misses objects or mislabels them, those errors are not merely ignored; they are actively learned as the target distribution. This is more fundamental than the other issues noted in the reader's rationale. The importance-ratio simplification in Appendix E is mathematically unjustified, but the final training objective is a well-defined filtered maximum-likelihood procedure, so a flawed derivation does not by itself falsify the empirical claim. Missing error bars are a reporting weakness, but the reported effect sizes, especially on referring tasks, are large enough that a pure noise explanation is unlikely. The grounding-reliability concern, by contrast, directly attacks whether the measured gains reflect improved vision-language alignment or improved mimicry of a particular detector's biases. The paper's own limitation section flags dependence on grounding quality, but the central generalization claim would be materially weakened if the method only aligns to GroundingDINO's coverage. The concrete test I propose would settle this by measuring whether the adapted model's object recall drops on objects the detector misses, and by comparing against an oracle-grounding variant. Because the paper already conditions the verdict on this kind of internal-consistency and reproducibility check, I recommend no change to the reader's CONDITIONAL verdict.","tokens_in":115,"tokens_out":16079,"duration_ms":165596,"concrete_test":"Take a stratified sample of images with ground-truth object annotations (e.g., COCO val2014 plus rare-category images from LVIS or ODinW). For each image: (1) run GroundingDINO and partition ground-truth objects into detected and missed; (2) generate base captions, grounded captions z_q, and SVP-adapted captions; (3) measure the mention rate (recall) for each stratum. If the mention rate on the missed stratum is lower for the adapted model than for the base model, the detector's omissions have been incorporated into the training objective. As a second arm, run SVP with oracle ground-truth boxes in place of GroundingDINO and compare POPE F1 and object recall to the GroundingDINO-based run; the gap directly quantifies the cost of imperfect grounding.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing step is Section 3's grounding feedback g ← g(zp, c), which defines the training target. SVP selects the K-samples with the largest log-ratio between the grounded and base distributions and then fine-tunes the VLM to maximize log p_theta(z|c) on those samples. Because the grounded captions are generated while conditioning on GroundingDINO's detections, the detector's misses and mislabels become the reference the model is trained to reproduce. The pipeline in Fig. 17 illustrates the mechanism: the grounded caption's object list is exactly the detector's output, so any object that GroundingDINO fails to output (e.g., 'door' or 'ottoman' from the base list, if present in the image) is absent from the training target. The paper does not measure how often the detector misses real objects that the base model correctly mentions. The reported gains on POPE object recall and hallucination are measured on COCO-derived object lists, which overlap heavily with categories GroundingDINO detects well; the benchmarks do not reveal whether SVP degrades recall on rare, small, or otherwise missed objects. Since the detector is not consulted at inference time, the adapted model cannot recover from these omissions. The paper's limitation section acknowledges dependence on grounding quality, but the central claim—general improvement in vision-language alignment with minimal supervision—requires the feedback to be a reliable target, not an imperfect heuristic.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SVP (Sampling-based Visual Projection), a three-stage procedure consisting of inner-loop sampling with grounding feedback from a pretrained detector (GroundingDINO), scoring and top-k selection of grounded captions, and outer-loop LoRA adaptation of the base VLM. The method requires no curated image-text pairs or preference annotations, only a small set of unlabeled images and self-captioning. The authors evaluate SVP on LLaVA and other VLM families across captioning, referring, VQA, multitasking, hallucination, and object-recall benchmarks, reporting consistent gains on captioning and referring, reduced POPE hallucinations, and improved object recall after up to three iterations. The paper also includes ablations of scoring mechanisms, iterations, sample size, and a comparison against DPO.","tokens_in":36153,"tokens_out":5667,"duration_ms":51835,"significance":"If the results hold, SVP offers a cheap, task-agnostic alignment method that could reduce dependence on human annotation and curated data, with potential applicability across VLM architectures. The empirical coverage is broad (10 benchmarks, multiple model scales, multiple text encoders), the qualitative comparisons are instructive, and the pseudocode and prompt templates facilitate reproduction. However, the theoretical derivation in Appendix E contains an unjustified step, and the empirical claims lack statistical support, so the significance can only be fully assessed after those issues are resolved.","major_comments":[{"comment":"The derivation drops the importance ratio pθ(zi|c)/q(zi|c,g) by stating that it is 'constant' for selected samples. This is not justified: selection is based on S(q,p) = log q - log p, so the retained samples are exactly those with large log-ratios, meaning pθ/q is systematically small and varies across the selected set. The claimed equivalence between the MLE gradient in Eq. (10) and the off-policy REINFORCE gradient in Eq. (15) is therefore not established. Because the 'principled formulation' in Section 3 and the Introduction rests on this equivalence, this step needs a rigorous justification or an explicit acknowledgment that Eq. (15) is a heuristic approximation.","section":"Appendix E, Eqs. (14)-(15)"},{"comment":"The paper is internally inconsistent about the selection ratio. The Implementation Details paragraph states 'selecting the top 10% using our scoring mechanisms', while Fig. 7's caption says 'selecting the top 20%' and Table 11 lists topk = 0.2 for LLaVA-1.5-13b, LLaVA-1.6-7b, and LLaVA-1.6-13b, but topk = 0.1 for LLaVA-OV. Since the selection ratio is a free parameter that affects the training set and results, the ratio used for each reported experiment must be stated unambiguously.","section":"Section 4 (Implementation Details), Fig. 7, Table 11, Algorithms 1-2"},{"comment":"All results are single-run point estimates with no error bars, variance, or significance tests. Several improvements are small (e.g., Table 1: LLaVA-1.6-13b w/SVP all-split F1 is 87.30 vs. 86.24 for the base; Table 3: MMMU for 13b w/SVP(CVQ) is 35.66 vs. 35.22 for the base). Given the small seed-image count (C=1000) and the stochastic sampling procedure, the wording 'significant improvements' in the Abstract and Section 4 is not supported without multiple seeds with standard deviations or a statistical test over evaluation subsets.","section":"Section 4, Tables 1-5 and Figures 2-5"},{"comment":"The training target is defined by the detector output: the grounded caption is generated from the box list produced by GroundingDINO, and the adapted model is trained to imitate the selected grounded captions. The paper does not measure how often GroundingDINO omits or mislabels objects that the base model correctly mentions, nor does it evaluate SVP on rare, small, or out-of-distribution objects. POPE's object categories overlap heavily with categories GroundingDINO detects well, so the reported object-recall and hallucination gains may not generalize to objects the detector misses. Since the detector is not used at final inference, the adapted model cannot recover such omissions. I recommend adding a detector-recall analysis on the seed images and reporting POPE-style recall split by object frequency or on a rare-object benchmark to substantiate the general claim.","section":"Section 3, step (2) and Section 4 (Hallucination and Object Recall)"}],"minor_comments":[{"comment":"The sentence 'When grounding provides no additional information, q(z|g, c) ≈ p(z|zp, c), and Eq. 3 reduces to 1' is unclear; presumably the ratio q/p reduces to 1, but this should be stated explicitly.","section":"Section 3, after Eq. (3)"},{"comment":"Algorithm 1 line 26 includes a normalization factor 1/|k(c)| in the loss, while Algorithm 2 line 26 omits it. The two objectives should be consistent, or the difference should be explained.","section":"Algorithms 1 and 2"},{"comment":"The entries 'w/oiSVP' and 'w/iSVP' appear to be typographical errors for 'w/o iSVP' and 'w/ iSVP'.","section":"Table 8"},{"comment":"The Introduction says grounding information is 'not shown during the fine-tuning process but is utilized during inference (Fig. 7.(iii))', but Section 4 later clarifies that grounding is used only during inner-loop sampling and not at final inference. This wording should be corrected to avoid confusion.","section":"Section 1, Introduction"},{"comment":"The abstract's '14% average improvement in captioning tasks' and 'up to 12% increase in object recall' should specify the exact datasets and metrics over which the averages are computed, since gains vary substantially across benchmarks in Section 4.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The manuscript reports a solid empirical study, but the Appendix E derivation and the detector-reliance concern are load-bearing. I recommend revision rather than rejection; the authors can address these points with additional analysis and a careful rewrite of the theory section."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"SVP is a genuinely new combination: self-captioning, a grounding model (GroundingDINO) as external feedback, sample scoring via log-ratio or weighted difference, top-k selection, and LoRA adaptation — no curated image-text pairs or preference annotations. That specific pipeline is not in the cited literature, and the paper runs it across several LLaVA variants, model sizes (0.5B to 13B), and ten benchmarks. The gains are consistent and often large: roughly +14 average CIDEr on captioning, up to +12 object recall, and hallucination F1 improvements that let a 7B model approach models five times larger. I believe the results.\n\nThe main weaknesses are statistical and reporting ones. There are no error bars, no significance tests, and a few runs are single-seed. The implementation details say top-10% selection while Figure 7 and Table 11 say 20%; that discrepancy needs fixing. The adaptation objective derivation in Appendix E contains an unjustified step — treating the importance ratio p/q as constant after claiming to reduce variance. The two derivations (MLE and off-policy RL) may agree in spirit, but the current write-up glosses over a real approximation. Also no code is released, which matters for a method whose core value is its training recipe.\n\nThe stress-test concern about detector misses is legitimate and worth taking seriously. The grounded captions are conditioned on GroundingDINO's output, so any object the detector omits is absent from the training target, and the detector isn't used at inference. That said, the paper is not definitionally circular: the grounding model is external, and evaluation is on independent benchmarks. The open question is empirical: does SVP trade recall on rare/small objects for gains on COCO-popular categories? The paper doesn't measure this, and POPE's object lists are COCO-derived, which likely favors detector strengths. The limitation section does acknowledge dependence on grounding quality, but a failure-mode analysis with detector miss rates would strengthen the claim considerably.\n\nOverall, this is a solid, broad empirical paper with a new method and a coherent story. The central argument holds up, but it needs a revised derivation, error bars, and an explicit analysis of grounding failures before I'd rely on its numbers. I would send it to peer review — it deserves referee time, and the concerns are addressable in revision.","headline":"SVP is a genuinely new and broadly evaluated method for VLM alignment without curated pairs; the empirical case is strong, but missing error bars and the unmeasured risk of detector-missed objects need addressing before I'd fully trust the numbers.","tokens_in":36654,"tokens_out":2154,"would_cite":true,"duration_ms":20995,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SVP improves vision-language alignment without curated image-text pairs, using the model's own captions filtered through a pre-trained grounding model.","keywords":["vision-language models","visual grounding","hallucination reduction","self-improvement","image captioning","referring expressions","object recall","feedback-driven alignment"],"falsifier":"Sample a set of images where a stronger detector or human labelers show the grounding model makes systematic false negatives or false positives, and run SVP on those images alone. If hallucination scores get worse or object recall does not improve relative to the base model on images where the detector is wrong, the feedback signal is carrying detector error rather than latent alignment.","tokens_in":35650,"feed_emoji":"🖼️","tokens_out":6588,"duration_ms":58316,"temperature":0.7,"pith_summary":"This paper argues that a vision-language model can be made to see more accurately without any curated image-text pairs or preference annotations. The proposed framework, SVP, uses a small set of plain images, asks the model to caption them, passes those captions through a pre-trained open-set grounding model to obtain object boxes, and then fine-tunes the model only on the grounded captions that score highest. Across ten benchmarks the authors report a 14% average gain in captioning, up to 12% higher object recall, and substantial hallucination reduction, with a 7-billion-parameter model reaching hallucination-avoidance scores comparable to models several times larger. The reason this matters is that alignment data is currently the bottleneck: if the claim holds, alignment becomes a self-supervised loop over images plus one frozen detector.","feed_headline":"Grounding feedback lets a small VLM match models five times larger","feed_subtitle":"Self-captioning plus a detector's boxes cuts hallucinations, doubles referring skill, with no curated pairs or preference labels.","key_machinery":"The central object is the visual projection z, a latent 'generalized caption' bridging image and text. The mechanism is a three-stage loop: (1) prior sampling z_p ~ p_θ(z_p|c); (2) grounding feedback g ← g(z_p, c) from a pre-trained open-set detector; (3) guided sampling z ~ q(z|g,c) that re-prompts the VLM with textual boxes. A scoring function S(z) = log q(z|c,g) - log p_θ(z|c), or a weighted-difference variant, ranks samples so that only grounded captions that surprise or inform the base model are kept. The outer loop is a reweighted maximum-likelihood or off-policy gradient that selects the top-k samples per image and fine-tunes with low-rank adaptation. The grounding text is stripped away before adaptation, so the model never sees boxes at test time.","core_discovery":"On its own terms, the paper's discovery is that grounding feedback can act as a training signal to elicit latent alignment already present in a VLM. The base model first samples a 'visual projection' z, a detailed caption treated as a latent variable. A grounding model g(z_p, c) converts that draft into detected objects and boxes, and the VLM is prompted again with this textual grounding to produce a guided caption z. The divergence between the guided distribution q and the prior p, measured by a log-ratio or weighted-difference score, identifies which samples carry new visual information; the top fraction becomes the fine-tuning set. Because the grounding boxes are used only inside this inner loop and never at inference or in the loss, the improvement in referring, captioning, and hallucination behavior is attributed to better vision-language alignment itself rather than to memorized boxes.","pith_inferences":["Editorial extension: the same sampling-scoring loop could in principle work with any differentiable 'world model' feedback channel, not just object grounding; depth or segmentation maps would be natural alternatives.","The approach implicitly assumes the detector's object vocabulary matches what a caption should mention, so on domains with rare or novel objects, the feedback would suppress valid descriptions the detector cannot ground.","Because grounding is applied only at data-generation time, future improvements in open-set detection should transfer directly to SVP-trained models without retraining them.","A testable extension is to apply SVP to video frames with a tracking-based grounding model to see whether spatial-temporal feedback improves temporal consistency of generated captions."],"forward_implications":["Captioning improves by roughly 14% on average across COCO2017, Flickr30k, and NoCaps, with larger gains on models that start weak.","Referring expression generation can more than double, letting a 7B model approach a 13B model without ever training on bounding boxes.","POPE hallucination F1 rises (for example, from 86.7 to 88.3 for a 7B model), matching or approaching models five times larger.","Object recall rises by up to 12%, while VQA and multitasking scores are roughly preserved.","The method needs only about 1,000 seed images and K=20 samples per image, replacing curated pair annotation with compute."],"supporting_citations":[{"why":"Supplies the pre-trained open-set grounding model whose object boxes and labels act as the feedback signal g(z_p, c).","marker":"[57]"},{"why":"Provides the COCO2014 training images used as the small seed set (1,000 images) for building visual projections.","marker":"[53]"},{"why":"Defines the LLaVA-1.5 architecture family used as base model for experiments and as a high-baseline comparison.","marker":"[54]"},{"why":"Defines LLaVA-NeXT (1.6), the primary base models at 7B/13B that SVP adapts.","marker":"[55]"},{"why":"Defines LLaVA-OneVision, the additional base model family at 0.5B/7B used to test scale transfer.","marker":"[45]"},{"why":"DPO preference optimization; serves as the main preference-based baseline that SVP compares against.","marker":"[70]"},{"why":"STaR self-improvement paradigm that inspires the inner-loop sampling and fine-tuning structure of SVP.","marker":"[108]"},{"why":"Factually augmented RLHF alignment baseline for hallucination reduction and VQA comparison.","marker":"[85]"},{"why":"POPE benchmark that measures hallucination F1 and object recall across adversarial, popular, and random splits.","marker":"[49]"}],"fun_headline_variants":["Small VLM matches 5x larger using grounding feedback","Detector feedback cuts hallucinations, doubles referring skill","Grounding feedback elicits latent alignment with minimal labels","Self-captioning plus detector boxes boosts VLM alignment","Feedback-driven alignment: small VLM rivals 5x larger"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the pre-trained grounding model returns accurate and sufficiently complete object and spatial information for natural images; if it misses, mislabels, or overshoots objects, those errors are baked into the selected grounded captions and then into the fine-tuned model.","fun_headline_variants_meta":{"raw":{"variants":["Small VLM matches 5x larger using grounding feedback","Detector feedback cuts hallucinations, doubles referring skill","Grounding feedback elicits latent alignment with minimal labels","Self-captioning plus detector boxes boosts VLM alignment","Feedback-driven alignment: small VLM rivals 5x larger"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00034,"raw_usage":{"total_tokens":1863,"prompt_tokens":921,"completion_tokens":942,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":537,"completion_tokens_details":{"reasoning_tokens":863}},"tokens_in":537,"tokens_out":942,"duration_ms":8242,"temperature":1.0,"reasoning_tokens":863,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:29:05.465622+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Sample a set of images where a stronger detector or human labelers show the grounding model makes systematic false negatives or false positives, and run SVP on those images alone. If hallucination scores get worse or object recall does not improve relative to the base model on images where the detector is wrong, the feedback signal is carrying detector error rather than latent alignment.","supporting_citations":[{"cited_title":"Rafailov, A","cited_arxiv_id":null,"evidence_quote":"DPO preference optimization; serves as the main preference-based baseline that SVP compares against."},{"cited_title":"Zelikman, Y","cited_arxiv_id":null,"evidence_quote":"STaR self-improvement paradigm that inspires the inner-loop sampling and fine-tuning structure of SVP."}],"review_version":1}