{"id":"c97d1021-803a-418b-8b8b-1274a08163ea","arxiv_id":"2412.07518","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"HCOENet combines multi-model entity cross-checking with open-set detection to delete hallucinated objects and add descriptions of overlooked traffic objects, with reported F1 gains on a modified POPE evaluation.","lead":"This paper introduces HCOENet, a plug-in correction pipeline that removes hallucinated objects from vision-language model descriptions in traffic scenes and adds descriptions of missed objects. It reports large F1-score gains on a modified POPE benchmark and releases two traffic description datasets.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"POPE evaluation feeds the corrected description to the LVLM as prompt text, leaking which objects exist; the reported F1 gains may measure text compliance rather than reduced visual hallucination.","rationale":"The reader's weakest assumption identifies exactly the same load-bearing concern: the evaluation in Section IV-A.3 is confounded by feeding the corrected text, which encodes external object decisions, to the LVLM during POPE questioning. This is the single most load-bearing issue because every quantitative headline claim (F1 improvements, comparison to GPT-4o, recall gains) derives from this protocol. If the concern lands, the central claim is unsupported, because the measured behavior is text-driven rather than vision-driven. The proposed concrete test directly settles the matter by removing the leaking text prompt and rerunning the identical benchmark. I find no additional concern that is more fundamental: other issues (missing error bars, non-standard CODA-based POPE, the unexplained HCNet drop on MiniGPT-4, overclaimed 'hallucination-free' datasets) are secondary to this evaluation confound. Since the reader already rejected on this basis, my stress-test does not change the verdict.","tokens_in":29492,"tokens_out":2891,"duration_ms":31782,"concrete_test":"Re-run Table V using the standard POPE protocol: provide only the image and the yes/no question, without any description text in the prompt, for both vanilla models and HCOENet-processed models. If the F1 gains for Mini-InternVL-4B and mPLUG-Owl3 largely disappear or reverse, the headline improvements are an artifact of answer leakage through the corrected text.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim rests on a confounded evaluation. In Section IV-A.3, the paper states that 'the descriptions before and after correction, along with the designed questions are provided as prompts to the LVLM.' Thus, when a POPE question like 'Is there a car in the image?' is asked, the LVLM is also given the corrected description, which explicitly lists objects detected by RAM++ and Grounding-DINO-B. The model can answer yes/no by reading the text, not by perceiving the image. The baseline is equally contaminated in the opposite direction: it receives the uncorrected description containing hallucinations, so the LVLM may repeat those false objects. The reported F1 improvements (e.g., +12.58% for Mini-InternVL-4B, +4.28% for mPLUG-Owl3) therefore largely measure how faithfully the LVLM follows the externally filtered text, plus how accurate the external detection pipeline is, not any reduction in the LVLM's own visual hallucination. This also invalidates the 'comparable to GPT-4o' claim, since GPT-4o is evaluated without receiving such a pre-answered text prompt. The method may be a useful post-processing pipeline, but the evidence as presented does not establish the stated claim of eliminating object hallucinations in LVLMs.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes HCOENet, a training-free chain-of-thought correction framework for reducing object hallucinations in LVLM-generated traffic-scene descriptions. The pipeline splits the LVLM's initial response into sentences, extracts entity words via Llama-3.1, verifies each entity with BLIP-2 and two InstructBLIP variants, removes hallucinated entities, and adds descriptions of objects identified by RAM++ and Grounding-DINO-B. The final text is produced by merging the corrected response with the additional object descriptions. The authors report POPE F1 improvements of up to 12.58% for Mini-InternVL-4B and 4.28% for mPLUG-Owl3, claim performance comparable to GPT-4o, and contribute two traffic-scene caption datasets (CODA_desc, nuScenes_desc).","tokens_in":29753,"tokens_out":8287,"duration_ms":75603,"significance":"If valid, the method would be a practical, training-free alternative to model fine-tuning for traffic-scene captioning, and the public code and datasets would be useful community resources. The proposed pipeline is clearly described, and the qualitative examples (Figs. 8 and 10) suggest the method can remove obvious hallucinated objects from long descriptions. However, the central quantitative claim is undermined by the evaluation protocol, which supplies the corrected description as a prompt to the LVLM during POPE testing. As a result, the reported F1 gains are not evidence of reduced visual hallucination in the LVLM; they are evidence of the external pipeline's object-detection accuracy and the LVLM's ability to read the provided text.","major_comments":[{"comment":"The reported POPE improvements are confounded because the corrected description is provided to the LVLM as part of the prompt. The paper states in Section IV-A.3 that \"the descriptions before and after correction, along with the designed questions are provided as prompts to the LVLM.\" In the \"w/Ours\" condition, the corrected text explicitly lists objects verified by RAM++ and Grounding-DINO-B, so the LVLM can produce yes/no answers by reading the text rather than by perceiving the image. In the baseline, the uncorrected description may contain the very hallucinations being probed, prompting false \"yes\" answers. The F1 gains in Table V therefore largely measure the accuracy of the external object-extraction pipeline and the LVLM's instruction-following, not a reduction in the LVLM's visual hallucination. The authors should evaluate the LVLM without injecting the corrected text (e.g., ask the POPE questions directly after correction) or should evaluate the corrected output text against ground-truth object annotations.","section":"Section IV-A.3, Table V"},{"comment":"The comparison with GPT-4o is not controlled. Table X reports that HCOENet-equipped models are competitive with GPT-4o, but GPT-4o is evaluated with the standard POPE protocol (image and question only), while the HCOENet-equipped models receive the externally filtered corrected description as an additional prompt. Any performance gap or parity is therefore not attributable to the LVLM's own visual grounding. The claim of \"comparable descriptive performance at lower cost\" requires an identical evaluation protocol.","section":"Section IV-D.1, Table X"},{"comment":"The modified POPE protocol is not validated. The authors replace the standard MSCOCO images with CODA images and restrict question words to traffic-related terms, but they do not show that this modified benchmark preserves POPE's psychometric properties or that the filtering does not alter the difficulty of negative samples. No error bars or statistical significance tests are reported for any F1-score difference in Tables V–IX, so it is unclear whether the reported improvements exceed run-to-run variability.","section":"Section IV-A.3"},{"comment":"The ablation studies inherit the same confounding. Because every listed variant is evaluated by feeding its output text to the LVLM, the differences in F1-score across Model A–D and across cross-checking variants reflect differences in the textual content of the supplied prompts rather than differences in the model's ability to ground objects in the image. On their own, these ablations cannot support the conclusion that the proposed cross-checking and enhancement stages reduce visual hallucination.","section":"Tables II–IV"},{"comment":"The two released datasets are described as \"hallucination-free semantic descriptions\", but no evaluation of their quality is provided (e.g., human evaluation or comparison to human-written captions). Given that the descriptions are generated by the same external models whose errors the pipeline is meant to correct, this claim is unsupported. The authors should report at least a sample-based human evaluation or a comparison with established captioning metrics.","section":"Section V"}],"minor_comments":[{"comment":"The text introduces the acronym \"HEORNet\" in Section III-A, but the method is named HCOENet throughout the rest of the paper; the acronym should be consistent.","section":"Section III-A"},{"comment":"The parameter count for \"Mini-InternVL-4B w/Ours\" is listed as 47B, but the base model is 4.2B and the additional components are not itemized; please explain how this number is obtained.","section":"Table VIII"},{"comment":"Several figure captions and surrounding text contain garbled characters (e.g., \"和CPT-4o对比的可视化结果：\" in Fig. 8 and hex-like sequences in the full text). These appear to be font-encoding artifacts and must be repaired before submission.","section":"Figures 6–9"},{"comment":"The stage-column alignment in Table II is difficult to follow, particularly for Model B, Model C, and Model D; please clarify which stages are active in each row.","section":"Table II"},{"comment":"The parameter count for Mini-InternVL-4B is given as approximately 4.0B in the text and 4.2B in Table I; please unify the values.","section":"Section IV-A.2"}],"recommendation":"major_revision","confidential_remarks":"The circularity in the POPE protocol is severe enough that the abstract's headline numbers should not be interpreted as measuring LVLM hallucination. The datasets and code release are useful, but the paper needs a fundamentally revised evaluation and a reframing of its claims. I would not recommend rejection outright, as a post-hoc correction pipeline can be a legitimate contribution if properly evaluated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The headline result is not supported by the protocol. In Section IV-A.3 the authors state that the descriptions before and after correction, along with the designed questions, are provided as prompts to the LVLM during POPE evaluation. That means the corrected text, which explicitly lists which objects passed the cross-check, is sitting in the prompt when the model answers 'Is there a car in the image?' The F1 gains (12.58% for Mini-InternVL-4B, etc.) are largely a measure of how faithfully the LVLM follows the externally filtered text, not of reduced visual hallucination. The baseline is equally contaminated in the other direction: it gets the uncorrected description full of hallucinations. And the GPT-4o comparison is unfair, since GPT-4o is not given such a pre-answered text prompt.\n\nThat said, the paper is not without substance. HCOENet is a sensible training-free post-hoc pipeline: entity extraction with Llama-3.1, cross-checking with three BLIP-family models, hallucination correction, then RAM++ plus Grounding-DINO-B to identify and describe critical objects missed in the initial response. The two new datasets, CODA_desc and nuScenes_desc, are a useful resource, and the code and data are public. The ablation studies are extensive: different stages, different cross-checking model sets, different detection backends. The comparisons span a dozen LVLMs across the mPLUG-Owl, LLaVA, and InternVL families, and the inference-time breakdown is honest about the ~22-second overhead.\n\nThe soft spots beyond the confound are real but secondary. There are no error bars or significance tests anywhere. The benchmark is a modified POPE with CODA images and traffic-only question words, but there's no validation that this matches the original POPE's calibration. The result that HCNet alone actually hurts MiniGPT-4 (F1 drops from 43.36 to 34.65 under random sampling) is left unexplained, which is odd given that adding the enhancement stage then recovers it. And the claim that the datasets are 'hallucination-free' is overclaimed, since the pipeline itself is the thing under test and was never validated for completeness on the dataset images.\n\nWho gets value from this? People working on LVLM hallucination correction in traffic settings, and anyone who wants a worked example of how easy it is to accidentally give the answer away in a POPE-style evaluation. I would not cite it for the headline numbers, but the datasets and the ablation matrix might be worth a look after the evaluation is redone.\n\nThis deserves a serious referee. The flaw is load-bearing but fixable: run POPE with image-only questions, or evaluate the corrected description's object consistency separately. The engineering is plausible and the release is generous. I'd send it to review with a clear request to re-verify the central claim.","headline":"The POPE evaluation leaks the corrected object list into the prompt, so the headline F1 gains don't establish reduced hallucination; the paper is a plausible engineering contribution with good ablations and a fixable flaw.","tokens_in":30226,"tokens_out":2951,"would_cite":false,"duration_ms":27884,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"HCOENet is a training-free correction pipeline that lifts vision-language F1-scores on traffic hallucination benchmarks by up to 12.58 percentage points and rivals GPT-4o at far lower cost.","keywords":["vision-language models","hallucination elimination","chain-of-thought correction","traffic scene understanding","POPE benchmark","critical object enhancement","autonomous driving","image-text annotation"],"falsifier":"Run the POPE protocol with the corrected caption paired with a different image of the same scene category; if the model still answers \"yes\" to objects mentioned in the caption that are absent from the paired image, the pipeline's gains come from text compliance rather than image verification.","tokens_in":1899,"feed_emoji":"🚗","tokens_out":1955,"duration_ms":64534,"temperature":0.7,"pith_summary":"The paper argues that object hallucination in vision-language models can be treated as a post-processing problem rather than a training problem: an external chain-of-thought pipeline can read the model's description, verify each named object against the image with several independent vision-language checkers, delete what is not there, and add descriptions of critical objects the model missed. The authors call this pipeline HCOENet and show that applying it to small open models such as Mini-InternVL-4B and mPLUG-Owl3 raises their F1-scores on the POPE object-hallucination benchmark by 12.58% and 4.28%, respectively, while bringing their descriptive quality close to that of GPT-4o at roughly one thirtieth the model size. The larger point is that traffic safety depends on avoiding both hallucinated objects and false negatives, and the proposed method addresses both without fine-tuning.","feed_headline":"Plug-in fix lifts small vision model F1 by 12.58 points","feed_subtitle":"HCOENet strips hallucinated objects and adds missed ones, rivaling GPT-4o at a fraction of the cost.","key_machinery":"The central object is HCOENet, a two-stage correction pipeline. First, the hallucination cross-checking framework splits the model's caption into sentences, extracts entity words with Llama-3.1-8B, and verifies each entity against the image by asking BLIP-2-FlanT5XXL and InstructBLIP-FlanT5XXL \"Is there a {word} in the image?\", with InstructBLIP-Vicuna-13B as tie-breaker; entities judged absent are stripped from the text by an LLM rewrite. Second, the critical-object enhancement framework tags the image with RAM++, verifies each tag with the open-set detector Grounding-DINO-B, and uses BLIP-2 to write one-sentence descriptions for newly confirmed objects, appending them to the corrected caption. The chain-of-thought character comes from the step-by-step entity extraction, cross-check, correction, and enrichment, each stage inspectable.","core_discovery":"The central claim is that a training-free, interpretable correction loop can eliminate most object-level hallucinations and simultaneously recover overlooked traffic participants, and that this is enough to make small LVLMs competitive with much larger proprietary ones on scene description. On the POPE benchmark with CODA traffic images, HCOENet improves Mini-InternVL-4B's average F1-score from 68.88% to 81.46% under random sampling and mPLUG-Owl3's from 73.37% to 77.65% under popular sampling; across ten LVLMs the pipeline raises F1-scores and, especially, recall, which the paper reads as fewer false negatives. The authors further claim that mPLUG-Owl3 plus HCOENet reaches an F1-score of 87.38% on a 200-image random subset, slightly above GPT-4o's 86.77%, at a fraction of the cost, and that the pipeline can serve as an automatic annotator to create the CODA_desc (9,695 pairs) and nuScenes_desc (40,157 pairs) datasets.","pith_inferences":["The reported gains may partly measure how faithfully the LVLM follows a cleaned text prompt rather than how well it grounds perception: the POPE protocol feeds the corrected description and the question together to the LVLM, so a model that trusts the supplied text will answer \"yes\" to objects the pipeline inserted; an experiment presenting the same corrected text with a mismatched image would sep","The same cross-checking machinery could be pointed at attributes and relations, not just object existence; the authors list this as future work, so a concrete next step is to replace the yes/no existence probe with property probes such as \"Is the traffic light red?\"","Because the critical-object stage relies on open-set tagging and detection thresholds, its benefit likely depends on the domain: scenes with many small or distant objects may show larger recall gains, while cluttered scenes may accumulate false positives; this is testable by sweeping the Grounding-DINO-B detection threshold."],"forward_implications":["Smaller LVLMs equipped with HCOENet can reach or exceed the POPE hallucination scores of far larger open models; Mini-InternVL-4B with the pipeline outperforms InternVL2-40B on recall and matches its F1-score in the random setting.","The pipeline raises recall more than precision across models, which the paper interprets as reducing false negatives on traffic participants, a property the authors tie directly to driving-safety risk.","Because HCOENet needs no training, it can be wrapped around any LVLM, including future models, as a plug-and-play correction stage.","The same pipeline can act as an automatic image-text annotator, generating hallucination-free descriptions at scale; the authors release 9,695 CODA_desc and 40,157 nuScenes_desc image-text pairs."],"supporting_citations":[{"why":"Supplies the POPE polling benchmark, the evaluation protocol that all F1-score and accuracy comparisons in the paper are built on.","marker":"[45]"},{"why":"Provides BLIP-2-FlanT5XXL, one of the two primary entity-existence checkers and the model that writes one-sentence descriptions for newly detected objects.","marker":"[14]"},{"why":"Provides InstructBLIP-FlanT5XXL and InstructBLIP-Vicuna-13B, the other entity checker and the tie-breaker used in the cross-checking algorithm.","marker":"[30]"},{"why":"Provides Llama-3.1-8B, which extracts entity words from each sentence and rewrites sentences to remove hallucinatory content.","marker":"[40]"},{"why":"Provides RAM++, the open-set image tagging model that proposes candidate critical objects in the enhancement stage.","marker":"[41]"},{"why":"Provides Grounding-DINO-B, the open-set detector that verifies RAM++ tags and filters them by detection score.","marker":"[42]"},{"why":"Woodpecker is a baseline hallucination-correction method the paper compares against on the POPE benchmark.","marker":"[25]"},{"why":"ReCaption is a fine-tuning-based baseline compared on the same benchmark, representing the data-centric alternative to the paper's training-free approach.","marker":"[32]"},{"why":"The GPT-4 technical report is the high-cost proprietary model the paper compares its descriptive performance and cost against.","marker":"[3]"},{"why":"CODA is the real-world traffic corner-case dataset whose images are used for POPE evaluation and for constructing the CODA_desc annotation set.","marker":"[43]"}],"fun_headline_variants":["Plug-and-play fix cuts hallucinations in traffic vision models","Training-free fix boosts small LVLMs by 12.58 F1 points in traffic","Small vision model rivals GPT-4o after plug-in correction","HCOENet: eliminate hallucinations and enhance traffic scene descriptions","Traffic vision model fix: 12.58 F1 point gain from plug-in"],"cache_read_input_tokens":32384,"weakest_assumption_plain":"The evaluation assumes that asking the LVLM to answer object-existence questions while also giving it the corrected description measures the model's visual hallucination, rather than its readiness to echo the externally filtered text.","fun_headline_variants_meta":{"raw":{"variants":["Plug-and-play fix cuts hallucinations in traffic vision models","Training-free fix boosts small LVLMs by 12.58 F1 points in traffic","Small vision model rivals GPT-4o after plug-in correction","HCOENet: eliminate hallucinations and enhance traffic scene descriptions","Traffic vision model fix: 12.58 F1 point gain from plug-in"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001119,"raw_usage":{"total_tokens":4698,"prompt_tokens":1024,"completion_tokens":3674,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":640,"completion_tokens_details":{"reasoning_tokens":3580}},"tokens_in":640,"tokens_out":3674,"duration_ms":23971,"temperature":1.0,"reasoning_tokens":3580,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T18:45:12.685890+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the POPE protocol with the corrected caption paired with a different image of the same scene category; if the model still answers \"yes\" to objects mentioned in the caption that are absent from the paired image, the pipeline's gains come from text compliance rather than image verification.","supporting_citations":[{"cited_title":"Instructblip: Towards general-purpose vision- language models with instruction tuning,","cited_arxiv_id":null,"evidence_quote":"Provides InstructBLIP-FlanT5XXL and InstructBLIP-Vicuna-13B, the other entity checker and the tie-breaker used in the cross-checking algorithm."},{"cited_title":"Open-set image tagging with multi-grained text supervision,","cited_arxiv_id":null,"evidence_quote":"Provides RAM++, the open-set image tagging model that proposes candidate critical objects in the enhancement stage."},{"cited_title":"Mitigating fine-grained hallucination by fine-tuning large vision-language models with caption rewrites,","cited_arxiv_id":null,"evidence_quote":"ReCaption is a fine-tuning-based baseline compared on the same benchmark, representing the data-centric alternative to the paper's training-free approach."},{"cited_title":"Coda: A real-world road corner case dataset for object detection in autonomous driving,","cited_arxiv_id":null,"evidence_quote":"CODA is the real-world traffic corner-case dataset whose images are used for POPE evaluation and for constructing the CODA_desc annotation set."}],"review_version":1}