Hallucination Elimination and Semantic Enhancement Framework for Vision-Language Models in Traffic Scenarios

Bingzhao Gao; Hongqing Chu; Jianhua Wu; Jiaqi Fan; Quanbo Ge

REVIEW 5 major objections 5 minor 51 references

Hallucination Elimination and Semantic Enhancement Framework for Vision-Language Models in Traffic Scenarios

T0 review · 5 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read HCOENet is a training-free correction pipeline that lifts vision-language F1-scores on traffic hallucination benchmarks by up to 12.58 percentage points and rivals GPT-4o at far lower cost.

desk verdict The POPE evaluation leaks the corrected object list into the prompt, so the headline F1 gains don't establish reduced hallucination; the paper is a plausible engineering contribution with good ablations and a fixable flaw. read the letter →

arxiv 2412.07518 v1 pith:IJGRN43M submitted 2024-12-10 cs.CV

Jiaqi Fan , Jianhua Wu , Hongqing Chu , Quanbo Ge , Bingzhao Gao This is my paper · ORCID

classification cs.CV

keywords vision-languagemodelshallucinationeliminationchain-of-thoughtcorrectiontrafficsceneunderstandingPOPEbenchmarkcriticalobjectenhancementautonomousdrivingimage-textannotation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that object hallucination in vision-language models can be treated as a post-processing problem rather than a training problem: an external chain-of-thought pipeline can read the model's description, verify each named object against the image with several independent vision-language checkers, delete what is not there, and add descriptions of critical objects the model missed. The authors call this pipeline HCOENet and show that applying it to small open models such as Mini-InternVL-4B and mPLUG-Owl3 raises their F1-scores on the POPE object-hallucination benchmark by 12.58% and 4.28%, respectively, while bringing their descriptive quality close to that of GPT-4o at roughly one thirtieth the model size. The larger point is that traffic safety depends on avoiding both hallucinated objects and false negatives, and the proposed method addresses both without fine-tuning.

What carries the argument

The central object is HCOENet, a two-stage correction pipeline. First, the hallucination cross-checking framework splits the model's caption into sentences, extracts entity words with Llama-3.1-8B, and verifies each entity against the image by asking BLIP-2-FlanT5XXL and InstructBLIP-FlanT5XXL "Is there a {word} in the image?", with InstructBLIP-Vicuna-13B as tie-breaker; entities judged absent are stripped from the text by an LLM rewrite. Second, the critical-object enhancement framework tags the image with RAM++, verifies each tag with the open-set detector Grounding-DINO-B, and uses BLIP-2 to write one-sentence descriptions for newly confirmed objects, appending them to the corrected caption. The chain-of-thought character comes from the step-by-step entity extraction, cross-check, correction, and enrichment, each stage inspectable.

What would settle it

Run the POPE protocol with the corrected caption paired with a different image of the same scene category; if the model still answers "yes" to objects mentioned in the caption that are absent from the paired image, the pipeline's gains come from text compliance rather than image verification.

Watch

Extended reading notes

Core claim

The central claim is that a training-free, interpretable correction loop can eliminate most object-level hallucinations and simultaneously recover overlooked traffic participants, and that this is enough to make small LVLMs competitive with much larger proprietary ones on scene description. On the POPE benchmark with CODA traffic images, HCOENet improves Mini-InternVL-4B's average F1-score from 68.88% to 81.46% under random sampling and mPLUG-Owl3's from 73.37% to 77.65% under popular sampling; across ten LVLMs the pipeline raises F1-scores and, especially, recall, which the paper reads as fewer false negatives. The authors further claim that mPLUG-Owl3 plus HCOENet reaches an F1-score of 87.38% on a 200-image random subset, slightly above GPT-4o's 86.77%, at a fraction of the cost, and that the pipeline can serve as an automatic annotator to create the CODA_desc (9,695 pairs) and nuScenes_desc (40,157 pairs) datasets.

Load-bearing premise

The evaluation assumes that asking the LVLM to answer object-existence questions while also giving it the corrected description measures the model's visual hallucination, rather than its readiness to echo the externally filtered text.

Editorial extensions

If this is right

Smaller LVLMs equipped with HCOENet can reach or exceed the POPE hallucination scores of far larger open models; Mini-InternVL-4B with the pipeline outperforms InternVL2-40B on recall and matches its F1-score in the random setting.
The pipeline raises recall more than precision across models, which the paper interprets as reducing false negatives on traffic participants, a property the authors tie directly to driving-safety risk.
Because HCOENet needs no training, it can be wrapped around any LVLM, including future models, as a plug-and-play correction stage.
The same pipeline can act as an automatic image-text annotator, generating hallucination-free descriptions at scale; the authors release 9,695 CODA_desc and 40,157 nuScenes_desc image-text pairs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The reported gains may partly measure how faithfully the LVLM follows a cleaned text prompt rather than how well it grounds perception: the POPE protocol feeds the corrected description and the question together to the LVLM, so a model that trusts the supplied text will answer "yes" to objects the pipeline inserted; an experiment presenting the same corrected text with a mismatched image would sep
The same cross-checking machinery could be pointed at attributes and relations, not just object existence; the authors list this as future work, so a concrete next step is to replace the yes/no existence probe with property probes such as "Is the traffic light red?"
Because the critical-object stage relies on open-set tagging and detection thresholds, its benefit likely depends on the domain: scenes with many small or distant objects may show larger recall gains, while cluttered scenes may accumulate false positives; this is testable by sweeping the Grounding-DINO-B detection threshold.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The headline result is not supported by the protocol. In Section IV-A.3 the authors state that the descriptions before and after correction, along with the designed questions, are provided as prompts to the LVLM during POPE evaluation. That means the corrected text, which explicitly lists which objects passed the cross-check, is sitting in the prompt when the model answers 'Is there a car in the image?' The F1 gains (12.58% for Mini-InternVL-4B, etc.) are largely a measure of how faithfully the LVLM follows the externally filtered text, not of reduced visual hallucination. The baseline is equally contaminated in the other direction: it gets the uncorrected description full of hallucinations. And the GPT-4o comparison is unfair, since GPT-4o is not given such a pre-answered text prompt.

That said, the paper is not without substance. HCOENet is a sensible training-free post-hoc pipeline: entity extraction with Llama-3.1, cross-checking with three BLIP-family models, hallucination correction, then RAM++ plus Grounding-DINO-B to identify and describe critical objects missed in the initial response. The two new datasets, CODA_desc and nuScenes_desc, are a useful resource, and the code and data are public. The ablation studies are extensive: different stages, different cross-checking model sets, different detection backends. The comparisons span a dozen LVLMs across the mPLUG-Owl, LLaVA, and InternVL families, and the inference-time breakdown is honest about the ~22-second overhead.

The soft spots beyond the confound are real but secondary. There are no error bars or significance tests anywhere. The benchmark is a modified POPE with CODA images and traffic-only question words, but there's no validation that this matches the original POPE's calibration. The result that HCNet alone actually hurts MiniGPT-4 (F1 drops from 43.36 to 34.65 under random sampling) is left unexplained, which is odd given that adding the enhancement stage then recovers it. And the claim that the datasets are 'hallucination-free' is overclaimed, since the pipeline itself is the thing under test and was never validated for completeness on the dataset images.

Who gets value from this? People working on LVLM hallucination correction in traffic settings, and anyone who wants a worked example of how easy it is to accidentally give the answer away in a POPE-style evaluation. I would not cite it for the headline numbers, but the datasets and the ablation matrix might be worth a look after the evaluation is redone.

This deserves a serious referee. The flaw is load-bearing but fixable: run POPE with image-only questions, or evaluate the corrected description's object consistency separately. The engineering is plausible and the release is generous. I'd send it to review with a clear request to re-verify the central claim.

Referee Report

5 major / 5 minor

Summary. The manuscript proposes HCOENet, a training-free chain-of-thought correction framework for reducing object hallucinations in LVLM-generated traffic-scene descriptions. The pipeline splits the LVLM's initial response into sentences, extracts entity words via Llama-3.1, verifies each entity with BLIP-2 and two InstructBLIP variants, removes hallucinated entities, and adds descriptions of objects identified by RAM++ and Grounding-DINO-B. The final text is produced by merging the corrected response with the additional object descriptions. The authors report POPE F1 improvements of up to 12.58% for Mini-InternVL-4B and 4.28% for mPLUG-Owl3, claim performance comparable to GPT-4o, and contribute two traffic-scene caption datasets (CODA_desc, nuScenes_desc).

Significance. If valid, the method would be a practical, training-free alternative to model fine-tuning for traffic-scene captioning, and the public code and datasets would be useful community resources. The proposed pipeline is clearly described, and the qualitative examples (Figs. 8 and 10) suggest the method can remove obvious hallucinated objects from long descriptions. However, the central quantitative claim is undermined by the evaluation protocol, which supplies the corrected description as a prompt to the LVLM during POPE testing. As a result, the reported F1 gains are not evidence of reduced visual hallucination in the LVLM; they are evidence of the external pipeline's object-detection accuracy and the LVLM's ability to read the provided text.

major comments (5)

[Section IV-A.3, Table V] The reported POPE improvements are confounded because the corrected description is provided to the LVLM as part of the prompt. The paper states in Section IV-A.3 that "the descriptions before and after correction, along with the designed questions are provided as prompts to the LVLM." In the "w/Ours" condition, the corrected text explicitly lists objects verified by RAM++ and Grounding-DINO-B, so the LVLM can produce yes/no answers by reading the text rather than by perceiving the image. In the baseline, the uncorrected description may contain the very hallucinations being probed, prompting false "yes" answers. The F1 gains in Table V therefore largely measure the accuracy of the external object-extraction pipeline and the LVLM's instruction-following, not a reduction in the LVLM's visual hallucination. The authors should evaluate the LVLM without injecting the corrected text (e.g., ask the POPE questions directly after correction) or should evaluate the corrected output text against ground-truth object annotations.
[Section IV-D.1, Table X] The comparison with GPT-4o is not controlled. Table X reports that HCOENet-equipped models are competitive with GPT-4o, but GPT-4o is evaluated with the standard POPE protocol (image and question only), while the HCOENet-equipped models receive the externally filtered corrected description as an additional prompt. Any performance gap or parity is therefore not attributable to the LVLM's own visual grounding. The claim of "comparable descriptive performance at lower cost" requires an identical evaluation protocol.
[Section IV-A.3] The modified POPE protocol is not validated. The authors replace the standard MSCOCO images with CODA images and restrict question words to traffic-related terms, but they do not show that this modified benchmark preserves POPE's psychometric properties or that the filtering does not alter the difficulty of negative samples. No error bars or statistical significance tests are reported for any F1-score difference in Tables V–IX, so it is unclear whether the reported improvements exceed run-to-run variability.
[Tables II–IV] The ablation studies inherit the same confounding. Because every listed variant is evaluated by feeding its output text to the LVLM, the differences in F1-score across Model A–D and across cross-checking variants reflect differences in the textual content of the supplied prompts rather than differences in the model's ability to ground objects in the image. On their own, these ablations cannot support the conclusion that the proposed cross-checking and enhancement stages reduce visual hallucination.
[Section V] The two released datasets are described as "hallucination-free semantic descriptions", but no evaluation of their quality is provided (e.g., human evaluation or comparison to human-written captions). Given that the descriptions are generated by the same external models whose errors the pipeline is meant to correct, this claim is unsupported. The authors should report at least a sample-based human evaluation or a comparison with established captioning metrics.

minor comments (5)

[Section III-A] The text introduces the acronym "HEORNet" in Section III-A, but the method is named HCOENet throughout the rest of the paper; the acronym should be consistent.
[Table VIII] The parameter count for "Mini-InternVL-4B w/Ours" is listed as 47B, but the base model is 4.2B and the additional components are not itemized; please explain how this number is obtained.
[Figures 6–9] Several figure captions and surrounding text contain garbled characters (e.g., "和CPT-4o对比的可视化结果：" in Fig. 8 and hex-like sequences in the full text). These appear to be font-encoding artifacts and must be repaired before submission.
[Table II] The stage-column alignment in Table II is difficult to follow, particularly for Model B, Model C, and Model D; please clarify which stages are active in each row.
[Section IV-A.2] The parameter count for Mini-InternVL-4B is given as approximately 4.0B in the text and 4.2B in Table I; please unify the values.

Circularity Check

1 steps flagged · score 7.0 of 10

POPE evaluation feeds the corrected description to the LVLM as prompt text, leaking which objects exist; the reported F1 gains may measure text compliance rather than reduced visual hallucination.

self definitional [Section IV-A.3, Hallucination Evaluation Benchmark]
"Besides, the descriptions before and after correction, along with the designed questions are provided as prompts to the LVLM."

The POPE benchmark is designed to probe the LVLM's visual grounding by asking yes/no questions such as 'Is there a car in the image?' with the image as visual evidence. In this paper, the corrected description, which HCOENet generates from external cross-checking and detection models (BLIP-2/InstructBLIP, RAM++, Grounding-DINO-B), is itself included in the prompt alongside the POPE question. The corrected description explicitly states which objects are present and which are absent, so the LVLM can answer the probe by reading the text rather than by perceiving the image.

full rationale

The main circularity is in the POPE evaluation protocol. By supplying the corrected description as a prompt, the paper converts the hallucination probe into a text-following task, so the F1 gains are forced by the external correction pipeline rather than by any change in the LVLM's visual grounding. The comparison with GPT-4o is also asymmetric, since GPT-4o is not given a pre-answered corrected description. No load-bearing self-citation or imported uniqueness theorem appears in the paper; the circularity is confined to the evaluation construction, but it affects the paper's headline quantitative claim, warranting a score of 7.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The central claim rests on trusting off-the-shelf VQA and detection models as ground truth, plus an evaluation protocol that hands the corrected text to the LVLM. No new physical or mathematical entities are introduced.

free parameters (1)

bounding box threshold alpha = 0.35
Ad hoc threshold for Grounding-DINO-B detection, with no sensitivity analysis in Section IV-B.3.

assumptions (5)

domain assumption BLIP-2 and InstructBLIP yes/no answers reliably indicate object existence in the image
Entity cross-checking treats agreement of these models as ground truth (Algorithm 1).
ad hoc to paper Providing the corrected text as a prompt during POPE evaluation does not bias the LVLM's answers
Section IV-A.3 states descriptions before and after correction are provided as prompts; the paper assumes this measures visual grounding.
domain assumption RAM++ plus Grounding-DINO with threshold alpha identifies all critical traffic objects
Critical-object enhancement treats these detections as true positives without human validation.
ad hoc to paper The modified POPE protocol on CODA images with only traffic-related words is a valid hallucination benchmark
Deviates from the standard POPE MSCOCO protocol; no validation against the original benchmark is provided.
domain assumption Llama-3.1-8B few-shot entity extraction and sentence correction accurately preserve meaning
Used in sentence split and correction stages with few-shot prompts (Fig. 4).

how reviews work

0 comments

Cite this review

Pith. "Pith review of Hallucination Elimination and Semantic Enhancement Framework for Vision-Language Models in Traffic Scenarios." pith.science (2026). https://pith.science/paper/IJGRN43M

@misc{pith2026241207518,
  author       = {Pith},
  title        = {Pith review of: Hallucination Elimination and Semantic Enhancement Framework for Vision-Language Models in Traffic Scenarios},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IJGRN43M}},
  note         = {Machine review of arXiv:2412.07518}
}

read the original abstract

Large vision-language models (LVLMs) have demonstrated remarkable capabilities in multimodal understanding and generation tasks. However, these models occasionally generate hallucinatory texts, resulting in descriptions that seem reasonable but do not correspond to the image. This phenomenon can lead to wrong driving decisions of the autonomous driving system. To address this challenge, this paper proposes HCOENet, a plug-and-play chain-of-thought correction method designed to eliminate object hallucinations and generate enhanced descriptions for critical objects overlooked in the initial response. Specifically, HCOENet employs a cross-checking mechanism to filter entities and directly extracts critical objects from the given image, enriching the descriptive text. Experimental results on the POPE benchmark demonstrate that HCOENet improves the F1-score of the Mini-InternVL-4B and mPLUG-Owl3 models by 12.58% and 4.28%, respectively. Additionally, qualitative results using images collected in open campus scene further highlight the practical applicability of the proposed method. Compared with the GPT-4o model, HCOENet achieves comparable descriptive performance while significantly reducing costs. Finally, two novel semantic understanding datasets, CODA_desc and nuScenes_desc, are created for traffic scenarios to support future research. The codes and datasets are publicly available at https://github.com/fjq-tongji/HCOENet.

Figures

Figures reproduced from arXiv: 2412.07518 by the authors.

**Figure 2.** (a) Process of the image-text annotation of HCOENet. (b) The results [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗

**Figure 3.** The overall structure of the proposed HCOENet method. Here, the LVLM represents the LLaVA-1.5, mPLUG-Owl2, MiniGPT-4, Mini-InternVL-4B, [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

Figures from the paper (7 more)

**Figure 4.** Figure 4: Few-shot prompting design for the key entity extraction stage and [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

**Figure 5.** Figure 5: Hallucination evaluation process of the POPE benchmark. [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]

**Figure 6.** Figure 6: The number of false negatives on the LLaVA-1.5-7B and LLaVA-1.5- [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

**Figure 7.** Figure 7: Inference time and F1-score of different InternVL models and LLaVA [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]

**Figure 9.** Figure 9: Inference time of each module in the HCOENet, with a total time of [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]

**Figure 8.** Figure 8: Hallucination elimination results in comparison with the GPT-4o [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]

**Figure 10.** Figure 10: Hallucination correction cases of campus scene images on the mPLUG-Owl3 model. Hallucinatory texts are highlighted in red, and texts generated [PITH_FULL_IMAGE:figures/full_fig_p013_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

51 extracted references · 23 canonical work pages

[1]

Traffic sign interpretation via natural language description,

C. Yang, K. Zhuang, M. Chen, H. Ma, X. Han, T. Han, C. Guo, H. Han, B. Zhao, and Q. Wang, “Traffic sign interpretation via natural language description,” IEEE Trans. Intell. Transp. Syst. , 2024

2024
[2]

Vision-language models can identify distracted driver behavior from naturalistic videos,

M. Z. Hasan, J. Chen, J. Wang, M. S. Rahman, A. Joshi, S. Velipasalar, C. Hegde, A. Sharma, and S. Sarkar, “Vision-language models can identify distracted driver behavior from naturalistic videos,” IEEE Trans. Intell. Transp. Syst. , 2024

work page 2024
[3]

Gpt-4 technical report,

J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al. , “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774 , 2023

arXiv 2023
[4]

Llava- o1: Let vision language models reason step-by-step,

G. Xu, P. Jin, L. Hao, Y . Song, L. Sun, and L. Yuan, “Llava- o1: Let vision language models reason step-by-step,” arXiv preprint arXiv:2411.10440, 2024

arXiv 2024
[5]

Traffic scenario understanding and video captioning via guidance attention captioning network,

C. Liu, X. Zhang, F. Chang, S. Li, P. Hao, Y . Lu, and Y . Wang, “Traffic scenario understanding and video captioning via guidance attention captioning network,” IEEE Trans. Intell. Transp. Syst. , 2023

work page 2023
[6]

Nle-dm: Natural-language explanations for decision making of autonomous driving based on semantic scene understanding,

Y . Feng, W. Hua, and Y . Sun, “Nle-dm: Natural-language explanations for decision making of autonomous driving based on semantic scene understanding,” IEEE Trans. Intell. Transp. Syst. , vol. 24, no. 9, pp. 9780–9791, 2023

work page 2023
[7]

The crossroads of llm and traffic control: A study on large language models in adaptive traffic signal control,

M. Movahedi and J. Choi, “The crossroads of llm and traffic control: A study on large language models in adaptive traffic signal control,” IEEE Trans. Intell. Transp. Syst

work page
[8]

Adapt: Action-aware driving caption transformer,

B. Jin, X. Liu, Y . Zheng, P. Li, H. Zhao, T. Zhang, Y . Zheng, G. Zhou, and J. Liu, “Adapt: Action-aware driving caption transformer,” in 2023 IEEE International Conference on Robotics and Automation (ICRA) , 2023, pp. 7554–7561

work page 2023

Show all 51 references

[9]

Hallucination of multimodal large language models: A survey,

Z. Bai, P. Wang, T. Xiao, T. He, Z. Han, Z. Zhang, and M. Z. Shou, “Hallucination of multimodal large language models: A survey,” arXiv preprint arXiv:2404.18930, 2024

2024 arXiv
[10]

Mitigating hallucination in large multi-modal models via robust instruction tuning,

F. Liu, K. Lin, L. Li, J. Wang, Y . Yacoob, and L. Wang, “Mitigating hallucination in large multi-modal models via robust instruction tuning,” in Proc. Int. Conf. Learn. Represent. (ICLR) , 2023

2023
[11]

Hallucidoctor: Mitigating hallucinatory toxicity in visual instruction data,

Q. Yu, J. Li, L. Wei, L. Pang, W. Ye, B. Qin, S. Tang, Q. Tian, and Y . Zhuang, “Hallucidoctor: Mitigating hallucinatory toxicity in visual instruction data,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2024, pp. 12 944–12 953

2024
[12]

Hallucination augmented contrastive learning for multimodal large language model,

C. Jiang, H. Xu, M. Dong, J. Chen, W. Ye, M. Yan, Q. Ye, J. Zhang, F. Huang, and S. Zhang, “Hallucination augmented contrastive learning for multimodal large language model,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR) , 2024, pp. 27 036–27 046

2024
[13]

Aligning large multimodal models with factually augmented rlhf,

Z. Sun, S. Shen, S. Cao, H. Liu, C. Li, Y . Shen, C. Gan, L.-Y . Gui, Y .-X. Wang, Y . Yanget al. , “Aligning large multimodal models with factually augmented rlhf,” arXiv preprint arXiv:2309.14525 , 2023

2023 arXiv
[14]

Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models,

J. Li, D. Li, S. Savarese, and S. Hoi, “Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models,” in Proc. Int. Conf. Mach. Learn. (ICML) , 2023, pp. 19 730– 19 742

2023
[15]

Improved baselines with visual instruction tuning,

H. Liu, C. Li, Y . Li, and Y . J. Lee, “Improved baselines with visual instruction tuning,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2024, pp. 26 296–26 306

2024
[16]

mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration,

Q. Ye, H. Xu, J. Ye, M. Yan, A. Hu, H. Liu, Q. Qian, J. Zhang, and F. Huang, “mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2024, pp. 13 040–13 051

2024
[17]

Vigc: Visual instruction generation and correction,

B. Wang, F. Wu, X. Han, J. Peng, H. Zhong, P. Zhang, X. Dong, W. Li, W. Li, J. Wang et al. , “Vigc: Visual instruction generation and correction,” in Proc. AAAI Conf. Artif. Intell. (AAAI) , vol. 38, no. 6, 2024, pp. 5309–5317

2024
[18]

A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions,

L. Huang, W. Yu, W. Ma, W. Zhong, Z. Feng, H. Wang, Q. Chen, W. Peng, X. Feng, B. Qin et al. , “A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions,” ACM T. Inform. Syst. , 2023

2023
[19]

Haloquest: A visual hallucination dataset for advancing multimodal reasoning,

Z. Wang, G. Bingham, A. W. Yu, Q. V . Le, T. Luong, and G. Ghiasi, “Haloquest: A visual hallucination dataset for advancing multimodal reasoning,” in Proc. Eur. Conf. Comput. Vis. (ECCV) , 2024, pp. 288– 304

2024
[20]

Incorporating visual experts to resolve the information loss in multimodal large language models,

X. He, L. Wei, L. Xie, and Q. Tian, “Incorporating visual experts to resolve the information loss in multimodal large language models,”arXiv preprint arXiv:2401.03105, 2024

2024 arXiv
[21]

Exploiting semantic reconstruction to mitigate hallucinations in vision-language models,

M. Kim, M. Kim, J. Bae, S. Choi, S. Kim, and B. Chang, “Exploiting semantic reconstruction to mitigate hallucinations in vision-language models,” in Proc. Eur. Conf. Comput. Vis. (ECCV) , 2025, pp. 236–252

2025
[22]

Vdgd: Mitigating lvlm hallucinations in cognitive prompts by bridging the visual perception gap,

S. Ghosh, C. K. R. Evuru, S. Kumar, U. Tyagi, O. Nieto, Z. Jin, and D. Manocha, “Vdgd: Mitigating lvlm hallucinations in cognitive prompts by bridging the visual perception gap,”arXiv preprint arXiv:2405.15683, 2024

2024 arXiv
[23]

Multi-modal hallucination control by visual information grounding,

A. Favero, L. Zancato, M. Trager, S. Choudhary, P. Perera, A. Achille, A. Swaminathan, and S. Soatto, “Multi-modal hallucination control by visual information grounding,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2024, pp. 14 303–14 312

2024
[24]

Vcoder: Versatile vision encoders for multimodal large language models,

J. Jain, J. Yang, and H. Shi, “Vcoder: Versatile vision encoders for multimodal large language models,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2024, pp. 27 992–28 002

2024
[25]

Woodpecker: Hallucination correction for multimodal large language models,

S. Yin, C. Fu, S. Zhao, T. Xu, H. Wang, D. Sui, Y . Shen, K. Li, X. Sun, and E. Chen, “Woodpecker: Hallucination correction for multimodal large language models,” arXiv preprint arXiv:2310.16045 , 2023

2023 arXiv
[26]

Llava-next: Improved reasoning, ocr, and world knowledge,

H. Liu, C. Li, Y . Li, B. Li, Y . Zhang, S. Shen, and Y . J. Lee, “Llava-next: Improved reasoning, ocr, and world knowledge,” January 2024. [Online]. Available: https://llava-vl.github.io/blog/2024-01-30-llava-next/

2024
[27]

How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites,

Z. Chen, W. Wang, H. Tian, S. Ye, Z. Gao, E. Cui, W. Tong, K. Hu, J. Luo, Z. Ma et al. , “How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites,” arXiv preprint arXiv:2404.16821, 2024

2024 arXiv
[28]

Minigpt-4: Enhancing vision-language understanding with advanced large language models,

D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny, “Minigpt-4: Enhancing vision-language understanding with advanced large language models,” arXiv preprint arXiv:2304.10592 , 2023

2023 arXiv
[29]

mplug-owl3: Towards long image-sequence understanding in 15 multi-modal large language models,

J. Ye, H. Xu, H. Liu, A. Hu, M. Yan, Q. Qian, J. Zhang, F. Huang, and J. Zhou, “mplug-owl3: Towards long image-sequence understanding in 15 multi-modal large language models,” arXiv preprint arXiv:2408.04840 , 2024

2024 arXiv
[30]

Instructblip: Towards general-purpose vision- language models with instruction tuning,

W. Dai, J. Li, D. Li, A. M. H. Tiong, J. Zhao, W. Wang, B. Li, P. Fung, and S. C. H. Hoi, “Instructblip: Towards general-purpose vision- language models with instruction tuning,” in Proc. Adv. Neural Inf. Process. Syst. (NeurIPS) , 2023

2023
[31]

Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond,

J. Bai, S. Bai, S. Yang, S. Wang, S. Tan, P. Wang, J. Lin, C. Zhou, and J. Zhou, “Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond,” 2023

2023
[32]

Mitigating fine-grained hallucination by fine-tuning large vision-language models with caption rewrites,

L. Wang, J. He, S. Li, N. Liu, and E.-P. Lim, “Mitigating fine-grained hallucination by fine-tuning large vision-language models with caption rewrites,” in Proc. Int. Conf. Multi. Model. (ICMM) , 2024, pp. 32–45

2024
[33]

Lion: Empowering multimodal large language model with dual-level visual knowledge,

G. Chen, L. Shen, R. Shao, X. Deng, and L. Nie, “Lion: Empowering multimodal large language model with dual-level visual knowledge,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR) , 2024, pp. 26 540–26 550

2024
[34]

Ibd: Alleviating hallu- cinations in large vision-language models via image-biased decoding,

L. Zhu, D. Ji, T. Chen, P. Xu, J. Ye, and J. Liu, “Ibd: Alleviating hallu- cinations in large vision-language models via image-biased decoding,” arXiv preprint arXiv:2402.18476 , 2024

2024 arXiv
[35]

Halc: Object hallucination reduction via adaptive focal-contrast decoding,

Z. Chen, Z. Zhao, H. Luo, H. Yao, B. Li, and J. Zhou, “Halc: Object hallucination reduction via adaptive focal-contrast decoding,” arXiv preprint arXiv:2403.00425, 2024

2024 arXiv
[36]

Analyzing and mitigating object hallucination in large vision- language models,

Y . Zhou, C. Cui, J. Yoon, L. Zhang, Z. Deng, C. Finn, M. Bansal, and H. Yao, “Analyzing and mitigating object hallucination in large vision- language models,” arXiv preprint arXiv:2310.00754 , 2023

2023 arXiv
[37]

Mit- igating object hallucinations in large vision-language models through visual contrastive decoding,

S. Leng, H. Zhang, G. Chen, X. Li, S. Lu, C. Miao, and L. Bing, “Mit- igating object hallucinations in large vision-language models through visual contrastive decoding,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2024, pp. 13 872–13 882

2024
[38]

Logical closed loop: Uncovering object hallucinations in large vision-language models,

J. Wu, Q. Liu, D. Wang, J. Zhang, S. Wu, L. Wang, and T. Tan, “Logical closed loop: Uncovering object hallucinations in large vision-language models,” arXiv preprint arXiv:2402.11622 , 2024

2024 arXiv
[39]

V olcano: mitigating multimodal hallucination through self-feedback guided revision,

S. Lee, S. H. Park, Y . Jo, and M. Seo, “V olcano: mitigating multimodal hallucination through self-feedback guided revision,” arXiv preprint arXiv:2311.07362, 2023

2023 arXiv
[40]

The llama 3 herd of models,

A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan et al. , “The llama 3 herd of models,” arXiv preprint arXiv:2407.21783 , 2024

2024 arXiv
[41]

Open-set image tagging with multi-grained text supervision,

X. Huang, Y .-J. Huang, Y . Zhang, W. Tian, R. Feng, Y . Zhang, Y . Xie, Y . Li, and L. Zhang, “Open-set image tagging with multi-grained text supervision,” arXiv e-prints, pp. arXiv–2310, 2023

2023
[42]

Grounding dino: Marrying dino with grounded pre- training for open-set object detection,

S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, C. Li, J. Yang, H. Su, J. Zhu et al., “Grounding dino: Marrying dino with grounded pre- training for open-set object detection,” arXiv preprint arXiv:2303.05499, 2023

2023 arXiv
[43]

Coda: A real-world road corner case dataset for object detection in autonomous driving,

K. Li, K. Chen, H. Wang, L. Hong, C. Ye, J. Han, Y . Chen, W. Zhang, C. Xu, D.-Y . Yeunget al., “Coda: A real-world road corner case dataset for object detection in autonomous driving,” in Proc. Eur. Conf. Comput. Vis. (ECCV), 2022, pp. 406–423

2022
[44]

mplug-owl: Modularization empowers large language models with multimodality,

Q. Ye, H. Xu, G. Xu, J. Ye, M. Yan, Y . Zhou, J. Wang, A. Hu, P. Shi, Y . Shi et al. , “mplug-owl: Modularization empowers large language models with multimodality,” arXiv preprint arXiv:2304.14178 , 2023

2023 arXiv
[45]

Evaluating object hallucination in large vision-language models,

Y . Li, Y . Du, K. Zhou, J. Wang, W. X. Zhao, and J.-R. Wen, “Evaluating object hallucination in large vision-language models,” arXiv preprint arXiv:2305.10355, 2023

2023 arXiv
[46]

Segment everything everywhere all at once,

X. Zou, J. Yang, H. Zhang, F. Li, L. Li, J. Wang, L. Wang, J. Gao, and Y . J. Lee, “Segment everything everywhere all at once,” vol. 36, 2024

2024
[47]

Yolov9: Learning what you want to learn using programmable gradient information,

C.-Y . Wang, I.-H. Yeh, and H.-Y . M. Liao, “Yolov9: Learning what you want to learn using programmable gradient information,” arXiv preprint arXiv:2402.13616, 2024

2024 arXiv
[48]

Yolov10: Real-time end-to-end object detection,

A. Wang, H. Chen, L. Liu, K. Chen, Z. Lin, J. Han, and G. Ding, “Yolov10: Real-time end-to-end object detection,” arXiv preprint arXiv:2405.14458, 2024

2024 arXiv
[49]

Yolov11: An overview of the key architectural enhancements,

R. Khanam and M. Hussain, “Yolov11: An overview of the key architectural enhancements,” arXiv preprint arXiv:2410.17725 , 2024

2024 arXiv
[50]

Tag2text: Guiding vision-language model via image tagging,

X. Huang, Y . Zhang, J. Ma, W. Tian, R. Feng, Y . Zhang, Y . Li, Y . Guo, and L. Zhang, “Tag2text: Guiding vision-language model via image tagging,” arXiv preprint arXiv:2303.05657 , 2023

2023 arXiv
[51]

Recognize anything: A strong image tagging model,

Y . Zhang, X. Huang, J. Ma, Z. Li, Z. Luo, Y . Xie, Y . Qin, T. Luo, Y . Li, S. Liu et al. , “Recognize anything: A strong image tagging model,” arXiv preprint arXiv:2306.03514 , 2023

2023 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

Traffic sign interpretation via natural language description,

C. Yang, K. Zhuang, M. Chen, H. Ma, X. Han, T. Han, C. Guo, H. Han, B. Zhao, and Q. Wang, “Traffic sign interpretation via natural language description,” IEEE Trans. Intell. Transp. Syst. , 2024

2024

[2] [2]

Vision-language models can identify distracted driver behavior from naturalistic videos,

M. Z. Hasan, J. Chen, J. Wang, M. S. Rahman, A. Joshi, S. Velipasalar, C. Hegde, A. Sharma, and S. Sarkar, “Vision-language models can identify distracted driver behavior from naturalistic videos,” IEEE Trans. Intell. Transp. Syst. , 2024

work page 2024

[3] [3]

Gpt-4 technical report,

J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al. , “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774 , 2023

arXiv 2023

[4] [4]

Llava- o1: Let vision language models reason step-by-step,

G. Xu, P. Jin, L. Hao, Y . Song, L. Sun, and L. Yuan, “Llava- o1: Let vision language models reason step-by-step,” arXiv preprint arXiv:2411.10440, 2024

arXiv 2024

[5] [5]

Traffic scenario understanding and video captioning via guidance attention captioning network,

C. Liu, X. Zhang, F. Chang, S. Li, P. Hao, Y . Lu, and Y . Wang, “Traffic scenario understanding and video captioning via guidance attention captioning network,” IEEE Trans. Intell. Transp. Syst. , 2023

work page 2023

[6] [6]

Nle-dm: Natural-language explanations for decision making of autonomous driving based on semantic scene understanding,

Y . Feng, W. Hua, and Y . Sun, “Nle-dm: Natural-language explanations for decision making of autonomous driving based on semantic scene understanding,” IEEE Trans. Intell. Transp. Syst. , vol. 24, no. 9, pp. 9780–9791, 2023

work page 2023

[7] [7]

The crossroads of llm and traffic control: A study on large language models in adaptive traffic signal control,

M. Movahedi and J. Choi, “The crossroads of llm and traffic control: A study on large language models in adaptive traffic signal control,” IEEE Trans. Intell. Transp. Syst

work page

[8] [8]

Adapt: Action-aware driving caption transformer,

B. Jin, X. Liu, Y . Zheng, P. Li, H. Zhao, T. Zhang, Y . Zheng, G. Zhou, and J. Liu, “Adapt: Action-aware driving caption transformer,” in 2023 IEEE International Conference on Robotics and Automation (ICRA) , 2023, pp. 7554–7561

work page 2023