REVIEW 4 major objections 6 minor 43 references
ImageRef-VL: Enabling Contextual Image Referencing in Vision-Language Models
T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read The paper claims that supervised fine-tuning on roughly 7,600 curated samples teaches a vision-language model when and where to insert relevant images into answers, letting an 8B model outscore GPT-4o, Claude-3.5-Sonnet, and open-source…
desk verdict A useful new task and benchmark for contextual image referencing in RAG chatbots, but the evaluation protocol needs tightening before the headline numbers can be taken at face value. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The mechanism that carries the argument is a three-stage data-construction pipeline followed by standard supervised fine-tuning. First, a text-only response is generated from the retrieved document text without images. Second, each image receives a context-aware caption in two stages: a purely visual description, then a supplemental pass that adds surrounding context. Third, an LLM inserts image references at chosen positions in the response, with manual filtering. The resulting samples form CIR-Interleave (7,645 samples with 73,833 images) and CIR-Caption (5,633 samples), mixed with InternVL2's own SFT data before fine-tuning InternVL2-8B and InternVL2-26B with a next-token generative loss. On the evaluation side, CIR-Test labels every potential insertion point with a 0-3 relevance score, from unrelated to perfectly matching, and defines precision, recall3, F1, plus human 5-point Likert ratings of text, image, and overall quality.
What would settle it
Build a fresh CIR-Test whose insertion-point labels are assigned by independent human annotators who never see the training data or the generation code, then re-run ImageRef-VL-8B and InternVL2-26B; if the human-overall gap of 3.52 versus 1.87 shrinks below noise or reverses, the reported capability is not separable from label style.
Extended reading notes
Core claim
The paper's central claim is that the missing ingredient in current VLMs is training signal for when and where to reference an image, not visual understanding or retrieval itself. The authors report that a relatively small volume of manually curated interleaved image-text responses transfers this ability: ImageRef-VL-8B reaches a human overall score of 3.52 on a 5-point scale, above GPT-4o (3.17), Claude-3.5-Sonnet (3.32), and InternVL2-26B (1.87), with a human image-quality score of 4.05. They also claim their evaluation framework is valid: the automated F1 for image placement correlates with the human image score at 0.9854, and the automated text score correlates with the human text score at 0.9033. The intended upshot is that the capability and a measurement for it have been established together.
Load-bearing premise
The headline numbers assume that CIR-Test's 0-3 insertion-point labels and the human 5-point ratings measure contextually appropriate image use, not just resemblance to the authors' own data-construction pipeline.
Editorial extensions
If this is right
- Open-source VLM developers can add contextual image referencing without changing architecture or retrieval: a curated interleaved SFT set on the order of ten thousand samples is claimed to be sufficient.
- End-to-end ImageRef-VL inference is cheaper than the three-stage pipeline, with the paper reporting 1.83 seconds per sample for the 26B model versus 9.77 seconds for the three-stage variant.
- CIR-Test and its F1 metric provide an automatic screening signal for future work, because the paper reports a 0.9854 correlation between F1 and human image scores.
- The authors state that starting from a well-pretrained VLM and including the collected dataset in supervised fine-tuning might yield better results, and that bad cases remain, so the method is a mechanism proof rather than a complete package.
Reading between the lines
- A testable extension the paper does not run: apply the same data-construction recipe to a VLM without InternVL2's interleaved-image pretraining, to see whether the SFT data or the base model is the active ingredient.
- The 0-3 insertion-point labels could be converted into a reward model, letting follow-up work use reinforcement learning to optimize the same image-placement objective and reduce the bad cases the authors acknowledge.
- Since the training data comes from LLM/VLM generation plus human filtering, the pipeline could be bootstrapped: ImageRef-VL's own outputs, after filtering, could seed the next round of training data and lower the manual curation burden.
- An independent-annotation version of CIR-Test, with labels written by people blind to the generation pipeline, would test whether the measured gap reflects contextual appropriateness rather than stylistic mimicry of the training-data generation process.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces Contextual Image Reference, a new capability for vision-language models (VLMs) to insert contextually relevant images from retrieved documents into their responses, and proposes ImageRef-VL, a supervised fine-tuning recipe built on InternVL2. The authors construct two training datasets (CIR-Interleave with 7,645 conversations and CIR-Caption with 5,633 samples) via LLM/VLM generation followed by manual filtering, mix these with the InternVL2-SFT data, and fine-tune InternVL2-8B and 26B. They also construct a test set (CIR-Test) with 456 samples and propose automated metrics (text evaluation, image position precision/recall/F1) plus a 5-point Likert human evaluation. The main claim is that ImageRef-VL-8B achieves an 88% improvement over state-of-the-art open-source VLMs in human overall score (3.52 vs 1.87 for InternVL2-26B), and that it outperforms proprietary closed-source models (GPT-4o, Claude-3.5-Sonnet) on the same human metrics.
Significance. If the claimed results are valid, this is a useful contribution to multimodal conversational AI and retrieval-augmented generation. The paper identifies a real and previously unaddressed gap—contextually appropriate image referencing—and provides a practical SFT method that appears to transfer this skill to open-source VLMs with a large human-evaluation margin. The evaluation framework, including the labeled insertion-point scores and the metric-validation correlations, is a useful starting point for future work. The paper also evaluates computational efficiency, showing an advantage over a three-stage generation pipeline. However, the significance is conditional on the evaluation protocol being trustworthy; the current manuscript lacks sufficient detail to rule out test-set contamination and selection-on-test bias, and the human evaluation is under-reported.
major comments (4)
- [Sections 5.1, 6.1, Table 1] The construction of CIR-Test is not described anywhere. The paper states that CIR-Test is constructed 'as outlined in Section 5' (Section 6.1), but Section 5 only defines the metric formulas and the scoring rubric; it does not describe how the 456 samples, 3,767 images, prompts, or retrieved documents were sampled, nor who assigned the 0-3 position labels. The paper also never reports any overlap or deduplication analysis between CIR-Test and the training sets CIR-Interleave (7,645 conversations, 73,833 images) and CIR-Caption (5,633 samples, 29,558 images). Because both training and test data appear to come from the same retrieved-document conversation format, the headline 88% improvement on human overall evaluation (Table 2) could be inflated by memorized page-image associations rather than by generalized contextual referencing. Please provide a full construction protocol and an overlap analysis (e.g., exact or near-duplicate text and image detection) to support the generalization claim.
- [Section 6.4, Figure 4, Section 6.1] The final hyperparameters (data mixture ratio 1:4 for the 8B model and 1:1 for the 26B model; training steps 950 and 550, respectively) are selected by evaluating the text evaluation score and image F1 on CIR-Test. Since CIR-Test is the same benchmark on which the main comparison in Table 2 is reported, the reported numbers are the best of a small hyperparameter sweep rather than a fixed-protocol prediction, which introduces selection-on-test bias. Please introduce a validation split for hyperparameter tuning, or report results across all sweep points so that the reported values are not cherry-picked from the test set.
- [Section 5.2, Table 2, Figure 3] The human evaluation reports mean 5-point Likert scores (e.g., 3.52 for ImageRef-VL-8B versus 1.87 for InternVL2-26B) without specifying the number of annotators, the annotation instructions, whether annotators were blinded to model identity, or any inter-annotator agreement measure. Without this information and without error bars or significance tests, the 88% improvement claim is not statistically substantiated. Please add these experimental details and report per-item variance or bootstrap confidence intervals for the reported means.
- [Section 6.2, Table 2] The image position evaluation is not applied uniformly across methods. Open-source baselines are evaluated with 'controlled sampling' to insert image references, ImageRef-VL is evaluated end-to-end, and closed-source models are not evaluated on this metric at all. The paper itself acknowledges that 'a fair comparison is not feasible' for the three-stage approaches, yet it still reports these numbers and uses them for hyperparameter selection (Section 6.4) and for the metric-validity Pearson correlation (0.9854). This makes the automated-metric validation and the position scores non-comparable across systems. Please either apply a single evaluation protocol for all models or report the position metric only for end-to-end models and explicitly justify its use in metric validation.
minor comments (6)
- [Sections 4.1, 6.1, Figure 2] The naming is inconsistent: 'IMI-interleave', 'MI-Interleave', 'IMI-Caption', and 'IMI-VL' are used interchangeably; please unify the terminology (e.g., CIR-Interleave, CIR-Caption, ImageRef-VL).
- [Equation (3), Section 5.1] In the Recall3 definition, the indicator 'I(sp = 1)' should read 'I(sp = 3)' to match the text description that follows.
- [Section 6.2, Figure 3] The text mentions 'ImageRef-VL-8B and -32B' when discussing Figure 3, but the experiments only use the 8B and 26B models; please correct the typo.
- [Section 5.1, Equation (2)] The symbol N is used both for the number of test samples in Eq. (1) and for the total number of inserted images in Eq. (2); please use distinct symbols for clarity.
- [Section 6.2] The 'controlled sampling' procedure used for open-source baselines is not described; please provide the prompt template and sampling mechanism so the evaluation is reproducible.
- [Abstract and Section 6.2] The '88% performance improvement' should be qualified as a relative gain on the human overall score over InternVL2-26B on the proposed test set, since it is not an aggregate across all reported metrics.
Circularity Check
No significant circularity: the paper is an empirical SFT recipe whose reported gains are not forced by construction or by a self-citation chain.
full rationale
This paper presents an empirical method: training data for contextual image referencing is generated with existing LLMs/VLMs and manual filtering (Section 4.1), a VLM is fine-tuned with standard SFT (Section 4.2), and performance is measured with automated metrics and human Likert ratings (Section 5). There is no claimed derivation, uniqueness theorem, or fitted parameter that is renamed as a prediction. The 88% improvement over open-source VLMs is an arithmetic comparison of reported human-evaluation scores in Table 2, not a quantity defined into existence by the method. The main concerns noted by a skeptical reader are about evaluation validity — how CIR-Test was constructed, whether it overlaps with the training data, and the lack of annotator agreement or a fixed validation protocol before the hyperparameter sweep in Section 6.4. These are legitimate scientific-validity concerns, but they are not circularity as defined here: no quoted passage shows a test quantity or headline result being reduced by construction to a training input, and the paper does not invoke a load-bearing self-citation or an author-imported uniqueness result. Accordingly, the appropriate circularity score is 0.
Assumptions & free parameters
free parameters (3)
- data mixture ratio (CIR data : InternVL2-SFT) =
1:1 (26B), 1:4 (8B)
- training steps =
550 (26B), 950 (8B)
- image position score thresholds (0/1/2/3) =
categories 3, 2, 1, 0
assumptions (4)
- domain assumption The CIR training data, generated by LLMs/VLMs and manually filtered, is sufficiently high quality to teach appropriate image referencing.
- domain assumption CIR-Test and its insertion-point labels reflect the true task of contextual image referencing.
- domain assumption The LLM-as-judge text score is a valid proxy for response quality even though it does not see the images.
- domain assumption Human Likert ratings are reliable without reported annotator agreement.
Cite this review
Pith. "Pith review of ImageRef-VL: Enabling Contextual Image Referencing in Vision-Language Models." pith.science (2026). https://pith.science/paper/JFIYFLMH
@misc{pith2026250112418,
author = {Pith},
title = {Pith review of: ImageRef-VL: Enabling Contextual Image Referencing in Vision-Language Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/JFIYFLMH}},
note = {Machine review of arXiv:2501.12418}
}
read the original abstract
Vision-Language Models (VLMs) have demonstrated remarkable capabilities in understanding multimodal inputs and have been widely integrated into Retrieval-Augmented Generation (RAG) based conversational systems. While current VLM-powered chatbots can provide textual source references in their responses, they exhibit significant limitations in referencing contextually relevant images during conversations. In this paper, we introduce Contextual Image Reference -- the ability to appropriately reference relevant images from retrieval documents based on conversation context -- and systematically investigate VLMs' capability in this aspect. We conduct the first evaluation for contextual image referencing, comprising a dedicated testing dataset and evaluation metrics. Furthermore, we propose ImageRef-VL, a method that significantly enhances open-source VLMs' image referencing capabilities through instruction fine-tuning on a large-scale, manually curated multimodal conversation dataset. Experimental results demonstrate that ImageRef-VL not only outperforms proprietary models but also achieves an 88% performance improvement over state-of-the-art open-source VLMs in contextual image referencing tasks. Our code is available at https://github.com/bytedance/ImageRef-VL.
Figures
Reference graph
Works this paper leans on
-
[1]
Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. 2024. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219
arXiv 2024
-
[2]
Anthropic. 2024. https://www.anthropic.com/news/claude-3-5-sonnet Claude 3.5 sonnet
2024
-
[3]
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. 2024. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In CVPR, pages 24185--24198
work page 2024
-
[4]
Wenqi Fan, Yujuan Ding, Liangbo Ning, Shijie Wang, Hengyun Li, Dawei Yin, Tat-Seng Chua, and Qing Li. 2024. A survey on rag meeting llms: Towards retrieval-augmented large language models. In KDD, pages 6491--6501
work page 2024
-
[5]
Constanza Fierro, Reinald Kim Amplayo, Fantine Huot, Nicola De Cao, Joshua Maynez, Shashi Narayan, and Mirella Lapata. 2024. Learning to plan and generate text with citations. arXiv preprint arXiv:2404.03381
arXiv 2024
-
[6]
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. 2024. https://arxiv.org/abs/2306.13394 Mme: A comprehensive evaluation benchmark for multimodal large language models . Preprint, arXiv:2306.13394
arXiv 2024
-
[7]
Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. 2023 a . Enabling large language models to generate text with citations. arXiv preprint arXiv:2305.14627
arXiv 2023
-
[8]
Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. 2023 b . Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997
arXiv 2023
Show all 43 references
-
[9]
Shailja Gupta, Rajesh Ranjan, and Surya Narayan Singh. 2024. https://arxiv.org/abs/2410.12837 A comprehensive survey of retrieval-augmented generation (rag): Evolution, current landscape and future directions . Preprint, arXiv:2410.12837
2024 arXiv
-
[10]
Lucas Torroba Hennigen, Shannon Shen, Aniruddha Nrusimha, Bernhard Gapp, David Sontag, and Yoon Kim. 2023. Towards verifiable text generation with symbolic references. arXiv preprint arXiv:2311.09188
2023 arXiv
-
[11]
Chengyu Huang, Zeqiu Wu, Yushi Hu, and Wenya Wang. 2024. Training language models to generate text with citations via fine-grained rewards. arXiv preprint arXiv:2402.04315
2024 arXiv
-
[12]
Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. 2023. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. TOIS
2023
-
[13]
Drew A Hudson and Christopher D Manning. 2019. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, pages 6700--6709
2019
-
[14]
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276
2024 arXiv
-
[15]
Ankur Joshi, Saket Kale, Satish Chandel, and D Kumar Pal. 2015. Likert scale: Explored and explained. British journal of applied science & technology, 7(4):396--403
2015
-
[16]
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. NIPS, 35:22199--22213
2022
-
[17]
Dongyub Lee, Taesun Whang, Chanhee Lee, and Heuiseok Lim. 2023. Towards reliable and fluent large language models: Incorporating feedback learning loops in qa systems. arXiv preprint arXiv:2309.06384
2023 arXiv
-
[18]
Dongfang Li, Zetian Sun, Baotian Hu, Zhenyu Liu, Xinshuo Hu, Xuebo Liu, and Min Zhang. 2024. Improving attributed text generation of large language models via preference learning. arXiv preprint arXiv:2403.18381
2024 arXiv
-
[19]
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, pages 19730--19742. PMLR
2023
-
[20]
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In ICML, pages 12888--12900
2022
-
[21]
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024 a . Improved baselines with visual instruction tuning. In CVPR, pages 26296--26306
2024
-
[22]
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2024 b . Visual instruction tuning. NIPS, 36
2024
-
[23]
Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, et al. 2024. Deepseek-vl: towards real-world vision-language understanding. arXiv preprint arXiv:2403.05525
2024 arXiv
-
[24]
Meta. 2024. https://ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-mobile-devices/ Llama 3.2
2024
-
[25]
Seyed Mahed Mousavi, Simone Alghisi, and Giuseppe Riccardi. 2024. Is your llm outdated? benchmarking llms & alignment algorithms for time-sensitive knowledge. arXiv preprint arXiv:2404.08700
2024 arXiv
-
[26]
Masayasu Muraoka, Ryosuke Kohita, and Etsuko Ishii. 2020. Image position prediction in multimodal documents. In LREC, pages 4265--4274
2020
-
[27]
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. NIPS, 35:27730--27744
2022
-
[28]
Gabriel Poesia, Oleksandr Polozov, Vu Le, Ashish Tiwari, Gustavo Soares, Christopher Meek, and Sumit Gulwani. 2022. Synchromesh: Reliable code generation from pre-trained language models. arXiv preprint arXiv:2201.11227
2022 arXiv
-
[29]
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In ICML, pages 8748--8763
2021
-
[30]
Jiajun Shen, Tong Zhou, Suifeng Zhao, Yubo Chen, and Kang Liu. 2024. Citekit: A modular toolkit for large language model citation generation. arXiv preprint arXiv:2408.04662
2024 arXiv
-
[31]
Marzieh Tahaei, Aref Jafari, Ahmad Rashid, David Alfonso-Hermelo, Khalil Bibi, Yimeng Wu, Ali Ghodsi, Boxing Chen, and Mehdi Rezagholizadeh. 2024. Efficient citer: Tuning large language models for enhanced answer quality and verification. In Findings of NAACL, pages 4443--4450
2024
-
[32]
SM Tonmoy, SM Zaman, Vinija Jain, Anku Rani, Vipula Rawte, Aman Chadha, and Amitava Das. 2024. A comprehensive survey of hallucination mitigation techniques in large language models. arXiv preprint arXiv:2401.01313
2024 arXiv
-
[33]
A Vaswani. 2017. Attention is all you need. NIPS
2017
-
[34]
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. 2024. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409.12191
2024 arXiv
-
[35]
Ziwei Xu, Sanjay Jain, and Mohan Kankanhalli. 2024. Hallucination is inevitable: An innate limitation of large language models. arXiv preprint arXiv:2401.11817
2024 arXiv
-
[36]
Xi Ye, Ruoxi Sun, Sercan Arik, and Tomas Pfister. 2024. Effective large language model adaptation for improved grounding and citation generation. In NAACL, pages 6237--6251
2024
-
[37]
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. 2023. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490
2023 arXiv
-
[38]
Biao Zhang, Barry Haddow, and Alexandra Birch. 2023. Prompting large language model for machine translation: A case study. In ICML, pages 41092--41110
2023
-
[39]
Jiajie Zhang, Yushi Bai, Xin Lv, Wanjun Gu, Danqing Liu, Minhao Zou, Shulin Cao, Lei Hou, Yuxiao Dong, Ling Feng, et al. 2024 a . Longcite: Enabling llms to generate fine-grained citations in long-context qa. arXiv preprint arXiv:2409.02897
2024 arXiv
-
[40]
Jingyu Zhang, Marc Marone, Tianjian Li, Benjamin Van Durme, and Daniel Khashabi. 2024 b . Verifiable by design: Aligning language models to quote from pre-training data. arXiv preprint arXiv:2404.03862
2024 arXiv
-
[41]
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. NIPS, 36:46595--46623
2023
-
[42]
online" 'onlinestring :=
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
-
[43]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.