Pith. sign in

REVIEW 4 major objections 6 minor 14 references

VisCon-100K: Leveraging Contextual Web Data for Fine-tuning Vision Language Models

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Web-page context improves vision-language model fine-tuning, a new 100K dataset shows.

desk verdict A useful dataset paper whose headline 'leaky modality mix' claim is shakier than it looks, but the contextual-data finding and released resources are worth engaging with. read the letter →

arxiv 2502.10250 v2 pith:7CKV3UHJ submitted 2025-02-14 cs.CL cs.CV

Gokul Karthik Kumar , Iheb Chaabane , Kebin Wu This is my paper · ORCID
classification cs.CLcs.CV
keywords VisCon-100Kcontextualcaptioningvision-languagemodelsfine-tuningdataleakymodalitymixwebOBELICSGPT-4V
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces VisCon-100K, a dataset of 100K image-conversation samples built from 45K interleaved image-text web documents in OBELICS. It claims that adding this dataset to the fine-tuning mix of vision-language models improves their scores on six benchmarks, and that the improvement comes from contextual web text rather than from added data volume. The central design choice is a 'leaky modality mix': in each sample, captions and Q&A pairs coexist, with questions answerable from both the image and its contextual caption. If the claims hold, web context is a cheap, scalable supplement to hand-curated VQA data, and the released contextual captioner removes the need for paid captioning APIs at scale.

What carries the argument

The pipeline is the machinery. It filters OBELICS documents to under 2000 tokens, then uses GPT-4V to produce contextual captions conditioned on webpage URL, alt-text, and surrounding text with image placeholders. OpenChat 3.5 converts these captions into free-form and multiple-choice Q&A pairs, deduplicated via AnglE sentence embeddings. The operating mechanism is the 'leaky modality mix': each conversation sample combines the contextual caption with Q&A pairs whose answers draw on both the image content and the caption's contextual information, so the model can integrate visual and textual evidence within one exchange. This controlled cross-modal leakage is what the paper argues drives the performance gains.

What would settle it

Re-run the six data-configuration comparisons with multiple seeds and hold out one benchmark for selection: if the leaky modality mix does not beat contextual captions alone on both SEED and a second benchmark such as MMBench, the leaky-benefit claim is not robust.

Watch

Extended reading notes

Core claim

The paper claims that fine-tuning vision-language models on image conversations generated from interleaved image-text web documents improves performance across multiple benchmarks. It introduces VisCon-100K, a dataset of 100K samples derived from 45K OBELICS web documents, where each sample pairs an image with a GPT-4V-generated contextual caption—a description enriched by the surrounding web text, alt-text, and URL—and OpenChat 3.5-generated free-form and multiple-choice Q&A pairs. The central discovery is the 'leaky modality mix': within one conversation sample, questions are answerable from either the image or its contextual caption, and this mixture outperforms captions alone, Q&A alone, or non-leaky combinations. With ShareGPT4V-7b, the leaky mix scores 67.62 on SEED versus 66.24 for the base model, and the contextual mix averages 60.81 across six benchmarks versus 60.35 for the base and 59.51 for the non-contextual mix. With IDEFICS2-8b, the contextual mix averages 68.21 versus 65.50 for non-contextual and 63.31 for the base model.

Load-bearing premise

The leaky modality mix was chosen as the best data configuration from a single run on the SEED benchmark using ShareGPT4V-7b; if that one comparison is not reproducible, the paper's cross-benchmark gains lack support.

Editorial extensions

If this is right

  • Fine-tuning with VisCon-100K raises ShareGPT4V-7b's average score across six benchmarks to 60.81, above the base model's 60.35.
  • The leaky modality mix (captions plus image- and context-answerable Q&A in one sample) outperforms captions alone, Q&A alone, and non-leaky mixes on SEED.
  • IDEFICS2-8b gains more from contextual data (68.21 average) than from non-contextual data (65.50) or no extra data (63.31).
  • The released contextual captioner generates GPT-4V-style contextual captions without the paid API, enabling larger datasets such as VisCon-1M.
  • Because the non-contextual mix did not help on average, the value of VisCon-100K comes from web context, not from adding data volume alone.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If contextual grounding is the active ingredient, then other context sources (e.g., alt-text alone, surrounding text, or knowledge bases) could yield similar gains when added to fine-tuning data.
  • The leaky mix likely works by preventing the model from learning a single-output bias (always describing vs always answering); this suggests conversation-level data mixing might help other multimodal aptitudes like reasoning and instruction following.
  • The non-contextual mix performing no better than base models hints that many existing VQA datasets are redundant with base fine-tuning data; VisCon-style contextual data adds information rather than volume.
  • A direct test: scale the pipeline to 1M samples with the trained contextual captioner and check whether gains on SEED and MMBench continue to grow.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces VisCon-100K, a vision-language fine-tuning dataset derived from 45K OBELICS web documents. The pipeline uses GPT-4V to generate contextual captions that incorporate surrounding web text and OpenChat 3.5 to convert these captions into free-form and multiple-choice Q&A pairs. Fine-tuning ShareGPT4V-7b and IDEFICS2-8b with VisCon-100K is reported to improve performance across six benchmarks. The authors also propose a 'leaky modality mix' where captions and Q&A appear in the same conversation sample, claiming it outperforms non-leaky combinations. In addition, they release a contextual captioner fine-tuned on the dataset and a larger VisCon-1M dataset.

Significance. If the central claims hold, the manuscript contributes a useful, publicly released dataset and a scalable pipeline for generating contextual vision-language instruction data. The central idea of using web context to enrich captions is timely, and the 'leaky modality mix' is a thought-provoking design choice that could influence future dataset construction. The paper also evaluates its data on two model families, which strengthens the practical relevance. However, the experimental evidence for the headline design recommendation is fragile, as detailed in the major comments. The release of VisCon-100K and VisCon-1M, along with the contextual captioner, is a concrete contribution to the community.

major comments (4)
  1. [Section 5.3, Table 1] The reported advantage of the 'leaky modality mix' over non-leaky variants is confounded with content composition. The leaky condition is the only configuration in Table 1 that includes both contextual captions and Q&A rounds in the same conversation sample; the 'Separated Samples' control splits them into different samples, and the other controls omit captions entirely. Since 'Contextual Captions Alone' improves over the base model (66.9 vs. 66.24) while the Q&A-only conditions are flat or worse, a combined condition could be expected to win simply because it exposes the model to more useful content, not because of cross-modal 'leakage'. No control matches total samples, turns, or tokens across conditions. The causal attribution of the gain to explicit information leakage is therefore unsupported by the presented experiments.
  2. [Section 5.3, McNemar p-values] The statistical significance of the leaky-mix advantage is fragile. The p-value of 0.027 for leaky mix vs. captions alone is the smallest among seven configurations compared on the SEED benchmark using a single training run per configuration. With a Bonferroni correction over seven comparisons, the adjusted p-value is roughly 0.19, which is not significant. The paper also reports no error bars or multiple-seed runs. Because this configuration selection is performed on the same benchmark used for the headline improvement, the model-selection aspect must be accounted for in the significance claim.
  3. [Sections 5.4 and 5.5] All cross-benchmark and cross-model results are obtained with the single configuration selected on SEED (Section 5.3), and each condition is a single run with no variance estimate. This is load-bearing because the paper's key claims—that contextual data helps and that the leaky mix transfers—depend on these results. For ShareGPT4V-7b, the contextual mix outperforms the base by 0.46 on average but wins on only 3 out of 6 benchmarks; without per-benchmark numbers with error bars, this could be within run-to-run noise. The IDEFICS2-8b results show a larger average gap (68.21 vs. 63.31), but again with no variance estimate. The authors should report multiple seeds or, at minimum, detail the variance across benchmark samples and training runs.
  4. [Section 5.6] The evaluation of the contextual captioner is conducted against a held-out set of 1894 GPT-4V-generated contextual captions. Since the fine-tuning data for the captioner was also generated by GPT-4V, this measures agreement with the teacher model, not caption quality in any absolute sense. A small BLEU/ROUGE-L gain of 4/3 points is reported without significance testing, human evaluation, or downstream task impact. The claim that the trained captioner 'facilitates scalable fine-tuning data generation' would be better supported by showing that captions from the captioner lead to similar VLM fine-tuning performance as GPT-4V captions when used in the VisCon pipeline.
minor comments (6)
  1. [Abstract and throughout] The dataset name is written inconsistently as 'VisCon-100K', 'VisCon-100k', and 'VisCon-100k'; please standardize.
  2. [Figure 1] The extracted web text in Figure 1 contains the typo 'a a video'; while this is likely from the original source, it should be noted or corrected if it is part of the figure's illustrative text.
  3. [Section 5.1] The fine-tuning hyperparameters (learning rate, batch size, epochs, optimizer settings) are not given. Stating that the original papers' hyperparameters were followed is insufficient for reproducibility, especially because the dataset is augmented with 100K additional samples.
  4. [Section 5.4, Figure 4] The claim that the contextual mix outperforms on 3 out of 6 benchmarks would be easier to assess if the per-benchmark accuracy numbers were reported in the text or a table, rather than only in a figure with unknown error bars.
  5. [Section 3.2] The qualitative validation of GPT-4V over open-source captioners is described as '100 samples' with no rubric or inter-annotator agreement; a brief description of the evaluation criteria would help.
  6. [Section 8, Limitations] The limitation about 'potentially harmful content' is acknowledged, but the paper does not quantify how much of VisCon-100K might be affected or describe how content filtering was performed; a simple heuristic or example would be informative.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: claims rest on external benchmarks and empirical ablations.

full rationale

VisCon-100K is an empirical dataset-construction paper. Its central claims (contextual web captions improve VLMs; leaky modality mix outperforms alternatives) are established by fine-tuning ShareGPT4V-7b and IDEFICS2-8b on fixed, published benchmarks (SEED, MMBench, MMMU, AI2D, ScienceQA, LLaVA-Bench). No quantity in the paper is defined in terms of the outcome it is used to explain. The leaky modality mix is not a fitted parameter: it is one of seven discrete dataset compositions compared in Table 1, and no equation or construction makes the winning configuration equivalent to the comparison metric. The McNemar p-values and the cross-benchmark averages are empirical outcomes, not derivations. Section 5.6 evaluates the in-house contextual captioner against held-out GPT-4V captions; this measures imitation fidelity rather than external VLM performance, but it is explicitly a teacher-student consistency check, not a claim that the captioner's outputs are definitionally 'high quality,' so no circularity is introduced. No load-bearing argument rests on a self-citation: citations to OBELICS, ShareGPT4V, IDEFICS2, LLaVA, and benchmarks are external prior work, and none is invoked as a uniqueness theorem or ansatz that predefines the paper's findings. The principal methodological risks (single-run comparisons, configuration selection on SEED without multiple-comparison correction, and the confound between leakage and content volume in Table 1) are experimental-design concerns rather than circularity.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The paper is an empirical dataset paper, so the ledger is dominated by design choices and domain assumptions about data quality and benchmark validity rather than mathematical axioms. The main unverified premises are caption accuracy and the generalizability of the SEED-based configuration choice.

free parameters (4)
  • Maximum document token length = 2000
    Used to filter OBELICS documents; hand-chosen, affects which images are included. Section 3.1.
  • Q&A conversation rounds = 3 to 5
    Prompt constraint for OpenChat generation; affects dataset structure. Tables 3 and 4.
  • Deduplication minimum count per Q&A type = not reported
    The deduplication step maintains a minimum count for free-form and multiple-choice types but the value is not specified. Section 3.4.
  • Evaluation sample for model selection = 100 samples
    Qualitative selection of GPT-4V and OpenChat was based on 100 samples; this is a small hand-picked evaluation. Sections 3.2 and 3.3.
assumptions (3)
  • domain assumption GPT-4V produces accurate contextual captions from web context without hallucinating details not present in the image.
    The whole dataset depends on caption quality; the prompt instructs GPT-4V to ignore non-image info, but no quantitative verification is provided. Sections 3.2 and Appendix Table 2.
  • domain assumption Fine-tuning on data derived from OBELICS does not cause harmful overfitting or contamination with the evaluation benchmarks.
    The authors note IDEFICS2 was pretrained on OBELICS (Section 5.5) but do not test for benchmark overlap. This is a load-bearing premise for the reported improvements.
  • ad hoc to paper The optimal data composition on the SEED benchmark with ShareGPT4V-7b transfers to other benchmarks and to IDEFICS2-8b.
    The leaky modality mix was selected using SEED only, then applied to all other evaluations; no separate validation of the choice was performed. Section 5.3.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VisCon-100K: Leveraging Contextual Web Data for Fine-tuning Vision Language Models." pith.science (2026). https://pith.science/paper/7CKV3UHJ

@misc{pith2026250210250,
  author       = {Pith},
  title        = {Pith review of: VisCon-100K: Leveraging Contextual Web Data for Fine-tuning Vision Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7CKV3UHJ}},
  note         = {Machine review of arXiv:2502.10250}
}
read the original abstract

Vision-language models (VLMs) excel in various visual benchmarks but are often constrained by the lack of high-quality visual fine-tuning data. To address this challenge, we introduce VisCon-100K, a novel dataset derived from interleaved image-text web documents. Our approach transforms 45K web documents from the OBELICS dataset into 100K image conversation samples. We utilize GPT-4V to generate image-contextual captions and OpenChat 3.5 model to convert these captions into diverse free-form and multiple-choice question-answer pairs. Integrating this dataset for fine-tuning considerably enhances VLM performance across multiple benchmarks. Unlike methods that focus solely on fine-grained visual content, our approach leverages accompanying web context, yielding superior results. We also discover that a 'leaky modality mix', where conversation samples contain questions answerable from both the image and its contextual caption, outperforms non-leaky combinations of captions and Q&A pairs. VisCon-100k dataset shows strong performance with two popular VLM approaches: text-only large language model (LLM) aligned with a vision encoder using image captions data (ShareGPT4V-7b) and multimodally pretrained LLM (IDEFICS2-8b) using interleaved image-text data. In addition to releasing the VisCon-100K dataset, we provide a contextual captioner trained on this dataset, facilitating scalable fine-tuning data generation for future research and open-source applications. Using the same pipeline, but substituting our trained contextual captioner for GPT-4V, we also release the larger VisCon-1M dataset.

Figures

Figures reproduced from arXiv: 2502.10250 by the authors.

Figure 1
Figure 1. An OBELICS web document with generated contextual and non-contextual captions. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Data generation pipeline for creating the VisCon-100K dataset. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Examples from the VisCon-100K dataset. The text, highlighted in red, shows contextual [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (4 more)
Figure 5
Figure 5. Figure 5: Performance of IDEFICS2-8b model across 6 benchmarks for different data configu￾rations The performance for each configuration is shown in [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Textual Characteristics of Source Context and their transformed Contextual Captions [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 7
Figure 7. Figure 7: Distributions of Q&A Types The plot in Figure 7a shows that the majority of samples contain 4 free-form Q&A pairs, which aligns with the dataset’s design to provide detailed conversational data. Figure 7b illustrates most samples also contain 4 multiple-choice Q&A pair…
Figure 8
Figure 8. Figure 8: Word Cloud of Captions. suggests a focus on detailed visual descriptions, which is critical for enhancing visual understanding in VLMs. Few examples of the VisCon-100K dataset in [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

14 extracted references · 3 canonical work pages

  1. [3]

    Mistral 7b

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825,

    arXiv

  2. [6]

    Angle-optimized text embeddings

    Xianming Li and Jing Li. Angle-optimized text embeddings. arXiv preprint arXiv:2309.12871 ,

    arXiv

  3. [7]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pp. 26296–26306, 2024a. 11 Accepted at PAKDD 2025 Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural informatio...

    arXiv 2025

  4. [8]

    FuseCap: Leveraging Large Language Models for Enriched Fused Image Captions

    Noam Rotstein, David Bensaid, Shaked Brody, Roy Ganz, and Ron Kimmel. Fusecap: Lever- aging large language models to fuse visual data into enriched image captions. arXiv preprint arXiv:2305.17718,

    work page Pith review arXiv

  5. [9]

    Gemma: Open models based on gemini research and technology

    Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivi `ere, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295,

    arXiv

  6. [10]

    Llama 2: Open founda- tion and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288,

    arXiv

  7. [11]

    Openchat: Ad- vancing open-source language models with mixed-quality data.arXiv preprint arXiv:2309.11235,

    Guan Wang, Sijie Cheng, Xianyuan Zhan, Xiangang Li, Sen Song, and Yang Liu. Openchat: Ad- vancing open-source language models with mixed-quality data.arXiv preprint arXiv:2309.11235,

    arXiv

  8. [12]

    Meta-transformer: A unified framework for multimodal learning

    Yiyuan Zhang, Kaixiong Gong, Kaipeng Zhang, Hongsheng Li, Yu Qiao, Wanli Ouyang, and Xi- angyu Yue. Meta-transformer: A unified framework for multimodal learning. arXiv preprint arXiv:2307.10802,

    arXiv

Show all 14 references
  1. [13]

    Minigpt-4: En- hancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592,

    arXiv

  2. [14]

    (a) Distribution of Number of Tokens in the Source Context

    12 Accepted at PAKDD 2025 A A PPENDIX A.1 V ISCON-100K P ROPERTIES This section presents various data properties of the VisCon-100K dataset. (a) Distribution of Number of Tokens in the Source Context. (b) Distribution of Word Overlap Ratio between Con- textual Caption and Sour...

    2025

  3. [2016]

    Obelics: An open web-scale filtered dataset of interleaved image-text documents

    Hugo Laurenc ¸on, Lucile Saulnier, L´eo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander Rush, Douwe Kiela, et al. Obelics: An open web-scale filtered dataset of interleaved image-text documents. Advances in Neural Information...

    arXiv

  4. [2022]

    Allava: Harnessing gpt4v-synthesized data for a lite vision-language model

    Guiming Hardy Chen, Shunian Chen, Ruifei Zhang, Junying Chen, Xiangbo Wu, Zhiyi Zhang, Zhi- hong Chen, Jianquan Li, Xiang Wan, and Benyou Wang. Allava: Harnessing gpt4v-synthesized data for a lite vision-language model. arXiv preprint arXiv:2402.11684,

    arXiv

  5. [2023]

    A diagram is worth a dozen images

    Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In Computer Vision–ECCV 2016: 14th Euro- pean Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14 , pp. 235–251....

    2016

  6. [2024]

    Sharegpt4v: Improving large multi-modal models with better captions

    Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.12793,

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.