Pith. sign in

REVIEW 4 major objections 4 minor 71 references

MM-GEN: Enhancing Task Performance Through Targeted Multimodal Data Curation

T0 review · 4 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Synthetic task-specific data from a stronger VLM lifts small vision-language models by up to 29% on specialized tasks.

desk verdict Sensible synthetic-data pipeline, but headline gains for ChartQA and SpatialMap rest on validation-set leakage; fix the eval protocol and the paper becomes much more credible. read the letter →

arxiv 2501.04155 v1 pith:LSNNIBDS submitted 2025-01-07 cs.CV cs.CLcs.LG

Siddharth Joshi , Besmira Nushi , Vidhisha Balachandran , Varun Chandrasekaran , Vibhav Vineet , Neel Joshi , Baharan Mirzasoleiman This is my paper · ORCID
classification cs.CVcs.CLcs.LG
keywords vision-languagemodelssyntheticdatagenerationcurationtask-specificfine-tuningin-contextlearningperplexityfilteringchartunderstandingspatialreasoning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the main reason small vision-language models underperform on specialized tasks such as chart reading, diagram understanding, and spatial reasoning is the training text: generic captions omit the specific details the task requires. It proposes MM-GEN, an automated three-stage pipeline that uses a stronger VLM to generate task-relevant question-answer pairs for candidate images, with a small set of reference examples from the target task specifying what counts as relevant. Fine-tuning Llava-1.5 (7B) on this synthetic data yields absolute accuracy gains of 15% on chart understanding, 14% on diagram understanding, and 29% on spatial reasoning, and on AI2D the gain is 1.6 times that of human-curated i.i.d. training data. If these numbers hold, task-specific synthetic data can substitute for expensive human curation across specialized multimodal benchmarks, at the cost of only a handful of reference examples and a pool of candidate images.

What carries the argument

The load-bearing mechanism is reference-sample-conditioned in-context generation: a stronger VLM is given one image-question-answer triple from the task's reference set and asked to generate similar, diverse question-answer pairs for a new candidate image. The reference example, rather than a hand-written task description, is what communicates which visual details matter, and the text is generated separately for each image-type subgroup obtained via CLIP partitioning. The second mechanism is a midpoint-perplexity filter: using the target VLM's own masked-language-modeling perplexity, MM-GEN discards low-perplexity examples the model already knows and high-perplexity examples that are likely noisy, keeping the middle band that adds new training signal.

What would settle it

Re-run the MM-GEN pipeline on ChartQA and SpatialMap with reference samples drawn from the training split or from a separate task-survey dataset, evaluate on a truly held-out test set, and compare the gains to the paper's reported 15% and 29%; if the gains shrink substantially, the validation-set conditioning is the source of the improvement.

Watch

Extended reading notes

Core claim

MM-GEN's central discovery is that the target task can be specified to a teacher VLM data-centrically, by showing it one reference image-question-answer example, rather than by writing a natural-language task description. The pipeline first partitions both the reference set and the candidate image pool into subgroups by image type using CLIP zero-shot classification, so that each generated example is conditioned on a reference of the same type. It then prompts a stronger VLM (GPT-4o) with a reference sample and a candidate image to produce a diverse set of task-specific questions and answers, and finally filters the generated pairs by the perplexity of the target VLM, retaining the middle-perplexity band that represents non-trivial but learnable examples. The paper reports that models fine-tuned on the resulting data improve by 15%, 14%, and 29% absolute on ChartQA, AI2D, and SpatialMap respectively, outperform task-agnostic captions and natural-language-description baselines, and match or exceed the gains of human-curated i.i.d. training data on AI2D.

Load-bearing premise

The results assume that conditioning the synthetic training data on reference samples drawn from the evaluation benchmarks' validation sets does not inflate the accuracy later reported on those same benchmarks.

Editorial extensions

If this is right

  • If the reported gains are real, any specialized VLM task can be improved automatically with just a small reference set and a pool of candidate images, removing the need for manual task-specific dataset construction.
  • The perplexity filter cuts the generated data volume by up to 50% with no or minimal loss in accuracy, so half as much fine-tuning compute can deliver essentially the same improvement.
  • The method also improves the larger Llava-1.5 (13B) model and, when data from all three tasks is combined, improves all three benchmarks without degrading performance on a general multimodal benchmark (MMMU), suggesting it can be applied per-task and merged.
  • The ablation showing that one in-context reference sample works better than three suggests that current VLMs' multi-image limitations make data-centric task specification both cheaper and more effective than scaling the number of demonstrations.
  • The comparison with natural-language task descriptions indicates that data-centric task specification reliably captures task-relevant details that text descriptions omit, extending the paper's case study on MS COCO captions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reference samples for ChartQA and SpatialMap are taken from the validation sets of those benchmarks, and accuracy is reported on the same benchmarks; the reported gains may therefore partly reflect conditioning the synthetic training distribution on the evaluation distribution, and a cleaner estimate would draw references from a separate split.
  • The approach could be tested on other detail-hungry multimodal tasks, such as medical imaging or diagrams in specialized documents, where reference examples exist but human-curated task data is scarce.
  • The perplexity filter's success suggests that the target model's own prediction confidence is a useful selection signal; this principle could be explored for self-training or curriculum design, where the model generates its own training examples and selects them by its own uncertainty.
  • Because the teacher is a frontier model (GPT-4o), an open question is how the gains scale as the teacher becomes weaker; if a smaller open-source VLM can serve as teacher, the method becomes far more accessible.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes MM-GEN, a pipeline for generating task-specific synthetic question-answer annotations for images using a stronger VLM (GPT-4o) as a teacher. The pipeline has three stages: partitioning candidate images and reference samples by image type via CLIP, generating annotations by conditioning the teacher on in-context reference samples, and filtering synthesized examples by middle perplexity using the student VLM. The authors fine-tune LLaVA-1.5 7B and 13B on the resulting synthetic data for ChartQA, AI2D, and SpatialMap, and report large absolute gains over the base model (15%, 14%, and 29% for the 7B model), with comparisons against task-agnostic captions, task descriptions without reference images, and a human-curated or programmatic i.i.d. skyline. They also report ablations and a control experiment on MMMU.

Significance. If the headline results are valid, the paper makes a useful practical contribution: a largely automated, task-targeted synthetic data pipeline with modest human input and a released codebase. The AI2D experiment, which draws reference samples from the training split, is a genuinely positive signal that the method can improve a specialized VLM benchmark, and the perplexity-based filtering idea is simple and potentially reusable. However, as presented, the evaluation protocol for two of the three headline tasks conditions the teacher on the same validation benchmarks used for scoring, so the 15%, 14%, and 29% gains cannot be accepted as evidence of transferable data curation without a re-run under a clean held-out protocol.

major comments (4)
  1. [Sec. 5, Fig. 5] The evaluation protocol leaks information from the benchmark used for scoring into the training-data generation process. For ChartQA, the reference samples are 'the validation set of ChartQA' and the evaluation is reported on the same benchmark; for SpatialMap, the reference samples are 'the validation set of SpatialMap' and the skyline is generated with the same code that creates the evaluation set. Because the teacher prompt in Appendix A contains the reference samples verbatim, the synthetic training distribution is conditioned on the exact question/answer style and, for ChartQA, on the exact images that define the evaluation split. This does not make the reported 15%, 14%, and 29% absolute gains valid evidence for transferable data curation. The central claim can be restored by regenerating with reference samples drawn from a disjoint training or held-out split, as already done for AI2D, and reporting evaluation on the official test/holdout set; at minimum, an analysis of the overlap between reference samples and evaluation samples is needed.
  2. [Sec. 5.1, Fig. 5(c)] The claim in Sec. 5.1 that MM-GEN can 'close the gap with or exceed the skyline performance across all three tasks' is contradicted by the reported numbers on SpatialMap: the filtered MM-GEN result is 0.55 versus a 0.98 skyline, i.e., only 0.4x of the skyline's absolute improvement (0.29 over 0.72). This overstatement should be corrected; the results should be described as a partial gap reduction on ChartQA and SpatialMap and a full closing only on AI2D.
  3. [Table 1] The statement that training on MM-GEN data 'does not hurt performance on other tasks' is not supported by Table 1: MMMU accuracy drops from 35.8 (base) to 33.6 when using the ChartQA-generated data and to 34.1 when using the SpatialMap-generated data. Please either report variance and statistical significance, soften the claim to 'does not substantially degrade,' or explain why these drops are within noise.
  4. [Sec. 5, Fig. 5] All reported accuracies appear to be single-run validation numbers with no error bars or seeds. Differences that drive the ranking of filtering and ablation variants (e.g., 33.0 vs. 32.8 in Table 3, and 0.58 vs. 0.55 in Fig. 5c) are within the range that typically varies across fine-tuning runs. Please report the number of runs and standard deviation or confidence intervals for the headline comparisons, or identify which numbers are averaged.
minor comments (4)
  1. [Sec. 5.2, Table 3] The text says partitioning contributes 'a non-trivial 2%' to the improvement, but Table 3 shows 33.0 - 31.6 = 1.4 percentage points; please correct the number.
  2. [Sec. 4.2.3] The filtering step is described only as selecting 'middle perplexity' examples with 50% retained; please specify the exact percentile interval (e.g., 25th-75th) and how ties or boundary cases are handled, so the pipeline is reproducible.
  3. [Appendix A] The prompt text contains typos such as 'understadning' and 'Refererence Sample'; please proofread the appendix.
  4. [Abstract and Conclusion] There are minor wording and consistency issues, including 'taksks' in the conclusion and inconsistent capitalization of 'Llava/LLava'; please fix these throughout.

Circularity Check

1 steps flagged · score 4.0 of 10

Headline gains on ChartQA and SpatialMap are partially circular because MM-GEN's teacher is conditioned on the validation split of the same benchmark used for evaluation.

  1. fitted input called prediction [Section 5, 'Chart Understanding and Reasoning' and 'Spatial Reasoning on Maps'; Appendix A prompt; Fig. 5.]
    "Reference Samples: the validation set of ChartQA ( ≈ 1K samples); ... Candidate Image Pool: 15K images of charts taken from the ChartQA training set. ... Reference Samples: the validation set of SpatialMap; ... Candidate Image Pool: 1K images of maps retrieved from DataComp-Small [16] using CLIP embedding search. ... Given example image-question-answer tuples, your task is to generate diverse high-quality question-answer pairs relevant to this skill similar to the provided examples."

    The teacher prompt instructs GPT-4o to generate Q/A pairs 'similar to the provided examples.' For ChartQA and SpatialMap, those provided examples are the validation split of the benchmark on which Fig. 5 reports accuracy. By construction, the synthetic training distribution is a function of the evaluation distribution: question style, answer format, and image-type mix are conditioned on validation-set samples from the same benchmark. The reported 15% and 29% gains are therefore partly a measure of how well MM-GEN expands the evaluation distribution, not a clean transferable estimate.

full rationale

This is an empirical systems paper rather than a mathematical derivation, so the circularity question hinges on evaluation design. The only load-bearing circularity is the use of the validation split of ChartQA and SpatialMap as reference samples for the teacher, followed by evaluation on those same benchmarks; this makes part of the reported improvement an artifact of conditioning on the evaluation distribution. The AI2D setup uses training-set references and still shows gains, which provides some independent evidence for the method. The perplexity filter is an external data-selection heuristic (from Marion et al.) applied before evaluation and is not itself circular. There are no self-citations used as load-bearing evidence, no uniqueness theorems, and no renamed known results. The central derivation chain (partition, generate with reference samples, filter) is otherwise self-contained; the score reflects the partial circularity of the two headline evaluations, not a fully circular method.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The framework rests on several hand-chosen budgets (50% retention, number of in-context samples, reference set sizes, epoch counts) and on assumptions that GPT-4o produces accurate task-specific questions, that CLIP partitioning assigns images to correct types, and that middle-perplexity examples are the most useful training signal. No new physical or conceptual entities are introduced, and no derivation is attempted; the contribution is empirical.

free parameters (5)
  • retained_fraction_r = 0.5
    Middle-perplexity selection keeps 50% of generated data for all tasks; chosen by hand "for simplicity" in Sec 4.2.3 and used in all main results.
  • in_context_sample_count = 1
    One reference sample per generation prompt; ablation in Table 3 shows 3 in-context samples hurt performance, so 1 is a hand-chosen design point.
  • reference_set_size_n = ChartQA ~1000 (validation), AI2D 100, SpatialMap ~validation
    Set per task in Sec 5; a 10x smaller set is ablated only on ChartQA, so the effect on other tasks is unknown.
  • epochs_per_task = 6 (ChartQA), 6 (AI2D), 3 (SpatialMap)
    Tuned per task so training loss converges (App D); the tuning protocol is not specified, so this is a free choice affecting results.
  • generated_budget_Ngen = 150K (ChartQA), 100K (AI2D), 50K (SpatialMap)
    Total number of synthetic samples generated before filtering; chosen per task in Sec 5 without a sensitivity study.
assumptions (5)
  • domain assumption GPT-4o generates accurate, diverse, and well-formed question-answer pairs for candidate images when given one in-context reference sample.
    Invoked in Sec 4.2.2 and App A; no verification step is applied to the generated answers.
  • domain assumption CLIP zero-shot text-image matching assigns each image to the correct image type subgroup (e.g., bar chart vs line chart).
    Used in Sec 4.2.1 Eq. (1) to partition Sref and Vpool; mispartitioning would mix reference styles across subgroups.
  • domain assumption Middle-perplexity examples, as measured by the target VLM, are the most useful training signal and low/high perplexity examples are already learned or noisy.
    Adapted from LLM pretraining [31] in Sec 4.2.3; the paper does not validate this assumption on VLM fine-tuning.
  • ad hoc to paper Reference samples from the validation set are representative of the task and do not give the generation process information specific to the evaluation split.
    Sec 5 uses the ChartQA and SpatialMap validation sets as reference samples and also evaluates on those sets; this assumption is questionable.
  • domain assumption Fine-tuning on synthetic task-specific QA data transfers to the real test distribution.
    The entire evaluation in Sec 5 assumes that improvements measured on the chosen benchmarks reflect task capability rather than overfitting to synthetic question styles.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MM-GEN: Enhancing Task Performance Through Targeted Multimodal Data Curation." pith.science (2026). https://pith.science/paper/LSNNIBDS

@misc{pith2026250104155,
  author       = {Pith},
  title        = {Pith review of: MM-GEN: Enhancing Task Performance Through Targeted Multimodal Data Curation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LSNNIBDS}},
  note         = {Machine review of arXiv:2501.04155}
}
read the original abstract

Vision-language models (VLMs) are highly effective but often underperform on specialized tasks; for example, Llava-1.5 struggles with chart and diagram understanding due to scarce task-specific training data. Existing training data, sourced from general-purpose datasets, fails to capture the nuanced details needed for these tasks. We introduce MM-Gen, a scalable method that generates task-specific, high-quality synthetic text for candidate images by leveraging stronger models. MM-Gen employs a three-stage targeted process: partitioning data into subgroups, generating targeted text based on task descriptions, and filtering out redundant and outlier data. Fine-tuning VLMs with data generated by MM-Gen leads to significant performance gains, including 29% on spatial reasoning and 15% on diagram understanding for Llava-1.5 (7B). Compared to human-curated caption data, MM-Gen achieves up to 1.6x better improvements for the original models, proving its effectiveness in enhancing task-specific VLM performance and bridging the gap between general-purpose datasets and specialized requirements. Code available at https://github.com/sjoshi804/MM-Gen.

Figures

Figures reproduced from arXiv: 2501.04155 by the authors.

Figure 1
Figure 1. Examples of general text captions vs. task-specific text annotations generated by MM-G [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Even high-quality human curated captions (MS COCO) [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Examples of different text perplexity mapping to easy cases (low perplexity), potential noise and outliers in difficulty (highest [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Comparing different baselines for multimodal data generation with MM-G [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Comparing performance of MM-GEN across Tasks against Contributed Baselines and Skyline baseline 1 can be attributed to the varying difficulties of each task for the base model, shown by the range of ac￾curacies that the base model achieves on these tasks. Ad￾ditionally…
Figure 6
Figure 6. Figure 6: Evaluation on Llava-1.5 (13B Parameters) [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Examples Generated by MM-GEN for Chart Understanding Q: Which organism is at the base of the food web? A. Grass B. Rabbit C. Snake D. Hawk A: A. Grass Q: What is the primary prey for the hawk in this food web? A. Rabbit B. Snake C. Rat D. Caterpillar A: B. Snake Q: Whi…
Figure 8
Figure 8. Figure 8: Examples Generated by MM-GEN for Diagram Understanding 14 [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: Examples Generated by MM-GEN for Spatial Reasoning on Maps 15 [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: Task: Spatial Reasoning on Maps; Highly Repetitive Generated Text for Baseline 3: Task-Specific Text Annotations, Generated Using Natural Language Task Description Task-Agnostic Annotation The map depicts the island of Great Britain during a historical period characte…
Figure 11
Figure 11. Figure 11: Task: Spatial Reasoning on Maps; Highly Repetitive Generated Text for Baseline 2: Task-Agnostic Captions 17 [PITH_FULL_IMAGE:figures/full_fig_p017_11.png]
Figure 12
Figure 12. Figure 12: Task: Chart Understanding; Highly Repetitive Generated Text for Baseline 3: Task-Specific Text Annotations, Generated Using Natural Language Task Description 18 [PITH_FULL_IMAGE:figures/full_fig_p018_12.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

71 extracted references · 52 canonical work pages

  1. [1]

    Amro Abbas, Kushal Tirumala, D ´aniel Simig, Surya Gan- guli, and Ari S. Morcos. Semdedup: Data-efficient learning at web-scale through semantic deduplication, 2023. 3

    work page 2023

  2. [2]

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadal- lah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Ben- haim, Misha Bilenko, Johan Bjorck, S ´ebastien Bubeck, Martin Cai, Qin Cai, Vishrav Chaudhary, Dong Chen, Dongdong Chen, Weizhu Chen, Yen-Chun Chen, Yi-Ling Chen, Hao Cheng, Parul Chopra, Xiyang Dai...

    work page 2024

  3. [3]

    Eureka: Eval- uating and understanding large foundation models

    Vidhisha Balachandran, Jingya Chen, Neel Joshi, Besmira Nushi, Hamid Palangi, Eduardo Salinas, Vibhav Vineet, James Woffinden-Luey, and Safoora Yousefi. Eureka: Eval- uating and understanding large foundation models. arXiv preprint arXiv:2409.10566, 2024. 1

    arXiv 2024

  4. [4]

    Introduction to Natural Language Pro- cessing

    Peter Brown et al. Introduction to Natural Language Pro- cessing. Morgan Kaufmann, 1992. 2, 5

    work page 1992

  5. [5]

    Conceptual 12m: Pushing web-scale image-text pre- training to recognize long-tail visual concepts

    Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre- training to recognize long-tail visual concepts. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3558–3568, 2021. 1, 3

    work page 2021

  6. [6]

    Sharegpt4v: Improving large multi-modal models with better captions,

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions,

    work page

  7. [7]

    Alpagasus: Training a better alpaca with fewer data

    Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gu- naratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, et al. Alpagasus: Training a better alpaca with fewer data. arXiv preprint arXiv:2307.08701, 2023. 3

    arXiv 2023

  8. [8]

    Selection via proxy: Efficient data se- lection for deep learning

    Cody Coleman, Christopher Yeh, Stephen Mussmann, Baha- ran Mirzasoleiman, Peter Bailis, Percy Liang, Jure Leskovec, and Matei Zaharia. Selection via proxy: Efficient data se- lection for deep learning. arXiv preprint arXiv:1906.11829,

    arXiv 1906

Show all 71 references
  1. [9]

    A survey on in-context learning, 2024

    Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Tianyu Liu, Baobao Chang, Xu Sun, Lei Li, and Zhifang Sui. A survey on in-context learning, 2024. 4

    2024

  2. [10]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 ,

    arXiv

  3. [11]

    Tinystories: How small can language models be and still speak coherent english?, 2023

    Ronen Eldan and Yuanzhi Li. Tinystories: How small can language models be and still speak coherent english?, 2023. 2

    2023

  4. [12]

    Data curation via joint example selec- tion further accelerates multimodal learning

    Talfan Evans, Nikhil Parthasarathy, Hamza Merzic, and Olivier J Henaff. Data curation via joint example selec- tion further accelerates multimodal learning. arXiv preprint arXiv:2406.17711, 2024. 3

    2024 arXiv

  5. [13]

    Improving clip training with language rewrites

    Lijie Fan, Dilip Krishnan, Phillip Isola, Dina Katabi, and Yonglong Tian. Improving clip training with language rewrites. Advances in Neural Information Processing Sys- tems, 36, 2024. 2

    2024

  6. [14]

    Data fil- tering networks, 2023

    Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander Toshev, and Vaishaal Shankar. Data fil- tering networks, 2023. 3

    2023

  7. [15]

    Blink: Multimodal large language models can see but not perceive

    Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. Blink: Multimodal large language models can see but not perceive. ECCV, 2024. 1

    2024

  8. [16]

    Dat- acomp: In search of the next generation of multimodal datasets

    Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. Dat- acomp: In search of the next generation of multimodal datasets. Advances in Neural Information Processing Sys...

    2024

  9. [17]

    Textbooks are all you need, 2023

    Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio C´esar Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, S´ebastien Bubeck, Ronen Eldan, Adam Tauman Kalai,...

    2023

  10. [18]

    Statistical Methods for Speech Recogni- tion

    Frederick Jelinek. Statistical Methods for Speech Recogni- tion. MIT Press, 1997. 5

    1997

  11. [19]

    Data-efficient contrastive self-supervised learning: Most beneficial exam- ples for supervised learning contribute the least

    Siddharth Joshi and Baharan Mirzasoleiman. Data-efficient contrastive self-supervised learning: Most beneficial exam- ples for supervised learning contribute the least. In Proceed- ings of the 40th International Conference on Machine Learn- ing, pages 15356–15370. PMLR, 2023. 2

    2023

  12. [20]

    Data-efficient contrastive language-image pre- training: Prioritizing data quality over quantity, 2024

    Siddharth Joshi, Arnav Jain, Ali Payani, and Baharan Mirza- soleiman. Data-efficient contrastive language-image pre- training: Prioritizing data quality over quantity, 2024. 3 10

    2024

  13. [21]

    What’s ”up” with vision-language models? investigating their strug- gle with spatial reasoning, 2023

    Amita Kamath, Jack Hessel, and Kai-Wei Chang. What’s ”up” with vision-language models? investigating their strug- gle with spatial reasoning, 2023. 1

    2023

  14. [22]

    Not all sam- ples are created equal: Deep learning with importance sam- pling

    Angelos Katharopoulos and Franc ¸ois Fleuret. Not all sam- ples are created equal: Deep learning with importance sam- pling. In International conference on machine learning , pages 2525–2534. PMLR, 2018. 2

    2018

  15. [23]

    A diagram is worth a dozen images

    Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14 , pages 235–

    2016

  16. [24]

    Grad-match: Gradient matching based data subset selection for efficient deep model training

    Krishnateja Killamsetty, Sivasubramanian Durga, Ganesh Ramakrishnan, Abir De, and Rishabh Iyer. Grad-match: Gradient matching based data subset selection for efficient deep model training. In International Conference on Ma- chine Learning, pages 5464–5474. PMLR, 2021. 2

    2021

  17. [25]

    Revisit large-scale image-caption data in pre-training multi- modal foundation models, 2024

    Zhengfeng Lai, Vasileios Saveris, Chen Chen, Hong-You Chen, Haotian Zhang, Bowen Zhang, Juan Lao Tebar, Wenze Hu, Zhe Gan, Peter Grasch, Meng Cao, and Yinfei Yang. Revisit large-scale image-caption data in pre-training multi- modal foundation models, 2024. 1

    2024

  18. [26]

    Veclip: Improving clip training via visual-enriched captions,

    Zhengfeng Lai, Haotian Zhang, Bowen Zhang, Wentao Wu, Haoping Bai, Aleksei Timofeev, Xianzhi Du, Zhe Gan, Jiu- long Shan, Chen-Nee Chuah, Yinfei Yang, and Meng Cao. Veclip: Improving clip training via visual-enriched captions,

  19. [27]

    M3it: A large-scale dataset towards multi-modal multilingual instruction tuning

    Lei Li, Yuwei Yin, Shicheng Li, Liang Chen, Peiyi Wang, Shuhuai Ren, Mukai Li, Yazheng Yang, Jingjing Xu, Xu Sun, Lingpeng Kong, and Qi Liu. M3it: A large-scale dataset towards multi-modal multilingual instruction tuning. arXiv preprint arXiv:2306.04387, 2023. 2

    2023 arXiv

  20. [28]

    Textbooks are all you need ii: phi-1.5 technical report, 2023

    Yuanzhi Li, S ´ebastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks are all you need ii: phi-1.5 technical report, 2023. 2

    2023

  21. [29]

    Visual instruction tuning, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023. 1, 2, 3, 6

    2023

  22. [30]

    T-MARS: Improving visual representations by circumventing text feature learning

    Pratyush Maini, Sachin Goyal, Zachary Chase Lipton, J Zico Kolter, and Aditi Raghunathan. T-MARS: Improving visual representations by circumventing text feature learning. In The Twelfth International Conference on Learning Represen- tations, 2024. 3

    2024

  23. [31]

    When less is more: In- vestigating data pruning for pretraining llms at scale

    Max Marion, Ahmet ¨Ust¨un, Luiza Pozzobon, Alex Wang, Marzieh Fadaee, and Sara Hooker. When less is more: In- vestigating data pruning for pretraining llms at scale. arXiv preprint arXiv:2309.04564, 2023. 3, 5

    2023 arXiv

  24. [32]

    Joty, and Enamul Hoque

    Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq R. Joty, and Enamul Hoque. Chartqa: A benchmark for question an- swering about charts with visual and logical reasoning. In Findings of the Association for Computational Linguistics: ACL 2022, Dublin, Ireland, May 22-27, 2022 , p...

    2022

  25. [33]

    Chartinstruct: Instruction tuning for chart comprehension and reasoning,

    Ahmed Masry, Mehrad Shahmohammadi, Md Rizwan Parvez, Enamul Hoque, and Shafiq Joty. Chartinstruct: Instruction tuning for chart comprehension and reasoning,

  26. [34]

    Mmiu: Multimodal multi- image understanding for evaluating large vision-language models, 2024

    Fanqing Meng, Jin Wang, Chuanhao Li, Quanfeng Lu, Hao Tian, Jiaqi Liao, Xizhou Zhu, Jifeng Dai, Yu Qiao, Ping Luo, Kaipeng Zhang, and Wenqi Shao. Mmiu: Multimodal multi- image understanding for evaluating large vision-language models, 2024. 8

    2024

  27. [35]

    Coresets for data-efficient training of machine learning mod- els

    Baharan Mirzasoleiman, Jeff Bilmes, and Jure Leskovec. Coresets for data-efficient training of machine learning mod- els. In International Conference on Machine Learning, pages 6950–6960. PMLR, 2020. 2

    2020

  28. [36]

    Orca 2: Teaching small language models how to reason, 2023

    Arindam Mitra, Luciano Del Corro, Shweti Mahajan, An- dres Codas, Clarisse Simoes, Sahaj Agarwal, Xuxi Chen, Anastasia Razdaibiedina, Erik Jones, Kriti Aggarwal, Hamid Palangi, Guoqing Zheng, Corby Rosset, Hamed Khanpour, and Ahmed Awadallah. Orca 2: Teaching small language mo...

    2023

  29. [37]

    Agentinstruct: Toward generative teaching with agentic flows, 2024

    Arindam Mitra, Luciano Del Corro, Guoqing Zheng, Shweti Mahajan, Dany Rouhana, Andres Codas, Yadong Lu, Wei ge Chen, Olga Vrousgos, Corby Rosset, Fillipe Silva, Hamed Khanpour, Yash Lara, and Ahmed Awadallah. Agentinstruct: Toward generative teaching with agentic flows, 2024. 2

    2024

  30. [38]

    Orca: Progressive learning from complex explanation traces of gpt- 4, 2023

    Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sa- haj Agarwal, Hamid Palangi, and Ahmed Awadallah. Orca: Progressive learning from complex explanation traces of gpt- 4, 2023. 2

    2023

  31. [39]

    Improving multimodal datasets with image captioning

    Thao Nguyen, Samir Yitzhak Gadre, Gabriel Ilharco, Se- woong Oh, and Ludwig Schmidt. Improving multimodal datasets with image captioning. Advances in Neural Infor- mation Processing Systems, 36, 2024. 1, 2, 3

    2024

  32. [40]

    Gpt-4 technical report

    OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. 3, 6, 19

    2023 arXiv

  33. [41]

    Deep learning on a data diet: Finding important ex- amples early in training

    Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziu- gaite. Deep learning on a data diet: Finding important ex- amples early in training. Advances in neural information processing systems, 34:20596–20607, 2021. 2

    2021

  34. [42]

    Adaptive second order coresets for data-efficient machine learning

    Omead Pooladzandi, David Davini, and Baharan Mirza- soleiman. Adaptive second order coresets for data-efficient machine learning. In International Conference on Machine Learning, pages 17848–17869. PMLR, 2022. 2

    2022

  35. [43]

    Learning transferable visual models from natural language supervision, 2021

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021. 4

    2021

  36. [44]

    Fusecap: Leveraging large language models for enriched fused image captions

    Noam Rotstein, David Bensaid, Shaked Brody, Roy Ganz, and Ron Kimmel. Fusecap: Leveraging large language models for enriched fused image captions. arXiv preprint arXiv:2305.17718, 2023. 2

    2023 arXiv

  37. [45]

    Is a caption worth a thousand im- ages? a study on representation learning

    Shibani Santurkar, Yann Dubois, Rohan Taori, Percy Liang, and Tatsunori Hashimoto. Is a caption worth a thousand im- ages? a study on representation learning. In The Eleventh In- ternational Conference on Learning Representations , 2023. 3

    2023

  38. [46]

    Laion-5b: An open large-scale dataset for training next generation image-text models

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Worts- man, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural In- f...

    2022

  39. [47]

    Conceptual captions: A cleaned, hypernymed, im- age alt-text dataset for automatic image captioning

    Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, im- age alt-text dataset for automatic image captioning. In Pro- ceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Paper...

    2018

  40. [48]

    Math- llava: Bootstrapping mathematical reasoning for multimodal large language models, 2024

    Wenhao Shi, Zhiqiang Hu, Yi Bin, Junhua Liu, Yang Yang, See-Kiong Ng, Lidong Bing, and Roy Ka-Wei Lee. Math- llava: Bootstrapping mathematical reasoning for multimodal large language models, 2024. 1, 2

    2024

  41. [49]

    Chatgpt-4 vision struggles with radiologic image interpretation

    Keri Stephens. Chatgpt-4 vision struggles with radiologic image interpretation. AXIS Imaging News, 2024. 1

    2024

  42. [50]

    Dataset cartography: Mapping and diag- nosing datasets with training dynamics

    Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi, Noah A Smith, and Yejin Choi. Dataset cartography: Mapping and diag- nosing datasets with training dynamics. arXiv preprint arXiv:2009.10795, 2020. 2

    2009 arXiv

  43. [51]

    Kushal Tirumala, Daniel Simig, Armen Aghajanyan, and Ari S. Morcos. D4: Improving LLM pretraining via document de-duplication and diversification. In Thirty- seventh Conference on Neural Information Processing Sys- tems Datasets and Benchmarks Track, 2023. 3

    2023

  44. [52]

    An empirical study of example forget- ting during deep neural network learning

    Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geof- frey J Gordon. An empirical study of example forget- ting during deep neural network learning. arXiv preprint arXiv:1812.05159, 2018. 2

    2018 arXiv

  45. [53]

    Dynamic data selection for efficient ssl via coarse-to-fine re- finement

    Aditay Tripathi, Pradeep Shenoy, and Anirban Chakraborty. Dynamic data selection for efficient ssl via coarse-to-fine re- finement. 2

  46. [54]

    Show and tell: Lessons learned from the 2015 mscoco image captioning challenge

    Oriol Vinyals, Alexander Toshev, Samy Bengio, and Du- mitru Erhan. Show and tell: Lessons learned from the 2015 mscoco image captioning challenge. IEEE transactions on pattern analysis and machine intelligence , 39(4):652–663,

    2015

  47. [55]

    Is a picture worth a thou- sand words? delving into spatial reasoning for vision lan- guage models

    Jiayu Wang, Yifei Ming, Zhenmei Shi, Vibhav Vineet, Xin Wang, Yixuan Li, and Neel Joshi. Is a picture worth a thou- sand words? delving into spatial reasoning for vision lan- guage models. NeurIPS, 2024. 6

    2024

  48. [56]

    Decoding data quality via synthetic corruptions: Embedding-guided pruning of code data

    Yu Yang, Aaditya K Singh, Mostafa Elhoushi, Anas Mahmoud, Kushal Tirumala, Fabian Gloeckle, Baptiste Rozi`ere, Carole-Jean Wu, Ari S Morcos, and Newsha Ardalani. Decoding data quality via synthetic corruptions: Embedding-guided pruning of code data. arXiv preprint arXiv:2312.0...

    2023 arXiv

  49. [57]

    Capsfu- sion: Rethinking image-text data at scale, 2024

    Qiying Yu, Quan Sun, Xiaosong Zhang, Yufeng Cui, Fan Zhang, Yue Cao, Xinlong Wang, and Jingjing Liu. Capsfu- sion: Rethinking image-text data at scale, 2024. 1, 2

    2024

  50. [58]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for ex- pert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for ex- pert agi. In Proceedings of the IEEE/CVF Conference on...

    2024

  51. [59]

    Multimodal self-instruct: Synthetic abstract image and visual reasoning instruction using language model, 2024

    Wenqi Zhang, Zhenglin Cheng, Yuanyu He, Mengna Wang, Yongliang Shen, Zeqi Tan, Guiyang Hou, Mingqian He, Yanna Ma, Weiming Lu, and Yueting Zhuang. Multimodal self-instruct: Synthetic abstract image and visual reasoning instruction using language model, 2024. 1

    2024

  52. [60]

    Lima: Less is more for alignment

    Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36, 2024. 3

    2024

  53. [61]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023. 2 12 Appendix A. Exact Input to Stronger VLM and Generated Text Annotations Exa...

    2023 arXiv

  54. [64]

    Analyze the Example: Review the provided example question-answer pair to understand the structure, focus, and context

  55. [65]

    Understand the New Image: Infer relevant details, objects, and themes in the new image, considering how they relate to the skill

  56. [66]

    Generate Questions: Create questions that reflect the context and content of the new image, ensuring they align with the skill and follow the example’s style

  57. [67]

    If the question is a multiple-choice question, make sure to include the options in the question

  58. [68]

    Q": The generated question (include options if it’s multiple-choice). -

    Formulate Answers: Generate accurate and concise answers to the questions. Ensure each answer directly corresponds to the content of the new image. Output Format: Return the results as a JSON list of objects. Each object should include: - "Q": The generated question (include o...

    2011

  59. [69]

    Chart Understanding (ChartQA): 6 epochs

  60. [70]

    Diagram Understanding (AI2D): 6 epochs

  61. [71]

    Spatial Reasoning on Map (SpatialMap): 3 epochs 19

  62. [251]

    Springer, 2016. 5, 6

    2016

  63. [2279]

    3, 4, 5, 6

    Association for Computational Linguistics, 2022. 3, 4, 5, 6

    2022

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.