REVIEW 5 major objections 5 minor 2 cited by
TaskGalaxy: Scaling Multi-modal Instruction Fine-tuning with Tens of Thousands Vision Task Types
T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Fine-tuning with a dataset of 19,227 hierarchical task types and 413,648 samples improves LLaVA-v1.5 and InternVL-Chat-v1.0 on all 16 benchmarks, with average gains of 3.0 to 4.5 points.
desk verdict A genuinely large task-diversity dataset with an automated pipeline and consistent gains, but the missing train/eval image-overlap check leaves the central diversity claim less secure than the paper's 'proves' language suggests. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the hierarchical task-type taxonomy and the automated pipeline that produces it. A handful of manually defined seed tasks (OCR, image description, logical reasoning, and others) is expanded by GPT-4o into three levels connected by '~' notation, yielding 19,227 distinct task labels, with prompts that explicitly ask for categories not overlapping existing ones. This taxonomy is the independent variable of the paper's main ablation: holding the training set at 100,000 samples while varying the task-type count from 2,000 to 19,227 raises average benchmark performance steadily. The matching stage uses the CLIP cosine similarity $s_j = I(x_i) \cdot T(t_j)$ between the image embedding and the task-label embedding to select the top $k = 10$ candidate task types per image, followed by GPT-4o pruning, GPT-4o question-answer generation, and a three-model referee vote that keeps samples scoring at least 2 out of 3, which is the quality-control gate that lets the pipeline run without human labeling.
What would settle it
Compute image-level overlap between the TaskGalaxy training pool and each evaluation benchmark, then re-run the fine-tuning with every overlapping image removed. If the reported gains (3.0 to 4.5 points on average, and the monotone improvement from 2,000 to 19,227 task types) persist after deduplication, the task-diversity claim stands; if they shrink or vanish, training-set contamination was a carrier of the effect.
Extended reading notes
Core claim
The paper's central claim is that the range of task types in supervised fine-tuning data is a bottleneck for vision-language models, and that a dataset spanning roughly 19,000 hierarchical task types removes that bottleneck. TaskGalaxy organizes 413,648 image-question-answer samples under 19,227 task types produced by a five-step pipeline: GPT-4o expands dozens of hand-written seed tasks into a three-level taxonomy (115 first-level, 2,796 second-level, and 14,370 third-level types); CLIP scores each image against each task label by cosine similarity and keeps the top 10; GPT-4o prunes labels that do not fit the image content; GPT-4o writes a question-answer pair for each surviving label; and three open-source multimodal models independently score each image-task-question triple, keeping samples that score at least 2 out of 3. Fine-tuning LLaVA-v1.5 (7B and 13B) and InternVL-Chat-v1.0 (7B and 13B) with TaskGalaxy added to the original fine-tuning data improved every one of the 16 benchmarks, with average gains of 4.5, 3.83, 3.0, and 3.64 points on the 15 non-MME benchmarks and a 68-point gain for LLaVA-v1.5-13B on MME. The paper states that these results prove that expanding the diversity of visual task types, together with the associated high-quality question-answer pairs, significantly enhances the generalization capabilities of multimodal models.
Load-bearing premise
The paper credits the benchmark gains to task-type diversity, but it never verifies that its training images are absent from the evaluation sets, and several training sources (COCO images used by POPE and other benchmarks, TextVQA training images, and DocVQA images) overlap with evaluation imagery, so part of the reported improvement could come from the model having seen evaluation images during fine-tuning rather than from task diversity.
Editorial extensions
If this is right
- Task-type coverage becomes a primary design axis for fine-tuning datasets: the paper's ablation shows that, at a fixed 100,000 samples, expanding from 2,000 to 19,227 task types raises average benchmark performance.
- Manual task labeling is no longer the limiting cost: with a small seed taxonomy and the expand-match-generate-score pipeline, a dataset with hundreds of times more task types than previous human-labeled efforts can be built almost automatically.
- The diversity benefit transfers across model families and scales: all four tested LLaVA-v1.5 and InternVL-Chat-v1.0 variants improve on all 16 benchmarks, and the appendix reports 14-of-16 benchmark gains on the newer InternVL-Chat-V2.0-8B.
- Task diversity helps more than reasoning-heavy tasks: the paper reports gains on hallucination and perception benchmarks (POPE, HallusionBench, Q-Bench, MME) as well as on chart, diagram, and math benchmarks.
- The dataset extends without re-annotation: adding new images and task types can be done through the same pipeline, so coverage can grow with new domains.
Reading between the lines
- If task coverage is the carrier, a data-scaling law in 'task space' is plausible: for a fixed fine-tuning budget, spreading samples across more distinct task labels should beat concentrating them in fewer labels, a hypothesis that could be tested by varying the taxonomy while holding images and answer content fixed.
- Because the taxonomy's long tail includes refusal and safety-related task types, the same pipeline could encode safety behaviors into models through task labeling; the paper does not analyze this effect directly.
- A controlling experiment the paper does not run would attach task-type labels at random to a fixed set of question-answer pairs; if the diversity benefit survives random labeling, the active ingredient is the answer-content distribution rather than the taxonomy structure.
- The comparison against equally sized synthetic datasets (ShareGPT-4V, ALLaVA, Cambrian-1) leaves open whether the gains come from task-type count or from characteristics of GPT-4o-generated answers, so an equal-size human-annotated dataset at the same task-type count would separate the two factors.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. TaskGalaxy is a multimodal instruction fine-tuning dataset containing 19,227 hierarchical task types and 413,648 question-answer samples. The construction pipeline expands a small set of manually defined task types using GPT-4o, matches task types to images with CLIP and GPT-4o, generates QA pairs with GPT-4o, and filters them with three open-source vision-language models acting as referees. The paper evaluates the dataset by fine-tuning LLaVA-v1.5-7B/13B and InternVL-Chat-v1.0-7B/13B on the original SFT mix plus TaskGalaxy, reporting improvements on 16 benchmarks relative to the original-data baselines. Ablations vary the number of task types and the sample budget, and the appendix includes an equal-sample-size comparison with other instruction-tuning datasets and an experiment on InternVL-Chat-V2.0-8B.
Significance. If the causal interpretation is accepted, the paper makes a strong practical contribution: TaskGalaxy is two orders of magnitude larger in task-type coverage than prior instruction-tuning datasets, the generation pipeline is largely automated, the dataset is publicly released, and the gains are shown across multiple base model families. The equal-sample-size comparison in Appendix A.7 and the extension to a newer model in Appendix A.8 are useful controls, as is the chain-of-thought ablation. However, the central attribution of the benchmark gains to task diversity is weakened by the absence of a train/eval image-overlap analysis and by the reliance on an uncontrolled primary comparison; both are fixable empirically rather than fundamental flaws in the method.
major comments (5)
- [§2.2, Table A-1; §3.1] The TaskGalaxy image pool shares image families with the evaluation benchmarks, and the paper does not check or control for this. Table A-1 explicitly lists 'coco train2017' (118,326 images), 'text vqa train images' (3,485), 'ocr vqa images' (12,614), and 'share textvqa images' (55), while Table 1 lists Visual Genome, ShareGPT4V, and MathV360K, which are known to contain COCO, SA, SBU, and DocVQA images. POPE is built from COCO subsets and TextVQA uses TextVQA images, so training images may overlap with evaluation images in two of the sixteen benchmarks. The paper reports no exact-duplicate or near-duplicate overlap statistics and no evaluation restricted to images disjoint from the training pool. Because the central claim is that task diversity, not image familiarity, drives the improvements, this gap is load-bearing. A concrete fix is to compute overlap between TaskGalaxy images and each evaluation benchmark's image set, and to re-report the affected benchmarks on the disjoint subset; ideally this should be done for all benchmarks.
- [§3.2, Table 2; §A.7] The primary comparison in Table 2 adds all 413k TaskGalaxy samples on top of the original SFT mix, so the average gain conflates increased data quantity with increased task diversity. Appendix A.7 is the correct control and TaskGalaxy still wins at matched sample size on most benchmarks, but the main text and abstract continue to treat the uncontrolled comparison as the primary evidence, and the sentence in Section 3.2 that this 'proves' the diversity claim is stronger than the experiment supports. The authors should either move the matched-sample-size comparison into the main paper as the headline result or add an explicit data-quantity-matched ablation; at minimum the wording should be softened to 'consistent with' rather than 'proves.'
- [§2.2, Referee Screening; §3.1] Two of the three referee models used to filter TaskGalaxy samples (InternVL-Chat-v1.5 and InternVL2-26B) belong to the InternVL family, and one of the evaluated models is InternVL-Chat-v1.0. Consequently, the selected samples may be preferentially well-matched to what InternVL-family models can score and answer, which could partly explain the InternVL-specific improvements. This concern is mitigated by the fact that the same dataset also helps LLaVA, which is outside the judge family, but the paper should acknowledge the potential selection-family effect and, ideally, run the referee filtering with at least one judge outside the InternVL family to quantify its influence.
- [§3.2, Table 2] All results in Table 2 appear to be single runs without variance estimates or significance tests. Several reported gains are small (e.g., LLaVA-v1.5-13B POPE 86.03→86.20; InternVL-Chat-v1.0-13B MME 1525→1534; LLaVA-v1.5-13B LLaVA-Wild 62.6→63.1), and the claim of improvement on all 16 benchmarks would be more convincing with multiple random seeds or per-benchmark significance testing. At minimum, the paper should state whether the differences are stable across seeds and report standard deviations for the main table.
- [§A.7, Table A-5] The equal-sample-size comparison contains implausible baseline numbers that undermine its role as a control. For example, fine-tuning LLaVA-v1.5-7B with LLaVA-OneVision data is reported with a SEED score of 30.06 and with ALLaVA-4V data as 40.52, far below the Baseline's 58.62; the InternVL row shows a similar pattern (40.87 and 48.36 vs. Baseline 59.06). These values are not consistent with expected performance levels and suggest a possible evaluation or sampling error. The authors should verify the evaluation code and data-sampling procedure for these rows before the matched-sample-size comparison can be used as evidence.
minor comments (5)
- [Throughout] There are several typos and inconsistencies: 'comparision' in Table A-4, 'tunning' in Introduction and Related Work, 'properities' in Section 2.2, 'leval' in Appendix A.2, and 'the third part ... screen the ten images' in Appendix A.4 should presumably read 'ten task types'. The abstract says 413,648 samples while Table A-4 says 431K; the numbers should be reconciled.
- [§A.6] The benchmark name is written inconsistently as 'MM-VeT' and 'MMVet' across the paper and appendix; please unify the notation.
- [Table A-6] The table title says 'InternVL-Chat-v1.0-8B' while the surrounding text and Section A.8 refer to 'InternVL-Chat-V2.0-8B'; the table header should match the model actually used.
- [Figure 6 and §3.3] The horizontal-axis labels in Figure 6 are not fully self-explanatory: the left panel reports the number of task types at a constant 100k sample budget, and the right panel reports the per-task-type sample cap (5 to 55) resulting in different total sample sizes. Please make these axes explicit and describe in the caption how samples are selected when the number of task types is reduced.
- [§6, Ethics Statement] The ethics statement is generic boilerplate about informed consent and human participants, which is not directly applicable to a dataset generated from public image collections and synthetic QA pairs. The licensing discussion in Appendix A.5 is more relevant; consider replacing the generic paragraph with a dataset-specific statement about image provenance, licensing, and content risks.
Circularity Check
No significant circularity: TaskGalaxy is an empirically evaluated dataset, with no fitted parameter or self-citation chain that makes its benchmark conclusions equivalent to its construction inputs.
full rationale
The paper's derivation chain is empirical rather than definitional: TaskGalaxy is constructed by expanding manually seeded task types with GPT-4o, matching them to open-source images with CLIP, and filtering generated Q&A with three external open-source judges. The central claim, that incorporating TaskGalaxy improves LLaVA-v1.5 and InternVL-Chat-v1.0 across 16 benchmarks, is then tested on held-out standardized benchmarks. No equation in the paper reduces a predicted quantity to a fitted parameter or to the dataset's own definition. The only same-family element is that InternVL-Chat-v1.5 and InternVL2-26B are used as referee filters while InternVL-Chat-v1.0 is the evaluated model; this is not load-bearing for the benchmark claim and the paper itself acknowledges judge fallibility, e.g., 'The Internvl model, however, provided a score, which may reflect a lack of mathematical reasoning capabilities, leading to an incorrect evaluation.' The train/evaluation image-overlap issue (COCO and TextVQA images appearing in Visual Genome, ShareGPT4V, and MathV360K) is a real contamination risk for causal attribution, but it is a validity concern, not a circularity: it does not make any benchmark result equivalent to the dataset's construction by definition. There is no load-bearing self-citation chain, no uniqueness theorem imported from the authors' prior work, and no ansatz smuggled in via citation. Accordingly, the honest circularity finding is no significant circularity.
Assumptions & free parameters
free parameters (4)
- CLIP top-k =
10
- Referee score threshold =
2 (out of 3)
- Max samples per task type =
55
- Number of hierarchy levels =
3
assumptions (4)
- domain assumption GPT-4o can generate non-overlapping, comprehensive task types from seed prompts covering all multimodal content-understanding scenarios.
- domain assumption CLIP image-text cosine similarity is a useful first-pass signal for matching task types to images.
- domain assumption The three open-source VLM judges provide reliable relevance scores, and a majority threshold of 2 correctly selects high-quality samples.
- domain assumption The image sources do not overlap with evaluation benchmark images, or any overlap does not materially affect benchmark scores.
Cite this review
Pith. "Pith review of TaskGalaxy: Scaling Multi-modal Instruction Fine-tuning with Tens of Thousands Vision Task Types." pith.science (2026). https://pith.science/paper/P2YD3RDR
@misc{pith2026250209925,
author = {Pith},
title = {Pith review of: TaskGalaxy: Scaling Multi-modal Instruction Fine-tuning with Tens of Thousands Vision Task Types},
year = {2026},
howpublished = {\url{https://pith.science/paper/P2YD3RDR}},
note = {Machine review of arXiv:2502.09925}
}
read the original abstract
Multimodal visual language models are gaining prominence in open-world applications, driven by advancements in model architectures, training techniques, and high-quality data. However, their performance is often limited by insufficient task-specific data, leading to poor generalization and biased outputs. Existing efforts to increase task diversity in fine-tuning datasets are hindered by the labor-intensive process of manual task labeling, which typically produces only a few hundred task types. To address this, we propose TaskGalaxy, a large-scale multimodal instruction fine-tuning dataset comprising 19,227 hierarchical task types and 413,648 samples. TaskGalaxy utilizes GPT-4o to enrich task diversity by expanding from a small set of manually defined tasks, with CLIP and GPT-4o filtering those that best match open-source images, and generating relevant question-answer pairs. Multiple models are employed to ensure sample quality. This automated process enhances both task diversity and data quality, reducing manual intervention. Incorporating TaskGalaxy into LLaVA-v1.5 and InternVL-Chat-v1.0 models shows substantial performance improvements across 16 benchmarks, demonstrating the critical importance of task diversity. TaskGalaxy is publicly released at https://github.com/Kwai-YuanQi/TaskGalaxy.
Figures
Figures from the paper (4 more)
Forward citations
Cited by 2 Pith papers
-
Kwai Keye-VL 1.5 Technical Report
Keye-VL-1.5 combines similarity-based Slow-Fast video token allocation with progressive context extension and iterative RL, reporting leading video-understanding results among 8B-scale multimodal models.
-
Kwai Keye-VL Technical Report
Kwai Keye-VL shows that a five-mode chain-of-thought cold-start plus mix-mode reinforcement learning can push an 8B multimodal model to strong short-video and general vision-language performance.
Reference graph
Works this paper leans on
-
[1]
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774,
-
[4]
Language models are few-shot learners
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901,
1901
-
[5]
in benchmarks such as MME, LLaV A-in-the-wild, and Q-Bench. Additionally, the average performance across 15 benchmarks, excluding MME, increased by approx- imately 1.3 points with CoT. These findings underscore the value of incorporating CoT prompting into multimodal models. Table A-7: Performance comparison of CoT validity verification. +max 5 refers to ...
work page 2025
-
[7]
Mme: A comprehensive evaluation benchmark for multimodal large language models
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.13394,
-
[8]
Chatglm: A family of large language models from glm-130b to glm-4 all tools
Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, et al. Chatglm: A family of large language models from glm-130b to glm-4 all tools. arXiv preprint arXiv:2406.12793,
-
[9]
11 Published as a conference paper at ICLR 2025 Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, et al. Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. In Pro- ceedings of the IEEE/CVF Confere...
work page 2025
-
[10]
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276,
-
[14]
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730–19742. PMLR, 2023c. Lei Li, Yuwei Yin, Shicheng Li, Liang Chen, Peiyi Wang, Shuhuai Ren, Mukai Li, Yazheng Yang, Jingjing Xu, Xu Sun, et al...
arXiv 2025
Show all 42 references
-
[15]
Clevr-math: A dataset for compositional language, visual and mathematical reasoning
Adam Dahlgren Lindstr ¨om and Savitha Sam Abraham. Clevr-math: A dataset for compositional language, visual and mathematical reasoning. arXiv preprint arXiv:2208.05358,
-
[16]
Improved baselines with visual instruction tuning
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pp. 26296–26306, 2024a. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Vi...
-
[17]
Learn to explain: Multimodal reasoning via thought chains for science question answering
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507–25...
-
[19]
Ocr-vqa: Visual question answering by reading text in images
Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Ocr-vqa: Visual question answering by reading text in images. In 2019 international conference on document analysis and recognition (ICDAR), pp. 947–952. IEEE,
2019
-
[20]
Vicente Ordonez, Girish Kulkarni, and Tamara Berg
Available at https://openai.com/index/ gpt-4v-system-card (accessed on February 14, 2025). Vicente Ordonez, Girish Kulkarni, and Tamara Berg. Im2text: Describing images using 1 million captioned photographs. Advances in neural information processing systems, 24,
2025
-
[22]
Solving geometry problems: Combining text and diagram interpretation
Minjoon Seo, Hannaneh Hajishirzi, Ali Farhadi, Oren Etzioni, and Clint Malcolm. Solving geometry problems: Combining text and diagram interpretation. In Proceedings of the 2015 conference on empirical methods in natural language processing, pp. 1466–1476,
2015
-
[23]
Math-llava: Bootstrapping mathematical reasoning for multimodal large language models
Wenhao Shi, Zhiqiang Hu, Yi Bin, Junhua Liu, Yang Yang, See-Kiong Ng, Lidong Bing, and Roy Ka-Wei Lee. Math-llava: Bootstrapping mathematical reasoning for multimodal large language models. arXiv preprint arXiv:2406.17294,
-
[24]
Aligning large multimodal models with factually augmented rlhf
Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, et al. Aligning large multimodal models with factually augmented rlhf. arXiv preprint arXiv:2309.14525,
-
[25]
Cambrian-1: A fully open, vision-centric exploration of multimodal llms
Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, et al. Cambrian-1: A fully open, vision-centric exploration of multimodal llms. arXiv preprint arXiv:2406.16860,
-
[26]
Llama: Open and efficient foundation language models
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth ´ee Lacroix, Baptiste Rozi `ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023a. Hugo Touvron...
-
[27]
Q-bench: A benchmark for general-purpose foundation models on low-level vision
Haoning Wu, Zicheng Zhang, Erli Zhang, Chaofeng Chen, Liang Liao, Annan Wang, Chunyi Li, Wenxiu Sun, Qiong Yan, Guangtao Zhai, et al. Q-bench: A benchmark for general-purpose foundation models on low-level vision. arXiv preprint arXiv:2309.14181,
-
[28]
Visionllm v2: An end-to-end generalist multimodal large language model for hundreds of vision-language tasks
Jiannan Wu, Muyan Zhong, Sen Xing, Zeqiang Lai, Zhaoyang Liu, Wenhai Wang, Zhe Chen, Xizhou Zhu, Lewei Lu, Tong Lu, et al. Visionllm v2: An end-to-end generalist multimodal large language model for hundreds of vision-language tasks. arXiv preprint arXiv:2406.08394,
-
[29]
Vision-flan: Scaling human-labeled tasks in visual instruction tuning
Zhiyang Xu, Chao Feng, Rulin Shao, Trevor Ashby, Ying Shen, Di Jin, Yu Cheng, Qifan Wang, and Lifu Huang. Vision-flan: Scaling human-labeled tasks in visual instruction tuning. arXiv preprint arXiv:2402.11690,
-
[30]
Baichuan 2: Open large-scale language models
14 Published as a conference paper at ICLR 2025 Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, et al. Baichuan 2: Open large-scale language models. arXiv preprint arXiv:2309.10305,
2025 arXiv
-
[31]
mplug-owl: Modularization empowers large language models with multimodality
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304.14178,
-
[32]
Mm-vet: Evaluating large multimodal models for integrated capabilities.arXiv preprint arXiv:2308.02490,
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities.arXiv preprint arXiv:2308.02490,
-
[33]
Cocot: Contrastive chain-of-thought prompting for large multimodal models with multiple image inputs
Daoan Zhang, Junming Yang, Hanjia Lyu, Zijian Jin, Yuan Yao, Mingkai Chen, and Jiebo Luo. Cocot: Contrastive chain-of-thought prompting for large multimodal models with multiple image inputs. arXiv preprint arXiv:2401.02582, 2024a. Pan Zhang, Xiaoyi Dong Bin Wang, Yuhang Cao, ...
-
[34]
Retrieval-augmented mixture of lora experts for uploadable machine learning.arXiv preprint arXiv:2406.16989,
Ziyu Zhao, Leilei Gan, Guoyin Wang, Yuwei Hu, Tao Shen, Hongxia Yang, Kun Kuang, and Fei Wu. Retrieval-augmented mixture of lora experts for uploadable machine learning.arXiv preprint arXiv:2406.16989,
-
[35]
Minigpt-4: En- hancing vision-language understanding with advanced large language models
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592,
-
[36]
Model tailor: Mitigating catastrophic forgetting in multi-modal large language models
Didi Zhu, Zhongyi Sun, Zexi Li, Tao Shen, Ke Yan, Shouhong Ding, Kun Kuang, and Chao Wu. Model tailor: Mitigating catastrophic forgetting in multi-modal large language models. ICML 2024,
2024
-
[37]
The approximate data sources and their corre- sponding sample sizes are presented in Table 1 of the main text
15 Published as a conference paper at ICLR 2025 A A PPENDIX A.1 O VERALL REVIEW OF IMAGE SOURCES Considering the accessibility of data sources and the task-related nature of the image data we aim to mine, we have opted for open-source image data. The approximate data sources a...
2025
-
[38]
data fusion,
Please only output your final score without any other characters. p CoT You are a multimodal content understanding expert and you are very good at solving visual question answering tasks, I will give you an image and a question related to this image, the question is: {question...
2023
-
[39]
flooding inundates the marina and affects nearby buildings and facilities
Thus, the answer is incorrect. The task requires logical reasoning and complex algebraic calculations to solve the problem, but the answer does not meet the expected level of complexity required for this task type. Additionally, the image content appears to be more relevant to...
2023
-
[41]
The best results are highlighted in bold. Model Method BenchmarksMME MMB MMBCN POPE LLaV AW MMVet TQA SQA MathVista InternVL-Chat-V2.0-8B Baseline 1536 68.52 66.46 86.30 63.20 46.17 66.24 90.58 50.10TaskGalaxy1565 73.88 70.79 86.90 62.85 48.86 70.49 92.71 52.31ChartQA AI2D Q-B...
2023
-
[2000]
The best results are highlighted in bold. Model Method BenchmarksMME MMB MMBCN POPE LLaV AW MMVet TQA SQA MathVista LLaV A-v1.5-7B Baseline 1476 63.29 56.45 86.30 47.70 24.70 57.59 68.77 28.20ShareGPT-4V1501 65.97 59.10 86.29 49.20 29.00 57.56 70.60 28.20LLaV A-OneVision1251 5...
2025
-
[2014]
A diagram is worth a dozen images
Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In Computer Vision–ECCV 2016: 14th Euro- pean Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14 , pp. 235–251....
2016
-
[2017]
Mimic-it: Multi-modal in-context instruction tuning
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Fanyi Pu, Jingkang Yang, Chunyuan Li, and Ziwei Liu. Mimic-it: Multi-modal in-context instruction tuning. arXiv preprint arXiv:2306.05425, 2023a. Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, ...
-
[2018]
Referitgame: Referring to objects in photographs of natural scenes
Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 787–798,
2014
-
[2019]
Chartqa: A bench- mark for question answering about charts with visual and logical reasoning
Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A bench- mark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203.10244,
-
[2020]
Mapqa: A dataset for question answering on choropleth maps
Shuaichen Chang, David Palzer, Jialin Li, Eric Fosler-Lussier, and Ningchuan Xiao. Mapqa: A dataset for question answering on choropleth maps. arXiv preprint arXiv:2211.08545,
-
[2021]
Laion-400m: Open dataset of clip-filtered 400 million image-text pairs
13 Published as a conference paper at ICLR 2025 Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXi...
2025 arXiv
-
[2022]
Allava: Harnessing gpt4v-synthesized data for a lite vision-language model
Guiming Hardy Chen, Shunian Chen, Ruifei Zhang, Junying Chen, Xiangbo Wu, Zhiyi Zhang, Zhi- hong Chen, Jianquan Li, Xiang Wan, and Benyou Wang. Allava: Harnessing gpt4v-synthesized data for a lite vision-language model. arXiv preprint arXiv:2402.11684, 2024a. Keqin Chen, Zhao ...
2023 arXiv
-
[2023]
Direct preference optimization for suppressing hallucinated prior exams in radiology report gen- eration
Oishi Banerjee, Hong-Yu Zhou, Subathra Adithan, Stephen Kwak, Kay Wu, and Pranav Rajpurkar. Direct preference optimization for suppressing hallucinated prior exams in radiology report gen- eration. arXiv preprint arXiv:2406.06496,
-
[2024]
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou
Available at https://www.anthropic.com/news/ claude-3-5-sonnet (accessed on February 14, 2025). Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abi...
2025 arXiv
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.