{"id":"204c0057-5626-4a20-920e-4f6b9755cf0c","arxiv_id":"2411.10557","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Training a multimodal LLM on mostly text-only instructions with a small vision-language tail matches or beats vision-heavy instruction tuning on held-out text and vision tasks at about half the token cost.","lead":"MLAN tests whether multimodal language models can be instruction-tuned mostly on text-only data instead of image-text pairs. The authors find that a 75 percent text, 25 percent vision mixture matches or beats vision-heavy recipes on held-out benchmarks while using roughly half the training tokens.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Single-run results and small vision margins leave the 'matches or exceeds' claim underdetermined; a seed analysis and validation split for the 75% ratio are needed.","rationale":"The paper's controlled experimental design is a genuine strength: fixing the 186k-instance budget and drawing all mixtures from the same two data sources isolates the modality ratio, and Table 3's token accounting directly supports the efficiency claim. The text-only transfer effect is plausible and, if robust, practically important. However, the central empirical claim has a soft spot: the vision-language comparisons are extremely close, particularly for Llama-3.1-8B where MLAN's vision average is 0.33 points below MIX-Cambrian-1. Without repeated runs, there is no way to distinguish true parity from a favorable draw. The ratio-selection procedure compounds this: Figure 3 was presumably produced using the same evaluation benchmarks that later serve as the main result, and no validation split is mentioned. If the 75% value was chosen by inspecting these curves, the reported numbers are selected rather than predicted. The paper's Limitations do mention single-run and lack of code/data, but the tightness of the vision margins makes this more than a routine caveat. I would not reject the paper: the text benchmark improvements are large and consistent, and the token efficiency is real. But the 'on-par' claim on vision should be provisional until seed variance is measured and the ratio-selection procedure is made explicit.","tokens_in":19701,"tokens_out":17872,"duration_ms":171185,"concrete_test":"Run MLAN, MIX-LLaVA-1.5, and MIX-Cambrian-1 for Llama-3.1-8B with at least 5 independent seeds (varying both data sampling and optimizer seed), keeping the 186k-instance budget and all hyperparameters fixed. Report the mean and 95% bootstrap confidence interval of the vision-language average over the five benchmarks in Table 1. If MLAN's CI overlaps MIX-Cambrian-1's CI and the point difference is within ±1.0, the 'matches or exceeds' phrasing should be softened to 'comparable in expectation'; if MLAN is significantly below, the central claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is that MLAN (75% text-only / 25% vision-language) is on par with vision-heavy mixtures on both modalities while using fewer tokens. The text-side advantage is large and consistent, but the vision-side comparison is extremely close: for Llama-3.1-8B, MLAN's vision average is 62.25 vs 62.58 for MIX-Cambrian-1, and individual benchmarks trade wins and losses by several points. All results are from a single run with no error bars, no repeated seeds, and no confidence intervals. Because the paper's headline uses 'matches or exceeds', a 0.33-point deficit could easily become a multi-point deficit under a different seed or data sample, undermining the 'on-par' claim. Additionally, the 75% ratio appears to have been selected after inspecting the knowledge-transfer curves (Figure 3) on the same evaluation benchmarks used in the main tables; no validation split is described. If the ratio was chosen by peeking at these results, the reported comparison is optimistically selected rather than a prediction. These issues are partially acknowledged in the Limitations, but the magnitude of the vision margins makes them more than a routine reproducibility caveat.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MLAN, a visual instruction tuning strategy that replaces most vision-language instruction data with text-only data (75% text, 25% vision) under a fixed 186k-instance budget after multimodal pretraining. Using LLaVA-style models built on Llama-3.2-3B and Llama-3.1-8B, the authors compare MLAN against simplified mixtures with LLaVA-1.5-style (6% text) and Cambrian-1-style (25% text) ratios on seven text-only and five vision-language held-out benchmarks. The headline finding is that MLAN matches or exceeds these vision-heavy proxy mixtures on vision benchmarks, clearly improves text benchmarks, and processes roughly 40-50% fewer training tokens. Additional experiments explore the language-data ratio curve, task diversity, pretraining data, and instruction-tuned versus base backbones.","tokens_in":19927,"tokens_out":7501,"duration_ms":67522,"significance":"If the result is robust to seed variation and does not come from selection on the evaluation benchmarks, this is a valuable and cost-saving empirical contribution. The paper addresses an underexplored design choice (modality composition in MLLM instruction tuning) with controlled fixed-budget comparisons at two model scales, includes token-level efficiency accounting, and provides several ablations. It is also candid about limitations, including the narrow architecture scope and the absence of specialized tasks such as OCR and captioning. However, the central 'matches or exceeds' claim currently rests on small vision-side margins from single runs, and the chosen 75% ratio appears to have been selected from the same evaluation curves used in the main tables. These issues need to be addressed before the quantitative conclusion can be considered reliable.","major_comments":[{"comment":"All results in the main comparison are single runs, with no error bars, standard deviations, or repeated-seed analysis reported anywhere in the manuscript. The vision-side difference that supports 'matches or exceeds' is extremely small for the larger model: MLAN's vision average is 62.25 versus 62.58 for MIX-Cambrian-1 in Table 1, a 0.33-point deficit, while individual benchmarks trade by several points (MMMU 34.44 vs 36.00; MMBench 72.51 vs 73.50; POPE 81.84 vs 82.57). Under sampling or optimization noise, that deficit could easily become a multi-point gap, which would change the conclusion from 'matches or exceeds' to 'slightly worse on vision with large text gains.' Please provide at least three seeds with error bars or confidence intervals, or explicitly restrict the claim to 'comparable on vision' with the uncertainty stated. The Limitations section does not currently address this single-run variance.","section":"§3.1, Tables 1–2"},{"comment":"The paper selects the 75% text-only ratio for MLAN after inspecting the knowledge-transfer curves in Figure 3, which are computed on the same held-out evaluation benchmarks used in Tables 1 and 2; no separate validation split is described. If the ratio was chosen by peeking at these curves, the reported comparison is optimistically selected rather than a prediction, and the 0.33-point vision deficit could reflect overfitting to the chosen evaluation suite. Please either document that the ratio was fixed before evaluation, or introduce a validation split (for example, hold out a subset of the 12 benchmarks for ratio selection and report the remaining benchmarks as the headline results).","section":"§3.3, Figure 3"},{"comment":"The baselines called 'MIX-LLaVA-1.5' and 'MIX-Cambrian-1' are not the actual LLaVA-1.5 or Cambrian-1 instruction tuning recipes: they are simplified mixtures sampled from the same two datasets (Super-NaturalInstructions and Vision-Flan) with 6% and 25% text-only ratios at a fixed 186k-instance budget. Actual LLaVA-1.5 uses 665k instances and Cambrian-1 uses millions of instances with different data sources, as Table 8 itself shows. Therefore, the conclusion that MLAN 'matches or better performance' on downstream vision-language tasks compared with these state-of-the-art recipes overstates what is directly supported. Please either compare against the real recipes (ideally at matched token budgets) or rename the baselines to 'proxy mixtures with LLaVA-1.5/Cambrian-1 text ratios' and avoid the state-of-the-art claim.","section":"§3.1, Table 3 and Appendix D.1"}],"minor_comments":[{"comment":"The treatment of ScienceQA is inconsistent: the Table 1 note says it is 'included in Vision-Flan but excluded in experiments,' whereas Appendix A says it is removed from the training set for evaluation, and Table 7 lists it as an evaluation benchmark. Please clarify whether it was excluded from training or from evaluation, or both, and correct the wording accordingly.","section":"Table 1, Appendix A, Table 7"},{"comment":"The statistical tests for cosine similarity report only p-values; please also report effect sizes and confidence intervals, since with 100k samples a 'significantly non-negative mean' is a weak statement that does not quantify how similar the instructions actually are.","section":"§2.1, Figure 4"},{"comment":"The knowledge-transfer curve would be more informative with per-benchmark values and error bars; currently the text 'peaks and then slightly declines' cannot be quantitatively verified from the figure.","section":"§3.3, Figure 3"},{"comment":"Because most production MLLMs use an instruction-tuned chat backbone, the 'Instruct LLM' row is an important caveat: the main comparison in Tables 1–2 uses non-instruction base models, so the practical significance of MLAN for standard recipes remains unclear. Please discuss whether the main conclusions hold with chat backbones.","section":"§3.4, Table 4"},{"comment":"The Cambrian-1 row in Table 8 is garbled ('Cambrian-1 (Tong et al., 2024) – Cambrian-7M 1.68M ∼7M 23.8%'); please fix the formatting so the dataset size and text-only percentage are unambiguous.","section":"Appendix D.1, Table 8"},{"comment":"The text states '12 comprehensive benchmarks' while Tables 6–7 list 13 datasets because ARC-E and ARC-C are reported separately; please align the count.","section":"§2.3, Tables 6–7"},{"comment":"No code, configuration, or checkpoint release is mentioned; given that the headline depends on exact data sampling and the 75% ratio, releasing the data mixture and training configuration would substantially improve reproducibility.","section":"Appendix B / Data Availability"}],"recommendation":"major_revision","confidential_remarks":"The paper is honest about several limitations, but the ratio-selection and single-run issues are more serious than the Limitations section suggests. I would ask the authors to add a validation split or otherwise justify that the 75% ratio was not chosen on the reported benchmarks, and to report seed variance. The baseline naming should also be corrected. If those are addressed, the paper could be acceptable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: MLAN's controlled fixed-budget modality sweep is a real contribution, but the \"matches or exceeds\" vision claim is not yet proven—single runs and a post-hoc ratio choice make it conditional, not settled.\n\nWhat's actually new: a systematic sweep over text-only vs. vision-language instruction data under a fixed 186k-instance budget, with token counts reported. Prior work mixed in text-only data as regularization or forgetting prevention, but didn't measure the ratio effect with token efficiency as a headline metric. That framing is useful and the text-side results are strong and consistent across both model sizes. The token savings (roughly half) are real and well documented.\n\nThe soft spots are in proportion to how much they matter. The vision-side comparison is the load-bearing one: for Llama-3.1-8B, MLAN trails MIX-Cambrian-1 by 0.33 average points, with individual benchmarks trading wins and losses by several points. All results are single-run, no seeds, no intervals. A 0.33-point gap can easily flip under another seed, so \"matches or exceeds\" overstates what the data support. That's a genuine underdetermination, and the paper's limitations section doesn't acknowledge it.\n\nThe second issue is ratio selection. The 75% figure appears to have been chosen after inspecting the knowledge-transfer curve (Figure 3) on the same benchmarks used in the main tables. No validation split is described. That makes the headline number optimistically selected. The qualitative finding—text-heavy helps, vision peaks then slightly declines—is robust across the curve, but the exact 75/25 recommendation shouldn't be treated as a prediction.\n\nThe proxy baselines (6% and 25% text using the same two datasets rather than actual LLaVA/Cambrian data) are a fair controlled choice, but they only isolate the ratio effect; the paper shouldn't be read as beating the real recipes. That framing is a minor overreach. The cosine-similarity motivation is also weak—non-negative mean similarity is a very low bar—but it's motivational, not load-bearing.\n\nWho it's for: anyone working on multimodal instruction tuning or MLLM training efficiency. It deserves a serious referee: the experimental setup is clear and the token-efficiency claim is practically relevant. But I'd send it back for multiple seeds, confidence intervals, and a validation split for ratio selection, or a revised claim that doesn't sell 75/25 as optimal. I'd also ask for code or data artifacts; right now none are released.","headline":"Useful controlled sweep on modality ratios with token accounting; the vision-parity claim needs seeds and a validation split before it's believed.","tokens_in":20461,"tokens_out":3359,"would_cite":true,"duration_ms":33252,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that instruction tuning of multimodal language models with a text-heavy mixture (75% text-only, 25% vision-language) under a fixed 186,000-instance budget matches or outperforms vision-heavy mixtures on both text and…","keywords":["instruction tuning","multimodal large language models","zero-shot generalization","text-only data","vision-language transfer","training efficiency","data mixture","catastrophic forgetting"],"falsifier":"Evaluate MLAN on vision tasks with no close text-only analogue, such as OCR, fine-grained object grounding, or object counting in images, under the same 186,000-instance budget; if the 75% text-heavy mixture falls clearly below the vision-heavy Cambrian-1 mixture on those benchmarks, the claim that language-based tuning generally preserves and transfers vision knowledge is falsified.","tokens_in":19531,"feed_emoji":"🤖","tokens_out":9922,"duration_ms":76052,"temperature":0.7,"pith_summary":"The paper claims that, after multimodal pretraining, the instruction tuning stage of a multimodal language model does not need to be dominated by image-text data. It proposes MLAN, a fixed-budget mixture that is 75% text-only and 25% vision-language instruction data, trained on 186,000 instances sampled from Super-NaturalInstructions and Vision-Flan. Across 12 held-out text and vision benchmarks, this text-heavy mixture matches or slightly beats the vision-heavy mixtures used by LLaVA-1.5 and Cambrian-1 on two Llama-based models, while seeing fewer than half the images and processing up to about half the training tokens. The paper's explanation is that instruction-following ability and domain knowledge are mostly language-borne once vision-language alignment is done, so a small amount of vision data suffices for grounding. If correct, this would make multimodal instruction tuning substantially cheaper and more reliant on diverse text-only task supervision.","feed_headline":"75% text-only data beats vision-heavy tuning at half the tokens","feed_subtitle":"A 75% text-only mixture matches vision-heavy mixtures on 12 benchmarks and cuts training tokens by up to half.","key_machinery":"The load-bearing mechanism is task-level semantic alignment between text-only and vision-language instruction data. The paper samples 100,000 instruction prompts from Super-NaturalInstructions and Vision-Flan, embeds them with a pretrained sentence transformer, and reports a significantly non-negative mean cosine similarity between the two modalities, reasoning that tasks are defined by their instructions and that shared task semantics transfer once vision-language pretraining aligns image tokens with text tokens. The method itself is a controlled training recipe: a fixed 186,000-instance budget, FLAN-style formatting, a CLIP-ViT-L/14@336 visual encoder with a two-layer MLP projector, and a fully unfrozen LLM, with only the data composition varied across experiments.","core_discovery":"The central discovery is that instruction tuning of a multimodal LLM can be re-oriented around text without losing vision performance. Under a fixed budget of 186,000 training instances, the MLAN mixture of 75% text-only and 25% vision-language instructions yields held-out zero-shot performance on both modalities that is on par with or better than vision-heavy recipes (MIX-LLaVA-1.5 with 6% text and MIX-Cambrian-1 with 25% text). On the language benchmarks, MLAN averages 64.50 versus 63.18 for MIX-Cambrian-1 on Llama-3.2-3B and 71.57 versus 67.76 on Llama-3.1-8B; on vision benchmarks it averages 59.13 versus 58.57 on the 3B model and trails MIX-Cambrian-1 by 0.33 points on the 8B model. The text-heavy model processes 60.1 million training tokens, compared with 101.5 million for the Cambrian-1 mix and 117.2 million for the LLaVA-1.5 mix, a reduction of roughly 40% to 49%. The paper argues the transfer is possible because text-only and vision-language instructions share task-level semantics, and its controlled ablation shows that even 12.5% text-only data sharply raises both text and vision scores.","pith_inferences":["If text-heavy tuning generalizes beyond the two models tested here, the cost bottleneck of multimodal instruction tuning shifts from collecting image-text pairs to assembling diverse text-only task mixtures; a testable extension is to select text-only tasks whose instruction embeddings are most similar to a target vision benchmark and measure the resulting vision score.","The cosine-similarity analysis implies a practical data-selection tool: embed candidate text-only and vision-language instructions, then choose cross-modally similar subsets instead of fixing ratios by hand.","The asymmetric forgetting pattern — text abilities erode under vision-heavy tuning while vision abilities improve under text tuning — suggests language knowledge is the fragile resource in multimodal models, a prediction that could be checked on other architectures and other pretraining corpora.","The paper's own limitation section notes that OCR, captioning, and other specialized out-of-distribution vision tasks were not evaluated; a natural next test is whether the 75/25 mixture holds up when those tasks are added to the benchmark suite."],"forward_implications":["A fixed-budget instruction tuning set can be made 75% text-only without sacrificing held-out vision performance, on both a 3B and an 8B Llama-based multimodal model.","Vision-heavy instruction mixtures (6% to 25% text) erode language knowledge on datasets like CommonsenseQA and CosmosQA by up to 20 percentage points, while the text-heavy mixture largely avoids that degradation.","Because CLIP converts each image into 576 visual tokens, replacing vision instances with text instances at the same instance budget cuts the number of training tokens processed by roughly half.","The mixture ratio is not arbitrary: 12.5% text-only data already produces a sharp gain on both text and vision axes, and vision performance peaks at a moderate language share, showing that neither pure modality is sufficient."],"supporting_citations":[{"why":"Supplies the LLaVA architecture and the 6% text-only mixture baseline used as the vision-heavy comparison.","marker":"Liu et al., 2023a"},{"why":"Supplies the Cambrian-1 mixture baseline with 25% text-only data that MLAN is compared against.","marker":"Tong et al., 2024"},{"why":"Source of the 1,600+ text-only tasks sampled for the language-heavy training mixture.","marker":"Wang et al., 2022b"},{"why":"Source of the 187 vision-language tasks and the counterpart dataset in the cross-modal similarity analysis.","marker":"Xu et al., 2024"},{"why":"Provides the FLAN-style instruction formatting used for both training and evaluation prompts.","marker":"Wei et al., 2022"},{"why":"Provides the CLIP-ViT-L/14@336 encoder whose 576 visual tokens drive the training-token cost comparison.","marker":"Radford et al., 2021"},{"why":"Provides the Llama-3.2-3B and Llama-3.1-8B base language models used in all experiments.","marker":"Dubey et al., 2024"},{"why":"Provides the all-mpnet-base-v2 sentence transformer used to embed instructions for the similarity evidence.","marker":"Song et al., 2020"}],"fun_headline_variants":["Text-only data cuts tokens by half and matches vision-heavy tuning","75% text beats 50% vision for multimodal LLM training","Multimodal LLMs learn better with mostly text instructions","Language-heavy tuning transfers knowledge across modalities","Half the tokens, same performance: text-heavy instruction tuning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The transfer mechanism assumes that cosine similarity between embedded text-only and vision-language instruction prompts is a reliable proxy for whether abilities learned from text-only data will transfer to image-grounded tasks; if that similarity is not the right proxy, the motivation for the text-heavy mixture weakens even if the fixed-budget empirical results still hold.","fun_headline_variants_meta":{"raw":{"variants":["Text-only data cuts tokens by half and matches vision-heavy tuning","75% text beats 50% vision for multimodal LLM training","Multimodal LLMs learn better with mostly text instructions","Language-heavy tuning transfers knowledge across modalities","Half the tokens, same performance: text-heavy instruction tuning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000707,"raw_usage":{"total_tokens":3209,"prompt_tokens":989,"completion_tokens":2220,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":605,"completion_tokens_details":{"reasoning_tokens":2140}},"tokens_in":605,"tokens_out":2220,"duration_ms":15416,"temperature":1.0,"reasoning_tokens":2140,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T19:34:14.267083+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Evaluate MLAN on vision tasks with no close text-only analogue, such as OCR, fine-grained object grounding, or object counting in images, under the same 186,000-instance budget; if the 75% text-heavy mixture falls clearly below the vision-heavy Cambrian-1 mixture on those benchmarks, the claim that language-based tuning generally preserves and transfers vision knowledge is falsified.","supporting_citations":[],"review_version":1}