{"id":"ba4196bf-3e71-4764-93cc-0c482d92b442","arxiv_id":"2412.08307","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Multimodal language models trained with a medium number of instruction templates (5,000 for 7B, 100 for 13B) outperform both fewer and many more templates, with gains up to 10 points on small benchmark samples.","lead":"Training a picture-answering AI on many differently worded questions can help, but only up to a point: this study finds a medium number of wordings works best. The practical upshot is that smarter wording reuse may improve multimodal models without collecting more data, though the result needs stronger statistical validation.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Holding total dataset size fixed while varying template count confounds template scale with per-template exposure frequency, so the reported non-monotonic curve may be a repetition–diversity tradeoff rather than a template-scaling law.","rationale":"The paper makes a real contribution by introducing a scalable 15K-template generator and systematically sweeping template counts on two model sizes, and the qualitative claim that more templates do not monotonically help is plausible. However, the experimental design fixes total training size at 665K while varying template count, making template count and per-template exposure frequency inversely related. With N=10 each template appears roughly 66,500 times; with N=15K only about 44 times. The 'optimal scale' (5K for 7B, 100 for 13B) may thus be the point where repetition and diversity balance, not a scaling law in template count itself. The paper's statement that the fixed-size construction isolates the effect of template scale is therefore the weakest step, and the reader correctly identifies it. The additional in-domain evaluation inconsistency strengthens this: the 100 'in-domain' templates are described as encountered during training, but for N=10 only 10 of those 100 were actually used in training, so the in-domain curves mix training overlap with template-scale effects. The proposed matched-repetition experiment would settle whether the non-monotonic curve survives when exposure frequency is controlled. Because the reader's verdict was already CONDITIONAL and this is the same central weakness, I recommend leaving the verdict unchanged; the quantitative scaling claims should not be accepted unless the authors provide this control or an equivalent analysis.","tokens_in":28049,"tokens_out":5593,"duration_ms":64449,"concrete_test":"Retrain the 7B model at N = 10, 100, 1K, 5K, 10K, 15K while holding expected per-template exposure constant, e.g., scale total training size as N × 665K / 100 so each template appears about 6,650 times at every N, and evaluate on the same out-of-domain and matched in-domain template sets. If the inverted-U peak persists under matched repetition, the template-scale interpretation survives; if the curve flattens or the peak moves, the original results are driven by the repetition–diversity confound.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central scaling result is produced by six training sets of the same total size (665K) using 10, 100, 1K, 5K, 10K, and 15K templates. This design makes template count and per-template exposure frequency inversely related: with N=10 each template is seen roughly 66,500 times, while with N=15K each is seen only about 44 times. Section 3.1 presents the fixed-size construction as a control ('maintain the same size as the original'), but it is exactly what introduces the confound. The observed rise-then-fall curve could therefore reflect a standard repetition–diversity tradeoff: very few templates cause overfitting to a handful of surface forms, while very many templates give each form too little exposure to be learned. This confound affects both the in-domain and the manually written out-of-domain evaluations, because all training runs share the same total-size constraint. A second, related inconsistency: the 100 'in-domain' evaluation templates are stated to have been 'encountered during training,' but for the 10-template training condition at least 90 of those 100 templates were never in the training set. That makes the in-domain curves partly a measure of training/evaluation overlap. Either issue alone prevents attributing the non-monotonicity to template scale per se rather than to exposure frequency.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates how the number of distinct instruction templates used during visual instruction tuning affects multimodal language model (MLM) performance. The authors build a programmatic generator that can produce roughly 15K instruction templates from 24 meta templates, and they fine-tune LLaVA-1.5-7B and LLaVA-1.5-13B on six versions of the same 665K-sample instruction dataset, using 10, 100, 1K, 5K, 10K, and 15K templates. Performance is measured on five VQA benchmarks, each evaluated with 100 in-domain templates, 25 manually written out-of-domain templates, and three simple templates. The central empirical claim is that MLM performance is non-monotonic in template scale, peaking at an intermediate number of templates: 5K for the 7B model and 100 for the 13B model. The paper further claims that models trained at these optimal scales outperform comparable models trained on up to 75 times more instruction data, while also reducing performance variance across instruction templates.","tokens_in":28329,"tokens_out":4766,"duration_ms":53317,"significance":"If the main claim were established, it would be practically significant: it would show that visual instruction tuning can be improved substantially without adding new training data, and that template diversity is not a quantity to be maximized blindly. The generator itself is a useful contribution, and the systematic sweep over six template scales across two model sizes is a reasonable way to probe the question. The variance-reduction analysis across templates is also valuable and is the kind of robustness evidence the field needs. However, the central scaling conclusion is currently undercut by a design confound between template count and per-template exposure frequency, by an evaluation whose statistical precision is too low for the claimed peaks, and by selection of the optimal scale on the same benchmarks used for the final comparison. The paper is therefore interesting but the headline result is not yet supported.","major_comments":[{"comment":"The fixed-size design confounds template scale with per-template exposure frequency. The text states that the six template-augmented versions 'maintain the same size as the original' 665K dataset while using 10, 100, 1K, 5K, 10K, and 15K templates. Consequently, with 10 templates each template appears roughly 66,500 times, while with 15K templates each appears only about 44 times. The rise-then-fall curve in Figure 4 may therefore reflect a repetition-diversity tradeoff rather than a template-scaling law. To support the central claim, the authors should either vary template count while holding per-template exposure roughly constant (for example, by scaling total dataset size with template count, or by controlling the number of samples per template) or report per-template exposure and show that the non-monotonic trend is not explained by it.","section":"Sec. 3.1"},{"comment":"The in-domain evaluation is not consistently 'in-domain' across training conditions. The paper says the 100 in-domain evaluation templates 'have encountered during training,' but in the 10-template training condition at most 10 of those templates can have been seen during training; for the 100-template condition the overlap depends on whether the evaluation templates are exactly the training templates. This means the in-domain curves in Figures 3 and 4 partly measure train/evaluation overlap rather than a systematic effect of template scale. The authors should specify how the 100 evaluation templates relate to the template pool used for each training run, and report the actual overlap for each condition.","section":"Sec. 3.1"},{"comment":"The evaluation has no error bars and no repeated sampling. The protocol randomly selects 100 data points per benchmark according to category distribution, once, with no seeds or repeated trials reported. For 100 binary-accuracy items the standard error can be as large as 5 points, and many of the between-condition differences in Table 1 are only a few points. The claims that the 7B model peaks at 5K templates and the 13B model peaks at 100 templates, and that these peaks are robust, require confidence intervals, repeated subsampling, or evaluation on the full benchmark sets. Without this, the specific peak locations may be noise.","section":"Sec. 3.1 and Sec. 3.2"},{"comment":"The optimal template scale is selected on the same benchmarks used to report the final gains. The peak scale is identified from average performance on the five benchmarks in Figure 4, and Table 1 then uses those same benchmarks to claim that the selected 5K/100-template models outperform baselines. This is a form of test-set selection and inflates the reported gains. The authors should either hold out a validation set for scale selection, report results for all six scales in the comparison table, or use an additional benchmark suite for the final claim that the optimal-scale model is the best overall.","section":"Sec. 3.2 and Sec. 4.2"}],"minor_comments":[{"comment":"The phrase 'randomly select 100 data points for each dataset according to their category distribution' is ambiguous: it should say whether this is a stratified sample of 100 total items or a fixed number per category, and it should report the random seed used.","section":"Sec. 3.1"},{"comment":"The column header 'OverallS ID OOD' is difficult to parse; the table would be clearer with a separate 'Overall' column followed by clearly separated 'S', 'ID', and 'OOD' subcolumns for each benchmark.","section":"Table 1"},{"comment":"The caption text 'Optimal template scale vary across different datasets' has a subject-verb agreement error and should be 'varies'; the same issue appears in the main text.","section":"Figures 3 and 4"},{"comment":"The related work cites the authors' own Provision paper (Zhang et al., 2024b) for the benefit of multi-template training, but the present paper's generator and augmentation recipe appear closely related to Provision; the authors should clarify the specific incremental contribution beyond that prior work.","section":"Sec. 5 and References"},{"comment":"The notation '50M' for Qwen-VL-7B is inconsistent with '665K' and '760k' used elsewhere; please use uniform units (e.g., 50M, 760K, 665K) and clarify whether the number refers to instruction-tuning samples.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The referee's main concern is not about the generator, which is a reasonable engineering contribution, but about whether the experimental design can support the scaling-law claim. The fixed-dataset-size confound and the low-precision evaluation are both fixable in revision, and I would be willing to look at a revised version that adds controlled experiments and error bars. I would also suggest the editors ask the authors to clarify the relationship with their own Provision paper, since the novelty boundary is currently unclear."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Hi [Name],\n\nThe thing to know: this is a useful empirical observation wrapped in a design that doesn't yet support the central claim. The authors build a programmatic generator (24 meta templates, synonym slots, weighted sampling) and show that for LLaVA-1.5-7B/13B, average accuracy across five VQA benchmarks rises then falls as the number of instruction templates grows, with peaks at 5K and 100 respectively. That non-monotonic curve is the paper's real contribution, and the generator itself is a reasonable, reusable tool.\n\nWhat the paper does well: the evaluation uses both in-domain and manually written out-of-domain templates, the training protocol is described in enough detail to reproduce (LoRA, same 665K set, six scales), and the claim that template augmentation can reduce performance variance across templates (Max-Min) is supported as far as it goes. The comparison against bigger-data baselines is an interesting frame, even if the headline 'beats 75x data' depends on the same confounded runs.\n\nThe soft spot is structural. Section 3.1 fixes total dataset size at 665K and varies template count from 10 to 15K. That makes per-template exposure inversely proportional to template count: at 10 templates each appears ~66,500 times, at 15K each appears ~44 times. So the rise-then-fall curve is exactly what a repetition–diversity tradeoff would look like; the design cannot separate template-scale effects from exposure frequency. The same issue contaminates the 'in-domain' evaluation: with 10 training templates, calling 100 generated templates 'encountered during training' is only true for at most 10 of them. There are also no error bars, no repeated seeds, and the optimal scale is selected on the same test benchmarks used for the final comparison. These are not minor quibbles: they are load-bearing for the headline claim.\n\nThat said, the qualitative direction is plausible, and the generator plus the evaluation protocol are worth building on. I would not take the specific optimum values (5K vs 100) as established.\n\nRecommendation: send it to review, but the reviewers should ask for a design that decouples template count from exposure, plus repeated seeds and a held-out validation set for choosing the optimum. Who it's for: people working on instruction tuning and robustness of MLMs, and anyone building template-diverse training data.","headline":"A useful empirical observation about medium template optima, but the fixed-dataset-size design cannot separate template count from per-template exposure, so the central scaling claim is not yet established.","tokens_in":28866,"tokens_out":1860,"would_cite":false,"duration_ms":19299,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Instruction-template scale has a sweet spot for multimodal AI: performance rises, then declines.","keywords":["instruction templates","template scaling","multimodal language models","visual instruction tuning","data augmentation","prompt sensitivity","LLaVA","LoRA"],"falsifier":"Train the 7B model with 5K templates but give each template the same exposure count as in the 10-template run (by shrinking the dataset or repeating the pool), and see whether the 5K peak persists; if the peak moves, the inverted-U shape is driven by exposure frequency rather than template count.","tokens_in":27855,"feed_emoji":"🧩","tokens_out":11556,"duration_ms":108620,"temperature":0.7,"pith_summary":"Multimodal language models do not get steadily better as more instruction templates are added to their training data; the paper argues there is a medium-scale optimum. Using a programmatic generator that can produce over 15,000 unique templates, the authors fine-tune LLaVA-1.5-7B and LLaVA-1.5-13B models on the same 665K-sample instruction set while varying only the number of templates from 10 to 15,000. Across five visual question-answering benchmarks the average performance rises, peaks, and then declines: the 7B model peaks at 5,000 templates and the 13B model at 100. If this holds, instruction tuning can be improved by up to 10 percent and made less sensitive to prompt phrasing without collecting any new training data.","feed_headline":"LLaVA-1.5 peaks at medium template scale, not larger","feed_subtitle":"7B peaks at 5K templates, 13B at 100; optimal tuning beats 75x-data models.","key_machinery":"The load-bearing object is a programmatic instruction template generator. It fills placeholders in 24 hand-designed meta templates with position-specific synonyms, and a sentence-pattern tree organizes those meta templates by grammatical structure; a weight-accumulation sampling algorithm then makes every one of the 15K generated instruction templates roughly equally likely to appear. This keeps the training set's size and content fixed while letting the number of distinct templates vary from 10 to 15K, which is what makes the scaling curves interpretable. All twelve training runs use Low-Rank Adaptation (LoRA) on the same LLaVA-1.5 base checkpoints.","core_discovery":"The paper's central discovery is that MLM capability is a non-monotonic function of instruction-template scale. With the training set held at 665K samples, increasing template count from 10 to 15K produces an inverted-U average-performance curve on five VQA benchmarks. The optimal scale depends on model size: LLaVA-1.5-7B peaks at 5K templates, while LLaVA-1.5-13B peaks at only 100 templates. Models trained at their optimal template scale outperform the original data-only LLaVA-1.5 models by up to 10 percent and achieve the best overall performance relative to similar-scale models trained on up to 75.19 times more instruction data, while also reducing the spread between best and worst accuracy across evaluation templates.","pith_inferences":["Because the total sample count is fixed, varying template scale also varies how often each template is seen during training; separating that repetition effect from template count would be a direct test of the paper's mechanism.","If this non-monotonic pattern extends to text-only language models, template count would become a cheap scaling axis for instruction tuning alongside data size and model size.","The generator varies lexical and syntactic surface form only; template families that change semantics, language, or reasoning style may have a different optimum.","Holding per-template exposure frequency constant (by subsampling or repeating templates) while changing the number of distinct templates could yield a sharper design rule than the current inverted-U curve."],"forward_implications":["Template count becomes a tunable hyperparameter of visual instruction tuning; adding more templates is not automatically better.","At the optimal scale, template augmentation can substitute for much larger instruction datasets, saving data collection and compute.","The optimal template scale is model-dependent, with stronger models needing fewer templates.","Training with the right template scale reduces performance fluctuation across prompt phrasings, which should make MLM evaluation and deployment more stable.","A medium-scale template pool is enough, so an exhaustive large-scale template search is unnecessary."],"supporting_citations":[{"why":"Supplies the LLaVA-1.5 base checkpoints and the 665K-sample instruction dataset used for every training run.","marker":"Liu et al., 2024a"},{"why":"Documents instruction-template sensitivity exceeding 40 percent for MLMs, motivating the question of how many templates to train with.","marker":"Zhang et al., 2024a"},{"why":"Shows that multi-template training improves MLM performance and reduces template sensitivity; this is the approach the paper scales up.","marker":"Zhang et al., 2024b"},{"why":"Provides the prompt-formatting sensitivity analysis and the Max-Min fluctuation metric used to measure robustness.","marker":"Sclar et al., 2023"},{"why":"Frames the scaling-law expectation that more training data helps, which the non-monotonic template result qualifies.","marker":"Kaplan et al., 2020"},{"why":"LoRA is the adaptation method used for all twelve trained models, keeping training runs comparable.","marker":"Hu et al., 2021"},{"why":"Supplies the Qwen-VL baseline trained on 50M instruction samples, the largest-data comparison supporting the data-efficiency claim.","marker":"Bai et al., 2023"},{"why":"Provides the SeedBench benchmark used in the main comparison figure and one of the five evaluation datasets.","marker":"Li et al., 2023b"}],"fun_headline_variants":["More templates don't help multimodal LMs","Optimal template scale beats 75x data","LLaVA-1.5: medium template scale wins","Template scaling: inverted-U curve for LMs","7B peaks at 5K templates, 13B at 100"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that holding the training set at 665K samples while varying the number of templates isolates the template-scaling effect; but with few templates each one is seen many more times, and with many templates most are seen only briefly, so the curve could instead reflect a repetition-diversity tradeoff or evaluation familiarity.","fun_headline_variants_meta":{"raw":{"variants":["More templates don't help multimodal LMs","Optimal template scale beats 75x data","LLaVA-1.5: medium template scale wins","Template scaling: inverted-U curve for LMs","7B peaks at 5K templates, 13B at 100"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000338,"raw_usage":{"total_tokens":1842,"prompt_tokens":891,"completion_tokens":951,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":507,"completion_tokens_details":{"reasoning_tokens":872}},"tokens_in":507,"tokens_out":951,"duration_ms":9853,"temperature":1.0,"reasoning_tokens":872,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:57:53.647578+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the 7B model with 5K templates but give each template the same exposure count as in the 10-template run (by shrinking the dataset or repeating the pool), and see whether the 5K peak persists; if the peak moves, the inverted-U shape is driven by exposure frequency rather than template count.","supporting_citations":[],"review_version":1}