Pith. sign in

REVIEW 4 major objections 5 minor 31 references

Investigating the Scaling Effect of Instruction Templates for Training Multimodal Language Model

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Instruction-template scale has a sweet spot for multimodal AI: performance rises, then declines.

desk verdict A useful empirical observation about medium template optima, but the fixed-dataset-size design cannot separate template count from per-template exposure, so the central scaling claim is not yet established. read the letter →

arxiv 2412.08307 v3 pith:VHTQRDDR submitted 2024-12-11 cs.CV

Shijian Wang , Linxin Song , Jieyu Zhang , Ryotaro Shimizu , Jiarui Jin , Ao Luo , Yuan Lu , Li Yao
show 4 more authors
Cunjian Chen Julian McAuley Wentao Zhang Hanqian Wu
This is my paper · ORCID
classification cs.CV
keywords instructiontemplatestemplatescalingmultimodallanguagemodelsvisualtuningdataaugmentationpromptsensitivityLLaVALoRA
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Multimodal language models do not get steadily better as more instruction templates are added to their training data; the paper argues there is a medium-scale optimum. Using a programmatic generator that can produce over 15,000 unique templates, the authors fine-tune LLaVA-1.5-7B and LLaVA-1.5-13B models on the same 665K-sample instruction set while varying only the number of templates from 10 to 15,000. Across five visual question-answering benchmarks the average performance rises, peaks, and then declines: the 7B model peaks at 5,000 templates and the 13B model at 100. If this holds, instruction tuning can be improved by up to 10 percent and made less sensitive to prompt phrasing without collecting any new training data.

What carries the argument

The load-bearing object is a programmatic instruction template generator. It fills placeholders in 24 hand-designed meta templates with position-specific synonyms, and a sentence-pattern tree organizes those meta templates by grammatical structure; a weight-accumulation sampling algorithm then makes every one of the 15K generated instruction templates roughly equally likely to appear. This keeps the training set's size and content fixed while letting the number of distinct templates vary from 10 to 15K, which is what makes the scaling curves interpretable. All twelve training runs use Low-Rank Adaptation (LoRA) on the same LLaVA-1.5 base checkpoints.

What would settle it

Train the 7B model with 5K templates but give each template the same exposure count as in the 10-template run (by shrinking the dataset or repeating the pool), and see whether the 5K peak persists; if the peak moves, the inverted-U shape is driven by exposure frequency rather than template count.

Watch

Extended reading notes

Core claim

The paper's central discovery is that MLM capability is a non-monotonic function of instruction-template scale. With the training set held at 665K samples, increasing template count from 10 to 15K produces an inverted-U average-performance curve on five VQA benchmarks. The optimal scale depends on model size: LLaVA-1.5-7B peaks at 5K templates, while LLaVA-1.5-13B peaks at only 100 templates. Models trained at their optimal template scale outperform the original data-only LLaVA-1.5 models by up to 10 percent and achieve the best overall performance relative to similar-scale models trained on up to 75.19 times more instruction data, while also reducing the spread between best and worst accuracy across evaluation templates.

Load-bearing premise

The load-bearing premise is that holding the training set at 665K samples while varying the number of templates isolates the template-scaling effect; but with few templates each one is seen many more times, and with many templates most are seen only briefly, so the curve could instead reflect a repetition-diversity tradeoff or evaluation familiarity.

Editorial extensions

If this is right

  • Template count becomes a tunable hyperparameter of visual instruction tuning; adding more templates is not automatically better.
  • At the optimal scale, template augmentation can substitute for much larger instruction datasets, saving data collection and compute.
  • The optimal template scale is model-dependent, with stronger models needing fewer templates.
  • Training with the right template scale reduces performance fluctuation across prompt phrasings, which should make MLM evaluation and deployment more stable.
  • A medium-scale template pool is enough, so an exhaustive large-scale template search is unnecessary.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the total sample count is fixed, varying template scale also varies how often each template is seen during training; separating that repetition effect from template count would be a direct test of the paper's mechanism.
  • If this non-monotonic pattern extends to text-only language models, template count would become a cheap scaling axis for instruction tuning alongside data size and model size.
  • The generator varies lexical and syntactic surface form only; template families that change semantics, language, or reasoning style may have a different optimum.
  • Holding per-template exposure frequency constant (by subsampling or repeating templates) while changing the number of distinct templates could yield a sharper design rule than the current inverted-U curve.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper investigates how the number of distinct instruction templates used during visual instruction tuning affects multimodal language model (MLM) performance. The authors build a programmatic generator that can produce roughly 15K instruction templates from 24 meta templates, and they fine-tune LLaVA-1.5-7B and LLaVA-1.5-13B on six versions of the same 665K-sample instruction dataset, using 10, 100, 1K, 5K, 10K, and 15K templates. Performance is measured on five VQA benchmarks, each evaluated with 100 in-domain templates, 25 manually written out-of-domain templates, and three simple templates. The central empirical claim is that MLM performance is non-monotonic in template scale, peaking at an intermediate number of templates: 5K for the 7B model and 100 for the 13B model. The paper further claims that models trained at these optimal scales outperform comparable models trained on up to 75 times more instruction data, while also reducing performance variance across instruction templates.

Significance. If the main claim were established, it would be practically significant: it would show that visual instruction tuning can be improved substantially without adding new training data, and that template diversity is not a quantity to be maximized blindly. The generator itself is a useful contribution, and the systematic sweep over six template scales across two model sizes is a reasonable way to probe the question. The variance-reduction analysis across templates is also valuable and is the kind of robustness evidence the field needs. However, the central scaling conclusion is currently undercut by a design confound between template count and per-template exposure frequency, by an evaluation whose statistical precision is too low for the claimed peaks, and by selection of the optimal scale on the same benchmarks used for the final comparison. The paper is therefore interesting but the headline result is not yet supported.

major comments (4)
  1. [Sec. 3.1] The fixed-size design confounds template scale with per-template exposure frequency. The text states that the six template-augmented versions 'maintain the same size as the original' 665K dataset while using 10, 100, 1K, 5K, 10K, and 15K templates. Consequently, with 10 templates each template appears roughly 66,500 times, while with 15K templates each appears only about 44 times. The rise-then-fall curve in Figure 4 may therefore reflect a repetition-diversity tradeoff rather than a template-scaling law. To support the central claim, the authors should either vary template count while holding per-template exposure roughly constant (for example, by scaling total dataset size with template count, or by controlling the number of samples per template) or report per-template exposure and show that the non-monotonic trend is not explained by it.
  2. [Sec. 3.1] The in-domain evaluation is not consistently 'in-domain' across training conditions. The paper says the 100 in-domain evaluation templates 'have encountered during training,' but in the 10-template training condition at most 10 of those templates can have been seen during training; for the 100-template condition the overlap depends on whether the evaluation templates are exactly the training templates. This means the in-domain curves in Figures 3 and 4 partly measure train/evaluation overlap rather than a systematic effect of template scale. The authors should specify how the 100 evaluation templates relate to the template pool used for each training run, and report the actual overlap for each condition.
  3. [Sec. 3.1 and Sec. 3.2] The evaluation has no error bars and no repeated sampling. The protocol randomly selects 100 data points per benchmark according to category distribution, once, with no seeds or repeated trials reported. For 100 binary-accuracy items the standard error can be as large as 5 points, and many of the between-condition differences in Table 1 are only a few points. The claims that the 7B model peaks at 5K templates and the 13B model peaks at 100 templates, and that these peaks are robust, require confidence intervals, repeated subsampling, or evaluation on the full benchmark sets. Without this, the specific peak locations may be noise.
  4. [Sec. 3.2 and Sec. 4.2] The optimal template scale is selected on the same benchmarks used to report the final gains. The peak scale is identified from average performance on the five benchmarks in Figure 4, and Table 1 then uses those same benchmarks to claim that the selected 5K/100-template models outperform baselines. This is a form of test-set selection and inflates the reported gains. The authors should either hold out a validation set for scale selection, report results for all six scales in the comparison table, or use an additional benchmark suite for the final claim that the optimal-scale model is the best overall.
minor comments (5)
  1. [Sec. 3.1] The phrase 'randomly select 100 data points for each dataset according to their category distribution' is ambiguous: it should say whether this is a stratified sample of 100 total items or a fixed number per category, and it should report the random seed used.
  2. [Table 1] The column header 'OverallS ID OOD' is difficult to parse; the table would be clearer with a separate 'Overall' column followed by clearly separated 'S', 'ID', and 'OOD' subcolumns for each benchmark.
  3. [Figures 3 and 4] The caption text 'Optimal template scale vary across different datasets' has a subject-verb agreement error and should be 'varies'; the same issue appears in the main text.
  4. [Sec. 5 and References] The related work cites the authors' own Provision paper (Zhang et al., 2024b) for the benefit of multi-template training, but the present paper's generator and augmentation recipe appear closely related to Provision; the authors should clarify the specific incremental contribution beyond that prior work.
  5. [Table 1] The notation '50M' for Qwen-VL-7B is inconsistent with '665K' and '760k' used elsewhere; please use uniform units (e.g., 50M, 760K, 665K) and clarify whether the number refers to instruction-tuning samples.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the scaling finding is an empirical measurement, not a derivation, and the self-citations are contextual rather than load-bearing.

full rationale

The paper's central claim is an empirical scaling curve: six training runs with 10/100/1K/5K/10K/15K templates on fixed 665K data are evaluated on five external benchmarks, and the paper reports the measured optimum (Sec 3.2, Figures 3-4). No quantity is fitted to a target and then announced as a prediction; the 'optimal template scale' is an observed peak, and the subsequent comparison in Sec 4 reports direct measurements. The only in-paper derivation, the uniform-sampling property of the sentence-pattern tree, is self-contained: Algorithm 1-2's weighted sampling makes each leaf probability w(leaf)/w(root), and the uniform placeholder choice gives each concrete template probability 1/w(root) (Sec 2.2). This does not rely on any cited result. The self-citations (Zhang et al. 2024b for multi-template training; Zhang et al. 2024a for template sensitivity/evaluation) are motivational and methodological; neither is used to derive the scaling law, which is established by the paper's own experiments. The reader-identified confound (fixed total size couples template count with per-template exposure frequency, and the 100 'in-domain' evaluation templates cannot all have been encountered by the 10-template training runs, Sec 3.1) is a validity/overfitting concern about what the curve isolates, not a circular reduction: the numbers are measurements, not definitional consequences. It should be fixed or discussed, but it does not make the derivation circular.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new physical or model entities. Its load-bearing assumptions are methodological: small benchmark samples stand in for full benchmarks, 25 manual templates stand in for all out-of-domain phrasings, and fixed dataset size supposedly isolates template scale even though per-template exposure frequency changes. The optimal template scales are selected on the same test benchmarks that are later used to claim success.

free parameters (3)
  • Optimal template scale (7B) = 5,000 templates
    Selected by comparing average accuracy across the five evaluation benchmarks; the same benchmarks are then used to report final results. This is a hand-selected hyperparameter with no held-out validation or repeated-seed estimate.
  • Optimal template scale (13B) = 100 templates
    Selected by the same procedure as the 7B optimum, on the same test benchmarks used for final claims.
  • Benchmark evaluation sample size = 100 questions per dataset
    Chosen for tractability in Sec 3.1; makes reported accuracies noisy and without error bars.
assumptions (5)
  • domain assumption The 100 randomly selected questions per benchmark are representative of the full benchmark.
    Sec 3.1: 'randomly select 100 data points for each dataset according to their category distribution'. All accuracy and Max-Min numbers in Table 1 rest on this sample.
  • domain assumption The 25 manually written templates represent a meaningful out-of-domain distribution.
    Sec 3.1: OOD evaluation uses only 25 templates written by the authors; no evidence is given that these are hard, diverse, or genuinely outside the training distribution.
  • ad hoc to paper Holding total training data size fixed while varying the template pool isolates the effect of template scale.
    Sec 3.1: 'maintain the same size as the original'. This is the main interpretive assumption; it also changes how many times each template is seen during training.
  • domain assumption The sentence-transformer fallback correctly maps model outputs to answer options.
    Sec 3.1 Evaluation Protocol: if string matching fails, embedding similarity selects the answer. No accuracy of this extraction method is reported.
  • standard math Weighted sampling in the sentence pattern tree produces a uniform distribution over generated templates.
    Algorithms 1 and 2; true if the tree weights are correctly accumulated, but no formal proof or code is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Investigating the Scaling Effect of Instruction Templates for Training Multimodal Language Model." pith.science (2026). https://pith.science/paper/VHTQRDDR

@misc{pith2026241208307,
  author       = {Pith},
  title        = {Pith review of: Investigating the Scaling Effect of Instruction Templates for Training Multimodal Language Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VHTQRDDR}},
  note         = {Machine review of arXiv:2412.08307}
}
read the original abstract

Current multimodal language model (MLM) training approaches overlook the influence of instruction templates. Previous research deals with this problem by leveraging hand-crafted or model-generated templates, failing to investigate the scaling effect of instruction templates on MLM training. In this work, we propose a programmatic instruction template generator capable of producing over 15K unique instruction templates by filling randomly sampled positional synonyms into weighted sampled meta templates, enabling us to comprehensively explore MLM's performance across various template scales in the training process. Our investigation into scaling instruction templates for MLM training demonstrates that MLM capabilities do not consistently improve with increasing template scale. Instead, optimal performance is achieved at a medium template scale. Models trained with data augmented at the optimal template scale achieve performance gains of up to 10% over those trained on the original data and achieve the best overall performance compared with the similar-scale MLMs tuned on at most 75 times the scale of our augmented dataset. The code will be publicly available at https://github.com/shijian2001/TemplateScaling.

Figures

Figures reproduced from arXiv: 2412.08307 by the authors.

Figure 1
Figure 1. Training with the optimal template scale significantly improves MLM’s perfor￾mance and reduces the performance variance. LLaVA-1.5-7B trained with 5K templates and LLaVA-1.5-13B trained with 100 templates achieve the highest average performance and the lowest performance variance among similar-scale MLMs on the SeedBench dataset, evaluated across 25 held-out instruction templates that are not included in training. p… view at source ↗
Figure 2
Figure 2. Example of the instruction template generation through a meta template. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Scaling trends of MLM performance with increasing template scale on each [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Scaling trend of MLM performance with increasing template scale on the average [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Sentence pattern trees with meta templates. Each tree uses distinct colors to denote [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]
Figure 6
Figure 6. Figure 6: Heat map illustrating the performance variations of eight MLMs on the BLINK [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

31 extracted references · 4 canonical work pages

  1. [1]

    Mc-llava: Multi-concept personalized vision-language model

    Ruichuan An, Sihan Yang, Ming Lu, Kai Zeng, Yulin Luo, Ying Chen, Jiajun Cao, Hao Liang, Qi She, Shanghang Zhang, et al. Mc-llava: Multi-concept personalized vision-language model. arXiv preprint arXiv:2411.11706,

    arXiv

  2. [3]

    Posix: A prompt sensitivity index for large language models

    Anwoy Chatterjee, HSVNS Kowndinya Renduchintala, Sumit Bhatia, and Tanmoy Chakraborty. Posix: A prompt sensitivity index for large language models. arXiv preprint arXiv:2410.02185,

    arXiv

  3. [5]

    Blink: Multimodal large language models can see but not perceive

    Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. Blink: Multimodal large language models can see but not perceive. arXiv preprint arXiv:2404.12390,

    arXiv

  4. [6]

    Sensitivity and robustness of large language models to prompt template in japanese text classification tasks

    Chengguang Gan and Tatsunori Mori. Sensitivity and robustness of large language models to prompt template in japanese text classification tasks. arXiv preprint arXiv:2305.08714,

    arXiv

  5. [7]

    Under review

    10 Preprint. Under review. Hila Gonen, Srini Iyer, Terra Blevins, Noah A Smith, and Luke Zettlemoyer. Demystifying prompts in language models via perplexity estimation. arXiv preprint arXiv:2212.04037,

    arXiv

  6. [11]

    The language of prompting: What linguistic properties make a prompt successful?

    Alina Leidinger, Robert Van Rooij, and Ekaterina Shutova. The language of prompting: What linguistic properties make a prompt successful? arXiv preprint arXiv:2311.01967 ,

    work page Pith review arXiv

  7. [12]

    Mimic-it: Multi-modal in-context instruction tuning, 2023a

    Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Fanyi Pu, Jingkang Yang, Chunyuan Li, and Ziwei Liu. Mimic-it: Multi-modal in-context instruction tuning, 2023a. Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326,

    arXiv

  8. [13]

    Seed- bench: Benchmarking multimodal llms with generative comprehension

    Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed- bench: Benchmarking multimodal llms with generative comprehension. arXiv preprint arXiv:2307.16125, 2023b. Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In ICML,

    arXiv

Show all 31 references
  1. [14]

    BLIP-2: bootstrapping language- image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: bootstrapping language- image pre-training with frozen image encoders and large language models. In ICML, 2023c. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhan...

    arXiv

  2. [15]

    Draw-and-understand: Leveraging visual prompts to enable mllms to comprehend what you want

    Weifeng Lin, Xinyu Wei, Ruichuan An, Peng Gao, Bocheng Zou, Yulin Luo, Siyuan Huang, Shanghang Zhang, and Hongsheng Li. Draw-and-understand: Leveraging visual prompts to enable mllms to comprehend what you want. arXiv preprint arXiv:2403.20271,

    arXiv

  3. [16]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26296–26306, 2024a. Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sh...

    2024 arXiv

  4. [17]

    Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity

    Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. arXiv preprint arXiv:2104.08786,

    arXiv

  5. [18]

    Llm as dataset analyst: Subpopulation structure discovery with large language model

    Yulin Luo, Ruichuan An, Bocheng Zou, Yiming Tang, Jiaming Liu, and Shanghang Zhang. Llm as dataset analyst: Subpopulation structure discovery with large language model. arXiv preprint arXiv:2405.02363,

    arXiv

  6. [19]

    m&m’s: A benchmark to evaluate tool-use for multi-step multi-modal tasks

    Zixian Ma, Weikai Huang, Jieyu Zhang, Tanmay Gupta, and Ranjay Krishna. m&m’s: A benchmark to evaluate tool-use for multi-step multi-modal tasks. In Synthetic Data for Computer Vision Workshop@ CVPR 2024,

    2024

  7. [20]

    What makes chain-of- thought prompting effective? a counterfactual study

    Aman Madaan, Katherine Hermann, and Amir Yazdanbakhsh. What makes chain-of- thought prompting effective? a counterfactual study. In Findings of the Association for Computational Linguistics: EMNLP 2023 , pp. 1448–1535,

    2023

  8. [21]

    Joty, Ran Xu, Silvio Savarese, Caiming Xiong, and Juan Carlos Niebles

    Artemis Panagopoulou, Le Xue, Ning Yu, Junnan Li, Dongxu Li, Shafiq R. Joty, Ran Xu, Silvio Savarese, Caiming Xiong, and Juan Carlos Niebles. X-instructblip: A framework for aligning x-modal instruction-aware representations to llms and emergent cross-modal rea- soning. ArXiv,...

    arXiv

  9. [22]

    Benchmarking prompt sensitivity in large language models

    Amirhossein Razavi, Mina Soltangheis, Negar Arabzadeh, Sara Salamat, Morteza Zihayat, and Ebrahim Bagheri. Benchmarking prompt sensitivity in large language models. arXiv preprint arXiv:2502.06065,

    arXiv

  10. [25]

    Mind your format: Towards consistent evaluation of in-context learning improvements

    Anton Voronov, Lena Wolf, and Max Ryabinin. Mind your format: Towards consistent evaluation of in-context learning improvements. arXiv preprint arXiv:2401.06766,

    arXiv

  11. [26]

    Qwen2-vl: Enhanc- ing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhanc- ing vision-language model’s ...

    arXiv

  12. [27]

    Task me anything

    Jieyu Zhang, Weikai Huang, Zixian Ma, Oscar Michel, Dong He, Tanmay Gupta, Wei-Chiu Ma, Ali Farhadi, Aniruddha Kembhavi, and Ranjay Krishna. Task me anything. arXiv preprint arXiv:2406.11775, 2024a. 13 Preprint. Under review. Jieyu Zhang, Le Xue, Linxin Song, Jun Wang, Weikai ...

    arXiv

  13. [28]

    On large language models’ selection bias in multi-choice questions

    Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. On large language models’ selection bias in multi-choice questions. arXiv preprint arXiv:2309.03882,

    arXiv

  14. [29]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592,

    arXiv

  15. [30]

    Prosa: Assessing and understanding the prompt sensitivity of llms

    Jingming Zhuo, Songyang Zhang, Xinyu Fang, Haodong Duan, Dahua Lin, and Kai Chen. Prosa: Assessing and understanding the prompt sensitivity of llms. arXiv preprint arXiv:2410.12405,

    arXiv

  16. [2019]

    Multitask prompted training enables zero-shot task generalization

    Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. Multitask prompted training enables zero-shot task generalization. arXiv preprint arXiv:2110.08207,

    arXiv

  17. [2020]

    What matters when building vision-language models? arXiv preprint arXiv:2405.02246,

    Hugo Laurenc ¸on, L ´eo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision-language models? arXiv preprint arXiv:2405.02246,

    arXiv

  18. [2021]

    Scaling laws for neural language models

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361,

    2001 arXiv

  19. [2022]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685,

    arXiv

  20. [2023]

    Qwen-vl: A frontier large vision-language model with versatile abilities

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966,

    arXiv

  21. [2024]

    Grounding- prompter: Prompting llm with multimodal information for temporal sentence grounding in long videos, 2023a

    Houlun Chen, Xin Wang, Hong Chen, Zihan Song, Jia Jia, and Wenwu Zhu. Grounding- prompter: Prompting llm with multimodal information for temporal sentence grounding in long videos, 2023a. Xi Chen, Xiao Wang, Lucas Beyer, Alexander Kolesnikov, Jialin Wu, Paul Voigtlaender, Basi...

    arXiv

  22. [2025]

    Sentence-bert: Sentence embeddings using siamese bert-networks

    N Reimers. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084,

    1908 arXiv

  23. [3360]

    - (weight: 1920): <verb><what_you_see>the<is_provided><image>,<is_please><answer> the<is_following>question:<is_line_breaking>{question} - (weight: 1440): <prep><what_you_see>the<is_provided><image>,<is_please><answer> the<is_following>question:<is_line_breaking>{question} + A...

    1920

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.