Pith. sign in

REVIEW 4 major objections 6 minor 88 references

Error-driven Data-efficient Large Multimodal Model Tuning

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A generic large multimodal model can be adapted to a new task without task-specific training samples: a teacher finds the first wrong step in the student's reasoning, names the missing skill, and retrieves targeted samples from a…

desk verdict A genuinely new pipeline for data-efficient LMM adaptation with a plausible central idea, but the mistake-identification evidence is compromised by train/test overlap and gold-answer leakage, so the quantitative claims need stronger validation. read the letter →

arxiv 2412.15652 v1 pith:X57NCA34 submitted 2024-12-20 cs.CL

Barry Menglong Yao (UC Davis) , Qifan Wang (Meta AI) , Lifu Huang (UC Davis) This is my paper · ORCID
classification cs.CL
keywords error-drivenlearningdata-efficienttuninglargemultimodalmodelsteacher-studentframeworkdataselectionskillgapanalysismistakeidentificationretrieval-basedfine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large multimodal models still need task-specific fine-tuning data, which is often scarce. This paper claims that a generic model can be adapted to a new task without any task-specific training samples: run the model on a small validation set, have a stronger teacher locate the first wrong step in its reasoning and name the missing skill, then retrieve training samples that exercise that skill from an existing task-agnostic dataset. Across seven tasks and three data scales, the authors report an average 7.01% accuracy gain over the untuned model with 100K retrieved samples, and report that 100K samples reproduce or beat fine-tuning on the full 1.55M-sample supporting dataset on five of seven tasks. The load-bearing idea is that capability gaps, not data similarity, should drive data selection.

What carries the argument

The answer-switch mistake identification: a teacher model, deprived of the image, receives the question, a prior hint that the correct option has 60% probability, and an incrementally growing prefix of the student's rationale; the first reasoning step after which the teacher's probability for the wrong answer exceeds the correct answer by a margin δ and stays there for λ steps is labeled the mistake step. That step is then converted by the same teacher into a missing-skill phrase, which BM25 matches against per-sample skill annotations of a supporting dataset to retrieve the tuning samples. The loop is rounded out by using the retrieved samples to fine-tune the student.

What would settle it

On a held-out task, compare the teacher's mistake-step labels with fresh human annotations (not the 100 ScienceQA examples used to tune the margin thresholds). If localization accuracy does not beat random selection, or if replacing the answer-switch step with a random intermediate step in the retrieval pipeline yields the same downstream accuracy, the central mechanism is not doing the work.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that the error pattern of a pretrained LMM can be converted into a training-data query. The teacher reads the student's chain of reasoning, tracks how the probability of the correct versus the wrong answer shifts as each reasoning sentence is appended, and takes the first step that flips the balance as the mistake step; it then names a skill whose absence explained that step. Retrieving samples from a large human-labeled supporting set by matching that skill name, rather than by surface similarity to the validation examples, produces a smaller, more effective tuning set. The paper reports that this selection beat random selection and INSTA-derived similarity selection at 10K, 30K, and 100K samples, and that the ordering held for two student models and two teacher models.

Load-bearing premise

The framework assumes the teacher can reliably locate the first genuinely faulty reasoning step from the shift in its own answer probabilities, and that this localization transfers across tasks.

Editorial extensions

If this is right

  • At 100K tuning samples, the average gain across seven tasks is 7.01 percentage points over the untuned model.
  • Using 100K samples (6% of the 1.55M-sample supporting set) reaches at least 94.57% of full-data performance and beats full-data fine-tuning on five of seven tasks.
  • The selection principle transfers: the same pipeline works with LLaVA-OneVision-72B as teacher and with Qwen2-VL-7B as student, with Qwen2-VL gaining up to 3.80%.
  • The answer-switch localization (65% accuracy on 100 ScienceQA labels) outperforms per-step verification (28%) and pseudo-rationale matching (59%).
  • Fine-tuning on the 1K validation samples of the target task underperforms the retrieval-based approach by an average of 5.11 percentage points.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One implication the paper leaves implicit: the skill vocabulary is generated by the same teacher that does the diagnosis, so the framework's success may depend on the teacher's skill annotations being consistent with the retrieval index it builds; a testable extension would use a different model to annotate the supporting set than to diagnose the student.
  • The paper reports only single-round tuning within one pass; iterating the three steps over multiple rounds, as the framework's description allows, is a straightforward extension that could reveal whether skills accumulate or saturate.
  • The answer-switch method requires token-level probabilities for candidate options, which are not available for all closed-source teachers; an open-weight teacher with accessible logits would make the method reproducible and allow the δ and λ thresholds to be set without held-out labels.
  • The gains suggest a reframing of data-efficient tuning: rather than maximizing similarity between retrieved samples and the target task, the right objective is maximizing coverage of the student's named capability gaps; this could be tested by ablating the skill-matching step against direct mistake-step retrieval.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The manuscript proposes a three-step, error-driven framework for data-efficient tuning of large multimodal models (LMMs) without task-specific training samples. A student LMM is evaluated on a small validation set; a teacher LMM identifies the first erroneous reasoning step in the student's rationale (Mistake Identification), summarizes the missing skill (Skill Analysis), and then retrieves related samples from the task-agnostic Vision-Flan supporting set for fine-tuning (Targeted Tuning). Experiments are reported for LLaVA-7B and Qwen2-VL-7B as students, GPT-4o-mini and LLaVA-OneVision-72B as teachers, across seven tasks and three data scales, with an abstract-level claim of an average 7.01% performance boost.

Significance. If the claims hold, the framework is practically valuable: it addresses the common situation where task-specific training data is scarce, while providing a transparent mechanism (erroneous-step localization, skill inference, targeted retrieval) that goes beyond simple similarity-based data selection. The paper includes component ablations, two student models, two teacher models, and removes the seven evaluation tasks from the supporting dataset, which are genuine strengths. However, the headline number is inconsistent with Table 1, and the evidence for the novel Mistake Identification component is weakened by hyperparameter tuning on the evaluation set and by a prompt that reveals the gold answer. These issues are load-bearing because Mistake Identification drives the entire skill-retrieval loop, and the reported downstream improvements are the central claim. The framework remains credible, but the evidence as presented needs substantial correction before the contribution can be accepted.

major comments (4)
  1. [Abstract; §4.2, Table 1] The headline claim of an average performance boost of 7.01% is not supported by Table 1. In the 100K setting, the differences between "Our Approach" and "Pre-trained LMM" across the seven columns are +0.11, +18.30, +18.70, +14.20, +2.00, +16.00, and +2.68 percentage points, which average to 10.28 percentage points. Please verify the arithmetic and clearly state which baseline and which averaging procedure produce 7.01%; the abstract, Section 4.2, and Table 1 must be made consistent.
  2. [§3.3, §4.4, Table 5] The hyperparameters δ and λ in Mistake Identification are tuned on the same 100 manually labeled ScienceQA error examples that are later used for the Table 5 evaluation. The footnote in §3.3 states this explicitly, and §4.4 says it samples the same 100 error samples. The reported 65.0% accuracy is therefore not a held-out estimate, and the central evidence for the mistake-localization mechanism is optimistically biased. Please evaluate on a separate set or use nested validation.
  3. [§3.3, Figure 4] The Mistake Identification prompt tells the teacher that there is a 60% probability that the gold option is correct (option B in Figure 4). The teacher is thus given the correct answer before reading the student's rationale, and the answer-switch method is effectively detecting when the student's steps disagree with a gold-answer prior. This makes the 65% accuracy difficult to interpret as evidence of independent mistake identification. Please re-run the comparison with a neutral prior, or include an ablation that holds the prior fixed and uninformative.
  4. [§4.2, Tables 1–4] No error bars or multiple runs are reported, and several differences are small relative to what would be expected from typical LMM fine-tuning variance (e.g., MM-Bench 64.41 vs. 64.30 in the 100K row of Table 1, or several 0.5–1 point gaps in Table 2). The phrase "significantly improves" needs statistical support; please report the mean and standard deviation over at least three random seeds, or state explicitly which independent runs were performed.
minor comments (6)
  1. [Section 5] The heading "Conclution" is a typo and should be corrected to "Conclusion".
  2. [Table 1] The label "Ramdom" is misspelled in several rows and should be "Random".
  3. [Abstract, Introduction, Limitations] The abstract and introduction say the method works without task-specific training samples, but the framework requires a 1K validation set of the target task; the Limitations section honestly acknowledges this, and the earlier statements should be qualified for precision.
  4. [Appendix E] The hyperparameter search space for learning rate and batch size is given, but the selection criterion is not stated; please clarify whether these hyperparameters were chosen on a held-out split rather than on the evaluation test set.
  5. [§3.3] The parenthetical remark that non-multiple-choice tasks can be handled by treating the gold answer as option B and the wrong prediction as option A is described for a two-choice setting; please clarify how open-ended or multi-class tasks are converted.
  6. [Appendix C] The annotation guidelines allow annotators to label a sample as "-1" when no reasoning step is erroneous, but the paper does not describe how such samples are handled by the subsequent Skill Analysis and retrieval steps; please clarify.

Circularity Check

1 steps flagged · score 2.0 of 10

No significant circularity in the central empirical claim: downstream gains are measured on external benchmarks; however, the reported mistake-identification accuracy is an in-sample estimate because the same 100 labeled ScienceQA examples are used to tune δ and λ and to produce Table 5.

  1. fitted input called prediction [Sec 3.3 footnote 5 and Sec 4.4 / Table 5]
    "We manually labeled the mistake step for 100 error examples from the validation set of ScienceQA and tuned the hyper-parameters δ and λ on the 100 examples. ... Since there are no gold labels for the mistake steps of the validation datasets, we sample 100 error samples from the validation set of ScienceQA and manually label the mistake step for each error sample."

    The mistake-identification accuracy of 65% in Table 5 is computed on the same 100 ScienceQA error examples used to tune the mistake-step rule (first step where P(wrong answer) - P(correct answer) >= δ for λ consecutive steps). The reported number is therefore an in-sample fit rather than an independent prediction of the teacher's localization ability. This does not make the downstream benchmark gains circular, since Tables 1-3 are evaluated on held-out task test sets, but it removes Table 5 as independent evidence for the central mistake-identification mechanism.

full rationale

The derivation chain is not circular at the level of the paper's central claim. The student is evaluated on a validation set, the teacher identifies mistakes and missing skills, BM25 retrieves support samples, and the student is fine-tuned; the resulting accuracies in Tables 1-3 are measured on external task test sets independent of the retrieval and skill-analysis machinery. The internal skill-matching loop is self-consistent (the same teacher writes missing-skill queries and the supporting-set skill index), but that is a modeling choice, not a reduction of the predicted accuracy to the input. No load-bearing self-citation or imported uniqueness theorem is present. The only issue touching circularity is the mistake-identification evaluation: δ and λ are tuned on the same 100 labeled ScienceQA examples later used as the evaluation set for Table 5, making the 65% accuracy an in-sample estimate. In addition, the Mistake Identification prompt leaks the gold answer by injecting "There is a probability of 60% that option B is correct"; this is label leakage rather than circular reasoning, but it further weakens Table 5 as evidence. Because the central benchmark results are independently evaluated and the main framework still works even when mistake selection is randomized (ablation Table 4), the overall circularity score remains low.

Assumptions & free parameters 3 free parameters · 4 assumptions · 1 invented entities

The central claim rests on teacher reliability, the skill-based transfer assumption, BM25 as a retrieval signal, and the availability of a labeled validation set. These are domain assumptions, not standard mathematical axioms. The only explicit free parameters are the mistake-identification thresholds and the 60% prior, plus unreported training hyperparameters. No new physical entities are introduced; 'missing skill' is a latent abstraction.

free parameters (3)
  • delta and lambda thresholds in Mistake Identification = not reported
    Tuned on 100 manually labeled ScienceQA error examples to define when the wrong answer probability overtakes the correct answer; the chosen values are not stated in the paper.
  • Prior probability of 60% for the correct answer = 60%
    Hand-chosen in the mistake identification prompt; based on a preliminary observation that the teacher favors the correct answer in 77% of samples, and it leaks the gold label into the teacher prompt.
  • Learning rate and batch size = not reported
    Searched over the given ranges but final values are not reported; standard hyperparameters but unreported in the paper.
assumptions (4)
  • domain assumption Teacher model can reliably identify erroneous reasoning steps and summarize missing skills.
    The entire pipeline hinges on teacher judgments; the paper validates this with one 100-sample ScienceQA evaluation that is itself affected by tuning and label leakage.
  • domain assumption The skill definition from Chen et al., 2023c applies: training on data associated with a skill improves performance on unseen samples of that skill.
    Used to justify that samples retrieved by skill similarity will fill capability gaps in the student model.
  • domain assumption BM25 lexical similarity between teacher-written skill descriptions is a valid ranking signal for retrieving helpful training samples.
    No comparison to semantic or gradient-based retrieval beyond the SBERT similarity baseline in INSTA*.
  • domain assumption 1K labeled validation samples are available and representative of each target task.
    The framework requires a task-specific validation set with gold labels; the paper acknowledges the cost in its Limitations section.
invented entities (1)
  • Missing skill (capability gap)
    purpose: Intermediate abstraction used to connect a student error to supporting-dataset samples via teacher judgment.
    The paper defines skills operationally following Chen et al. (2023c), but the specific 'missing skill' summaries are produced by the teacher model with no independent measurement, so their validity is only indirectly tested through downstream accuracy.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Error-driven Data-efficient Large Multimodal Model Tuning." pith.science (2026). https://pith.science/paper/X57NCA34

@misc{pith2026241215652,
  author       = {Pith},
  title        = {Pith review of: Error-driven Data-efficient Large Multimodal Model Tuning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/X57NCA34}},
  note         = {Machine review of arXiv:2412.15652}
}
read the original abstract

Large Multimodal Models (LMMs) have demonstrated impressive performance across numerous academic benchmarks. However, fine-tuning still remains essential to achieve satisfactory performance on downstream tasks, while the task-specific tuning samples are usually not readily available or expensive and time-consuming to obtain. To address this, we propose an error-driven data-efficient tuning framework that aims to efficiently adapt generic LMMs to newly emerging tasks without requiring any task-specific training samples. In our approach, a generic LMM, acting as a student model, is first evaluated on a small validation set of the target task, and then a more powerful model, acting as a teacher model, identifies the erroneous steps within the student model's reasoning steps and analyzes its capability gaps from fully addressing the target task. Based on these gaps, targeted training samples are further retrieved from existing task-agnostic datasets to tune the student model and tailor it to the target task. We perform extensive experiments across three different training data scales and seven tasks, demonstrating that our training paradigm significantly and efficiently improves LMM's performance on downstream tasks, achieving an average performance boost of 7.01%.

Figures

Figures reproduced from arXiv: 2412.15652 by the authors.

Figure 1
Figure 1. Overview of the error-driven data-efficient tuning paradigm. In [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Example for illustrating the process of mistake identification. At each iteration, we append one more [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. One example prompt for ScienceQA task to obtain the student model’s prediction. [PITH_FULL_IMAGE:figures/full_fig_p015_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: One example prompt to obtain the teacher model’s prediction by following the student model’s rationale [PITH_FULL_IMAGE:figures/full_fig_p015_4.png]
Figure 5
Figure 5. Figure 5: One example prompt to trigger the teacher model to analyze the missing skill based on the wrong rationale [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]
Figure 6
Figure 6. Figure 6: One example prompt to trigger the teacher model to analyse a sequence of required skills for each sample [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

88 extracted references · 20 canonical work pages

  1. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  2. [2]

    Afra Feyza Akyürek, Ekin Akyürek, Aman Madaan, Ashwin Kalyan, Peter Clark, Derry Wijaya, and Niket Tandon. 2023. https://doi.org/10.48550/arxiv.2305.08844 RL4F: Generating Natural Language Feedback with Reinforcement Learning for Repairing Model Outputs . arXiv

    arXiv doi:10.48550/arxiv.2305.08844 2023

  3. [3]

    Shengnan An, Zexiong Ma, Zeqi Lin, Nanning Zheng, Jian-Guang Lou, and Weizhu Chen. 2023. https://arxiv.org/abs/2310.20689 Learning From Mistakes Makes LLM Better Reasoner . arXiv

    arXiv 2023

  4. [4]

    Athanasios Angelakis and Andrey Rass. 2024. A data-centric approach to class-specific bias in image data augmentation. arXiv preprint arXiv:2403.04120

    work page Pith review arXiv 2024

  5. [5]

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966

    arXiv 2023

  6. [6]

    Paul Bambrick-Santoyo. 2010. Driven by data: A practical guide to improve instruction. John Wiley & Sons

    2010

  7. [7]

    Yoshua Bengio, J\' e r\^ o me Louradour, Ronan Collobert, and Jason Weston. 2009. https://doi.org/10.1145/1553374.1553380 Curriculum learning . In Proceedings of the 26th Annual International Conference on Machine Learning, ICML '09, page 41–48, New York, NY, USA. Association for Computing Machinery

    arXiv 2009

  8. [8]

    Tom B Brown. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165

    arXiv 2020

Show all 88 references
  1. [9]

    Gail A Carpenter and Stephen Grossberg. 1987. A massively parallel architecture for a self-organizing neural pattern recognition machine. Computer vision, graphics, and image processing, 37(1):54--115

    1987

  2. [10]

    Hailin Chen, Amrita Saha, Steven Hoi, and Shafiq Joty. 2023 a . https://doi.org/10.48550/arxiv.2310.18628 Personalised Distillation: Empowering Open-Sourced LLMs with Adaptive Learning for Code Generation . arXiv

    2023 doi arXiv

  3. [11]

    Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, et al. 2023 b . Alpagasus: Training a better alpaca with fewer data. arXiv preprint arXiv:2307.08701

    2023 arXiv

  4. [12]

    Mayee F Chen, Nicholas Roberts, Kush Bhatia, Jue Wang, Ce Zhang, Frederic Sala, and Christopher Ré. 2023 c . https://doi.org/10.48550/arxiv.2307.14430 Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Models . arXiv

    2023 doi arXiv

  5. [13]

    Haixing Dai, Zhengliang Liu, Wenxiong Liao, Xiaoke Huang, Yihan Cao, Zihao Wu, Lin Zhao, Shaochen Xu, Wei Liu, Ninghao Liu, Sheng Li, Dajiang Zhu, Hongmin Cai, Lichao Sun, Quanzheng Li, Dinggang Shen, Tianming Liu, and Xiang Li. 2023. https://doi.org/10.48550/arxiv.2302.13007 ...

    2023 doi arXiv

  6. [14]

    Jehanzeb Mirza, Wei Lin, Amit Alfassy, Assaf Arbelle, Shimon Ullman, and Leonid Karlinsky

    Sivan Doveh, Shaked Perek, M. Jehanzeb Mirza, Wei Lin, Amit Alfassy, Assaf Arbelle, Shimon Ullman, and Leonid Karlinsky. 2024. https://arxiv.org/abs/2403.12736 Towards multimodal in-context learning for vision & language models . Preprint, arXiv:2403.12736

    2024 arXiv

  7. [15]

    Ksenia Dmitrievna Dyatlova, Irina Mikchailovna Shvets, Elena Sergeevna Orlova, Yulia Vitalievna Sinitsyna, and Irina Valerievna Struchkova. 2018. Project-based learning as an instrument for the formation and development of research skills of biology students. In Handbook of Re...

    2018

  8. [16]

    A tale of tails: Model collapse as a change of scaling laws

    Yunzhen Feng, Elvis Dohmatob, Pu Yang, Francois Charton, and Julia Kempe. A tale of tails: Model collapse as a change of scaling laws. In ICLR 2024 Workshop on Navigating and Addressing Data Problems for Foundation Models

    2024

  9. [17]

    Francis Green. 2011. What is Skill?: An Inter-Disciplinary Synthesis. Centre for Learning and Life Chances in Knowledge Economies and Societies London

    2011

  10. [18]

    Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2021. Domain-specific language model pretraining for biomedical natural language processing. ACM Transactions on Computing for Healthcare (HEALTH), 3(1):1--23

    2021

  11. [19]

    Suchin Gururangan, Ana Marasovi \'c , Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. 2020. Don't stop pretraining: Adapt language models to domains and tasks. arXiv preprint arXiv:2004.10964

    2020 arXiv

  12. [20]

    Guy Hacohen and Daphna Weinshall. 2019. On the power of curriculum learning in training deep networks. In International conference on machine learning, pages 2535--2544. PMLR

    2019

  13. [21]

    Xiaochuang Han, Daniel Simig, Todor Mihaylov, Yulia Tsvetkov, Asli Celikyilmaz, and Tianlu Wang. 2023. Understanding in-context learning via supportive pretraining data. arXiv preprint arXiv:2306.15091

    2023 arXiv

  14. [22]

    Trevor Hastie, Andrea Montanari, Saharon Rosset, and Ryan J Tibshirani. 2022. Surprises in high-dimensional ridgeless least squares interpolation. Annals of statistics, 50(2):949

    2022

  15. [23]

    Doroth \'e e B Hoppe, Petra Hendriks, Michael Ramscar, and Jacolien van Rij. 2022. An exploration of error-driven learning in simple two-layer networks from a discriminative learning perspective. Behavior Research Methods, 54(5):2221--2251

    2022

  16. [24]

    Yixing Jiang, Jeremy Irvin, Ji Hun Wang, Muhammad Ahmed Chaudhry, Jonathan H Chen, and Andrew Y Ng. 2024. Many-shot in-context learning in multimodal foundation models. arXiv preprint arXiv:2405.09798

    2024 arXiv

  17. [25]

    Krishnateja Killamsetty, Durga Sivasubramanian, Ganesh Ramakrishnan, Abir De, and Rishabh Iyer. 2021. https://doi.org/10.48550/arxiv.2103.00123 GRAD-MATCH: Gradient Matching based Data Subset Selection for Efficient Deep Model Training . arXiv

    2021 doi arXiv

  18. [26]

    Tae-Hoon Kim and Jonghyun Choi. 2018. Screenernet: Learning self-paced curriculum for deep neural networks. arXiv preprint arXiv:1801.00904

    2018 arXiv

  19. [27]

    M Kumar, Benjamin Packer, and Daphne Koller. 2010. Self-paced learning for latent variable models. Advances in neural information processing systems, 23

    2010

  20. [28]

    Po-Nien Kung, Fan Yin, Di Wu, Kai-Wei Chang, and Nanyun Peng. 2023. https://doi.org/10.48550/arxiv.2311.00288 Active Instruction Tuning: Improving Cross-Task Generalization by Training on Prompt Sensitive Tasks . arXiv

    2023 doi arXiv

  21. [29]

    Changho Lee, Janghoon Han, Seonghyeon Ye, Stanley Jungkyu Choi, Honglak Lee, and Kyunghoon Bae. 2024 a . Instruction matters, a simple yet effective task selection approach in instruction tuning for specific tasks. arXiv preprint arXiv:2404.16418

    2024 arXiv

  22. [30]

    Nicholas Lee, Thanakul Wattanawong, Sehoon Kim, Karttikeya Mangalam, Sheng Shen, Gopala Anumanchipali, Michael W Mahoney, Kurt Keutzer, and Amir Gholami. 2024 b . Llm2llm: Boosting llms with novel iterative data enhancement. arXiv preprint arXiv:2403.15042

    2024 arXiv

  23. [31]

    Yong Jae Lee and Kristen Grauman. 2011. Learning the easy things first: Self-paced visual category discovery. In CVPR 2011, pages 1721--1728. IEEE

    2011

  24. [32]

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. 2024 a . Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326

    2024 arXiv

  25. [33]

    Fei-Fei Li, Marco Andreeto, Marc'Aurelio Ranzato, and Pietro Perona. 2022. https://doi.org/10.22002/D1.20086 Caltech 101

    2022 doi

  26. [34]

    Haoran Li, Qingxiu Dong, Zhengyang Tang, Chaojun Wang, Xingxing Zhang, Haoyang Huang, Shaohan Huang, Xiaolong Huang, Zeqiang Huang, Dongdong Zhang, et al. 2024 b . Synthetic data (almost) from scratch: Generalized instruction tuning for language models. arXiv preprint arXiv:2402.13064

    2024 arXiv

  27. [35]

    Wenyan Li, Jonas F Lotz, Chen Qiu, and Desmond Elliott. 2023. https://doi.org/10.48550/arxiv.2305.03610 Data Curation for Image Captioning with Text-to-Image Generative Models . arXiv

    2023 doi arXiv

  28. [36]

    Zhenwen Liang, Wenhao Yu, Tanmay Rajpurohit, Peter Clark, Xiangliang Zhang, and Ashwin Kalyan. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.889 Let GPT be a math tutor: Teaching math word problem solvers with customized exercise generation . In Proceedings of the 2023 Con...

    2023 doi

  29. [37]

    Chi-Heng Lin, Chiraag Kaushik, Eva L Dyer, and Vidya Muthukumar. 2024. The good, the bad and the ugly sides of data augmentation: An implicit spectral regularization perspective. Journal of Machine Learning Research, 25(91):1--85

    2024

  30. [38]

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll \'a r, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In Computer Vision--ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, ...

    2014

  31. [39]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2024 a . Visual instruction tuning. Advances in neural information processing systems, 36

    2024

  32. [40]

    Liangxin Liu, Xuebo Liu, Derek F Wong, Dongfang Li, Ziyi Wang, Baotian Hu, and Min Zhang. 2024 b . Selectit: Selective instruction tuning for large language models via uncertainty-aware self-reflection. arXiv preprint arXiv:2402.16705

    2024 arXiv

  33. [41]

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. 2024 c . https://arxiv.org/abs/2307.06281 Mmbench: Is your multi-modal model an all-around player? Preprint, arXiv:2307.06281

    2024 arXiv

  34. [42]

    Zikang Liu, Kun Zhou, Wayne Xin Zhao, Dawei Gao, Yaliang Li, and Ji-Rong Wen. 2024 d . https://doi.org/10.48550/arxiv.2403.09559 Less is More: Data Value Estimation for Visual Instruction Tuning . arXiv

    2024 doi arXiv

  35. [43]

    Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Processing Sy...

    2022

  36. [44]

    Renqian Luo, Liai Sun, Yingce Xia, Tao Qin, Sheng Zhang, Hoifung Poon, and Tie-Yan Liu. 2022. Biogpt: generative pre-trained transformer for biomedical text generation and mining. Briefings in bioinformatics, 23(6):bbac409

    2022

  37. [45]

    Fan Ma, Deyu Meng, Qi Xie, Zina Li, and Xuanyi Dong. 2017. Self-paced co-training. In International Conference on Machine Learning, pages 2275--2284. PMLR

    2017

  38. [46]

    Tambet Matiisen, Avital Oliver, Taco Cohen, and John Schulman. 2017. https://doi.org/10.48550/arxiv.1707.00183 Teacher-Student Curriculum Learning . arXiv

    2017 doi arXiv

  39. [47]

    Vidya Muthukumar, Kailas Vodrahalli, Vignesh Subramanian, and Anant Sahai. 2020. Harmless interpolation of noisy data in regression. IEEE Journal on Selected Areas in Information Theory, 1(1):67--83

    2020

  40. [48]

    Nihal V Nayak, Yiyang Nan, Avi Trost, and Stephen H Bach. 2024. Learning to generate instruction tuning datasets for zero-shot task adaptation. arXiv preprint arXiv:2402.18334

    2024 arXiv

  41. [49]

    Pascual P \'e rez-Paredes and Mar \' a S \'a nchez-Tornel. 2009. Understanding e-skills in the flt context. In Handbook of Research on E-Learning Methodologies for Language Acquisition, pages 1--21. IGI Global

    2009

  42. [50]

    Jeff M Phillips. 2016. https://doi.org/10.48550/arxiv.1601.00617 Coresets and Sketches . arXiv

    2016 doi arXiv

  43. [51]

    Jingyuan Qi, Zhiyang Xu, Ying Shen, Minqian Liu, Di Jin, Qifan Wang, and Lifu Huang. 2023. https://arxiv.org/abs/2305.14999 The art of socratic questioning: Recursive thinking with large language models . Preprint, arXiv:2305.14999

    2023 arXiv

  44. [52]

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9

    2019

  45. [53]

    Revant Gangi Reddy, Xilin Rui, Manling Li, Xudong Lin, Haoyang Wen, Jaemin Cho, Lifu Huang, Mohit Bansal, Avirup Sil, Shih-Fu Chang, et al. 2022. Mumuqa: Multimedia multi-hop news question answering via cross-media knowledge extraction and grounding. In Proceedings of the AAAI...

    2022

  46. [54]

    Nils Reimers and Iryna Gurevych. 2019. https://arxiv.org/abs/1908.10084 Sentence-bert: Sentence embeddings using siamese bert-networks . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics

    2019 arXiv

  47. [55]

    Stephen Robertson, Hugo Zaragoza, et al. 2009. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends in Information Retrieval , 3(4):333--389

    2009

  48. [56]

    David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. 1986. Learning representations by back-propagating errors. nature, 323(6088):533--536

    1986

  49. [57]

    Ozan Sener and Silvio Savarese. 2017. Active learning for convolutional neural networks: A core-set approach. arXiv preprint arXiv:1708.00489

    2017 arXiv

  50. [58]

    Ying Shen, Zhiyang Xu, Qifan Wang, Yu Cheng, Wenpeng Yin, and Lifu Huang. 2024. Multimodal instruction tuning with conditional mixture of lora. arXiv preprint arXiv:2402.15896

    2024 arXiv

  51. [59]

    Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, and Ross Anderson. 2023. The curse of recursion: Training on generated data makes models forget. arXiv preprint arXiv:2305.17493

    2023 arXiv

  52. [60]

    less is more

    Valentin I Spitkovsky, Hiyan Alshawi, and Dan Jurafsky. 2010. From baby steps to leapfrog: How “less is more” in unsupervised dependency parsing. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Lingu...

    2010

  53. [61]

    Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J Gordon. 2018. https://doi.org/10.48550/arxiv.1812.05159 An Empirical Study of Example Forgetting during Deep Neural Network Learning . arXiv

    2018 doi arXiv

  54. [62]

    Gladys Tyen, Hassan Mansoor, Victor C a rbune, Yuanzhu Peter Chen, and Tony Mak. 2024. Llms cannot find reasoning errors, but can correct them given the error location. In Findings of the Association for Computational Linguistics ACL 2024, pages 13894--13908

    2024

  55. [63]

    Danqing Wang and Lei Li. 2023. Learning from mistakes via cooperative study assistant for large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 10667--10685

    2023

  56. [64]

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. 2024. Qwen2-vl: Enhancing vision-language mode...

    2024 arXiv

  57. [65]

    Sijia Wang and Lifu Huang. 2024. Targeted augmentation for low-resource event extraction. arXiv preprint arXiv:2405.08729

    2024 arXiv

  58. [66]

    Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Chandu, David Wadden, Kelsey MacMillan, Noah A Smith, Iz Beltagy, et al. 2023. How far can camels go? exploring the state of instruction tuning on open resources. Advances in Neural Information Proc...

    2023

  59. [67]

    Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. 2022 a . Emergent abilities of large language models. arXiv preprint arXiv:2206.07682

    2022 arXiv

  60. [68]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. https://arxiv.org/abs/2201.11903 Chain-of-thought prompting elicits reasoning in large language models . Preprint, arXiv:2201.11903

    2023 arXiv

  61. [69]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022 b . Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824--24837

    2022

  62. [71]

    Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. 2024 b . https://arxiv.org/abs/2402.04333 LESS: Selecting Influential Data for Targeted Instruction Tuning . arXiv

    2024 arXiv

  63. [72]

    Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy S Liang. 2023. Data selection for language models via importance resampling. Advances in Neural Information Processing Systems, 36:34201--34227

    2023

  64. [73]

    Wenda Xu, Daniel Deutsch, Mara Finkelstein, Juraj Juraska, Biao Zhang, Zhongtao Liu, William Yang Wang, Lei Li, and Markus Freitag. 2023 a . https://arxiv.org/abs/2311.09336 Pinpoint, Not Criticize: Refining Large Language Models via Fine-Grained Actionable Feedback . arXiv

    2023 arXiv

  65. [74]

    Zhiyang Xu, Chao Feng, Rulin Shao, Trevor Ashby, Ying Shen, Di Jin, Yu Cheng, Qifan Wang, and Lifu Huang. 2024. Vision-flan: Scaling human-labeled tasks in visual instruction tuning. arXiv preprint arXiv:2402.11690

    2024 arXiv

  66. [75]

    Zhiyang Xu, Jay Yoon Lee, and Lifu Huang. 2023 b . Learning from a friend: Improving event extraction via self-training with feedback from abstract meaning representation. In Findings of the Association for Computational Linguistics: ACL 2023, pages 10421--10437

    2023

  67. [76]

    Zhiyang Xu, Ying Shen, and Lifu Huang. 2022. Multiinstruct: Improving multi-modal zero-shot learning via instruction tuning. arXiv preprint arXiv:2212.10773

    2022 arXiv

  68. [77]

    Zifan Xu, Haozhu Wang, Dmitriy Bespalov, Peter Stone, and Yanjun Qi. 2023 c . https://arxiv.org/abs/2312.04684 Latent Skill Discovery for Chain-of-Thought Reasoning . arXiv

    2023 arXiv

  69. [78]

    Zeyuan Yang, Peng Li, and Yang Liu. 2023. https://doi.org/10.48550/arxiv.2310.15746 Failures Pave the Way: Enhancing Large Language Models through Tuning-free Rule Accumulation . arXiv

    2023 doi arXiv

  70. [79]

    Barry Menglong Yao, Yu Chen, Qifan Wang, Sijia Wang, Minqian Liu, Zhiyang Xu, Licheng Yu, and Lifu Huang. 2023. Ameli: Enhancing multimodal entity linking with fine-grained attributes. arXiv preprint arXiv:2305.14725

    2023 arXiv

  71. [80]

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2024. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36

    2024

  72. [81]

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. 2023. https://arxiv.org/abs/2308.02490 Mm-vet: Evaluating large multimodal models for integrated capabilities . Preprint, arXiv:2308.02490

    2023 arXiv

  73. [82]

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. 2024. htt...

    2024 arXiv

  74. [83]

    Min Zhang, Zhongwei Yu, Hai Wang, Hongbo Qin, Wei Zhao, and Yan Liu. 2019. Automatic digital modulation classification based on curriculum learning. Applied Sciences, 9(10):2171

    2019

  75. [84]

    Chenyang Zhao, Xueying Jia, Vijay Viswanathan, Tongshuang Wu, and Graham Neubig. 2024 a . Self-guide: Better task-specific instruction following via self-synthetic finetuning. arXiv preprint arXiv:2407.12874

    2024 arXiv

  76. [85]

    Haozhe Zhao, Zefan Cai, Shuzheng Si, Xiaojian Ma, Kaikai An, Liang Chen, Zixuan Liu, Sheng Wang, Wenjuan Han, and Baobao Chang. 2024 b . https://arxiv.org/abs/2309.07915 Mmicl: Empowering vision-language model with multi-modal in-context learning . Preprint, arXiv:2309.07915

    2024 arXiv

  77. [86]

    Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. 2024. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36

    2024

  78. [87]

    Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei-Fei. 2016. Visual7w: Grounded question answering in images. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4995--5004

    2016

  79. [88]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  80. [89]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.