Pith. sign in

REVIEW 6 major objections 7 minor 37 references

Keep the General, Inject the Specific: Structured Dialogue Fine-Tuning for Knowledge Injection without Catastrophic Forgetting

T0 review · 6 major / 7 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Structured dialogue fine-tuning injects specialized knowledge without catastrophic forgetting.

desk verdict A practical dialogue-template recipe that likely helps retention more than it helps knowledge injection, and the evidence is not yet tight enough. read the letter →

arxiv 2505.00029 v1 pith:S4P2JCA3 submitted 2025-04-27 cs.CL cs.AI

Yijie Hong , Xiaofei Yin , Xinzhong Wang , Yi Tu , Ya Guo , Sufeng Duan , Weiqiang Wang , Lingyong Fang
show 2 more authors
Depeng Wang Huijia Zhu
This is my paper · ORCID
classification cs.CLcs.AI
keywords LargeVisionLanguageModelSupervisedFine-TuningKnowledgeInjectionCatastrophicForgettingDomainAdaptationMultimodalDialoguePersonalizationContrastiveDisambiguation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Structured Dialogue Fine-Tuning (SDFT) is a data-centric method for teaching a vision-language model a specialized concept—a particular pet, an abstract idea such as global warming, or a biomedical finding—from as few as three to five images while largely preserving the model's general abilities. Each training example is a three-turn conversation: a captioning turn anchors the model in its pre-trained behavior, a contrastive turn asks how the image relates to an unrelated topic to draw a semantic boundary, and a target turn injects the intended knowledge through chain-of-thought reasoning. The three turns are supervised with a weighted cross-entropy loss, placing the most weight on the target turn so the optimization can specialize without letting the other turns disappear. Reported results show the trade-off is favorable: abstract-concept weighted recognition on a 2B model rises from 40.3% to 69.3% while TextVQA falls by only 4.6%, and biomedical general retention reaches 69.2% versus 66.1% for the AdaMLLM baseline. If the claim is right, it gives a cheap, architecture-agnostic recipe for specializing open vision-language models without retrieval systems or per-concept training runs.

What carries the argument

The load-bearing object is the three-phase dialogue template combined with weighted multi-turn supervision. Phase one is a captioning task answered by the base model itself, anchoring training to the model's existing output distribution. Phase two is a counterfactual prompt—'How is this image related to an unrelated concept?'—whose negative answer is produced by majority vote over three base-model responses, drawing a boundary between target knowledge and general semantics. Phase three is a chain-of-thought question-answer pair about the target concept, generated by a teacher model and manually checked. The total loss is a weighted sum of the per-turn cross-entropy losses, with the target turn weighted highest, so the optimizer can favor specialization while the first two turns act as a regularizer. This structure replaces the usual one-turn supervised fine-tuning sample with a deliberately designed learning trajectory.

What would settle it

Take a set of images that contain no visual evidence of the target concept, generate the SDFT dialogues with a teacher that nonetheless reports the concept in its chain-of-thought answer, fine-tune, and measure whether the model learns to 'recognize' the concept. If it does, the method transfers whatever the teacher says and its success is capped by teacher reliability; if it does not, the dialogue structure itself discriminates true from false knowledge. A second test replaces the target-turn answers with random unrelated text while keeping the same three-turn structure, which would show whether any observed gains come from the content of the supervision or from the dialogue format alone.

Watch

Extended reading notes

Core claim

The central claim is that catastrophic forgetting during knowledge injection is a symptom of unstructured training data rather than an unavoidable property of fine-tuning. SDFT structures every sample as a three-turn dialogue with three explicit jobs: Foundation Preservation reinforces pre-trained visual-linguistic alignment using a generic caption query answered by the base model itself; Contrastive Disambiguation asks the model to relate the image to a distractor concept and records a majority-voted negative response, creating an explicit semantic boundary; Knowledge Specialization injects the target concept through a chain-of-thought answer generated by a larger teacher model and verified by hand. The three supervision signals are combined in one weighted cross-entropy loss with coefficients (0.2, 0.3, 0.5), and teacher outputs are produced by a three-pass majority vote to suppress noise. The paper argues this structure creates a dedicated pathway for the new concept while leaving representations of unrelated concepts close to the base model, and offers PCA visualizations of hidden states as supporting evidence. On the reported benchmarks the result is large gains in target-knowledge recognition and question answering with small, bounded drops on general capability tests.

Load-bearing premise

The method assumes that from three to five images without textual labels, the teacher model can correctly identify and articulate the target knowledge, and that manual verification catches every mistake the teacher makes; if the teacher hallucinates or lacks the fact, SDFT faithfully installs that error as the new knowledge.

Editorial extensions

If this is right

  • A single SDFT model can hold several personalized concepts at once: joint training loses only 3.4% weighted accuracy versus separate per-concept models, eliminating the need for per-concept training runs.
  • Abstract-concept understanding can be learned from a handful of images: on a 2B model, weighted recognition rises from 40.3% to 69.3% and QA accuracy from 42.7% to 57.8%.
  • The three-turn structure is itself a regularizer: ablations show removing the contrastive turn or the caption turn hurts both target recognition and general retention, with full three-phase training retaining 71.2% versus 58.9% for target-QA-only fine-tuning.
  • General capability loss stays bounded across model families and scales: TextVQA drops by at most a few points (e.g., -4.6% on Qwen2-VL-2B), while POPE and MME change by less than 2% on all tested models.
  • In biomedical settings SDFT achieves a higher General Retention (69.2%) than the AdaMLLM baseline (66.1%) while remaining competitive on the medical QA benchmarks themselves.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because SDFT changes only the training data format, it can be stacked on parameter-efficient methods such as LoRA; testing that combination would show whether the regularizing effect survives when most weights are frozen.
  • The contrastive disambiguation turn is conceptually a negative-sampling regularizer; comparing SDFT against a conventional contrastive objective on dialogue embeddings would reveal whether the dialogue form adds anything beyond the negative example itself.
  • The paper relies on a 72B teacher; a cheaper variant could use a smaller verified teacher or automatic consistency checks across multiple teachers, at some cost in supervision quality.
  • The same three-phase dialogue structure could transfer to audio or video inputs whenever a teacher can produce chain-of-thought answers, since catastrophic forgetting is not specific to the image modality.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

6 major / 7 minor

Summary. The paper introduces Structured Dialogue Fine-Tuning (SDFT), a data-centric method for injecting specialized knowledge into large vision-language models (LVLMs) while mitigating catastrophic forgetting. The method builds three-turn dialogues: a foundation-preserving caption turn, a contrastive disambiguation turn with unrelated concepts, and a knowledge-specialization turn with chain-of-thought QA, trained with a weighted cross-entropy objective (alpha_1=0.2, alpha_2=0.3, alpha_3=0.5). The first two turns use the base model's own outputs, with majority voting for the contrastive turn, while the questions and target-knowledge answers (A3) are generated by Qwen2-VL-72B-Instruct. Experiments cover personalized entities (Yo'LLaVA dataset), abstract concepts (InsightVision), and biomedical VQA, measuring recognition/QA accuracy and general retention on POPE, MME, and TextVQA. SDFT is reported to outperform baselines such as Yo'LLaVA, LLaVA-Med, PubMedVision, and AdaMLLM, and ablations show contributions from self-substitution, multi-round voting, and the three-phase dialogue structure.

Significance. If the results hold, SDFT is a useful data-centric recipe for few-shot knowledge injection: it requires only a few unlabeled images per concept, uses a strong teacher for target knowledge, and the self-distillation of caption and contrastive turns is a plausible mechanism for retention. Strengths of the manuscript are the breadth of evaluation across three knowledge types and several model families, the explicit ablation of each dialogue component, and the clear formulation of the weighted multi-turn loss. However, the current evidentiary base has important gaps: same-architecture baselines are missing for key comparisons, the majority-vote procedure is not defined, teacher-generated content is not verified with reported evidence, and statistical and training details are absent. These gaps are fixable, but they currently limit confidence in the quantitative claims.

major comments (6)
  1. [Section 3.2.2, Eq. (7)] The entire knowledge-injection signal A3 is produced by Qwen2-VL-72B-Instruct from only 3-5 images per concept with no textual labels, and the only safeguard described is a passing mention of 'manual verification' (also in Fig. 2). The paper does not report how many responses were verified, by whom, with what criteria, or what the rejection rate was. Because the dialogue structure cannot correct errors in A3, the reported recognition and QA gains are upper-bounded by the teacher's accuracy on these images. Please provide a concrete quality-control protocol with inter-annotator agreement, and ideally an analysis of how SDFT performance varies with measured teacher correctness.
  2. [Section 3.2.2, Eqs. (5)-(6)] The 'MajorityVote' over m=3 open-ended text generations is not defined. For free-form responses, there is no obvious consensus object; the procedure needs an explicit algorithm (e.g., similarity-based clustering, self-consistency decoding, or a fixed selection rule). Without this, the construction of A2 is not reproducible and the claimed stabilizing effect of voting cannot be independently verified.
  3. [Table 2 / Section 5.2] The comparison against Yo'LLaVA is cross-architecture: Yo'LLaVA is evaluated on a 7B LLaVA model while SDFT is run on Qwen2-VL-2B/7B and InternVL2-8B. Moreover, no standard SFT baseline is reported for InternVL2-8B and Qwen2-VL-7B in Table 2; only Table 4 provides a 'Target QA only' ablation, and only for Qwen2-VL-2B. To support the central claim of avoiding catastrophic forgetting relative to ordinary fine-tuning, please add same-backbone SFT and Yo'LLaVA-style baselines for all model variants.
  4. [Table 3 / Section 5.3] The 'LLaVA-v1.6-8B' block in Table 3 is unexplained: Section 4.3 names Qwen2-VL-2B/7B and InternVL2-8B as the experimental models, yet the table and text report SDFT results for LLaVA-v1.6-8B without giving training details. If these results are new experiments, the setup must be described; if they are comparisons to published baselines, the row should be labeled accordingly. In the same table, 'General Retention' is defined in Appendix C.2 as the average absolute POPE/MME/TextVQA accuracy, not a retention ratio relative to the base model, so the 69.2% versus 66.1% comparison is not a 3.1% relative retention gain. Please clarify the metric and recompute any percentage comparisons.
  5. [Section 4.1.1 / Tables 2-3] The main quantitative claims are reported as single numbers with no standard deviations, number of seeds, or significance tests, despite the abstract-concept evaluation using only 10 randomly sampled instances per subcategory. The headline +29.0% weighted recognition for Qwen2-VL-2B could be within sampling variability. Please report results over at least three seeds with error bars, and use significance tests for the key comparisons (SDFT versus SFT and SDFT versus baselines).
  6. [Section 4.3 / Appendix C] The paper states that full-parameter SFT is used but omits learning rate, batch size, epochs, optimizer, warmup, sequence length, and compute budget. In addition, Appendix C.1 refers to 'Table 5' for dataset statistics, but Table 5 in the main text is the domain-knowledge prompt template; no dataset-statistics table is present. Please add a complete training-details table and correct the cross-references.
minor comments (7)
  1. [Section 7.1] The sentence 'Figure 3 presents PCA visualizations of hidden state embeddings...' appears twice in consecutive paragraphs; one copy should be deleted.
  2. [Section 4.2] The phrase 'Representative examples from each dataset are provided shown in Fig 2' is ungrammatical; it should be 'are provided and shown in Fig. 2'.
  3. [Section 3.2.2] There is a double colon in 'depicted in the left portion of Figure 2: :'; this is a typographical error.
  4. [Table 4] The column 'Gen. Retention' should be renamed to 'Average POPE/MME/TextVQA' or explicitly defined as such, because the text in Section 7.3 interprets it as a retention percentage although it is an absolute average accuracy.
  5. [Appendix C.2] MME is reported as a normalized average score (e.g., 0.612), but the normalization formula is not given; please specify how raw MME scores are converted to the reported range.
  6. [Section 4.1.1] The abstract-concept evaluation uses the authors' own InsightVision benchmark [34]; please state this explicitly in the main text and confirm that the benchmark and exact splits are publicly available.
  7. [General] No code or data release is mentioned; an availability statement would improve reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: SDFT's reported knowledge-injection and retention numbers are measured on external or held-out benchmarks, not recovered from the supervision targets by construction.

full rationale

The paper's derivation chain is empirical rather than definitional. The knowledge-specialization target A3 is produced by a distinct teacher model, Qwen2-VL-72B-Instruct, via Eq. 7, while the foundation and contrastive responses A1 and A2 are generated by the base model B via Eqs. 4-6. Training the student to imitate B for captions and negative responses is an explicit anchoring regularizer, but the claimed general-capability preservation is measured on POPE, MME, and TextVQA, and the knowledge-injection claims are measured on Yo'LLaVA, InsightVision, and medical VQA benchmarks; none of these evaluation scores is defined as the training outputs themselves. The ablation in Section 6.1 showing that self-substitution improves retention is consistent with the intended mechanism, not with a tautology, because the retention metric still comes from held-out benchmark performance. The main validity risks are that the teacher's A3 answers may be erroneous or hallucinated and that the abstract-concept benchmark [34] has overlapping authorship; these are evidence-quality concerns, not circular reductions. There is no fitted parameter renamed as a prediction, no uniqueness theorem imported from the authors' prior work, and no equation in the paper that equates a claimed result to its own input by construction. Therefore the paper does not exhibit significant circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on two unverified premises: the teacher model's generated target responses are factually correct, and the base model's own captions and majority-voted negatives are a faithful regularizer for general capabilities. The method also relies on manually chosen loss weights and evaluation thresholds. No new entities are introduced.

free parameters (3)
  • Loss weights (alpha_1, alpha_2, alpha_3) = (0.2, 0.3, 0.5)
    Set empirically in Section 3.3 to yield favorable performance trade-offs; these weights directly control the training objective in Eq. 8 and no sensitivity analysis is shown.
  • Majority vote sample count m = 3
    Chosen for negative-response synthesis in Eq. 5-6; no ablation on this value is reported.
  • BERTScore threshold for open-ended medical QA = 0.85
    Used in Appendix C for semantic equivalence scoring of open-ended answers; no justification or sensitivity analysis is provided.
assumptions (4)
  • domain assumption The synthesis model Qwen2-VL-72B-Instruct produces factually correct, high-fidelity target-knowledge responses A3 for all three domains.
    Section 3.2.2 Eq. 7 and Section 4.2; if the teacher hallucinates, the injected knowledge is wrong and the method cannot succeed.
  • domain assumption Base model-generated captions A1 and majority-voted negative responses A2 are valid supervision for preserving foundational capabilities and semantic boundaries.
    Eq. 4-6; the forgetting-mitigation claim assumes the base model's own outputs are a good proxy for the general abilities to be preserved.
  • domain assumption The loss weights can be set once and transferred across tasks.
    Section 3.3 sets (0.2, 0.3, 0.5) for all tasks; the paper provides no analysis showing these weights are not overfit to the selected benchmarks.
  • domain assumption POPE, MME, and TextVQA collectively measure general visual-linguistic capability retention.
    Section 4.1.2 and Appendix C use these benchmarks as the definition of general capability; the paper does not validate this choice against other retention measures.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Keep the General, Inject the Specific: Structured Dialogue Fine-Tuning for Knowledge Injection without Catastrophic Forgetting." pith.science (2026). https://pith.science/paper/S4P2JCA3

@misc{pith2026250500029,
  author       = {Pith},
  title        = {Pith review of: Keep the General, Inject the Specific: Structured Dialogue Fine-Tuning for Knowledge Injection without Catastrophic Forgetting},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/S4P2JCA3}},
  note         = {Machine review of arXiv:2505.00029}
}
read the original abstract

Large Vision Language Models have demonstrated impressive versatile capabilities through extensive multimodal pre-training, but face significant limitations when incorporating specialized knowledge domains beyond their training distribution. These models struggle with a fundamental dilemma: direct adaptation approaches that inject domain-specific knowledge often trigger catastrophic forgetting of foundational visual-linguistic abilities. We introduce Structured Dialogue Fine-Tuning (SDFT), an effective approach that effectively injects domain-specific knowledge while minimizing catastrophic forgetting. Drawing inspiration from supervised fine-tuning in LLMs and subject-driven personalization in text-to-image diffusion models, our method employs a three-phase dialogue structure: Foundation Preservation reinforces pre-trained visual-linguistic alignment through caption tasks; Contrastive Disambiguation introduces carefully designed counterfactual examples to maintain semantic boundaries; and Knowledge Specialization embeds specialized information through chain-of-thought reasoning. Experimental results across multiple domains confirm SDFT's effectiveness in balancing specialized knowledge acquisition with general capability retention. Our key contributions include a data-centric dialogue template that balances foundational alignment with targeted knowledge integration, a weighted multi-turn supervision framework, and comprehensive evaluation across diverse knowledge types.

Figures

Figures reproduced from arXiv: 2505.00029 by the authors.

Figure 1
Figure 1. Structured multi-turn supervision enables knowl [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of the SDFT framework. Given domain-specific images across diverse categories (personalized entities, [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. PCA visualization of hidden states when responding [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

37 extracted references · 1 canonical work pages

  1. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

    arXiv 2023

  2. [2]

    Yuval Alaluf, Elad Richardson, Sergey Tulyakov, Kfir Aberman, and Daniel Cohen-Or. 2024. MyVLM: Personalizing VLMs for User-Specific Queries. arXiv:2403.14599 [cs.CV] https://arxiv.org/abs/2403.14599

    arXiv 2024

  3. [3]

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609 (2023)

    arXiv 2023

  4. [4]

    Junying Chen, Chi Gui, Ruyi Ouyang, Anningzhe Gao, Shunian Chen, Guim- ing Hardy Chen, Xidong Wang, Ruifei Zhang, Zhenyang Cai, Ke Ji, Guangjun Yu, Xiang Wan, and Benyou Wang. 2024. HuatuoGPT-Vision, Towards Injecting Med- ical Visual Knowledge into Multimodal LLMs at Scale. arXiv:2406.19280 [cs.CV] https://arxiv.org/abs/2406.19280

    arXiv 2024

  5. [5]

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, Ji Ma, Jiaqi Wang, Xiaoyi Dong, Hang Yan, Hewei Guo, Conghui He, Botian Shi, Zhenjiang Jin, Chao Xu, Bin Wang, Xingjian Wei, Wei Li, Wenjian Zhang, Bo Zhang, Pinlong Cai, Licheng Wen, Xiangchao Yan, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu...

    arXiv 2024

  6. [6]

    Daixuan Cheng, Shaohan Huang, Ziyu Zhu, Xintong Zhang, Wayne Xin Zhao, Zhongzhi Luan, Bo Dai, and Zhenliang Zhang. 2025. On Domain-Specific Post- Training for Multimodal Large Language Models. arXiv:2411.19930 [cs.CL] https://arxiv.org/abs/2411.19930

    arXiv 2025

  7. [7]

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. 2024. MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models. arXiv:2306.13394 [cs.CV] https://arxiv.org/abs/2306.13394

    arXiv 2024

  8. [8]

    Bermano, Gal Chechik, and Daniel Cohen-Or

    Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H. Bermano, Gal Chechik, and Daniel Cohen-Or. 2022. An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion. arXiv:2208.01618 [cs.CV] https://arxiv.org/abs/2208.01618

    arXiv 2022

Show all 37 references
  1. [9]

    Bermano, Gal Chechik, and Daniel Cohen-Or

    Rinon Gal, Moab Arar, Yuval Atzmon, Amit H. Bermano, Gal Chechik, and Daniel Cohen-Or. 2023. Encoder-based Domain Tuning for Fast Personalization of Text- to-Image Models. arXiv:2302.12228 [cs.CV] https://arxiv.org/abs/2302.12228

    2023 arXiv

  2. [10]

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. 2024. Retrieval-Augmented Generation for Large Language Models: A Survey. arXiv:2312.10997 [cs.CL] https://arxiv.org/abs/2312.10997

    2024 arXiv

  3. [11]

    Haoran Hao, Jiaming Han, Changsheng Li, Yu-Feng Li, and Xiangyu Yue. 2025. RAP: Retrieval-Augmented Personalization for Multimodal Large Language Models. arXiv:2410.13360 [cs.CV] https://arxiv.org/abs/2410.13360

    2025 arXiv

  4. [12]

    Hulingxiao He, Geng Li, Zijun Geng, Jinglin Xu, and Yuxin Peng. 2025. Analyzing and Boosting the Power of Fine-Grained Visual Recognition for Multi-modal Large Language Models. arXiv:2501.15140 [cs.CV] https://arxiv.org/abs/2501. 15140

    2025 arXiv

  5. [13]

    Zecheng He, Bo Sun, Felix Juefei-Xu, Haoyu Ma, Ankit Ramchandani, Vincent Cheung, Siddharth Shah, Anmol Kalia, Harihar Subramanyam, Alireza Zareian, Li Chen, Ankit Jain, Ning Zhang, Peizhao Zhang, Roshan Sumbaly, Peter Vajda, and Animesh Sinha. 2024. Imagine yourself: Tuning-F...

    2024 arXiv

  6. [14]

    Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell

    James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. 2017. Overcoming catastrophic forget...

    2017 doi

  7. [15]

    Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. 2023. Llava-med: Train- ing a large language-and-vision assistant for biomedicine in one day. Advances in Neural Information Processing Systems 36 (...

    2023

  8. [16]

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. 2023. Evaluating Object Hallucination in Large Vision-Language Models. arXiv:2305.10355 [cs.CV] https://arxiv.org/abs/2305.10355

    2023 arXiv

  9. [17]

    Chen Ling, Xujiang Zhao, Jiaying Lu, Chengyuan Deng, Can Zheng, Junxi- ang Wang, Tanmoy Chowdhury, Yun Li, Hejie Cui, Xuchao Zhang, Tianjiao Zhao, Amit Panalkar, Dhagash Mehta, Stefano Pasquali, Wei Cheng, Haoyu Wang, Yanchi Liu, Zhengzhang Chen, Haifeng Chen, Chris White, Qua...

    2024 arXiv

  10. [18]

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024. Improved Baselines with Visual Instruction Tuning. arXiv:2310.03744 [cs.CV] https://arxiv.org/abs/ 2310.03744

    2024 arXiv

  11. [19]

    Thao Nguyen, Haotian Liu, Yuheng Li, Mu Cai, Utkarsh Ojha, and Yong Jae Lee. 2024. Yo’LLaVA: Your Personalized Language and Vision Assistant. arXiv:2406.09400 [cs.CV] https://arxiv.org/abs/2406.09400

    2024 arXiv

  12. [20]

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schul- man, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Pe- ter Welinder, Paul Christiano, Jan Leike,...

    2022 arXiv

  13. [21]

    Oded Ovadia, Menachem Brief, Moshik Mishaeli, and Oren Elisha. 2023. Fine- tuning or retrieval? comparing knowledge injection in llms. arXiv preprint arXiv:2312.05934 (2023)

    2023 arXiv

  14. [22]

    Renjie Pi, Jianshu Zhang, Tianyang Han, Jipeng Zhang, Rui Pan, and Tong Zhang. 2024. Personalized Visual Instruction Tuning. arXiv:2410.07113 [cs.CV] https://arxiv.org/abs/2410.07113

    2024 arXiv

  15. [23]

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems 36 (2023), 53728–53741

    2023

  16. [24]

    Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. 2023. DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation. arXiv:2208.12242 [cs.CV] https://arxiv.org/abs/ 2208.12242

    2023 arXiv

  17. [25]

    Jing Shi, Wei Xiong, Zhe Lin, and Hyun Joon Jung. 2023. Instant- Booth: Personalized Text-to-Image Generation without Test-Time Finetuning. arXiv:2304.03411 [cs.CV] https://arxiv.org/abs/2304.03411

    2023 arXiv

  18. [26]

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. 2019. Towards VQA Models That Can Read. arXiv:1904.08920 [cs.CL] https://arxiv.org/abs/1904.08920

    2019 arXiv

  19. [27]

    Zirui Song, Bin Yan, Yuhan Liu, Miao Fang, Mingzhe Li, Rui Yan, and Xiuying Chen. 2025. Injecting Domain-Specific Knowledge into Large Language Models: A Comprehensive Survey. arXiv:2502.10708 [cs.CL] https://arxiv.org/abs/2502. 10708

    2025 arXiv

  20. [28]

    Katherine Tian, Eric Mitchell, Huaxiu Yao, Christopher D Manning, and Chelsea Finn. 2023. Fine-tuning language models for factuality. In The Twelfth Interna- tional Conference on Learning Representations

    2023

  21. [29]

    Markosyan, Luke Zettlemoyer, and Armen Agha- janyan

    Kushal Tirumala, Aram H. Markosyan, Luke Zettlemoyer, and Armen Agha- janyan. 2022. Memorization Without Overfitting: Analyzing the Training Dy- namics of Large Language Models. arXiv:2205.10770 [cs.CL] https://arxiv.org/ abs/2205.10770

    2022 arXiv

  22. [30]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. arXiv:2201.11903 [cs.CL] https: //arxiv.org/abs/2201.11903

    2023 arXiv

  23. [31]

    Peng Xia, Kangyu Zhu, Haoran Li, Tianze Wang, Weijia Shi, Sheng Wang, Linjun Zhang, James Zou, and Huaxiu Yao. 2025. MMed-RAG: Versatile Multimodal RAG System for Medical Vision Language Models. arXiv:2410.13085 [cs.LG] https://arxiv.org/abs/2410.13085

    2025 arXiv

  24. [32]

    Yan Xu, Mahdi Namazifar, Devamanyu Hazarika, Aishwarya Padmakumar, Yang Liu, and Dilek Hakkani-Tür. 2023. Kilm: Knowledge injection into encoder- decoder language models. arXiv preprint arXiv:2302.09170 (2023)

    2023 arXiv

  25. [33]

    Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. 2023. IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models. arXiv:2308.06721 [cs.CV] https://arxiv.org/abs/2308.06721

    2023 arXiv

  26. [34]

    Xiaofei Yin, Yijie Hong, Ya Guo, Yi Tu, Weiqiang Wang, Gongshen Liu, and Huijia zhu. 2025. InsightVision: A Comprehensive, Multi-Level Chinese-based Benchmark for Evaluating Implicit Visual Semantics in Large Vision Language Models. arXiv:2502.15812 [cs.LG] https://arxiv.org/a...

    2025 arXiv

  27. [35]

    Patel, Haochen Wang, Xun Huang, Ting-Chun Wang, Ming-Yu Liu, and Yogesh Balaji

    Yu Zeng, Vishal M. Patel, Haochen Wang, Xun Huang, Ting-Chun Wang, Ming-Yu Liu, and Yogesh Balaji. 2024. JeDi: Joint-Image Diffusion Models for Finetuning- Free Personalized Text-to-Image Generation. arXiv:2407.06187 [cs.CV] https: //arxiv.org/abs/2407.06187

    2024 arXiv

  28. [36]

    Lungren, Tristan Naumann, Sheng Wang, and Hoifung Poon

    Sheng Zhang, Yanbo Xu, Naoto Usuyama, Hanwen Xu, Jaspreet Bagga, Robert Tinn, Sam Preston, Rajesh Rao, Mu Wei, Naveen Valluri, Cliff Wong, Andrea Tupini, Yu Wang, Matt Mazzola, Swadheen Shukla, Lars Liden, Jianfeng Gao, Angela Crabtree, Brian Piening, Carlo Bifulco, Matthew P....

    2025 arXiv

  29. [37]

    global warming

    Da-Wei Zhou, Yuanhan Zhang, Yan Wang, Jingyi Ning, Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu. 2025. Learning Without Forgetting for Vision-Language Models. IEEE Transactions on Pattern Analysis and Machine Intelligence (2025), 1–16. https://doi.org/10.1109/tpami.2025.3540889 Ke...

    2025

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.