{"id":"dda58d04-168d-432b-8e36-e8afe7b17996","arxiv_id":"2505.00029","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Structured Dialogue Fine-Tuning (SDFT) injects domain knowledge into vision-language models through caption, contrastive, and specialization dialogue turns, reporting improved specialization with modest general-capability loss.","lead":"This paper proposes a three-part dialogue training recipe for vision-language models that adds specialized knowledge while trying not to erase general skills. Tests cover personalized objects, abstract concepts, and medical images, but the evidence lacks error bars, some controlled baselines, and public code.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Teacher-generated target answers are the sole source of injected knowledge; if Qwen2-VL-72B errs on the few unlabeled images, SDFT injects those errors and the reported gains are inherited teacher performance.","rationale":"I read the paper as proposing a data-centric fine-tuning strategy for injecting specialized knowledge into LVLMs without catastrophic forgetting. The load-bearing premise is that the teacher model Qwen2-VL-72B-Instruct generates correct target-knowledge answers A3, and that manual filtering ensures their quality. This premise is both necessary and insecure: the method has no internal mechanism to correct or validate the content of A3, so the ceiling of knowledge injection is set by the teacher's accuracy. The reader's weakest_assumption correctly identifies this issue. My independent check of the method and experiments did not reveal a different, more fundamental flaw. I agree with the reader's conditional verdict: the paper should not be accepted as-is because the teacher-quality question is not resolved, and the reported gains could largely reflect distillation rather than a novel ability to avoid forgetting. The proposed concrete test directly audits teacher outputs and would settle whether the concern lands. If the teacher is accurate, the conditional issue is resolved; if not, the main quantitative claims must be re-evaluated. Therefore, I recommend keeping the reader's CONDITIONAL verdict unchanged.","tokens_in":532,"tokens_out":4814,"duration_ms":145351,"concrete_test":"Manually audit a representative sample of the synthesized A3 responses. For each dataset (personalized, abstract, biomedical), randomly select 50 training images, obtain the teacher-generated A3 responses, and have three independent expert annotators judge (a) whether the response correctly identifies the target knowledge present in the image and (b) whether the response is free of factual errors. Report the proportion of correct responses and inter-annotator agreement. If the teacher error rate exceeds 5%, the paper should re-run the main experiments with corrected or human-verified A3 data; if the teacher error rate is negligible, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that SDFT injects knowledge while preserving general capabilities. That claim rests on the quality of the A3 responses generated by the teacher model Qwen2-VL-72B-Instruct in Eq. 7 (Section 3.2.2). For each concept, only 3–5 images without textual labels are available; the teacher must infer the target identity (e.g., 'my pet cat Max') or abstract concept (e.g., 'global warming') from the image and the prompt. The paper mentions 'manual verification' but gives no details on coverage, criteria, or inter-annotator reliability. If the teacher hallucinates or lacks the specialized knowledge, SDFT overtly trains the student to reproduce those errors. The structured dialogue and contrastive turns cannot correct the content of A3; they only shape how that content is learned. Consequently, the measured improvements in knowledge injection (e.g., +29.0% weighted recognition in Table 2) are bounded by the teacher's accuracy on the same images. The teacher is much larger than the student, so it may already recognize these concepts well; if so, SDFT is effectively a distillation method, and the paper's contribution reduces to a data-formatting scheme. If the teacher is frequently wrong, the reported gains are not reliable evidence of generalizable knowledge injection.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Structured Dialogue Fine-Tuning (SDFT), a data-centric method for injecting specialized knowledge into large vision-language models (LVLMs) while mitigating catastrophic forgetting. The method builds three-turn dialogues: a foundation-preserving caption turn, a contrastive disambiguation turn with unrelated concepts, and a knowledge-specialization turn with chain-of-thought QA, trained with a weighted cross-entropy objective (alpha_1=0.2, alpha_2=0.3, alpha_3=0.5). The first two turns use the base model's own outputs, with majority voting for the contrastive turn, while the questions and target-knowledge answers (A3) are generated by Qwen2-VL-72B-Instruct. Experiments cover personalized entities (Yo'LLaVA dataset), abstract concepts (InsightVision), and biomedical VQA, measuring recognition/QA accuracy and general retention on POPE, MME, and TextVQA. SDFT is reported to outperform baselines such as Yo'LLaVA, LLaVA-Med, PubMedVision, and AdaMLLM, and ablations show contributions from self-substitution, multi-round voting, and the three-phase dialogue structure.","tokens_in":16196,"tokens_out":8883,"duration_ms":80760,"significance":"If the results hold, SDFT is a useful data-centric recipe for few-shot knowledge injection: it requires only a few unlabeled images per concept, uses a strong teacher for target knowledge, and the self-distillation of caption and contrastive turns is a plausible mechanism for retention. Strengths of the manuscript are the breadth of evaluation across three knowledge types and several model families, the explicit ablation of each dialogue component, and the clear formulation of the weighted multi-turn loss. However, the current evidentiary base has important gaps: same-architecture baselines are missing for key comparisons, the majority-vote procedure is not defined, teacher-generated content is not verified with reported evidence, and statistical and training details are absent. These gaps are fixable, but they currently limit confidence in the quantitative claims.","major_comments":[{"comment":"The entire knowledge-injection signal A3 is produced by Qwen2-VL-72B-Instruct from only 3-5 images per concept with no textual labels, and the only safeguard described is a passing mention of 'manual verification' (also in Fig. 2). The paper does not report how many responses were verified, by whom, with what criteria, or what the rejection rate was. Because the dialogue structure cannot correct errors in A3, the reported recognition and QA gains are upper-bounded by the teacher's accuracy on these images. Please provide a concrete quality-control protocol with inter-annotator agreement, and ideally an analysis of how SDFT performance varies with measured teacher correctness.","section":"Section 3.2.2, Eq. (7)"},{"comment":"The 'MajorityVote' over m=3 open-ended text generations is not defined. For free-form responses, there is no obvious consensus object; the procedure needs an explicit algorithm (e.g., similarity-based clustering, self-consistency decoding, or a fixed selection rule). Without this, the construction of A2 is not reproducible and the claimed stabilizing effect of voting cannot be independently verified.","section":"Section 3.2.2, Eqs. (5)-(6)"},{"comment":"The comparison against Yo'LLaVA is cross-architecture: Yo'LLaVA is evaluated on a 7B LLaVA model while SDFT is run on Qwen2-VL-2B/7B and InternVL2-8B. Moreover, no standard SFT baseline is reported for InternVL2-8B and Qwen2-VL-7B in Table 2; only Table 4 provides a 'Target QA only' ablation, and only for Qwen2-VL-2B. To support the central claim of avoiding catastrophic forgetting relative to ordinary fine-tuning, please add same-backbone SFT and Yo'LLaVA-style baselines for all model variants.","section":"Table 2 / Section 5.2"},{"comment":"The 'LLaVA-v1.6-8B' block in Table 3 is unexplained: Section 4.3 names Qwen2-VL-2B/7B and InternVL2-8B as the experimental models, yet the table and text report SDFT results for LLaVA-v1.6-8B without giving training details. If these results are new experiments, the setup must be described; if they are comparisons to published baselines, the row should be labeled accordingly. In the same table, 'General Retention' is defined in Appendix C.2 as the average absolute POPE/MME/TextVQA accuracy, not a retention ratio relative to the base model, so the 69.2% versus 66.1% comparison is not a 3.1% relative retention gain. Please clarify the metric and recompute any percentage comparisons.","section":"Table 3 / Section 5.3"},{"comment":"The main quantitative claims are reported as single numbers with no standard deviations, number of seeds, or significance tests, despite the abstract-concept evaluation using only 10 randomly sampled instances per subcategory. The headline +29.0% weighted recognition for Qwen2-VL-2B could be within sampling variability. Please report results over at least three seeds with error bars, and use significance tests for the key comparisons (SDFT versus SFT and SDFT versus baselines).","section":"Section 4.1.1 / Tables 2-3"},{"comment":"The paper states that full-parameter SFT is used but omits learning rate, batch size, epochs, optimizer, warmup, sequence length, and compute budget. In addition, Appendix C.1 refers to 'Table 5' for dataset statistics, but Table 5 in the main text is the domain-knowledge prompt template; no dataset-statistics table is present. Please add a complete training-details table and correct the cross-references.","section":"Section 4.3 / Appendix C"}],"minor_comments":[{"comment":"The sentence 'Figure 3 presents PCA visualizations of hidden state embeddings...' appears twice in consecutive paragraphs; one copy should be deleted.","section":"Section 7.1"},{"comment":"The phrase 'Representative examples from each dataset are provided shown in Fig 2' is ungrammatical; it should be 'are provided and shown in Fig. 2'.","section":"Section 4.2"},{"comment":"There is a double colon in 'depicted in the left portion of Figure 2: :'; this is a typographical error.","section":"Section 3.2.2"},{"comment":"The column 'Gen. Retention' should be renamed to 'Average POPE/MME/TextVQA' or explicitly defined as such, because the text in Section 7.3 interprets it as a retention percentage although it is an absolute average accuracy.","section":"Table 4"},{"comment":"MME is reported as a normalized average score (e.g., 0.612), but the normalization formula is not given; please specify how raw MME scores are converted to the reported range.","section":"Appendix C.2"},{"comment":"The abstract-concept evaluation uses the authors' own InsightVision benchmark [34]; please state this explicitly in the main text and confirm that the benchmark and exact splits are publicly available.","section":"Section 4.1.1"},{"comment":"No code or data release is mentioned; an availability statement would improve reproducibility.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The manuscript appears to contain at least one serious table inconsistency: the LLaVA-v1.6-8B SDFT row in Table 3 is not supported by the model description in Section 4.3. Also, the abstract-concept benchmark [34] is from the same group; I recommend asking the authors to clarify its public availability and evaluation independence. The central idea is promising and the issues appear addressable, but the paper needs another round of experiments and substantially more reporting detail before it can be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, the new thing: a three-turn dialogue template for LVLM fine-tuning—caption, contrastive negative, target QA—with a weighted loss. That specific combination is not in Yo'LLaVA or AdaMLLM, and the ablation shows all three turns contribute. The self-substitution trick (using the base model's own captions instead of the teacher's) is a nice practical detail, and it makes the retention numbers more believable, not less. The spread across personalized, abstract, and biomedical knowledge is also a plus.\n\nThe soft spots are mostly in the evaluation. No error bars or significance tests; several margins are small (e.g., 69.2 vs 66.1 retention, 79.2 vs 78.6 on PathVQA closed). The missing raw-SFT baseline for most of Table 2 means we can't see the forgetting that SDFT is meant to cure. The Yo'LLaVA comparison is confounded by architecture—Yo'LLaVA-7B versus Qwen2-VL-2B is not an apples-to-apples comparison. And the LLaVA-v1.6-8B rows in Table 3 are not accounted for in the Models section; that needs an explanation. No code or data release either.\n\nThe load-bearing assumption is the teacher. All A3 responses come from Qwen2-VL-72B-Instruct, and the manual verification is described in one sentence. If the teacher is wrong about a concept, SDFT trains the student to reproduce the error. The stress-test note has it right: the knowledge-injection gains are bounded by the teacher's accuracy. That said, this is a limitation of the pipeline, not a flaw in the dialogue structure itself. The paper would be much stronger with a teacher pass-rate table and an error analysis.\n\nThe circularity concern is real but partially mitigated. Using the base model's own captions for the retention turns means part of the 'no forgetting' result is self-imitation. But the response-substitution ablation shows that substituting the teacher for those turns hurts both recognition and retention, which is evidence the self-distillation is doing useful regularization work. I'd still want a control where the retention turns contain the same content but are not self-generated.\n\nOverall: this is a solid, practical method with a coherent internal story. The gaps are addressable in revision. I'd send it out. If the authors add error bars, a proper raw-SFT baseline across models, clarify the LLaVA row, and report teacher verification numbers, I'd be fairly optimistic about the core claim.","headline":"A practical dialogue-template recipe that likely helps retention more than it helps knowledge injection, and the evidence is not yet tight enough.","tokens_in":16726,"tokens_out":4474,"would_cite":true,"duration_ms":44607,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Structured dialogue fine-tuning injects specialized knowledge without catastrophic forgetting.","keywords":["Large Vision Language Model","Supervised Fine-Tuning","Knowledge Injection","Catastrophic Forgetting","Domain Adaptation","Multimodal Dialogue","Personalization","Contrastive Disambiguation"],"falsifier":"Take a set of images that contain no visual evidence of the target concept, generate the SDFT dialogues with a teacher that nonetheless reports the concept in its chain-of-thought answer, fine-tune, and measure whether the model learns to 'recognize' the concept. If it does, the method transfers whatever the teacher says and its success is capped by teacher reliability; if it does not, the dialogue structure itself discriminates true from false knowledge. A second test replaces the target-turn answers with random unrelated text while keeping the same three-turn structure, which would show whether any observed gains come from the content of the supervision or from the dialogue format alone.","tokens_in":15719,"feed_emoji":"🧠","tokens_out":9343,"duration_ms":78235,"temperature":0.7,"pith_summary":"Structured Dialogue Fine-Tuning (SDFT) is a data-centric method for teaching a vision-language model a specialized concept—a particular pet, an abstract idea such as global warming, or a biomedical finding—from as few as three to five images while largely preserving the model's general abilities. Each training example is a three-turn conversation: a captioning turn anchors the model in its pre-trained behavior, a contrastive turn asks how the image relates to an unrelated topic to draw a semantic boundary, and a target turn injects the intended knowledge through chain-of-thought reasoning. The three turns are supervised with a weighted cross-entropy loss, placing the most weight on the target turn so the optimization can specialize without letting the other turns disappear. Reported results show the trade-off is favorable: abstract-concept weighted recognition on a 2B model rises from 40.3% to 69.3% while TextVQA falls by only 4.6%, and biomedical general retention reaches 69.2% versus 66.1% for the AdaMLLM baseline. If the claim is right, it gives a cheap, architecture-agnostic recipe for specializing open vision-language models without retrieval systems or per-concept training runs.","feed_headline":"Three-turn dialogue injects knowledge without catastrophic forgetting","feed_subtitle":"SDFT lifts abstract-concept recognition 29 points while cutting TextVQA by only 4.6 points.","key_machinery":"The load-bearing object is the three-phase dialogue template combined with weighted multi-turn supervision. Phase one is a captioning task answered by the base model itself, anchoring training to the model's existing output distribution. Phase two is a counterfactual prompt—'How is this image related to an unrelated concept?'—whose negative answer is produced by majority vote over three base-model responses, drawing a boundary between target knowledge and general semantics. Phase three is a chain-of-thought question-answer pair about the target concept, generated by a teacher model and manually checked. The total loss is a weighted sum of the per-turn cross-entropy losses, with the target turn weighted highest, so the optimizer can favor specialization while the first two turns act as a regularizer. This structure replaces the usual one-turn supervised fine-tuning sample with a deliberately designed learning trajectory.","core_discovery":"The central claim is that catastrophic forgetting during knowledge injection is a symptom of unstructured training data rather than an unavoidable property of fine-tuning. SDFT structures every sample as a three-turn dialogue with three explicit jobs: Foundation Preservation reinforces pre-trained visual-linguistic alignment using a generic caption query answered by the base model itself; Contrastive Disambiguation asks the model to relate the image to a distractor concept and records a majority-voted negative response, creating an explicit semantic boundary; Knowledge Specialization injects the target concept through a chain-of-thought answer generated by a larger teacher model and verified by hand. The three supervision signals are combined in one weighted cross-entropy loss with coefficients (0.2, 0.3, 0.5), and teacher outputs are produced by a three-pass majority vote to suppress noise. The paper argues this structure creates a dedicated pathway for the new concept while leaving representations of unrelated concepts close to the base model, and offers PCA visualizations of hidden states as supporting evidence. On the reported benchmarks the result is large gains in target-knowledge recognition and question answering with small, bounded drops on general capability tests.","pith_inferences":["Because SDFT changes only the training data format, it can be stacked on parameter-efficient methods such as LoRA; testing that combination would show whether the regularizing effect survives when most weights are frozen.","The contrastive disambiguation turn is conceptually a negative-sampling regularizer; comparing SDFT against a conventional contrastive objective on dialogue embeddings would reveal whether the dialogue form adds anything beyond the negative example itself.","The paper relies on a 72B teacher; a cheaper variant could use a smaller verified teacher or automatic consistency checks across multiple teachers, at some cost in supervision quality.","The same three-phase dialogue structure could transfer to audio or video inputs whenever a teacher can produce chain-of-thought answers, since catastrophic forgetting is not specific to the image modality."],"forward_implications":["A single SDFT model can hold several personalized concepts at once: joint training loses only 3.4% weighted accuracy versus separate per-concept models, eliminating the need for per-concept training runs.","Abstract-concept understanding can be learned from a handful of images: on a 2B model, weighted recognition rises from 40.3% to 69.3% and QA accuracy from 42.7% to 57.8%.","The three-turn structure is itself a regularizer: ablations show removing the contrastive turn or the caption turn hurts both target recognition and general retention, with full three-phase training retaining 71.2% versus 58.9% for target-QA-only fine-tuning.","General capability loss stays bounded across model families and scales: TextVQA drops by at most a few points (e.g., -4.6% on Qwen2-VL-2B), while POPE and MME change by less than 2% on all tested models.","In biomedical settings SDFT achieves a higher General Retention (69.2%) than the AdaMLLM baseline (66.1%) while remaining competitive on the medical QA benchmarks themselves."],"supporting_citations":[{"why":"Supplies the subject-binding mechanism (a unique identifier for a specific visual concept) that motivates SDFT's knowledge-isolation strategy.","marker":"[24]"},{"why":"Provides the personalized entity dataset and the Yo'LLaVA baseline that SDFT must match or beat in the concrete-knowledge scenario.","marker":"[19]"},{"why":"Provides the multi-level visual semantics benchmark and evaluation protocol for the abstract-concept scenario.","marker":"[34]"},{"why":"Provides the AdaMLLM two-round dialogue baseline and the domain-expertise setting, including the General Retention comparison.","marker":"[6]"},{"why":"Supplies both the base model family (Qwen2-VL) and the teacher synthesizer used to generate the target-turn dialogues with majority voting.","marker":"[3]"},{"why":"Provides the LLaVA-Med biomedical data and training baseline used for comparison in domain expertise injection.","marker":"[15]"}],"fun_headline_variants":["Structured dialogue fine-tuning balances knowledge gain and retention","Three-turn dialogue injects knowledge without losing base skills","SDFT: 29-point knowledge boost with only 4.6-point TextVQA drop","Dialogue-based fine-tuning prevents catastrophic forgetting during injection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that from three to five images without textual labels, the teacher model can correctly identify and articulate the target knowledge, and that manual verification catches every mistake the teacher makes; if the teacher hallucinates or lacks the fact, SDFT faithfully installs that error as the new knowledge.","fun_headline_variants_meta":{"raw":{"variants":["Structured dialogue fine-tuning balances knowledge gain and retention","Three-turn dialogue injects knowledge without losing base skills","SDFT: 29-point knowledge boost with only 4.6-point TextVQA drop","Dialogue-based fine-tuning prevents catastrophic forgetting during injection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000201,"raw_usage":{"total_tokens":1388,"prompt_tokens":961,"completion_tokens":427,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":577,"completion_tokens_details":{"reasoning_tokens":354}},"tokens_in":577,"tokens_out":427,"duration_ms":4619,"temperature":1.0,"reasoning_tokens":354,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:55:13.402993+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a set of images that contain no visual evidence of the target concept, generate the SDFT dialogues with a teacher that nonetheless reports the concept in its chain-of-thought answer, fine-tune, and measure whether the model learns to 'recognize' the concept. If it does, the method transfers whatever the teacher says and its success is capped by teacher reliability; if it does not, the dialogue structure itself discriminates true from false knowledge. A second test replaces the target-turn answers with random unrelated text while keeping the same three-turn structure, which would show whether any observed gains come from the content of the supervision or from the dialogue format alone.","supporting_citations":[],"review_version":1}