{"id":"fd643d2a-68cf-4342-a068-f35bab12c709","arxiv_id":"2412.12940","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Text-only fine-tuning of 7B VLMs on GPT-4o-generated descriptions rivals image-text fine-tuning for fine-grained visual VQA at substantially lower compute.","lead":"The authors fine-tuned two 7B vision-language models using only GPT-4o-written text descriptions, with no images, and found this text-only training roughly matches image-text training on butterfly species and Korean cultural visual understanding benchmarks. The result suggests a cheaper way to adapt VLMs to fine-grained domains, though the small, noisy experiments need independent replication.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Butterfly Type 2 evaluation is generated by GPT-4o from the same encyclopedic knowledge used to create the text-only training descriptions, and no image-free control is reported for this benchmark, so the headline gains may reflect text-pattern memorization rather than visual grounding.","rationale":"I read the paper's aim as testing whether rich textual descriptions can substitute for image-text pairs when adapting a VLM to fine-grained domains. The resource claim follows mechanically from omitting image preprocessing and is well supported by Figure 2. The performance claim is the load-bearing part. The cleanest support would be an image-free control on the same benchmark, as the authors provide for K-VISCUIT. That control shows large drops, which is genuine evidence against pure memorization for the Korean cultural dataset. But the authors did not run the corresponding control on Butterfly, where the training text and the Type 2 evaluation share a GPT-4o origin. Because the text-only model is trained on the same kind of encyclopedic butterfly facts that the Type 2 questions probe, a strong score without the image would indicate the model is retrieving text rather than using the image. Even with images, a human-written evaluation is needed to rule out distributional overlap. The reader's conditional verdict already captures this concern; my pass adds the precise missing experiment. I do not see an internal inconsistency, and I credit the K-VISCUIT no-image control and the resource measurements. The best-epoch selection is a further limitation because it can only overstate performance, but it affects both training conditions similarly, so it does not by itself overturn the comparison; it does warrant lower confidence in the small reported margins.","tokens_in":8960,"tokens_out":4885,"duration_ms":48269,"concrete_test":"On the Butterfly benchmark, rerun the exact Table 2 protocol: evaluate the Original, Image+Text, and Text-only models with image inputs withheld, for both Type 1 and Type 2. Then replace the GPT-4o-generated Type 2 questions with human-written questions drawn from an independent butterfly biology source and compare with-image accuracies. If text-only models score near their with-image accuracy on Type 2 without images, or if their advantage over the original model disappears on human-written questions, the central claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that text-only training improves visual understanding, not just that it reproduces GPT-4o text. The butterfly training descriptions (Appendix B) are generated by prompting GPT-4o with the species name for sections on visual appearance, habitat, feeding, breeding, and behavior. The butterfly Type 2 evaluation questions are also GPT-4o-generated from the same ecological and biological content, with the species name withheld. A model fine-tuned on these descriptions can answer a Type 2 question by classifying the species from the image and recalling the memorized fact; the reported 'visual understanding' gain is then inseparable from LLM knowledge transfer. The authors' contamination check (Table 2) is run only on K-VISCUIT, whose questions come from a pre-existing benchmark, and is not run on Butterfly. The Type 1 results, where the evaluation options are species names, are less vulnerable to this overlap, but the differences there are small (Qwen2-VL: +3.0 on Type 1; LLaVA: +2.5) and are reported without error bars or multiple seeds. Appendix A also states that the best epoch is selected on the evaluation set, which further inflates the reported numbers. Thus the decisive evidence that text-only training improves fine-grained visual understanding in the butterfly domain is missing.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a text-only training strategy for 7B vision-language models, in which the model is fine-tuned on detailed GPT-4o-generated textual descriptions of butterfly species and Korean cultural objects, without any image inputs. The authors compare this with fine-tuning on the same descriptions paired with single images, reporting comparable or better accuracy on two evaluation suites: a newly created GPT-4o-based Butterfly VQA (Types 1 and 2) and the K-VISCUIT benchmark. They also report reduced training time, peak memory, and energy consumption. They argue that the gains reflect improved visual-linguistic alignment rather than contamination, based on an image-free control on K-VISCUIT.","tokens_in":9157,"tokens_out":7860,"duration_ms":72702,"significance":"If the performance claims were robust, the paper would offer a practically valuable result: fine-tuning a VLM on text alone could improve visual tasks at much lower compute cost, which is especially relevant for low-resource settings. The resource measurements are a concrete strength, as are the authors' attempts to address contamination with an image-free control. However, the current evidence does not establish that text-only training improves fine-grained visual understanding, because the evaluation protocol is partly circular (GPT-4o-generated training captions and Butterfly Type 2 questions) and the statistical basis is a single run with best-epoch selection on the evaluation set. The claim is therefore plausible but unproven.","major_comments":[{"comment":"The Butterfly Type 2 evaluation is generated by prompting GPT-4o with ecological and biological content, and the text-only training descriptions are also generated by GPT-4o using the same topical sections (visual description, habitat, feeding, breeding, behavior). A model that fine-tunes on the training descriptions and then identifies the species from the image can answer Type 2 questions by retrieving memorized text; no image-free control is reported for the Butterfly benchmark. As a result, the Type 2 gains reported in Table 1 (e.g., LLaVA +7.5 points) do not isolate improved visual understanding from knowledge transfer from the shared text distribution.","section":"Evaluation Datasets / Appendix B"},{"comment":"All results come from a single training run in which the best epoch is selected on the evaluation set (Appendix A: \"we report the best performance achieved during the training process\"). With 200 Butterfly questions, a one-run gain of +1.5 points (Qwen2-VL Total) is well within sampling error (binomial SE approximately 3.5 points at 50% accuracy), and no confidence intervals or significance tests are provided. Multiple seeds with held-out validation, or at least error bars, are necessary to support the claim that text-only training is comparable to image-text training.","section":"Appendix A / Table 1"},{"comment":"The image-free control shows that text-only training also improves no-image accuracy on K-VISCUIT (LLaVA: 36.83 to 42.77; Qwen2-VL: 45.97 to 47.18). This demonstrates that part of the benefit is a language-side gain, not visual grounding. The paper's argument that the persistent drop without images rules out contamination is not sufficient; the relevant quantity for visual understanding is the differential improvement (with-image minus no-image) relative to the original model, which is not reported and is negative for LLaVA (3.80 minus 5.94 equals -2.14 points). An image-free control on the Butterfly benchmark is also missing.","section":"Evidence Against Data Contamination / Table 2"}],"minor_comments":[{"comment":"The text-only gain for Qwen2-VL Butterfly Total is stated as +1.05%p, whereas Table 1 reports +1.50%p; please correct the inconsistency.","section":"Main Results"},{"comment":"The Qwen2-VL Butterfly Type 2 comparison is given as 50.50 vs 50.00, but Table 1 reports 60.50 vs 60.00; the Type 1 comparison is given as 77.50 vs 78.50, but Table 1 reports 76.50 vs 78.00 for Image+Text vs Text-only.","section":"Analysis by Question Types"},{"comment":"The image-text baseline uses only one randomly sampled image per species (100 images total); the abstract's phrase 'conventional image-text training' should be qualified to reflect this small-data regime.","section":"Methodology / Datasets"},{"comment":"The statement about 'training instability starting from 13B' lacks supporting details; please provide the observations or remove the claim.","section":"Methodology / Models"},{"comment":"Figure 2 does not specify the measurement setup (e.g., GPU model, software, ambient conditions, whether energy includes CPU/GPU); please describe the methodology for the reported Wh values.","section":"Resource Comparison / Figure 2"},{"comment":"The paper refers to LLaVA-1.6 but cites Li et al. 2024 (LLaVA-Next-Interleave); please clarify the exact model variant and checkpoint used.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper reads like an early-stage submission with several numerical inconsistencies and missing statistical rigor. The core idea is timely, but the evaluation must be substantially strengthened before it could meet the journal's bar. If the authors can provide image-free controls for Butterfly, multiple seeds with proper validation, and evaluation on an external benchmark, the paper could become a solid contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The punchline: this is a reasonable empirical study of an idea that is not new—text-only training for vision—applied to a concrete setting where it hasn't been tested before. The authors compare text-only fine-tuning against image-text fine-tuning on two fine-grained domains with 7B open VLMs, and they measure training time, memory, and energy. That measurement set is the contribution, and it's useful for practitioners who want a cheap adaptation recipe.\n\nThe paper does a few things right. The image-free control on K-VISCUIT is the right check: both original and text-only models drop sharply when images are removed, which suggests the text-only model is not just memorizing surface text patterns. The resource numbers (16–33% time savings, 37–49% energy savings) are plausible, though single-GPU and pipeline-specific. The writing is clear and the related work is honestly placed: the LENS, LOVM, and Caron et al. citations correctly anticipate the core idea.\n\nNow the soft spots, in proportion. The biggest one is the butterfly Type 2 evaluation. Those questions are generated with GPT-4o from the same encyclopedic butterfly knowledge that produced the training descriptions. A model that identifies the species from the image and then recalls a memorized fact from the text-only training can answer Type 2 questions without any new visual reasoning. The contamination check in Table 2 is run only on K-VISCUIT, not on Butterfly, so the decisive evidence that text-only training improves visual understanding in that domain is simply missing. The stress-test note lands on this.\n\nThe second soft spot is statistical. Every condition is a single run, the best epoch is selected on the evaluation set, and the differences are small—Qwen2-VL butterfly total +1.5 points on 200 questions is sampling noise. The authors acknowledge the best-epoch selection in Appendix A, which is honest but doesn't fix the inflation. Multiple seeds, fixed checkpoints, and error bars are needed before the headline claim is established.\n\nThird, the circularity burden is partly real: both training captions and butterfly Type 2 questions come from GPT-4o. The Type 1 results are less vulnerable, but those differences are also small and noisy.\n\nWho is this for? People who build VLMs with limited compute and want a cheaper fine-tuning route. The idea is plausible and consistent with prior language-driven vision work, so the paper deserves a serious referee. My own verdict is conditional: the experiment is well-motivated and the efficiency data is useful, but the butterfly evaluation needs a rework with external questions and proper statistics. I'd recommend peer review with major revision, not desk rejection.","headline":"Plausible efficiency story, but the butterfly evaluation overlaps with the training data generator and the reported gains lack error bars.","tokens_in":9767,"tokens_out":1845,"would_cite":false,"duration_ms":16359,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning a vision-language model on text alone, with no images, matches or beats image-text fine-tuning on fine-grained visual tasks, while cutting training time by up to 33% and energy use by up to 49%.","keywords":["text-only training","vision-language models","fine-grained visual understanding","butterfly species classification","Korean cultural VQA","computational efficiency","GPT-4o descriptions","visual-linguistic alignment"],"falsifier":"Replace GPT-4o with a different description source, such as encyclopedic entries or human-written species guides, while keeping the same evaluation, or use a human-written butterfly question set; if text-only training no longer matches image-text training, the effect is tied to GPT-4o's phrasing rather than to rich textual description in general.","tokens_in":8675,"feed_emoji":"🦋","tokens_out":5465,"duration_ms":46778,"temperature":0.7,"pith_summary":"This paper asks whether a vision-language model can improve its fine-grained visual recognition without ever seeing images during fine-tuning. The authors train 7B VLMs on pure-text descriptions of butterflies and Korean cultural objects, generated by GPT-4o, and compare against standard image-text fine-tuning. They report that text-only training matches or slightly beats image-text training on species-identification and cultural VQA accuracy, while cutting training time by up to a third and energy use by up to 49 percent. The reason to care: if this holds, adapting a VLM to a niche visual domain no longer requires collecting and processing paired image data, only writing good descriptions.","feed_headline":"Text-only training rivals image-text fine-tuning on visual tasks","feed_subtitle":"On butterfly species and cultural VQA, text-only fine-tuning stays competitive while cutting energy use by up to 49 percent.","key_machinery":"The machinery is text-only fine-tuning on GPT-4o-generated description documents: structured, visually rich passages covering color patterns, wing shapes, habitat, and behavior for butterflies, and appearance, materials, history, and usage for Korean cultural objects. Each description is produced by prompting GPT-4o with only the species name or keyword and a section-by-section template, then the VLM is fine-tuned on the resulting text as a normal instruction-following sample, with the vision encoder frozen and no image placeholder present. The comparison that carries the argument is between this text-only pipeline and an identical pipeline where one real image per keyword accompanies the same description, holding the model, optimizer, and evaluation fixed.","core_discovery":"Fine-tuning a VLM on richly structured text descriptions alone, with no image tokens and no vision-encoder activations during training, produces fine-grained visual understanding comparable to, and in some settings better than, fine-tuning on the same concept with a real image. On Butterfly Type 1 with Qwen2-VL-7B, text-only training raises accuracy from 75.00 to 78.00 percent, while image-text training reaches 76.50; on the K-VISCUIT total, text-only reaches 71.39 versus 71.08 for image-text. The authors interpret this as evidence that textual descriptions can carry the visual and conceptual signal needed for classification. They add an image-free probe in which accuracy still collapses when images are withheld, arguing that the gain reflects genuine visual-linguistic alignment rather than memorized question-option patterns.","pith_inferences":["Because the butterfly Type 2 questions and the training descriptions are both generated by GPT-4o, part of the measured gain could come from a distributional match between description wording and question wording; a human-written evaluation would test whether the visual grounding is genuinely general.","The same recipe could transfer to other description-rich fine-grained domains such as bird identification, mushroom classification, or architectural styles, where structured text already exists, though the paper only demonstrates butterflies and Korean cultural objects.","A hybrid extension left implicit by the paper is to use a few real images plus many text descriptions, or to distill an image-rich teacher into a text-only student, potentially capturing the compute savings without relying entirely on generated text.","The dependence on GPT-4o means the approach inherits any gaps or errors in that model's world knowledge: if a description is wrong, the text-only model will confidently learn the wrong visual or ecological fact, which matters for deployment in biological and cultural heritage settings."],"forward_implications":["Text-only fine-tuning can serve as a low-cost adaptation path for fine-grained visual domains where collecting image data is hard, such as rare species or cultural heritage objects, with meaningful savings in training time and energy.","Fine-grained visual recognition during fine-tuning does not require image exposure: a pretrained vision encoder plus well-written textual descriptions can deliver comparable classification gains.","The reported gains survive an image-free probe: both text-only and image-text models degrade sharply when images are removed, indicating the improvement is tied to image-conditioned inference rather than to superficial text patterns.","On the cultural-understanding benchmark, text-only training with Qwen2-VL-7B slightly exceeds image-text training overall, suggesting description-heavy domains may benefit especially from this approach."],"supporting_citations":[{"why":"Supplies prior evidence that decomposing visual inputs into detailed text can match end-to-end vision-language performance, motivating the text-only training hypothesis.","marker":"Berrios et al. 2023"},{"why":"Shows that language-only evaluation can predict zero-shot VLM performance, supporting the claim that language understanding can carry much of the signal.","marker":"Zohar et al. 2024"},{"why":"GPT-4o is the model that generated every training description and the butterfly Type 2 evaluation questions.","marker":"Hurst et al. 2024"},{"why":"The Butterfly & Moths 100-species dataset supplies the species list, the training images for the image-text condition, and the test images for evaluation.","marker":"Osenga 2023"},{"why":"The K-VISCUIT benchmark supplies the Korean cultural VQA evaluation and defines the cultural-understanding task domain.","marker":"Baek et al. 2024"},{"why":"Qwen2-VL-7B is one of the two base models fine-tuned in the experiments.","marker":"Wang et al. 2024"},{"why":"LLaVA-1.6 is the other base model; its lower baseline makes the relative gains of both training approaches visible.","marker":"Li et al. 2024"}],"fun_headline_variants":["Text-only training rivals image fine-tuning for VLM visual tasks","VLMs gain fine-grained vision from text alone, matching image-text training","Text-only fine-tuning matches image-text for visual understanding, cuts energy","No images, just text: VLMs still nail fine-grained visual recognition","Text-based training sharpens VLM vision without image pairs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the GPT-4o-written descriptions faithfully capture the distinguishing visual and biological details, and that the GPT-4o-written evaluation questions do not reward a mere style match with those descriptions.","fun_headline_variants_meta":{"raw":{"variants":["Text-only training rivals image fine-tuning for VLM visual tasks","VLMs gain fine-grained vision from text alone, matching image-text training","Text-only fine-tuning matches image-text for visual understanding, cuts energy","No images, just text: VLMs still nail fine-grained visual recognition","Text-based training sharpens VLM vision without image pairs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000284,"raw_usage":{"total_tokens":1648,"prompt_tokens":894,"completion_tokens":754,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":510,"completion_tokens_details":{"reasoning_tokens":664}},"tokens_in":510,"tokens_out":754,"duration_ms":7011,"temperature":1.0,"reasoning_tokens":664,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:34:11.561991+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replace GPT-4o with a different description source, such as encyclopedic entries or human-written species guides, while keeping the same evaluation, or use a human-written butterfly question set; if text-only training no longer matches image-text training, the effect is tied to GPT-4o's phrasing rather than to rich textual description in general.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows that language-only evaluation can predict zero-shot VLM performance, supporting the claim that language understanding can carry much of the signal."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The Butterfly & Moths 100-species dataset supplies the species list, the training images for the image-text condition, and the test images for evaluation."}],"review_version":1}