{"id":"33e9f28b-e1e3-47b0-9bfd-317132177a74","arxiv_id":"2412.16158","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"HoVLE is a monolithic VLM whose holistic embedding module maps images and text into one shared space, letting a frozen LLM reach near-compositional performance.","lead":"The authors build a monolithic vision-language model, HoVLE, whose embedding module maps images and text into one shared space, so a frozen LLM processes both like text. The model outperforms prior monolithic VLMs by large margins and approaches compositional models on 17 benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Stage-III instruction tuning updates the entire LLM (Sec. 3.3), contradicting the paper's 'avoids tuning pre-trained LLMs / preserves language' claim; no language-only scores are reported, so the language-preservation advantage is unverified.","rationale":"The reader's weakest_assumption (teacher compatibility) is plausible but not the most load-bearing: even if the image and text teacher spaces are initially incompatible, the alignment stage could absorb the mismatch, so the central architectural claim would survive and only the 'unpaired distillation' story would be weakened. Evaluation contamination is also real: several benchmarks in Table 2 (TextVQA, ChartQA, InfoVQA, DocVQA, GQA, ScienceQA, AI2D) appear in the instruction-tuning data listed in Table 11. Those specific scores are suspect. However, the Table 1 general benchmarks (MMB, MME, MMMU, MM-Vet, POPE, SEED, MathVista, HallB, CCB) are not in the training list and already show large margins over EVE and Emu3, so the headline performance claim does not collapse. By contrast, the final instruction-tuning stage's explicit 'whole model' optimization directly contradicts the no-LLM-tuning premise that motivates the work and that the reader incorporated into the strongest claim. This is an internal inconsistency, not a disagreement with current consensus. It can be settled by a language-only evaluation and/or release of a stage-2 checkpoint. I therefore keep the reader's CONDITIONAL verdict: the core performance result may be reproducible, but a stated central benefit is unsupported as written.","tokens_in":23103,"tokens_out":9215,"duration_ms":82197,"concrete_test":"Run a standard language-only evaluation (e.g., MMLU and C-Eval, 5-shot) on the released HoVLE and on InternLM2-1.8B under identical decoding settings. If HoVLE trails the base LLM by more than a small tolerance, the language-preservation claim is falsified. Additionally, if the authors release the stage-2 alignment-only checkpoint, evaluate it on the same suite and compare with the final checkpoint to separate the effect of instruction tuning on language ability.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.3 states that in the instruction-tuning stage 'the whole model is optimized', and Sec. 4.1 confirms a learning rate of 4e-5 is applied to the full model; the ablation discussion explicitly credits 'the LLM has more trainable parameters'. Thus θLLM is not frozen in the final stage. This matters because the paper's motivation is that monolithic VLMs that 'require tuning pre-trained LLMs ... may degrade their language capabilities', and the claimed advantage of HoVLE is to 'avoid tuning pre-trained LLMs' while 'preserving their language proficiency'. The only frozen-LLM optimization is the alignment stage described by Eq. 6. By the time the released checkpoint is produced, InternLM2-1.8B has been further trained on roughly 5M visual instruction samples. No language-only benchmark (e.g., MMLU, C-Eval, HellaSwag) is reported anywhere, so the 'no language degradation' benefit is neither demonstrated nor isolated. The reader's strongest claim, 'while keeping the pre-trained LLM frozen', is not true of the submitted method as described. At minimum the claim must be weakened to 'the LLM is frozen during distillation and alignment, but is later instruction-tuned like standard compositional VLMs', which removes the stated differentiator over EVE and SOLO.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes HoVLE, a monolithic vision-language model built from a frozen pre-trained LLM and a new 'holistic embedding module' that projects both images and text into a shared embedding space. The module is trained in three stages: (1) distillation of visual features from a pre-trained vision encoder and text embeddings from the LLM on unpaired random images and random text tokens; (2) alignment of the embeddings via next-token prediction with a frozen LLM; and (3) instruction tuning of the entire model. The authors report large improvements over previous monolithic VLMs and performance close to compositional VLMs such as InternVL2 at the 2.6B scale, and they release the model on Hugging Face.","tokens_in":23306,"tokens_out":4355,"duration_ms":38821,"significance":"If the central claims hold, the paper is significant for the monolithic VLM line of research: it shows that a monolithic architecture without a vision encoder at inference can approach the accuracy of compositional VLMs, and the unpaired-data distillation idea is conceptually interesting. The contribution includes a released model, extensive evaluation on 17 benchmarks, ablations, and a data-scaling analysis. However, the significance is tempered by three unresolved issues: the final model is not actually trained with a frozen LLM, despite the paper's central claim; no language-only metrics are provided to support the language-preservation claim; and several evaluation benchmarks appear in the training mixture without a decontamination statement.","major_comments":[{"comment":"The abstract and introduction state that HoVLE 'avoids tuning pre-trained LLMs' and 'preserves their language proficiency', but Sec. 3.3 explicitly says 'the whole model is optimized' during instruction tuning, and Sec. 4.1 confirms a learning rate of 4e-5 is applied to the full model. The released checkpoint thus includes additional training of InternLM2-1.8B on roughly 5M visual instruction samples. The claimed differentiator over EVE and SOLO, which also tune their LLMs, is therefore not supported by the experimental setup as described. Please either weaken the claim to 'the LLM is frozen during distillation and alignment but is later instruction-tuned', or provide results for a version with a truly frozen LLM.","section":"Abstract and Sec. 3.3"},{"comment":"No language-only benchmark scores (e.g., MMLU, C-Eval, HellaSwag, GSM8K) are reported anywhere, despite the paper's motivation that monolithic VLMs 'may degrade their language capabilities.' Because the final model updates the LLM on visual instruction data, the reader cannot verify that HoVLE preserves language proficiency. Please report language-only accuracies for the base InternLM2-1.8B, the model after the alignment stage (frozen LLM), and the final HoVLE, to isolate the effect of the instruction-tuning stage.","section":"Sec. 4.1 / Evaluation"},{"comment":"Several of the evaluation benchmarks (GQA, ScienceQA, ChartQA, TextVQA, InfoVQA, DocVQA, and others) appear in the instruction-tuning data mixture listed in Table 11, yet the paper does not state whether any decontamination was performed (e.g., removal of test or validation splits). If the training data includes the same examples used for evaluation, the reported numbers are not comparable to those of models that did not train on these benchmarks. Please clarify the exact splits used and add a formal decontamination statement.","section":"Sec. 4.1 and Table 11"},{"comment":"The distillation loss is defined as negative cosine similarity between the holistic embedding outputs x̂I, x̂T and the teacher features zI (from InternViT-300M) and zT (from InternLM2's embedding layer). The dimensionalities of these features are not specified; InternViT and InternLM2 embeddings may have different dimensions from each other and from the holistic embedding's output dimension c=2048. Cosine similarity requires equal dimensions, and if projection layers are used they are not described. This is load-bearing because Table 4 shows the distillation stage is critical, and the reader needs to know how the features are matched.","section":"Sec. 3.2, Eq. (5)"}],"minor_comments":[{"comment":"The checkmark layout in Table 4 is ambiguous; it is unclear which rows include Distillation, Alignment, and Instruction Tuning. Please restate the configurations explicitly in the table or text.","section":"Table 4"},{"comment":"The attention-map analysis is qualitative. A quantitative sparsity measure (e.g., concentration or entropy of text-to-image attention) would make the claim more rigorous.","section":"Sec. 4.4 / Fig. 5"},{"comment":"The paper defines monolithic VLMs as those without modality-specific encoders, but the distillation stage uses a pre-trained vision encoder as teacher. Clarify that this is only a training-time teacher and not part of the inference architecture.","section":"Sec. 2"},{"comment":"The phrase 'random text tokens' is accurate, but the text says '100 random text indices' and later 'random texts'; use consistent terminology to avoid confusion with natural-language sentences.","section":"Sec. 3.2"},{"comment":"The limitations section only mentions computational scale; it should also acknowledge the absence of language-preservation validation and the potential issue of benchmark contamination.","section":"Limitations"}],"recommendation":"major_revision","confidential_remarks":"The paper is technically interesting and the model release is a plus, but the mismatch between the 'frozen LLM' claim and the actual training procedure, the absence of language-only metrics, and the lack of a decontamination statement are all load-bearing for the paper's central claims. I would encourage the editor to send the revision back to the authors rather than rejecting, as the core architecture and training idea are sound enough to warrant a careful revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"One thing to know: this paper delivers a real, recipe-level advance—a monolithic VLM that gets within striking distance of compositional models at 2B scale by distilling a shared causal-Transformer embedding module from a vision encoder and the LLM's own embedding layer, using unpaired random images and text. That is genuinely new relative to EVE, SOLO, Chameleon, and Emu3, and the gains over those baselines are large and internally consistent. The ablation showing distillation as the most important stage, and the distillation-data scaling curves, add credibility.\n\nWhat it does well: the training pipeline is clearly described, the controlled comparison against InternVL2 (same LLM, same supervised data) is the right way to isolate the architectural contribution, and the speed-performance tradeoff table is a useful practical addition. The attention-map analysis is suggestive rather than conclusive, but it is presented as a hypothesis, which is fine.\n\nThe soft spots, in order of severity. First, the paper's central framing—'avoids tuning pre-trained LLMs' and 'preserves language proficiency'—is contradicted by its own Section 3.3 and Section 4.1: at instruction tuning, 'the whole model is optimized' at lr 4e-5, which updates InternLM2's weights on ~5M visual instruction samples. The LLM is frozen only during distillation and alignment. That is still a meaningful difference from EVE and SOLO (which continual-pretrain the LLM), but the stated differentiator is overstated. Second, no language-only benchmark is reported anywhere, so the 'no language degradation' claim is not demonstrated. This is addressable: run MMLU or C-Eval on the base LLM and on HoVLE. Third, several evaluation benchmarks (GQA, TextVQA, ChartQA, DocVQA, ScienceQA, AI2D, and others) appear in the instruction-tuning mixture in Table 11, with no decontamination statement. Given that supervised training on these exact test sets is standard practice in this literature, a contamination check is needed before the benchmark numbers can be taken at face value. Fourth, there are no error bars or multiple seeds; for a paper whose claims are mostly benchmark deltas, that is a minor but real weakness.\n\nThe distillation teacher-compatibility assumption—that InternViT features and InternLM2 text embeddings can be pulled into one space by negative-cosine loss—is not analyzed directly, but the ablation table shows removing distillation sharply degrades performance, so empirically the step is load-bearing and it works. I would not call that a flaw, just an open mechanistic question.\n\nBottom line: this deserves a serious referee. The core result—a monolithic VLM at 2B scale competitive with compositional peers—appears true and important for the subfield. The paper needs revision on the language-preservation claim and the contamination statement, but the engineering is sound. I would send it to review.","headline":"Solid recipe-level advance showing monolithic VLMs can approach compositional ones at 2B scale, but the 'no LLM tuning / language preservation' headline is contradicted by the instruction-tuning stage and unbacked by any language-only scores.","tokens_in":23905,"tokens_out":3407,"would_cite":true,"duration_ms":26321,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A shared embedding module lets a monolithic vision-language model match compositional rivals while keeping the language model frozen.","keywords":["monolithic vision-language model","holistic embedding","shared embedding space","knowledge distillation","unpaired training data","frozen LLM","instruction tuning","multimodal understanding"],"falsifier":"Train the identical pipeline but replace the vision-teacher features with random fixed vectors while keeping everything else the same; if the model still reaches the reported MMBench and OCR scores, then distillation from a real vision encoder is not doing the claimed work, whereas a collapse toward the no-distillation ablation level, roughly 35 points lower on MMBench, would confirm the mechanism. A complementary check is to run alignment without any distillation and measure the same gap directly.","tokens_in":22861,"feed_emoji":"🧩","tokens_out":5666,"duration_ms":49803,"temperature":0.7,"pith_summary":"This paper argues that the performance gap between monolithic vision-language models (VLMs) and compositional ones does not come from needing to retrain the language model; it comes from lacking an embedding module that handles both images and text. HoVLE introduces such a holistic embedding module, built from the same causal Transformer layers as the frozen LLM, and trains it in three stages: distillation from a pre-trained vision encoder and the LLM's own text embeddings using unpaired random images and random text tokens, next-token alignment on multimodal data, and instruction tuning. The claim is that a monolithic model with this module can reach scores close to leading compositional models while preserving language ability, and the experiments report it beating earlier monolithic models by a large margin, about 15 points on MMBench. If correct, this points to simpler, unified multimodal architectures that reuse pre-trained LLMs without retraining them.","feed_headline":"Monolithic VLM rivals compositional ones via one shared embedding","feed_subtitle":"It keeps the language model frozen and still beats earlier encoder-free models by about 15 points on MMBench.","key_machinery":"The holistic embedding module is a stack of causal Transformer layers with the same architecture as the LLM (8 layers, hidden size 2048, 16 heads, in the reported configuration), preceded by a patch embedding for images and the LLM's own word-embedding layer for text. It plays the role that a vision encoder plus connector plays in compositional models: it projects both modalities into one space that the frozen LLM can consume. The load-bearing training device is the distillation objective, negative cosine similarity between the module's image and text outputs and teacher targets from a pre-trained vision encoder and the LLM's embedding layer. Because these targets are computed independently, the stage runs on unpaired random images and random text tokens, which the paper finds works better than using paired captions.","core_discovery":"The central discovery is that a shared, modality-agnostic embedding stack can replace the separate vision encoder and connector of a compositional VLM without hurting accuracy. Image patches and text tokens are both passed through the same causal Transformer layers, whose outputs feed directly into a frozen LLM as if they were text embeddings. The paper shows that this module can be taught by distillation to reproduce, in a common space, the image features of a pre-trained vision encoder and the text embeddings of the LLM, using unpaired random images and random vocabulary tokens; the alignment and instruction-tuning stages then finish the job. On 17 benchmarks, HoVLE (HD) scores 73.3 on MMBench and 49.2 on MathVista, close to leading compositional models of similar size and far above previous monolithic models, supporting the paper's claim that the missing piece in monolithic VLMs was the holistic embedding, not the LLM tuning.","pith_inferences":["The same unpaired-distillation recipe might extend to other modalities such as audio or video, since the trick only requires fixed teacher targets that can be computed without paired data.","A direct test of the compatibility assumption would be to swap the vision teacher for a weaker or randomly initialized encoder and measure how much downstream performance drops; the paper's ablation shows distillation is critical but does not isolate which teacher property matters.","The reported attention sparsity suggests a measurable diagnostic: text-to-image attention density in early LLM layers could predict whether a monolithic VLM has effectively offloaded visual feature extraction to its embedding module.","Scaling beyond the 2B-parameter frozen LLM is the untested corollary; if the holistic embedding transfers, the monolithic approach should improve roughly in step with the language model's own capabilities."],"forward_implications":["A monolithic VLM can match compositional VLMs without continual pre-training of the LLM, so language knowledge in the pre-trained model is preserved rather than overwritten.","Distillation data for the embedding module does not need image-text pairs; unpaired images and random text tokens suffice, decoupling the first training stage from costly paired-data collection.","The holistic embedding transfers to higher input resolutions, with HoVLE (HD) improving on visual question answering benchmarks as tile resolution rises and paying only a predictable speed cost.","If the claim holds, encoder-free VLM designs no longer carry an inherent performance penalty, making unified generation-and-recognition architectures more viable."],"supporting_citations":[{"why":"Supplies the dynamic high-resolution image tiling strategy and the pre-trained vision encoder used as the image-teacher during distillation.","marker":"[21]"},{"why":"Supplies the frozen LLM backbone and the text-embedding layer used as the text-teacher during distillation.","marker":"[11]"},{"why":"Provides the 500M unpaired random images used for large-scale distillation training.","marker":"[94]"},{"why":"Serves as the strongest previous monolithic baseline that the paper compares against and aims to surpass.","marker":"[26]"},{"why":"Acts as the compositional counterpart with the same LLM and supervised data, anchoring the claim that HoVLE narrows the gap to compositional models.","marker":"[20]"},{"why":"Supports the design choice of using 1D causal attention for image tokens inside the holistic embedding module.","marker":"[104]"},{"why":"Provides a monolithic VLM baseline that motivates the paper's focus on avoiding LLM retraining.","marker":"[105]"},{"why":"Provides another monolithic VLM baseline that the paper exceeds despite using fewer parameters.","marker":"[115]"}],"fun_headline_variants":["Shared embedding makes monolithic VLM competitive","Frozen LLM reads images via one shared embedding","Holistic embedding narrows monolithic VLM gap","One embedding module lifts monolithic VLM"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole method assumes that a pre-trained vision encoder's image features and a frozen LLM's text embeddings can pull a shared Transformer stack into one space, so that negative-cosine distillation on unpaired random inputs gives the LLM inputs it can actually consume; if those teacher spaces are inherently incompatible, the alignment stage would have to do all the cross-modal work and the claimed benefit of unpaired distillation would disappear.","fun_headline_variants_meta":{"raw":{"variants":["Shared embedding makes monolithic VLM competitive","Frozen LLM reads images via one shared embedding","Holistic embedding narrows monolithic VLM gap","One embedding module lifts monolithic VLM"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.002032,"raw_usage":{"total_tokens":7952,"prompt_tokens":1012,"completion_tokens":6940,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":628,"completion_tokens_details":{"reasoning_tokens":6884}},"tokens_in":628,"tokens_out":6940,"duration_ms":41306,"temperature":1.0,"reasoning_tokens":6884,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T10:44:23.911031+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the identical pipeline but replace the vision-teacher features with random fixed vectors while keeping everything else the same; if the model still reaches the reported MMBench and OCR scores, then distillation from a real vision encoder is not doing the claimed work, whereas a collapse toward the no-distillation ablation level, roughly 35 points lower on MMBench, would confirm the mechanism. A complementary check is to run alignment without any distillation and measure the same gap directly.","supporting_citations":[{"cited_title":"Laion-5b: An open large-scale dataset for train- ing next generation image-text models","cited_arxiv_id":null,"evidence_quote":"Provides the 500M unpaired random images used for large-scale distillation training."},{"cited_title":"Learning 1D Causal Visual Representation with De-focus Attention Networks","cited_arxiv_id":"2406.04342","evidence_quote":"Supports the design choice of using 1D causal attention for image tokens inside the holistic embedding module."}],"review_version":1}