{"id":"471e7a01-afa2-4661-893e-f575cfbca637","arxiv_id":"2504.20996","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"X-Fusion adds a trainable vision tower to a frozen LLM, enabling both image captioning and text-to-image generation while keeping the LLM's language ability intact.","lead":"This paper introduces X-Fusion, a way to add image understanding and generation to a frozen large language model by training a separate vision tower alongside it. It reports that this design beats several alternative architectures and that training choices, like showing clean images and mixing in captioning data, matter more than extra alignment losses.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claim that X-Fusion 'consistently outperforms alternative architectures' is not yet supported: Table 1 has no error bars and the paper's own Table A shows X-Fusion FID 11.5 trailing Transfusion 6.7, Janus 8.5, and Show-O 9.2.","rationale":"The reader's designated weakest assumption was distributional drift between the trainable vision tower and the frozen text tower. That is a real concern in principle, but the training objective partially addresses it: the image-to-text loss backpropagates through the frozen text tower into the vision tower, so the vision features are explicitly trained to remain interpretable by the frozen layers, and the initialization of the vision tower from the language tower gives a matched starting distribution. The more load-bearing weakness is evidentiary: the central claim of consistent superiority over alternative architectures rests on a single small-scale comparison without error bars, and the paper's own state-of-the-art table contradicts the unqualified wording on the generation axis. This does not invalidate the dual-tower idea, which remains plausible and well-motivated, but it means the paper's headline empirical claim is broader than what the reported experiments establish. The reader already set a conditional verdict and explicitly noted missing error bars and the abstract overclaim, so my critique reinforces that verdict rather than changing it. I therefore recommend UNCHANGED: the paper should be accepted only if the authors add variance estimates, clarify the scope of the comparison, and either release code/data or rerun the 8B evaluation against the same protocol as the cited unified models.","tokens_in":19820,"tokens_out":8428,"duration_ms":91554,"concrete_test":"Rerun the 1B architecture comparison from Table 1 with at least three random seeds and report mean and standard deviation for FID and BLIP, using identical evaluation code for every variant. If the BLIP margin between Dual Tower and Dual Projection falls within one standard deviation, the 'consistently outperforms on image-to-text' half of the claim is not established. Since the 8B FID in Table A already trails Transfusion, Janus, and Show-O, the abstract's unqualified superiority claim should in any case be narrowed to 'outperforms our ablation baselines' unless the 8B evaluation is rerun on the same prompts and FID protocol and improves accordingly.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing sentence is the abstract's 'X-Fusion consistently outperforms alternative architectures on both image-to-text and text-to-image tasks.' The only head-to-head evidence for this is Table 1, a single 1B run with no seeds, no error bars, and in-house data. On the understanding axis the margin is thin: Dual Tower BLIP 31.3 vs. Dual Projection 30.9 and Single Tower 30.2; a 0.4-point margin on a metric the authors chose because other metrics failed to separate caption quality is not demonstrable without variance estimates. On the generation axis the FID gap (14.20 vs. 19.10/20.22) is large, but the paper's own Table A undercuts the unqualified wording: X-Fusion 8B FID is 11.5, while Transfusion reports 6.7, Janus 8.5, and Show-O 9.2, and BLIP is 80.0 vs. InternVL 81.1. If 'alternative architectures' means the three ablation blocks, the evidence base is one small-scale comparison; if it means leading unified models, the claim is contradicted by the paper's own table. Language preservation at MMLU 32.2 is guaranteed by construction (the text tower is frozen and text-only prompts never touch the vision tower), so the nontrivial part of the contribution rests on visual-task metrics, where the evidence is weakest. The paper itself limits the claim in Section D: image quality has 'room for improvement' and vanilla X-Fusion doubles parameters, reducing training efficiency compared to Transfusion.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes X-Fusion, a framework that extends a frozen pretrained LLM with a trainable parallel vision tower for both image understanding and image generation. In each transformer layer, text tokens are processed by the frozen text block and image tokens by a trainable vision block, with outputs routed by token modality; an optional X-Fuse operation merges features across towers. The paper compares Dual Tower against Single Tower, Gated Tower, and Dual Projection variants on a 1B LLaMA-3.2 model (Table 1), ablates the effect of image noise in image-to-text samples (Fig. 6), the data ratio between text-to-image and image-to-text tasks (Fig. 5), and CLIP feature alignment at 1B/3B/8B scale (Fig. 7). It also demonstrates extensions: X-Fuse, initialization from a pretrained DiT, and fine-tuning for editing/VQA/localization. The central claims are that X-Fusion preserves language capability (MMLU at 32.2) while outperforming alternative architectures on both image-to-text and text-to-image tasks, and that its data ablations yield transferable training insights.","tokens_in":20192,"tokens_out":5035,"duration_ms":55186,"significance":"If the empirical claims are robust, X-Fusion is a practically useful recipe for adding modalities to frozen LLMs: the per-layer dual-tower design with token routing is clean, the idea of keeping the language tower untouched is appealing, and the ablations on clean images for understanding data and on asymmetric task synergy are potentially valuable for the multimodal training community. The paper is honest in places: it includes a supplementary comparison with other unified models (Table A) and a limitations section (Sec. D) acknowledging hallucinations, parameter overhead, and room for improvement in image quality. However, the paper's headline claim of outperforming alternative architectures is considerably broader than the evidence: all results come from single runs with no variance estimates, the understanding-task margin in Table 1 is thin, and the paper's own Table A shows X-Fusion trailing several state-of-the-art models on FID. The contribution is therefore best characterized as a promising architecture plus a useful ablation study, not as an established state-of-the-art result.","major_comments":[{"comment":"The abstract's claim that 'X-Fusion consistently outperforms alternative architectures on both image-to-text and text-to-image tasks' is not supported at the level claimed. Every number in Table 1 comes from a single training run with no multiple seeds, no standard deviations, and no significance test. On the understanding axis the margin is particularly thin: Dual Tower reaches BLIP 31.3 versus 30.9 for Dual Projection and 30.2 for Single Tower, so the 0.4-point advantage is not demonstrable without variance estimates. The FID gap (14.20 vs 19.10/20.22) is large, but it is still a single-run observation. Please report results over at least three seeds with mean and standard deviation, or, if that is impractical, explicitly qualify all comparative statements as single-run observations.","section":"Abstract and Table 1"},{"comment":"The scope of 'alternative architectures' is ambiguous. If it means only the three ablation blocks in Table 1, the claim should say so explicitly. If it means leading unified multimodal models, the paper's own supplementary Table A contradicts the unqualified wording: X-Fusion-8B reaches COCO FID 11.5, whereas Transfusion reports 6.7, Janus 8.5, and Show-O 9.2, and BLIP 80.0 is below InternVL's 81.1. The main text should define which comparison class the headline claim refers to, and the abstract should be revised to avoid the impression that X-Fusion outperforms state-of-the-art unified models, since the supplementary table itself does not support that reading.","section":"Table A (Supplementary) vs Abstract"},{"comment":"The validity of all visual-task metrics depends on a clean separation between the in-house licensed training data and the MS-COCO evaluation set, but the paper never states whether the training set overlaps with MS-COCO. Since the in-house data is described only as 'licensed' and COCO is a standard public dataset, the possibility of training/evaluation contamination must be addressed; if any COCO images or captions were used during training, the reported FID and BLIP numbers are uninterpretable. Please state explicitly whether the training data is disjoint from the evaluation set, and also report how many images are used for the 30K-image COCO evaluation and how the BLIP2-ITM score is averaged over such a sample without confidence intervals.","section":"Sec. 4, Data and Task paragraphs"},{"comment":"The paper frames the preservation of MMLU at 32.2 as a key advantage, but this preservation is guaranteed by construction: the text tower is frozen and pure text inputs never pass through the vision tower, so the text-only forward pass is identical to the original LLM. The nontrivial burden rests on showing that the frozen later text layers can interpret the vision tower's outputs in the interleaved setting, yet the paper does not analyze this distributional assumption. The only supporting evidence is in-house captioning quality and the linear-probe experiment in Fig. 8, which probes the vision tower features, not the frozen text tower's ability to condition on them. Please either add an analysis of the feature distribution seen by frozen text layers (e.g., linear probing of the text tower on visual input, or a check of attention statistics on mixed sequences) or soften the claims framed around 'preserving language capabilities,' since the architecture's real risk is not language forgetting but visual-feature interpretability.","section":"Sec. 4, Dual Tower and Sec. 5"}],"minor_comments":[{"comment":"The text says 'Dual Tower and Dual Projection share a common insight: modality-specific operations,' but the comparison would be easier to interpret if the table also reported the number of trainable parameters for each variant; the current text mentions the same 'number of training parameters' for Single Tower and Dual Tower only in passing.","section":"Sec. 5 and Table 1"},{"comment":"The reference to 'Figure 12 shows that this operation further enhances the model's capability' appears to be a figure-numbering error: Figure 12 is the X-Fuse ablation, whereas the pretrained-DiT transfer results are shown in Figure 9. Please correct the cross-reference.","section":"Sec. 8.2"},{"comment":"The Fig. 5 caption states '2.4 times acceleration,' but the definition of this acceleration is not given anywhere. Specify how the acceleration factor is computed and which curves it compares.","section":"Sec. 6.1 and Fig. 5"},{"comment":"The default loss weights are given as lambda_AR = 0.2 and lambda_DM = 1, but Sec. 7 states 'We use lambda_AR = 0.5 in this ablation study.' Please clarify which configuration is used for the main experiments and whether different lambda values across the REPA ablation affect the comparability of the 1B/3B/8B curves in Fig. 7.","section":"Sec. 4, Training paragraph"},{"comment":"Several typos remain: 'effiecient' in the conclusion, 'pretraind' in the Sec. 8.2 title, 'texture features' should presumably be 'text features' in Sec. 4, and stray '/edt' tokens appear in Sec. 6.1 and Sec. 6.2. These do not affect the technical content but should be cleaned up.","section":"Throughout"},{"comment":"References [44] and [62] are the same paper (Rombach et al., High-resolution image synthesis with latent diffusion models) cited twice with different venues; please merge them.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper has a useful architecture and a solid ablation suite, but the abstract and Sec. 5 overstate the evidence. I would urge the editor to require the de-overclaiming described in major comments 1 and 2, plus an explicit statement about train/eval data overlap. If the authors can add multi-seed variance estimates for the core comparison, the paper could become a strong contribution; without them, the headline claim is not journal-grade. The scope issue between 'internal architecture baselines' and 'state-of-the-art unified models' should be resolved in favor of the narrower interpretation, unless the authors are prepared to add direct comparisons under matched training budgets."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a serious look. The core idea is clean: freeze the LLM, add a parallel trainable vision tower, route tokens by modality per layer, and train with both AR text loss and diffusion image loss. That design is genuinely distinct from LMFusion's joint attention and from Playground-v3's generation-only setup. The paper's real value, though, is the data ablations. The finding that clean images for I2T samples improve both understanding and generation, and that adding I2T data helps T2I while the reverse does not hold, are concrete and plausible. Those are the kinds of results that end up influencing training recipes even after the architecture is superseded.\n\nThe main soft spot is exactly what the stress-test note flags: the abstract's claim of 'consistently outperforms alternative architectures' is broader than the evidence. Table 1 supports the dual-tower choice over the three ablation baselines, and the FID gap there is large. But on the understanding axis, a 0.4-point BLIP margin over Dual Projection with no error bars is not compelling. And the paper's own Table A shows X-Fusion 8B at FID 11.5 against Transfusion 6.7, Janus 8.5, and Show-O 9.2. So 'alternative architectures' needs qualification: it means the paper's own baselines, not the SOTA unified models. Also, the MMLU preservation at 32.2 is essentially guaranteed by construction for text-only prompts, so that part of the headline shouldn't be oversold. The missing seeds and in-house data are addressable but real. Minor editorial artifacts (the stray '/edt' markers) don't affect the science.\n\nOne point where I'd push back on the skeptic: the paper is honest about its limits. Section D explicitly says image quality has room for improvement and that vanilla X-Fusion doubles parameters. The claim is not that this beats Transfusion on efficiency; it is that you can bolt vision onto a frozen LLM without touching language weights, and the ablations support that. The dual-tower mechanism — frozen text blocks and trainable vision blocks exchanging hidden states — is plausible and the paper validates it at three scales. The linear probing and data-ratio figures are internally consistent.\n\nWho is this for? Anyone working on unified multimodal models, especially on training data strategies or parameter-efficient extension of frozen LLMs. It deserves a serious referee, with requests for error bars, a few seeds, and a toned-down abstract. I'd give it a conditional accept if the revision addresses those.\n\nRecommendation: engage with it. Send it out.","headline":"A useful, well-ablated architecture paper whose central claim is overstated in the abstract but whose data-centric findings hold up on inspection; worth refereeing, not desk-rejecting.","tokens_in":20777,"tokens_out":652,"would_cite":true,"duration_ms":8870,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A frozen pretrained LLM gains image understanding and generation from a parallel trainable vision tower, without losing its language knowledge.","keywords":["frozen large language models","multimodal understanding and generation","dual-tower architecture","image-to-text","text-to-image","per-layer token routing","diffusion loss","catastrophic forgetting"],"falsifier":"Train the dual-tower model with the vision tower initialized randomly instead of copied from the language layers, holding all data and training budget fixed; if the FID and BLIP gains over the single-tower baseline shrink or MMLU preservation breaks, the reported advantage depends on the initialization choice rather than on per-token routing.","tokens_in":1540,"feed_emoji":"🖼️","tokens_out":1533,"duration_ms":52534,"temperature":0.7,"pith_summary":"This paper tries to establish that a pretrained, frozen large language model can learn both to understand and to generate images while keeping its language abilities intact, by adding a parallel trainable vision tower. The key move is per-layer routing: at each transformer layer, text tokens go through the frozen text block and image tokens through a trainable vision block, then the streams are recombined before the next frozen layer. On LLaMA-3.2-1B, the authors report that this dual-tower design beats single-tower, gated, and dual-projection alternatives on image quality (FID 14.20 vs 19.10 for single tower) and captioning (BLIP 31.3 vs 30.2), while preserving MMLU at 32.2 instead of dropping to chance-level 25.0. They further claim that training with clean images for understanding tasks and a 2:1 generation-to-understanding data ratio improves both tasks, and that aligning with pretrained vision features helps small models but not large ones.","feed_headline":"Frozen LLMs can learn to see and draw with a second tower","feed_subtitle":"Parallel trainable vision layers add image understanding and generation without touching the language weights.","key_machinery":"The central object is the dual-tower transformer block. At each layer, the same input sequence runs through a frozen text block and a trainable vision block initialized by copying the language layer; the output sequence takes text-block outputs at text positions and vision-block outputs at image positions, so vision information is injected into the frozen language stream at every intermediate layer rather than only at the input. An optional X-Fuse operation merges features from both towers with learnable scalars, trading extra FLOPs for improved performance.","core_discovery":"X-Fusion's central claim is that modality-specific weights inside each block - a frozen language transformer block and a copied-then-trained vision transformer block - let a frozen LLM process interleaved image and text tokens in both directions. For each token, the output is taken from the block that matches its modality, so image tokens are computed by the vision tower and text tokens by the language tower; the next frozen language layer then sees text features produced by frozen weights and image features produced by trainable ones. The paper argues this beats the natural alternatives: single tower fine-tunes the LLM and sacrifices language knowledge, gated tower is too weak because it only adds a gated residual, and dual projection, which duplicates attention QKV and MLP but computes joint attention, cannot generate fresh text key/value representations for the image tokens. The reported consequence is that a frozen LLM gains image generation and understanding at the same FLOPs as these alternatives while its MMLU score stays exactly at the pretrained level.","pith_inferences":["A direct test of the routing mechanism would be to freeze the vision tower after initialization; if generation quality stays high, the per-layer routing is doing less work than claimed.","If the dual-tower recipe transfers, a similar frozen-model design could add audio or video tokens by placing a third tower alongside the text tower and applying the same per-token selection rule.","The reported asymmetry - understanding data helps generation but not vice versa - suggests an ordering heuristic for training schedules: spend early budget on clean understanding samples, then add generation pairs.","The paper's 8B comparison against other unified models uses different training budgets, so a matched-budget head-to-head with the closest transformer-level baselines would sharpen the claim."],"forward_implications":["Frozen LLMs can adopt a new output modality without degrading MMLU, so language knowledge can be preserved while adding image generation.","Understanding-focused data improves generation quality, so a skewed 2:1 generation-to-understanding data ratio should be preferred over a balanced mix.","Clean images in image-to-text samples improve both understanding and generation, challenging the practice of applying diffusion noise to understanding samples.","Feature alignment with pretrained vision representations accelerates convergence for small models but can slightly hurt large models, so its value shrinks with scale.","The dual-tower design reaches the same attention FLOPs as single-tower, gated, and dual-projection alternatives while giving the vision tower its own attention and MLP weights."],"supporting_citations":[{"why":"Supplies the hybrid training recipe that combines autoregressive language loss with diffusion image loss and the noisy-image baseline the paper improves upon.","marker":"[28]"},{"why":"Provides the LLaMA-3 family used as the frozen text tower and the source of the MMLU language-capability baseline.","marker":"[17]"},{"why":"Generates the detailed InternVL captions used as paired training data for both image-to-text and text-to-image tasks.","marker":"[35]"},{"why":"Provides the pretrained Stable Diffusion VAE used as the visual encoder producing latent image tokens.","marker":"[44]"},{"why":"Supplies the flow-matching diffusion scheduler used for image generation training.","marker":"[64]"},{"why":"Provides the MS-COCO prompts and images used to evaluate generation and captioning performance.","marker":"[66]"},{"why":"Defines the FID metric used to measure image generation quality.","marker":"[67]"},{"why":"Defines the BLIP2-ITM metric used to measure caption quality against detailed generated captions.","marker":"[68]"},{"why":"Provides the representation-alignment loss used to regularize vision features with a pretrained CLIP encoder.","marker":"[73]"},{"why":"Inspires the gated-tower baseline that multiplies a gated block output by a learnable value initialized to zero.","marker":"[71]"}],"fun_headline_variants":["Frozen LLM sees and draws via a dedicated vision tower","Add vision, keep LLM frozen: dual-tower fusion","Frozen language, fresh vision: multimodal without unfreezing","Two towers, one frozen mind: image skills for LLM"],"cache_read_input_tokens":22784,"weakest_assumption_plain":"That the frozen text layers can safely condition on features produced by the trainable vision tower, so the two towers never drift into incompatible feature distributions; the paper tests this on in-house data with InternVL captions but does not report how sensitive the result is to the vision tower's initialization or choice of image encoder.","fun_headline_variants_meta":{"raw":{"variants":["Frozen LLM sees and draws via a dedicated vision tower","Add vision, keep LLM frozen: dual-tower fusion","Frozen language, fresh vision: multimodal without unfreezing","Two towers, one frozen mind: image skills for LLM"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000698,"raw_usage":{"total_tokens":3103,"prompt_tokens":842,"completion_tokens":2261,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":458,"completion_tokens_details":{"reasoning_tokens":2190}},"tokens_in":458,"tokens_out":2261,"duration_ms":17725,"temperature":1.0,"reasoning_tokens":2190,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:14:08.894819+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the dual-tower model with the vision tower initialized randomly instead of copied from the language layers, holding all data and training budget fixed; if the FID and BLIP gains over the single-tower baseline shrink or MMLU preservation breaks, the reported advantage depends on the initialization choice rather than on per-token routing.","supporting_citations":[{"cited_title":"Transfusion: Pre- dict the next token and diffuse images with one multi-modal model","cited_arxiv_id":null,"evidence_quote":"Supplies the hybrid training recipe that combines autoregressive language loss with diffusion image loss and the noisy-image baseline the paper improves upon."},{"cited_title":"Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks","cited_arxiv_id":null,"evidence_quote":"Generates the detailed InternVL captions used as paired training data for both image-to-text and text-to-image tasks."},{"cited_title":"High-resolution image syn- thesis with latent diffusion models, 2022","cited_arxiv_id":null,"evidence_quote":"Provides the pretrained Stable Diffusion VAE used as the visual encoder producing latent image tokens."},{"cited_title":"Gans trained by a two time-scale update rule converge to a local nash equilib- rium","cited_arxiv_id":null,"evidence_quote":"Defines the FID metric used to measure image generation quality."}],"review_version":1}