Pith. sign in

REVIEW 22 cited by

LMFusion: Adapting Pretrained Language Models for Multimodal Generation

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2412.15188 v4 pith:CU7RBPWB submitted 2024-12-19 cs.CL cs.AIcs.CVcs.LG

Weijia Shi , Xiaochuang Han , Chunting Zhou , Weixin Liang , Xi Victoria Lin , Luke Zettlemoyer , Lili Yu This is my paper · ORCID
classification cs.CLcs.AIcs.CVcs.LG
keywords languagelmfusionmultimodalcapabilitiesgenerationmodelsmoduleswhile
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

We present LMFusion, a framework for empowering pretrained text-only large language models (LLMs) with multimodal generative capabilities, enabling them to understand and generate both text and images in arbitrary sequences. LMFusion leverages existing Llama-3's weights for processing texts autoregressively while introducing additional and parallel transformer modules for processing images with diffusion. During training, the data from each modality is routed to its dedicated modules: modality-specific feedforward layers, query-key-value projections, and normalization layers process each modality independently, while the shared self-attention layers allow interactions across text and image features. By freezing the text-specific modules and only training the image-specific modules, LMFusion preserves the language capabilities of text-only LLMs while developing strong visual understanding and generation abilities. Compared to methods that pretrain multimodal generative models from scratch, our experiments demonstrate that, LMFusion improves image understanding by 20% and image generation by 3.6% using only 50% of the FLOPs while maintaining Llama-3's language capabilities. We also demonstrate that this framework can adapt existing vision-language models with multimodal generation ability. Overall, this framework not only leverages existing computational investments in text-only LLMs but also enables the parallel development of language and vision capabilities, presenting a promising direction for efficient multimodal model development.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 22 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Demystifying Video Reasoning

    cs.CV 2026-03 conditional novelty 7.0 of 10

    Video diffusion models reason along the denoising trajectory (Chain-of-Steps), not primarily across frames, and this mechanism can be nudged by ensembling latent trajectories.

  2. Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipes

    cs.CV 2026-08 conditional novelty 6.0 of 10

    Multimodal pretraining transfers asymmetrically: language boosts vision, understanding boosts generation, generation is mostly neutral, and early unified training prevents vision laziness.

  3. Streaming Multi-Agent Autoregressive Diffusion Model with World State Registers

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Adding persistently updated, supervised world-state register tokens to streaming multi-agent diffusion improves cross-agent consistency and visual quality in two-agent Minecraft generation.

  4. ChatUMM: Robust Context Tracking for Conversational Interleaved Generation

    cs.CV 2026-02 conditional novelty 6.0 of 10

    A unified multimodal model trained on synthetic multi-turn dialogues that can interleave image generation with text across a conversation.

  5. UniVideo: Unified Understanding, Generation, and Editing for Videos

    cs.CV 2025-10 conditional novelty 6.0 of 10

    UniVideo combines a frozen MLLM and a video DiT to unify video understanding, generation, in-context editing, visual prompting, and zero-shot free-form video edits under one instruction interface.

  6. Reconstruction Alignment Improves Unified Multimodal Models

    cs.CV 2025-09 conditional novelty 6.0 of 10

    RECA, a self-supervised post-training objective that conditions unified multimodal models on their own visual understanding embeddings to reconstruct input images, improves text-to-image and editing benchmarks across ...

  7. Interleaving Reasoning for Better Text-to-Image Generation

    cs.CV 2025-09 conditional novelty 6.0 of 10

    A text-image-text-image multi-turn pipeline improves text-to-image generation, gaining up to 8 points over the base model on several benchmarks while adding an image-conditioned reflection step.

  8. Galaxea Open-World Dataset and G0 Dual-System VLA Model

    cs.RO 2025-08 conditional novelty 6.0 of 10

    A new open-world mobile manipulation dataset and a dual-system VLA model show that single-embodiment pre-training, not cross-embodiment pre-training, drives strong downstream task performance.

  9. X-Omni: Reinforcement Learning Makes Discrete Autoregressive Image Generative Models Great Again

    cs.CV 2025-07 conditional novelty 6.0 of 10

    GRPO reinforcement learning applied to a discrete autoregressive image generator with a diffusion decoder improves instruction following, image quality, and long-text rendering in a unified multimodal model.

  10. FreeLoRA: Enabling Training-Free LoRA Fusion for Autoregressive Multi-Subject Personalization

    cs.CV 2025-07 conditional novelty 6.0 of 10

    A method for multi-subject image personalization that fuses independently trained LoRA modules at inference time on visual autoregressive models.

  11. Rethinking Discrete Tokens: Treating Them as Conditions for Continuous Autoregressive Image Synthesis

    cs.CV 2025-07 conditional novelty 6.0 of 10

    DisCon treats discrete image tokens as conditioning signals rather than targets, letting a continuous autoregressive model refine details and reach gFID 1.38 on ImageNet-256.

  12. Dreamland: Controllable World Creation with Simulator and Generative Models

    cs.CV 2025-06 conditional novelty 6.0 of 10

    A three-stage hybrid pipeline uses an intermediate layered world representation to refine simulator-rendered driving scenes into realistic, controllable images and videos.

  13. LaTtE-Flow: Layerwise Timestep-Expert Flow-based Transformer

    cs.CV 2025-06 conditional novelty 6.0 of 10

    LaTtE-Flow partitions transformer layers into timestep-specific groups for flow matching, activating only one group per sampling step to speed up image generation in unified multimodal models.

  14. I Think, Therefore I Diffuse: Enabling Multimodal In-Context Reasoning in Diffusion Models

    cs.LG 2025-02 conditional novelty 6.0 of 10

    ThinkDiff aligns vision-language model features to a T5 decoder via captioning, then injects those features into a T5-based diffusion decoder, achieving 46.3% on the CoBSAT benchmark without reasoning-specific training data.

  15. UniMoD: Efficient Unified Multimodal Transformers with Mixture-of-Depths

    cs.CV 2025-02 conditional novelty 6.0 of 10

    UniMoD prunes tokens with task-specific routers in unified multimodal transformers, cutting training FLOPs by 15-40% while roughly maintaining benchmark performance.

  16. FasterVAR: Plug-and-Play Acceleration for Visual Autoregressive Models

    cs.CV 2025-12 conditional novelty 5.0 of 10

    Stage-aware pruning of late generation steps, using random projection and cached-feature restoration, speeds up VAR text-to-image models by up to 3.4x with minimal quality loss.

  17. SRUM: Fine-Grained Self-Rewarding for Unified Multimodal Models

    cs.CV 2025-10 conditional novelty 5.0 of 10

    A unified multimodal model can improve its own text-to-image generation by using its understanding module as a rewarder in a global-plus-local reward-weighted training loop.

  18. WordCon: Word-level Typography Control in Scene Text Rendering

    cs.CV 2025-06 conditional novelty 5.0 of 10

    WordCon uses grounding-model masks and two extra losses to fine-tune Flux so that typography can be controlled word by word.

  19. Knowledge Insulating Vision-Language-Action Models: Train Fast, Run Fast, Generalize Better

    cs.LG 2025-05 conditional novelty 5.0 of 10

    Knowledge insulation blocks gradients from a continuous action expert into a VLM backbone while training with discrete action tokens, yielding faster training, better language following, and strong real-robot results.

  20. UniRL: Self-Improving Unified Multimodal Models via Supervised and Reinforcement Learning

    cs.CV 2025-05 conditional novelty 5.0 of 10

    A self-improving post-training method that uses a model's own generated images as training data, with SFT and GRPO, improves generation and understanding and reduces task imbalance.

  21. MADFormer: Mixed Autoregressive and Diffusion Transformers for Continuous Image Generation

    cs.CV 2025-06 conditional novelty 4.0 of 10

    An empirical testbed shows that allocating more transformer layers to autoregressive conditioning improves FID under low inference budgets, while block size should scale with image resolution.

  22. OpenUni: A Simple Baseline for Unified Multimodal Understanding and Generation

    cs.CV 2025-05 conditional novelty 4.0 of 10

    A lightweight open-source connector between a frozen multimodal LLM and a diffusion model yields a unified model that matches larger systems on image generation and understanding benchmarks, with the caveat that headl...

Pith tools