Pith. sign in

REVIEW 9 cited by

ILLUME: Illuminating Your LLMs to See, Draw, and Self-Enhance

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2412.06673 v1 pith:R5H7DXU5 submitted 2024-12-09 cs.CV

Chunwei Wang , Guansong Lu , Junwei Yang , Runhui Huang , Jianhua Han , Lu Hou , Wei Zhang , Hang Xu This is my paper
classification cs.CV
keywords generationmultimodalunifiedillumelargemodelunderstandingalignment
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

In this paper, we introduce ILLUME, a unified multimodal large language model (MLLM) that seamlessly integrates multimodal understanding and generation capabilities within a single large language model through a unified next-token prediction formulation. To address the large dataset size typically required for image-text alignment, we propose to enhance data efficiency through the design of a vision tokenizer that incorporates semantic information and a progressive multi-stage training procedure. This approach reduces the dataset size to just 15M for pretraining -- over four times fewer than what is typically needed -- while achieving competitive or even superior performance with existing unified MLLMs, such as Janus. Additionally, to promote synergistic enhancement between understanding and generation capabilities, which is under-explored in previous works, we introduce a novel self-enhancing multimodal alignment scheme. This scheme supervises the MLLM to self-assess the consistency between text descriptions and self-generated images, facilitating the model to interpret images more accurately and avoid unrealistic and incorrect predictions caused by misalignment in image generation. Based on extensive experiments, our proposed ILLUME stands out and competes with state-of-the-art unified MLLMs and specialized models across various benchmarks for multimodal understanding, generation, and editing.

Discussion (0). Sign in to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Twins: Learn to Predict Unified Representations with Focal Loss

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Channel-wise concatenation of SigLIP2 and Flux VAE features into one token, trained with a focal-style flow-matching loss, yields a unified representation with 1.59 gFID on ImageNet 256 and VAE-level reconstruction.

  2. ICG: Improving Cover Image Generation via MLLM-based Prompting and Personalized Preference Alignment

    cs.CL 2026-04 conditional novelty 6.0 of 10

    A dual-path adapter bridges MLLM meta-token context and user embeddings into frozen diffusion models, optimized by multi-reward preference alignment without ground-truth covers.

  3. ChatUMM: Robust Context Tracking for Conversational Interleaved Generation

    cs.CV 2026-02 conditional novelty 6.0 of 10

    A unified multimodal model trained on synthetic multi-turn dialogues that can interleave image generation with text across a conversation.

  4. Interleaving Reasoning for Better Text-to-Image Generation

    cs.CV 2025-09 conditional novelty 6.0 of 10

    A text-image-text-image multi-turn pipeline improves text-to-image generation, gaining up to 8 points over the base model on several benchmarks while adding an image-conditioned reflection step.

  5. UniCode$^2$: Cascaded Large-scale Codebooks for Unified Multimodal Understanding and Generation

    cs.CV 2025-06 conditional novelty 6.0 of 10

    UniCode² builds a 500K-entry codebook from clustered SigLIP embeddings and uses a cascaded frozen-plus-trainable codebook to unify multimodal understanding and generation with stable training and high token utilization.

  6. SUDER: Self-Improving Unified Large Multimodal Models for Understanding and Generation with Dual Self-Rewards

    cs.AI 2025-06 conditional novelty 6.0 of 10

    SUDER uses the likelihood of reconstructing the original input from a sampled output as a self-reward, improving both understanding and generation in unified multimodal models without external supervision.

  7. UniGen: Enhanced Training & Test-Time Strategies for Unified Multimodal Understanding and Generation

    cs.CV 2025-05 conditional novelty 6.0 of 10

    UniGen shows a 1.5B model trained on open data can beat larger systems on image understanding and generation once it verifies its own outputs with chain-of-thought and Best-of-N selection.

  8. ReasonGen-R1: CoT for Autoregressive Image generation models through SFT and RL

    cs.CV 2025-05 reject novelty 4.0 of 10

    ReasonGen-R1 reports that adding a GPT-generated text rationale before image generation, trained with SFT and GRPO, improves Janus-Pro-7B on GenEval, DPG-Bench, and T2I-Benchmark.

  9. OpenUni: A Simple Baseline for Unified Multimodal Understanding and Generation

    cs.CV 2025-05 conditional novelty 4.0 of 10

    A lightweight open-source connector between a frozen multimodal LLM and a diffusion model yields a unified model that matches larger systems on image generation and understanding benchmarks, with the caveat that headl...

Pith tools