Pith. sign in

REVIEW 4 major objections 6 minor 52 references

DiffSensei: Bridging Multi-Modal LLMs and Diffusion Models for Customized Manga Generation

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper proposes customized manga generation and claims DiffSensei—a diffusion generator paired with a multimodal LLM adapter and masked layout attention—produces multi-character manga pages whose characters keep their identity while…

desk verdict MangaZero is the lasting contribution; DiffSensei works qualitatively, but thin quantitative evidence and unquantified character-ID residual error need addressing in review. read the letter →

arxiv 2412.07589 v2 pith:6RGGXSV7 submitted 2024-12-10 cs.CV

Jianzong Wu , Chao Tang , Jingbo Wang , Yanhong Zeng , Xiangtai Li , Yunhai Tong This is my paper · ORCID
classification cs.CV
keywords customizedmangagenerationstoryvisualizationcharactercustomizationmultimodallargelanguagemodeldiffusionmaskedcross-attentionlayoutcontrolZerodataset
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes a new task, customized manga generation: given reference images of characters, per-panel captions, and user-supplied layout boxes, the model must produce manga pages in which each character stays recognizable while changing expression, pose, or action to follow the story. DiffSensei couples a diffusion-based image generator with a multimodal large language model that acts as a text-compatible character adapter, so the caption, not just the reference image, decides how the character looks in that panel. Character positions are controlled by masked cross-attention rather than by copying pixels from the reference, and dialog placement is encoded as a trainable mask added to the latent, leaving the text itself to artists. The paper also introduces MangaZero, 43,264 manga pages and 427,147 panels with character IDs, boxes, captions, and dialog annotations, and reports that DiffSensei outperforms existing story-visualization and customization baselines on automatic metrics and human preference, including on Manga109 characters never seen in training. If the claims hold, this is a practical route to text-adaptable multi-character manga generation and a reusable dataset for the task.

What carries the argument

The load-bearing mechanism is the two-stage coupling of an SDXL diffusion generator with an MLLM adapter plus masked layout attention. In stage one, character images are encoded by CLIP and a manga-specific Magi encoder, resampled into a small set of tokens, and injected through separate character cross-attention layers where a mask restricts each character's tokens to its bounding box ($M[i,j]=0$ if token $i$ lies in character $j$'s box, $-\infty$ otherwise) while empty regions attend to the placeholder $q_{\text{void}}$; a trainable dialog embedding is expanded and masked into the noisy latent. In stage two, the MLLM—initialized from SEED-X—reads the panel caption and source character features and outputs adapted features, trained with LM loss on special tokens, MSE loss to ground-truth target features, and a diffusion loss through the frozen generator; at inference the final features blend source and adapted features as $c'_i=(1-\beta)c_i+\beta\hat{c}_i$ with $\beta=0.4$. Masked cross-attention is what prevents pixel copy-paste while enforcing layout; the MLLM is what makes characters text-adaptable.

What would settle it

Retrain the stage-2 adapter on MangaZero using the raw, uncorrected Magi character IDs instead of the human-calibrated ones: if DINO-C and human character-consistency scores do not fall, the paper's reliance on human calibration is not load-bearing; if they fall, that reliance is confirmed. Alternatively, have human judges identify which reference character appears in DiffSensei-generated panels across several expressions and poses; if they frequently cannot, the claimed text-adaptable identity preservation fails.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that an MLLM can serve as the missing adapter between a text prompt and a diffusion generator for multi-character narrative images: instead of freezing a character to the pixel content of its reference image, the model converts character images into semantic tokens, lets the MLLM read those tokens together with the panel caption, and predicts target tokens that express the caption's requested state while preserving identity. The masked character cross-attention makes layout control cheap and explicit: each character's tokens attend only inside its bounding box, and character-free regions attend to a learnable placeholder token. Dialog boxes are encoded by adding a trainable embedding, masked to the dialog region, into the noisy latent, so the model learns where speech bubbles go without having to render readable dialogue. Two-stage training—first the generator with layout and dialog controls, then the MLLM adapter with language-model, mean-squared-error, and diffusion losses—is what lets the same character appear in different states across panels. The paper reports consistent wins over StoryDiffusion, AR-LDM, StoryGen, SEED-Story, and MS-Diffusion, and holds up on Manga109 characters the model never trained on.

Load-bearing premise

The method depends on MangaZero's character-ID labels correctly pairing the same character across panels; the paper says the raw automatic IDs were low-accuracy before human correction, so any residual mismatches would make the stage-2 adapter learn from wrong character pairs and break identity preservation.

Editorial extensions

If this is right

  • A user can hand DiffSensei several reference images, a story summary, and per-panel layout boxes, and receive a manga page whose characters keep their look while adopting the expressions, poses, and actions written in each panel caption.
  • Dialog control is spatial rather than textual: artists specify where speech bubbles go, the model fills in bubble placement, and the human edits the words, which sidesteps the poor long-text rendering of current text-to-image models.
  • The ablation results tie text adaptivity to the MLLM stage: removing it lowers CLIP text-alignment and DINO-C character similarity, so caption-driven changes are not an automatic property of the diffusion model.
  • Because Manga109 characters are never seen during training, the Manga109 evaluation results support the claim that DiffSensei generalizes to new, user-supplied characters rather than memorizing the 48 training series.
  • The MangaZero dataset, with 427,147 annotated panels, provides the multi-character, multi-state pairs needed to train and evaluate this task, and is released for the community.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the same masked-attention-plus-MLLM-adapter recipe could transfer to other sequential narrative media—comics, webtoons, storyboards—since nothing except the Magi encoder and the manga-specific data ties it to black-and-white manga.
  • Editorial inference: because the dialog embedding controls placement only, a natural next step is to pair DiffSensei with a text-rendering model or a font-aligned generator that fills readable dialogue inside the predicted bubbles, which would remove the remaining human editing step.
  • Editorial inference: the paper's own admission that automatic character IDs were low-accuracy until human calibration suggests the dataset's ceiling is labeling cost; a scalable follow-up would test whether self-supervised character clustering across pages can replace most of the human correction without degrading the MLLM adapter.
  • Editorial inference: the reported $\beta$ tradeoff between text alignment and identity preservation implies the framework exposes a tunable knob; an adaptive $\beta$ that increases when the caption calls for a large pose change and decreases for close-up identity shots might beat the fixed 0.4 setting.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes a new task, customized manga generation, and introduces DiffSensei, a two-stage framework that combines an SDXL-based diffusion generator with an MLLM-based character feature adapter. Stage 1 trains a multi-character manga generator with masked cross-attention for character layout control and a dialog-embedding mechanism for dialog bounding boxes. Stage 2 fine-tunes LoRA and resampler weights of an MLLM (initialized from SEED-X) so that source character features are transformed into text-adapted target features, supervised by a combination of LM, MSE, and diffusion losses. The authors also introduce MangaZero, a dataset of 43,264 manga pages / 427,147 panels with character, panel, dialog, and caption annotations, and report quantitative, qualitative, and human-preference comparisons against story-visualization and customization baselines. The central claim is that DiffSensei enables text-adaptable character customization and outperforms existing models.

Significance. If the central claim is secured, this is a useful contribution to story visualization and manga generation. The task formulation is well motivated, and MangaZero appears to be the first large-scale manga dataset with full character, dialog, and caption annotations. The two-stage architecture is technically plausible: converting character images into a compact token space avoids naive copy-pasting, and the MLLM adapter is a reasonable way to inject text-dependent pose/expression changes. The paper reports extensive qualitative results, including unseen Manga109 characters, and appendix failure cases are candidly discussed. However, the quantitative evidence is currently weak: metric gaps over the strongest baseline are small and reported without variance, the human study is small and lacks statistical support, and the stage-2 MSE target is produced by the paper's own feature extractor. The released dataset and code, if fully open-sourced, would be valuable to the community and would enable independent verification.

major comments (4)
  1. [Section 3.2] The construction pipeline states that Magi's character ID labeling 'has relatively low accuracy' and that human annotators refine the labels, but no residual error rate after refinement is reported. This is load-bearing because Eq. (5) uses character IDs to pair source and target features of the same character across panels; any remaining cross-panel ID mismatch would train the MLLM adapter on wrong source-target pairs and create conflicting gradients with the diffusion loss. The paper should report post-refinement character-ID accuracy on a validation subset, or otherwise provide evidence that identity pairing is reliable. An independent evaluation on truly unseen identities (e.g., held-out series not in MangaZero) would also help separate identity preservation from memorization of training characters.
  2. [Tables 2a and 2b] The numerical advantages over MS-Diffusion are close to measurement noise: on MangaZero, FID is 0.407 vs 0.408, CLIP is 0.235 vs 0.229, DINO-I is 0.618 vs 0.610, and DINO-C is 0.651 vs 0.641; on Manga109, DINO-C is exactly tied at 0.600 and DINO-I differs by only 0.004. No error bars, multiple seeds, or significance tests are reported. Without these, the abstract's claim that DiffSensei 'outperforms existing models' is not statistically supported. Please add repeated evaluation runs with confidence intervals and, where feasible, paired significance tests on the same evaluation pages.
  3. [Section 4, Eq. (5)] The MSE loss in stage 2 compares the MLLM-predicted character features with target features produced by the same feature extractor (Eq. 1) that generates the input features. This means part of the training signal teaches the MLLM to imitate the frozen feature extractor's own embedding space, and the diffusion loss is the only image-level external objective. This is not a fatal circularity, but it weakens the interpretation of the reported CLIP and DINO gains: those gains could partly reflect adaptation to the extractor's feature geometry rather than genuine identity improvements. I would like to see an experiment where the MSE term is ablated, and/or an identity evaluation using a feature extractor not used anywhere in training.
  4. [Section 5.2 and Figure 5] The human preference study uses 15 raters and 10 anonymized page sets per method, and results are reported only as preference percentages without variance, per-item distributions, or inter-rater agreement. The appendix contains a second human evaluation of the MLLM module (Fig. 12) without any sample-size or statistical details. Human preference is a key part of the 'significant advancement' claim, so the paper should report confidence intervals and a significance test, or temper the claim to a descriptive preference trend.
minor comments (6)
  1. [Equation (2)] The notation for the character cross-attention projections is confusing: the text defines Wt_k and Wt_v as text projections but then uses the same symbols for the character key and value projections. Please use distinct symbols such as Wi_k and Wi_v.
  2. [Section 3.2] There is a typo: 'LLaV A-v1.6-34B' should be 'LLaVA-v1.6-34B'.
  3. [Table 1] 'publically' should be 'publicly' in the table caption or body text.
  4. [Section 5.2] Several baselines (AR-LDM, StoryGen, SEED-Story, StoryDiffusion) are not designed to accept character images and instead use reference images (marked with '*'), so the comparison is not fully apples-to-apples. An additional baseline that uses character images with box-level layout control, such as IP-Adapter with box conditioning, would make the contribution of the MLLM adapter clearer.
  5. [Abstract and Project Page] The abstract states that code, model, and dataset are open-sourced, but the paper gives only a project page URL and no license or availability details. Please provide a public repository or data-release statement with usage terms so the reproducibility claim can be verified.
  6. [Figure 5] The bar chart shows preference percentages without error bars or sample sizes in the figure itself. Please add confidence intervals or refer the reader to a table containing them, consistent with the major comment on statistical reporting.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the MLLM adapter's MSE target is self-referential in flavor, but the diffusion loss and external Manga109 benchmark keep the central claim grounded.

full rationale

The paper's core derivation is not circular. Stage 1 trains the diffusion-based image generator with a standard diffusion loss against real manga panels, an external pixel-level objective. Stage 2 trains the MLLM adapter using Eq. (5), where the MSE loss compares the predicted character features with target character embeddings extracted by the same frozen feature extractor; this gives the training a self-referential flavor, but it is not a definitional reduction because the target embedding is derived from real target images and the diffusion loss additionally supervises the final generated image. The central claim of text-adaptable character customization is evaluated on Manga109 characters that were unseen during training (Sec. 5.1), providing an external benchmark for identity preservation and generalization. No load-bearing self-citation, uniqueness theorem, or fitted-parameter-renamed-as-prediction is used. The admitted low raw character-ID accuracy in Sec. 3.2 is a data-quality and correctness risk, not a circularity, because the human refinement and subsequent external evaluation prevent the claim from reducing to its own inputs.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on learned resamplers, pretrained models, and a dataset whose character-ID quality is human-corrected. The most consequential hand-chosen values are the feature-blend weight beta, the attention weight alpha, the MSE loss weight, and the source-image sampling rate.

free parameters (4)
  • alpha (character attention weight) = 0.6
    Set at inference following IP-Adapter and MS-Diffusion; controls the strength of character cross-attention in Eq. 2.
  • beta (MLLM adapted feature blend weight) = 0.4
    Chosen by ablation in Table 4b to balance CLIP and DINO metrics; directly controls the identity versus text-adaptation tradeoff.
  • MLLM loss weight lambda_mse = 6.0
    Hand-set in Section 5.1; the MSE loss dominates the feature-adapter training and shapes how closely predicted features match extractor outputs.
  • source character sampling rate from target panel = 0.5
    Chosen via ablation in Table 4a; 50% of source images are crops of the target panel, which materially affects identity preservation metrics.
assumptions (4)
  • domain assumption Magi's automatic annotations, after human calibration, provide accurate panel boxes, character boxes, character IDs, and dialog boxes on MangaZero.
    Section 3.2 states raw character ID accuracy was low and required human refinement; if residual errors remain, training pairs for the MLLM adapter are corrupted.
  • ad hoc to paper Resampled CLIP and Magi image features preserve character identity while discarding pixel-level detail.
    Eq. 1 and Section 4 assume a few query tokens carry enough identity information to avoid copy-paste; Appendix E shows unclear input images break this assumption.
  • domain assumption LLaVA-v1.6-34B generated panel captions are suitable text supervision.
    Section 3.2 uses generated captions as panel prompts without human verification on the full dataset; caption errors propagate to the text-conditioning signal.
  • domain assumption SDXL and SEED-X pretrained weights transfer to black-and-white manga after fine-tuning.
    Implementation relies on SDXL as the generator and SEED-X as the MLLM; Appendix E notes style control fails without character input, showing the transfer is imperfect.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DiffSensei: Bridging Multi-Modal LLMs and Diffusion Models for Customized Manga Generation." pith.science (2026). https://pith.science/paper/6RGGXSV7

@misc{pith2026241207589,
  author       = {Pith},
  title        = {Pith review of: DiffSensei: Bridging Multi-Modal LLMs and Diffusion Models for Customized Manga Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6RGGXSV7}},
  note         = {Machine review of arXiv:2412.07589}
}
read the original abstract

Story visualization, the task of creating visual narratives from textual descriptions, has seen progress with text-to-image generation models. However, these models often lack effective control over character appearances and interactions, particularly in multi-character scenes. To address these limitations, we propose a new task: \textbf{customized manga generation} and introduce \textbf{DiffSensei}, an innovative framework specifically designed for generating manga with dynamic multi-character control. DiffSensei integrates a diffusion-based image generator with a multimodal large language model (MLLM) that acts as a text-compatible identity adapter. Our approach employs masked cross-attention to seamlessly incorporate character features, enabling precise layout control without direct pixel transfer. Additionally, the MLLM-based adapter adjusts character features to align with panel-specific text cues, allowing flexible adjustments in character expressions, poses, and actions. We also introduce \textbf{MangaZero}, a large-scale dataset tailored to this task, containing 43,264 manga pages and 427,147 annotated panels, supporting the visualization of varied character interactions and movements across sequential frames. Extensive experiments demonstrate that DiffSensei outperforms existing models, marking a significant advancement in manga generation by enabling text-adaptable character customization. The project page is https://jianzongwu.github.io/projects/diffsensei/.

Figures

Figures reproduced from arXiv: 2412.07589 by the authors.

Figure 1
Figure 1. Results of DiffSensei. (a) Customized manga generation with controllable character images, panel captions, and layout condi￾tions. Our DiffSensei successfully generates detailed character expressions and states following the panel captions. (b) Manga creation for real human images. The dialogues are post-edited by humans. The continuation is in the Appendix. We strongly recommend that the readers see the Appendix fo… view at source ↗
Figure 2
Figure 2. We construct MangaZero through three steps: 1) Down￾load manga pages from the internet. 2) Annotate manga panels autonomously with pre-trained models. 3) Human calibration for the character ID annotation. ries from before the year 2000 that are not featured in Manga109, such as Doraemon (1974). Construction pipeline. To build our dataset, we first down￾load manga pages from the internet, explicitly sourcing im￾ages … view at source ↗
Figure 3
Figure 3. The architecture of DiffSensei. In the first stage, we train a multi-character customized manga image generation model with layout control. The dialog embedding is added to the noised latent after the first convolution layer. All the parameters in the U-Net and feature extractor are trained. In the second stage, we finetune LoRA and resampler weights of an MLLM to adapt the source character features corresponding to… view at source ↗
Figures from the paper (15 more)
Figure 4
Figure 4. Figure 4: Qualitative comparison with baselines. Baselines followed by a “*” use reference images as input rather than character images. Methods marked by “†” means re-trained with dialog embedding. Our model excels at preserving the characters while following the text prompt. O…
Figure 5
Figure 5. Figure 5: Human preference study on MangaZero eval set. els across five key metrics. Our model improves 0.06 in the CLIP metrics compared to the multi-subject cus￾tomization baseline, MS-Diffusion [39], which struggles to modify characters’ states effectively in response to tex￾…
Figure 6
Figure 6. Figure 6: Qualitative results. Character images in red boxes are from Manga109 (The rightmost example). Our DiffSensei can generate vivid manga pages in various scenarios. Better viewed with zoom-in. More results can be found in the appendix [PITH_FULL_IMAGE:figures/full_fig_p0…
Figure 7
Figure 7. Figure 7: A complete long manga story about Hinton, LeCun, and Bengio winning the Nobel Prize. [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: More qualitative comparisons with baselines. Baselines followed by a “*” use reference images as input rather than character images. Methods marked by “†” means re-trained with dialog embedding. DiffSensei GT Input Character Images [PITH_FULL_IMAGE:figures/full_fig_p0…
Figure 10
Figure 10. Figure 10 [PITH_FULL_IMAGE:figures/full_fig_p013_10.png]
Figure 11
Figure 11. Figure 11: Qualitative ablation of the proposed modules. CM is character masked attention injection. DM is dialog masked encod￾ing. Magi means using Magi [30] image encoder. MLLM means using MLLM for stage 2 training. 83.4 53.8 54.7 70.6 81.3 16.6 46.2 45.3 29.4 18.7 0 10 20 30 …
Figure 12
Figure 12. Figure 12: Human evaluation on the MLLM module. presented in [PITH_FULL_IMAGE:figures/full_fig_p014_12.png]
Figure 13
Figure 13. Figure 13: Qualitative ablation of β. (a) Unclear Character Images Input Character Image (c) Style Control without Character Input “A detailed illustration of a flower garden, featuring a variety of blooms and foliage. The garden is lush and vibrant, with a sense of tranquility”…
Figure 14
Figure 14. Figure 14: Failure cases. ter research into style control for manga generation. Each manga series or artist possesses a distinctive drawing style, and our dataset is well-positioned to support training mod￾els in style controllability. This would enable fine-tuning or conditiona…
Figure 15
Figure 15. Figure 15: Details of the MangaZero dataset [PITH_FULL_IMAGE:figures/full_fig_p016_15.png]
Figure 16
Figure 16. Figure 16: DiffSensei generated results with inputs (Part1). [PITH_FULL_IMAGE:figures/full_fig_p017_16.png]
Figure 17
Figure 17. Figure 17: DiffSensei generated results with inputs (Part2). [PITH_FULL_IMAGE:figures/full_fig_p018_17.png]
Figure 18
Figure 18. Figure 18: Manga pages generated by DiffSensei (Part 1). [PITH_FULL_IMAGE:figures/full_fig_p019_18.png]
Figure 19
Figure 19. Figure 19: Manga pages generated by DiffSensei (Part 2). [PITH_FULL_IMAGE:figures/full_fig_p020_19.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

52 extracted references · 29 canonical work pages

  1. [2]

    manga109

    Kiyoharu Aizawa, Azuma Fujimoto, Atsushi Otsubo, Toru Ogawa, Yusuke Matsui, Koki Tsubota, and Hikaru Ikuta. Building a manga dataset “manga109” with annotations for multimedia applications. TMM, 2020. 3, 6, 8

    work page 2020

  2. [3]

    Meissonic: Revitalizing masked generative trans- formers for efficient high-resolution text-to-image synthesis

    Jinbin Bai, Tian Ye, Wei Chow, Enxin Song, Qing-Guo Chen, Xiangtai Li, Zhen Dong, Lei Zhu, and Shuicheng Yan. Meissonic: Revitalizing masked generative trans- formers for efficient high-resolution text-to-image synthesis. arXiv preprint arXiv:2410.08261, 2024. 3

    arXiv

  3. [4]

    Anydoor: Zero-shot object-level im- age customization

    Xi Chen, Lianghua Huang, Yu Liu, Yujun Shen, Deli Zhao, and Hengshuang Zhao. Anydoor: Zero-shot object-level im- age customization. In CVPR, 2024. 2

    2024

  4. [5]

    Au- tostudio: Crafting consistent subjects in multi-turn interac- tive image generation

    Junhao Cheng, Xi Lu, Hanhui Li, Khun Loun Zai, Baiqiao Yin, Yuhao Cheng, Yiqiang Yan, and Xiaodan Liang. Au- tostudio: Crafting consistent subjects in multi-turn interac- tive image generation. arXiv preprint arXiv:2406.01388 ,

    arXiv

  5. [6]

    Guiding instruction-based im- age editing via multimodal large language models

    Tsu-Jui Fu, Wenze Hu, Xianzhi Du, William Yang Wang, Yinfei Yang, and Zhe Gan. Guiding instruction-based im- age editing via multimodal large language models. In ICLR,

  6. [7]

    An image is worth one word: Personalizing text-to- image generation using textual inversion

    Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patash- nik, Amit H Bermano, Gal Chechik, and Daniel Cohen- Or. An image is worth one word: Personalizing text-to- image generation using textual inversion. arXiv preprint arXiv:2208.01618, 2022. 2

    arXiv 2022

  7. [8]

    Seed-x: Mul- timodal models with unified multi-granularity comprehen- sion and generation

    Yuying Ge, Sijie Zhao, Jinguo Zhu, Yixiao Ge, Kun Yi, Lin Song, Chen Li, Xiaohan Ding, and Ying Shan. Seed-x: Mul- timodal models with unified multi-granularity comprehen- sion and generation. arXiv preprint arXiv:2404.14396, 2024. 2, 3, 6

    arXiv 2024

  8. [9]

    Mix-of-show: Decentralized low- rank adaptation for multi-concept customization of diffusion models

    Yuchao Gu, Xintao Wang, Jay Zhangjie Wu, Yujun Shi, Yun- peng Chen, Zihan Fan, Wuyou Xiao, Rui Zhao, Shuning Chang, Weijia Wu, et al. Mix-of-show: Decentralized low- rank adaptation for multi-concept customization of diffusion models. NeurIPS, 2024. 2

    work page 2024

Show all 52 references
  1. [10]

    Imagine this! scripts to composi- tions to videos

    Tanmay Gupta, Dustin Schwenk, Ali Farhadi, Derek Hoiem, and Aniruddha Kembhavi. Imagine this! scripts to composi- tions to videos. In ECCV, 2018. 2, 3

    2018

  2. [11]

    A generalist facex via learning unified facial representation

    Yue Han, Jiangning Zhang, Junwei Zhu, Xiangtai Li, Yan- hao Ge, Wei Li, Chengjie Wang, Yong Liu, Xiaoming Liu, and Ying Tai. A generalist facex via learning unified facial representation. arXiv preprint arXiv:2401.00551, 2023. 2

    2023 arXiv

  3. [12]

    Face adapter for pre-trained diffusion models with fine- grained id and attribute control

    Yue Han, Junwei Zhu, Keke He, Xu Chen, Yanhao Ge, Wei Li, Xiangtai Li, Jiangning Zhang, Chengjie Wang, and Yong Liu. Face adapter for pre-trained diffusion models with fine- grained id and attribute control. ECCV, 2024. 2

    2024

  4. [13]

    Gans trained by a two time-scale update rule converge to a local nash equilib- rium

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilib- rium. NeurIPS, 2017. 6

    2017

  5. [14]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021. 6

    2021 arXiv

  6. [15]

    Visual storytelling

    Ting-Hao Huang, Francis Ferraro, Nasrin Mostafazadeh, Is- han Misra, Aishwarya Agrawal, Jacob Devlin, Ross Gir- shick, Xiaodong He, Pushmeet Kohli, Dhruv Batra, et al. Visual storytelling. In NAACL, 2016. 2

    2016

  7. [16]

    Smartedit: Exploring com- plex instruction-based image editing with multimodal large language models

    Yuzhou Huang, Liangbin Xie, Xintao Wang, Ziyang Yuan, Xiaodong Cun, Yixiao Ge, Jiantao Zhou, Chao Dong, Rui Huang, Ruimao Zhang, et al. Smartedit: Exploring com- plex instruction-based image editing with multimodal large language models. In CVPR, 2024. 2, 3

    2024

  8. [17]

    Announcing black forest labs, 2024

    Black Forest Labs. Announcing black forest labs, 2024. 5, 14

    2024

  9. [18]

    Storygan: A sequential conditional gan for story visu- alization

    Yitong Li, Zhe Gan, Yelong Shen, Jingjing Liu, Yu Cheng, Yuexin Wu, Lawrence Carin, David Carlson, and Jianfeng Gao. Storygan: A sequential conditional gan for story visu- alization. In CVPR, 2019. 1, 2, 3

    2019

  10. [19]

    Photomaker: Customizing re- alistic human photos via stacked id embedding

    Zhen Li, Mingdeng Cao, Xintao Wang, Zhongang Qi, Ming- Ming Cheng, and Ying Shan. Photomaker: Customizing re- alistic human photos via stacked id embedding. In CVPR,

  11. [20]

    Sketch2manga: Shaded manga screening from sketch with diffusion models

    Jian Lin, Xueting Liu, Chengze Li, Minshan Xie, and Tien- Tsin Wong. Sketch2manga: Shaded manga screening from sketch with diffusion models. In ICIP, 2024. 2, 3

    2024

  12. [21]

    Intelligent grimm-open-ended vi- sual storytelling via latent diffusion models

    Chang Liu, Haoning Wu, Yujie Zhong, Xiaoyun Zhang, Yan- feng Wang, and Weidi Xie. Intelligent grimm-open-ended vi- sual storytelling via latent diffusion models. In CVPR, 2024. 1, 2, 3, 4, 6, 7, 11

    2024

  13. [22]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arXiv:2304.08485,

    arXiv

  14. [23]

    Decoupled weight decay regularization

    I Loshchilov. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. 6

    2017 arXiv

  15. [24]

    Dinov2: Learning robust visual features without supervision

    Maxime Oquab, Timoth ´ee Darcet, Th ´eo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023. 6

    2023 arXiv

  16. [25]

    Synthesizing coherent story with auto-regressive la- tent diffusion models

    Xichen Pan, Pengda Qin, Yuhong Li, Hui Xue, and Wenhu Chen. Synthesizing coherent story with auto-regressive la- tent diffusion models. In WACV, 2024. 1, 2, 3, 6, 7, 11

    2024

  17. [26]

    Sdxl: Improving latent diffusion mod- els for high-resolution image synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas M ¨uller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion mod- els for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023. 5, 6, 8, 11, 14

    2023 arXiv

  18. [27]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. In ICCV, 2021. 6

    2021

  19. [28]

    High-resolution image syn- thesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj¨orn Ommer. High-resolution image syn- thesis with latent diffusion models. In CVPR, 2022. 5

    2022

  20. [29]

    Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation

    Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In CVPR, 2023. 2, 3

    2023

  21. [30]

    The manga whis- perer: Automatically generating transcriptions for comics

    Ragav Sachdeva and Andrew Zisserman. The manga whis- perer: Automatically generating transcriptions for comics. In CVPR, 2024. 3, 4, 6, 8, 12, 14

    2024

  22. [31]

    Tails tell tales: Chapter-wide manga transcriptions with char- acter names

    Ragav Sachdeva, Gyungin Shin, and Andrew Zisserman. Tails tell tales: Chapter-wide manga transcriptions with char- acter names. arXiv preprint arXiv:2408.00298, 2024. 3

    2024 arXiv

  23. [32]

    Rela- tionbooth: Towards relation-aware customized object gener- ation

    Qingyu Shi, Lu Qi, Jianzong Wu, Jinbin Bai, Jingbo Wang, Yunhai Tong, Xiangtai Li, and Ming-Husang Yang. Rela- tionbooth: Towards relation-aware customized object gener- ation. arXiv preprint arXiv:2410.23280, 2024. 2

    2024 arXiv

  24. [33]

    Mangagan: Unpaired photo-to-manga transla- tion based on the methodology of manga drawing

    Hao Su, Jianwei Niu, Xuefeng Liu, Qingfeng Li, Jiahe Cui, and Ji Wan. Mangagan: Unpaired photo-to-manga transla- tion based on the methodology of manga drawing. In AAAI,

  25. [34]

    Generative multimodal models are in-context learners

    Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiy- ing Yu, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal models are in-context learners. In CVPR, 2024. 2, 3

    2024

  26. [35]

    One missing piece in vision and language: A survey on comics understanding

    Emanuele Vivoli, Andrey Barsky, Mohamed Ali Souibgui, Artemis LLabres, Marco Bertini, and Dimosthenis Karatzas. One missing piece in vision and language: A survey on comics understanding. arXiv preprint arXiv:2409.09502 ,

    arXiv

  27. [36]

    Instantid: Zero-shot identity-preserving gener- ation in seconds

    Qixun Wang, Xu Bai, Haofan Wang, Zekui Qin, and An- thony Chen. Instantid: Zero-shot identity-preserving gener- ation in seconds. arXiv preprint arXiv:2401.07519 , 2024. 2

    2024 arXiv

  28. [37]

    Autostory: Generating di- verse storytelling images with minimal human effort

    Wen Wang, Canyu Zhao, Hao Chen, Zhekai Chen, Kecheng Zheng, and Chunhua Shen. Autostory: Generating di- verse storytelling images with minimal human effort. arXiv preprint arXiv:2311.11243, 2023. 1, 2

    2023 arXiv

  29. [38]

    Instancediffusion: Instance-level control for image generation

    Xudong Wang, Trevor Darrell, Sai Saketh Rambhatla, Rohit Girdhar, and Ishan Misra. Instancediffusion: Instance-level control for image generation. In CVPR, 2024. 4

    2024

  30. [39]

    Ms-diffusion: Multi-subject zero-shot im- age personalization with layout guidance

    X Wang, Siming Fu, Qihan Huang, Wanggui He, and Hao Jiang. Ms-diffusion: Multi-subject zero-shot im- age personalization with layout guidance. arXiv preprint arXiv:2406.07209, 2024. 2, 3, 4, 6, 7, 8, 11

    2024 arXiv

  31. [40]

    Emu3: Next-token prediction is all you need

    Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need. arXiv preprint arXiv:2409.18869, 2024. 2, 3

    2024 arXiv

  32. [41]

    Genartist: Multimodal llm as an agent for unified image gen- eration and editing

    Zhenyu Wang, Aoxue Li, Zhenguo Li, and Xihui Liu. Genartist: Multimodal llm as an agent for unified image gen- eration and editing. In NeurIPS, 2024

    2024

  33. [42]

    Towards language-driven video inpainting via multimodal large language models

    Jianzong Wu, Xiangtai Li, Chenyang Si, Shangchen Zhou, Jingkang Yang, Jiangning Zhang, Yining Li, Kai Chen, Yun- hai Tong, Ziwei Liu, et al. Towards language-driven video inpainting via multimodal large language models. In CVPR,

  34. [43]

    Towards open vocabulary learning: A survey

    Jianzong Wu, Xiangtai Li, Shilin Xu, Haobo Yuan, Henghui Ding, Yibo Yang, Xia Li, Jiangning Zhang, Yunhai Tong, Xudong Jiang, Bernard Ghanem, and Dacheng Tao. Towards open vocabulary learning: A survey. T-PAMI, 2024. 2

    2024

  35. [44]

    Mo- tionbooth: Motion-aware customized text-to-video genera- tion

    Jianzong Wu, Xiangtai Li, Yanhong Zeng, Jiangning Zhang, Qianyu Zhou, Yining Li, Yunhai Tong, and Kai Chen. Mo- tionbooth: Motion-aware customized text-to-video genera- tion. In NeurIPS, 2024. 2

    2024

  36. [45]

    Boxdiff: Text-to-image synthesis with training-free box-constrained diffusion

    Jinheng Xie, Yuexiang Li, Yawen Huang, Haozhe Liu, Wen- tian Zhang, Yefeng Zheng, and Mike Zheng Shou. Boxdiff: Text-to-image synthesis with training-free box-constrained diffusion. In ICCV, 2023. 4

    2023

  37. [46]

    Seed-story: Multimodal long story generation with large language model

    Shuai Yang, Yuying Ge, Yang Li, Yukang Chen, Yixiao Ge, Ying Shan, and Yingcong Chen. Seed-story: Multimodal long story generation with large language model. arXiv preprint arXiv:2407.08683, 2024. 1, 2, 3, 6, 7, 11

    2024 arXiv

  38. [47]

    Ip- adapter: Text compatible image prompt adapter for text-to- image diffusion models

    Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip- adapter: Text compatible image prompt adapter for text-to- image diffusion models. arXiv preprint arXiv:2308.06721,

    arXiv

  39. [48]

    Ai-driven background generation for manga illustrations: A deep generative model approach

    Zhang Yunqian. Ai-driven background generation for manga illustrations: A deep generative model approach. ORES,

  40. [49]

    Magicbrush: A manually annotated dataset for instruction- guided image editing

    Kai Zhang, Lingbo Mo, Wenhu Chen, Huan Sun, and Yu Su. Magicbrush: A manually annotated dataset for instruction- guided image editing. NeurIPS, 2024. 2, 3

    2024

  41. [50]

    Generating manga from illustrations via mimicking manga creation workflow

    Lvmin Zhang, Xinrui Wang, Qingnan Fan, Yi Ji, and Chun- ping Liu. Generating manga from illustrations via mimicking manga creation workflow. In CVPR, 2021. 2, 3

    2021

  42. [51]

    Adding conditional control to text-to-image diffusion models

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In ICCV, 2023. 4

    2023

  43. [52]

    Cus- tomization assistant for text-to-image generation

    Yufan Zhou, Ruiyi Zhang, Jiuxiang Gu, and Tong Sun. Cus- tomization assistant for text-to-image generation. In CVPR,

  44. [53]

    page results.pdf

    Yupeng Zhou, Daquan Zhou, Ming-Ming Cheng, Jiashi Feng, and Qibin Hou. Storydiffusion: Consistent self- attention for long-range image and video generation. In NeurIPS, 2024. 1, 2, 6, 7, 8, 11 DiffSensei: Bridging Multi-Modal LLMs and Diffusion Models for Customized Manga Gene...

    2024

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.