{"id":"9d52170a-90e2-4eaf-98ee-b256f935bbbe","arxiv_id":"2412.07589","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"DiffSensei combines an SDXL diffusion generator with a multimodal LLM adapter and masked attention to generate manga pages with multiple characters whose poses and expressions follow panel captions.","lead":"DiffSensei is a system that turns short story captions, character reference images, and layout boxes into manga pages while keeping characters recognizable across panels. It also introduces MangaZero, a dataset of more than 43,000 manga pages, to train and test this new task.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"MangaZero character-ID accuracy is the linchpin: Sec. 3.2 admits raw IDs were low-accuracy and only states humans refined them, without residual-error numbers; wrong source-target pairs would corrupt the stage-2 MLLM adapter (Eqs. 5-6) and break text-adaptable identity.","rationale":"The reader's weakest assumption accurately identifies the load-bearing dependency: the MLLM adapter's paired training is only as good as the character-ID labels. I considered the evaluation inflation from target-panel sampling, but Tab. 4a shows the rate-0.5 ablation is only slightly better than rate-0.0 (DINO-C 0.651 vs 0.646, CLIP 0.235 vs 0.233), so even a corrected protocol would likely preserve a small edge. The dataset-label question, by contrast, is binary: if residual mismatches are pervasive, the stage-2 objective is corrupted and the central contribution cannot be validated. The paper's own admission in Sec. 3.2 makes this the least-secure assumption. The Manga109 external benchmark and the diffusion loss in stage 2 provide some independent grounding, but they do not eliminate the dependency on correct ID pairs. A concrete residual-error measurement would settle it. The verdict should remain CONDITIONAL because the condition (verifying/correcting IDs) is exactly what the authors need to report; hence UNCHANGED relative to the reader's verdict.","tokens_in":19849,"tokens_out":9009,"duration_ms":73368,"concrete_test":"Sample 500 pages from MangaZero and have two independent annotators re-verify character-ID consistency across panels within each page; compute the residual mismatch rate and inter-annotator agreement. If the mismatch rate exceeds ~2%, retrain the stage-2 MLLM adapter using only high-confidence pairs (or corrected labels) and compare DINO-C and CLIP on the MangaZero eval set. A large metric change would confirm the sensitivity; a negligible change would indicate the concern does not materially affect the results.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2 reports that Magi's character ID labeling 'has relatively low accuracy' and then says human annotators refine the labels, but it never quantifies the residual error after refinement. The entire stage-2 training (Eq. 5) constructs paired samples [caption, source character features, target character features] using these IDs to identify the same character across panels. If any residual mismatch exists, the MSE loss trains the MLLM to map a source character's features to a different character's target features, and the diffusion loss simultaneously encourages generating the target panel; these conflicting gradients would degrade the adapter. The paper's own ablation (Tab. 4a) shows identity metrics are sensitive to the source-sampling strategy, and the Manga109 evaluation (Tab. 2b) shows DiffSensei ties MS-Diffusion on DINO-C (0.600 vs 0.600), so the only strong evidence for identity beyond copy-level is on MangaZero eval, where the model has seen the same characters in training. Without a residual-error estimate or an independent evaluation with truly unseen character identities, the central claim that DiffSensei achieves text-adaptable character customization is not yet secured.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a new task, customized manga generation, and introduces DiffSensei, a two-stage framework that combines an SDXL-based diffusion generator with an MLLM-based character feature adapter. Stage 1 trains a multi-character manga generator with masked cross-attention for character layout control and a dialog-embedding mechanism for dialog bounding boxes. Stage 2 fine-tunes LoRA and resampler weights of an MLLM (initialized from SEED-X) so that source character features are transformed into text-adapted target features, supervised by a combination of LM, MSE, and diffusion losses. The authors also introduce MangaZero, a dataset of 43,264 manga pages / 427,147 panels with character, panel, dialog, and caption annotations, and report quantitative, qualitative, and human-preference comparisons against story-visualization and customization baselines. The central claim is that DiffSensei enables text-adaptable character customization and outperforms existing models.","tokens_in":20158,"tokens_out":4717,"duration_ms":45298,"significance":"If the central claim is secured, this is a useful contribution to story visualization and manga generation. The task formulation is well motivated, and MangaZero appears to be the first large-scale manga dataset with full character, dialog, and caption annotations. The two-stage architecture is technically plausible: converting character images into a compact token space avoids naive copy-pasting, and the MLLM adapter is a reasonable way to inject text-dependent pose/expression changes. The paper reports extensive qualitative results, including unseen Manga109 characters, and appendix failure cases are candidly discussed. However, the quantitative evidence is currently weak: metric gaps over the strongest baseline are small and reported without variance, the human study is small and lacks statistical support, and the stage-2 MSE target is produced by the paper's own feature extractor. The released dataset and code, if fully open-sourced, would be valuable to the community and would enable independent verification.","major_comments":[{"comment":"The construction pipeline states that Magi's character ID labeling 'has relatively low accuracy' and that human annotators refine the labels, but no residual error rate after refinement is reported. This is load-bearing because Eq. (5) uses character IDs to pair source and target features of the same character across panels; any remaining cross-panel ID mismatch would train the MLLM adapter on wrong source-target pairs and create conflicting gradients with the diffusion loss. The paper should report post-refinement character-ID accuracy on a validation subset, or otherwise provide evidence that identity pairing is reliable. An independent evaluation on truly unseen identities (e.g., held-out series not in MangaZero) would also help separate identity preservation from memorization of training characters.","section":"Section 3.2"},{"comment":"The numerical advantages over MS-Diffusion are close to measurement noise: on MangaZero, FID is 0.407 vs 0.408, CLIP is 0.235 vs 0.229, DINO-I is 0.618 vs 0.610, and DINO-C is 0.651 vs 0.641; on Manga109, DINO-C is exactly tied at 0.600 and DINO-I differs by only 0.004. No error bars, multiple seeds, or significance tests are reported. Without these, the abstract's claim that DiffSensei 'outperforms existing models' is not statistically supported. Please add repeated evaluation runs with confidence intervals and, where feasible, paired significance tests on the same evaluation pages.","section":"Tables 2a and 2b"},{"comment":"The MSE loss in stage 2 compares the MLLM-predicted character features with target features produced by the same feature extractor (Eq. 1) that generates the input features. This means part of the training signal teaches the MLLM to imitate the frozen feature extractor's own embedding space, and the diffusion loss is the only image-level external objective. This is not a fatal circularity, but it weakens the interpretation of the reported CLIP and DINO gains: those gains could partly reflect adaptation to the extractor's feature geometry rather than genuine identity improvements. I would like to see an experiment where the MSE term is ablated, and/or an identity evaluation using a feature extractor not used anywhere in training.","section":"Section 4, Eq. (5)"},{"comment":"The human preference study uses 15 raters and 10 anonymized page sets per method, and results are reported only as preference percentages without variance, per-item distributions, or inter-rater agreement. The appendix contains a second human evaluation of the MLLM module (Fig. 12) without any sample-size or statistical details. Human preference is a key part of the 'significant advancement' claim, so the paper should report confidence intervals and a significance test, or temper the claim to a descriptive preference trend.","section":"Section 5.2 and Figure 5"}],"minor_comments":[{"comment":"The notation for the character cross-attention projections is confusing: the text defines Wt_k and Wt_v as text projections but then uses the same symbols for the character key and value projections. Please use distinct symbols such as Wi_k and Wi_v.","section":"Equation (2)"},{"comment":"There is a typo: 'LLaV A-v1.6-34B' should be 'LLaVA-v1.6-34B'.","section":"Section 3.2"},{"comment":"'publically' should be 'publicly' in the table caption or body text.","section":"Table 1"},{"comment":"Several baselines (AR-LDM, StoryGen, SEED-Story, StoryDiffusion) are not designed to accept character images and instead use reference images (marked with '*'), so the comparison is not fully apples-to-apples. An additional baseline that uses character images with box-level layout control, such as IP-Adapter with box conditioning, would make the contribution of the MLLM adapter clearer.","section":"Section 5.2"},{"comment":"The abstract states that code, model, and dataset are open-sourced, but the paper gives only a project page URL and no license or availability details. Please provide a public repository or data-release statement with usage terms so the reproducibility claim can be verified.","section":"Abstract and Project Page"},{"comment":"The bar chart shows preference percentages without error bars or sample sizes in the figure itself. Please add confidence intervals or refer the reader to a table containing them, consistent with the major comment on statistical reporting.","section":"Figure 5"}],"recommendation":"major_revision","confidential_remarks":"The paper's contribution is a solid dataset plus a plausible framework, and the appendix shows real effort on limitations and ablations. The main gap is evidentiary: the quantitative and human evaluations are too thin to support the headline 'outperforms existing models' claim, and the dependence on unquantified character-ID accuracy in MangaZero is a genuine risk. I would encourage the editor to ask for a revision with additional statistical reporting and a residual-error analysis rather than a rejection, provided the authors can also clarify the availability of code and data."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the real contribution is MangaZero, a large annotated manga dataset, and a clean task definition for customized manga generation. DiffSensei itself is a sensible assembly of published pieces, and the qualitative results are convincing. The soft spot is evaluation evidence: automatic metric gains are tiny and lack error bars, and the human study is small. The deeper issue to probe is dataset annotation quality. Section 3.2 admits Magi's character IDs were low-accuracy and says humans refined them, but never quantifies residual error. Wrong source-target pairs would corrupt stage-2 training (Eqs. 5-6). That said, the external Manga109 evaluation and the ablation of the target-panel sampling rate (Tab. 4a) partially defuse this: the model transfers to unseen characters, and rate 0.0 still gets DINO-C 0.646, close to the 0.651 at rate 0.5. So it is a real reporting gap, not a fatal flaw.\n\nCredit where earned: MangaZero is 43k pages and 427k panels with character IDs, boxes, captions, and dialog boxes, and the authors commit to releasing it. The dialog embedding and masked cross-attention are low-overhead, practical choices that show clear qualitative gains. Using an MLLM as a multi-character identity adapter is not brand new, but the application with a diffusion loss is reasonable, and the appendix is honest about failure cases.\n\nMinor circularity: the MLLM's MSE target is extracted by the paper's own feature extractor, so stage 2 partly imitates its own encoder. Low stakes, because the diffusion loss is an external image-level objective and Manga109 is an external benchmark. For the revision, I'd ask for error bars or significance tests on Tables 2-3, and a residual-error estimate or human-verified subset for MangaZero IDs.\n\nThis paper is for anyone in story visualization, subject-driven generation, or manga analysis. I'd bring it to reading group and cite the dataset. A serious editor should send it to review, not desk reject, but with a clear request for stronger evaluation evidence. My guess is the task definition and dataset are the lasting contributions; the architecture will be seen as an early solid example.","headline":"MangaZero is the lasting contribution; DiffSensei works qualitatively, but thin quantitative evidence and unquantified character-ID residual error need addressing in review.","tokens_in":20683,"tokens_out":4065,"would_cite":true,"duration_ms":33975,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proposes customized manga generation and claims DiffSensei—a diffusion generator paired with a multimodal LLM adapter and masked layout attention—produces multi-character manga pages whose characters keep their identity while…","keywords":["customized manga generation","story visualization","character customization","multimodal large language model","diffusion model","masked cross-attention","layout control","MangaZero dataset"],"falsifier":"Retrain the stage-2 adapter on MangaZero using the raw, uncorrected Magi character IDs instead of the human-calibrated ones: if DINO-C and human character-consistency scores do not fall, the paper's reliance on human calibration is not load-bearing; if they fall, that reliance is confirmed. Alternatively, have human judges identify which reference character appears in DiffSensei-generated panels across several expressions and poses; if they frequently cannot, the claimed text-adaptable identity preservation fails.","tokens_in":19647,"feed_emoji":"✒️","tokens_out":10578,"duration_ms":94385,"temperature":0.7,"pith_summary":"The paper proposes a new task, customized manga generation: given reference images of characters, per-panel captions, and user-supplied layout boxes, the model must produce manga pages in which each character stays recognizable while changing expression, pose, or action to follow the story. DiffSensei couples a diffusion-based image generator with a multimodal large language model that acts as a text-compatible character adapter, so the caption, not just the reference image, decides how the character looks in that panel. Character positions are controlled by masked cross-attention rather than by copying pixels from the reference, and dialog placement is encoded as a trainable mask added to the latent, leaving the text itself to artists. The paper also introduces MangaZero, 43,264 manga pages and 427,147 panels with character IDs, boxes, captions, and dialog annotations, and reports that DiffSensei outperforms existing story-visualization and customization baselines on automatic metrics and human preference, including on Manga109 characters never seen in training. If the claims hold, this is a practical route to text-adaptable multi-character manga generation and a reusable dataset for the task.","feed_headline":"DiffSensei draws manga pages whose characters obey captions","feed_subtitle":"The same character smiles, fights, or kneels across panels while staying recognizable.","key_machinery":"The load-bearing mechanism is the two-stage coupling of an SDXL diffusion generator with an MLLM adapter plus masked layout attention. In stage one, character images are encoded by CLIP and a manga-specific Magi encoder, resampled into a small set of tokens, and injected through separate character cross-attention layers where a mask restricts each character's tokens to its bounding box ($M[i,j]=0$ if token $i$ lies in character $j$'s box, $-\\infty$ otherwise) while empty regions attend to the placeholder $q_{\\text{void}}$; a trainable dialog embedding is expanded and masked into the noisy latent. In stage two, the MLLM—initialized from SEED-X—reads the panel caption and source character features and outputs adapted features, trained with LM loss on special tokens, MSE loss to ground-truth target features, and a diffusion loss through the frozen generator; at inference the final features blend source and adapted features as $c'_i=(1-\\beta)c_i+\\beta\\hat{c}_i$ with $\\beta=0.4$. Masked cross-attention is what prevents pixel copy-paste while enforcing layout; the MLLM is what makes characters text-adaptable.","core_discovery":"On its own terms, the paper's discovery is that an MLLM can serve as the missing adapter between a text prompt and a diffusion generator for multi-character narrative images: instead of freezing a character to the pixel content of its reference image, the model converts character images into semantic tokens, lets the MLLM read those tokens together with the panel caption, and predicts target tokens that express the caption's requested state while preserving identity. The masked character cross-attention makes layout control cheap and explicit: each character's tokens attend only inside its bounding box, and character-free regions attend to a learnable placeholder token. Dialog boxes are encoded by adding a trainable embedding, masked to the dialog region, into the noisy latent, so the model learns where speech bubbles go without having to render readable dialogue. Two-stage training—first the generator with layout and dialog controls, then the MLLM adapter with language-model, mean-squared-error, and diffusion losses—is what lets the same character appear in different states across panels. The paper reports consistent wins over StoryDiffusion, AR-LDM, StoryGen, SEED-Story, and MS-Diffusion, and holds up on Manga109 characters the model never trained on.","pith_inferences":["Editorial inference: the same masked-attention-plus-MLLM-adapter recipe could transfer to other sequential narrative media—comics, webtoons, storyboards—since nothing except the Magi encoder and the manga-specific data ties it to black-and-white manga.","Editorial inference: because the dialog embedding controls placement only, a natural next step is to pair DiffSensei with a text-rendering model or a font-aligned generator that fills readable dialogue inside the predicted bubbles, which would remove the remaining human editing step.","Editorial inference: the paper's own admission that automatic character IDs were low-accuracy until human calibration suggests the dataset's ceiling is labeling cost; a scalable follow-up would test whether self-supervised character clustering across pages can replace most of the human correction without degrading the MLLM adapter.","Editorial inference: the reported $\\beta$ tradeoff between text alignment and identity preservation implies the framework exposes a tunable knob; an adaptive $\\beta$ that increases when the caption calls for a large pose change and decreases for close-up identity shots might beat the fixed 0.4 setting."],"forward_implications":["A user can hand DiffSensei several reference images, a story summary, and per-panel layout boxes, and receive a manga page whose characters keep their look while adopting the expressions, poses, and actions written in each panel caption.","Dialog control is spatial rather than textual: artists specify where speech bubbles go, the model fills in bubble placement, and the human edits the words, which sidesteps the poor long-text rendering of current text-to-image models.","The ablation results tie text adaptivity to the MLLM stage: removing it lowers CLIP text-alignment and DINO-C character similarity, so caption-driven changes are not an automatic property of the diffusion model.","Because Manga109 characters are never seen during training, the Manga109 evaluation results support the claim that DiffSensei generalizes to new, user-supplied characters rather than memorizing the 48 training series.","The MangaZero dataset, with 427,147 annotated panels, provides the multi-character, multi-state pairs needed to train and evaluate this task, and is released for the community."],"supporting_citations":[{"why":"Magi supplies the manga-understanding annotations—panel boxes, character boxes, character IDs, dialog boxes—that MangaZero is built on, and is also used to predict dialog boxes during evaluation.","marker":"[30]"},{"why":"LLaVA-v1.6-34B generates the panel captions that serve as text prompts in MangaZero and throughout training.","marker":"[22]"},{"why":"SDXL is the base diffusion generator that DiffSensei extends with character cross-attention and dialog embedding.","marker":"[26]"},{"why":"IP-Adapter-Plus-SDXL provides the feature-extractor initialization and the image-prompt cross-attention design that DiffSensei's masked character attention builds on.","marker":"[47]"},{"why":"SEED-X initializes the MLLM that acts as DiffSensei's text-compatible character feature adapter in stage two.","marker":"[8]"},{"why":"MS-Diffusion, re-trained on MangaZero with dialog embedding, is the strongest multi-subject customization baseline DiffSensei compares against for character preservation and layout control.","marker":"[39]"},{"why":"AR-LDM is a story-visualization baseline re-trained on MangaZero and included in the quantitative and human-preference comparisons.","marker":"[25]"},{"why":"StoryGen is a story-visualization baseline re-trained on MangaZero, used as a comparison point for caption-following and character control.","marker":"[21]"},{"why":"SEED-Story is a baseline that also uses an MLLM for story generation; the comparison shows what the added layout and character adapters contribute.","marker":"[46]"},{"why":"StoryDiffusion is the training-free story-consistency baseline; the comparison tests DiffSensei against self-attention sharing across panels.","marker":"[53]"}],"fun_headline_variants":["MLLM adapter lets manga characters act on caption cues","DiffSensei maps captions to character poses via token adapter","Custom manga: text-controlled characters without pixel transfer","Same character, new actions: DiffSensei reads panel context"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method depends on MangaZero's character-ID labels correctly pairing the same character across panels; the paper says the raw automatic IDs were low-accuracy before human correction, so any residual mismatches would make the stage-2 adapter learn from wrong character pairs and break identity preservation.","fun_headline_variants_meta":{"raw":{"variants":["MLLM adapter lets manga characters act on caption cues","DiffSensei maps captions to character poses via token adapter","Custom manga: text-controlled characters without pixel transfer","Same character, new actions: DiffSensei reads panel context"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000611,"raw_usage":{"total_tokens":2881,"prompt_tokens":1018,"completion_tokens":1863,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":634,"completion_tokens_details":{"reasoning_tokens":1796}},"tokens_in":634,"tokens_out":1863,"duration_ms":16026,"temperature":1.0,"reasoning_tokens":1796,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T18:40:35.254097+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the stage-2 adapter on MangaZero using the raw, uncorrected Magi character IDs instead of the human-calibrated ones: if DINO-C and human character-consistency scores do not fall, the paper's reliance on human calibration is not load-bearing; if they fall, that reliance is confirmed. Alternatively, have human judges identify which reference character appears in DiffSensei-generated panels across several expressions and poses; if they frequently cannot, the claimed text-adaptable identity preservation fails.","supporting_citations":[{"cited_title":"The manga whis- perer: Automatically generating transcriptions for comics","cited_arxiv_id":null,"evidence_quote":"Magi supplies the manga-understanding annotations—panel boxes, character boxes, character IDs, dialog boxes—that MangaZero is built on, and is also used to predict dialog boxes during evaluation."},{"cited_title":"Synthesizing coherent story with auto-regressive la- tent diffusion models","cited_arxiv_id":null,"evidence_quote":"AR-LDM is a story-visualization baseline re-trained on MangaZero and included in the quantitative and human-preference comparisons."},{"cited_title":"Intelligent grimm-open-ended vi- sual storytelling via latent diffusion models","cited_arxiv_id":null,"evidence_quote":"StoryGen is a story-visualization baseline re-trained on MangaZero, used as a comparison point for caption-following and character control."},{"cited_title":"page results.pdf","cited_arxiv_id":null,"evidence_quote":"StoryDiffusion is the training-free story-consistency baseline; the comparison tests DiffSensei against self-attention sharing across panels."}],"review_version":1}