{"id":"fb1e1892-d28c-45ee-8f3e-900169fbb7db","arxiv_id":"2505.21660","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"PreGenie combines text summarization, image captioning, code generation, and visual page review in an agentic MLLM pipeline to produce editable Slidev presentations from multimodal documents.","lead":"PreGenie is a multi-agent system that turns text-and-image documents into slide decks by generating Slidev Markdown and then using both code review and visual page review to fix layout problems. The paper reports that the visual review step improves design quality and consistency compared with earlier presentation generators, but the evidence is based on a small, uneven evaluation.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The core human-preference claim in §4.3 is not statistically secured: 20 raters on 10 decks, no reported blinding or significance tests, and Table 1 does not independently corroborate the content-consistency half of the claim.","rationale":"I read the paper as an engineering contribution: a two-stage agentic pipeline on Slidev, with a code-review loop and a VLM-based page-review loop. The method description is unusually concrete, the prompts in Appendix A are detailed enough to reimplement, and the ablation dropping code review shows a large success-rate drop (91.26 vs 58.72), which is genuine internal evidence that the code-review mechanism does something. The page-review examples in Fig. 6 are plausible and the CLIP/LongCLIP gains are consistent with the claimed visual improvement. These facts make me unwilling to reject the framework.\n\nThe weak point is not the mechanism but the evidence for the headline claim. For the abstract's sentence to be true, the evaluation must reliably separate PreGenie from the baselines. The human and GPT-4o study in §4.3 is the only quantitative evidence about human design preferences, and it is too small and too lightly controlled to carry that load. Table 1 points in a related direction: text-similarity metrics favor KCTV, and the page-review loop does not improve Rough-L/Coverage, so the content-consistency part of the superiority claim is not independently supported by the numbers. The AutoPresent comparison has an additional confound, since the baseline is effectively instructed to reproduce a description of PreGenie's output.\n\nThe limitation section also narrows the claim: it admits that MLLM hallucination can produce unrelated content and that chart/graph understanding is weak, which further tempers the phrase 'excels in multimodal understanding.' None of this makes the framework uninteresting; it means the paper should be accepted conditional on artifact release and a stronger, blinded, statistically analyzed evaluation. Because this is exactly the reader's stated condition, I would keep the verdict unchanged as a conditional acceptance.","tokens_in":14483,"tokens_out":9521,"duration_ms":102924,"concrete_test":"Obtain the per-rater, per-deck scores from the Fig. 7 protocol, or independently collect 20 new raters scoring the same 10 slide pairs with method labels hidden and presentation order counterbalanced. Run a one-sided paired permutation test for each of the four dimensions, with Holm correction across dimensions, and report 95% bootstrap confidence intervals for the mean differences. If the Page Design and Page Consistency advantages over PPTAgent do not survive both the significance test and the blinded replication, the sentence in the abstract claiming closer alignment with human design preferences should be softened to a small, non-quantitative observation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that PreGenie outperforms existing models in aesthetics and content consistency while aligning more closely with human design preferences—is carried almost entirely by the evaluation in §4.3, not by the framework mechanics. Three features of that evaluation are load-bearing. First, the human-preference result (Fig. 7) uses 20 raters and 10 slide sets without specifying whether raters were blinded to method, without reporting confidence intervals, and without a significance test. The point differences favor PreGenie on Page Design (7.1 vs 6.2) and Page Consistency (8.4 vs 6.6), but the reported numbers alone cannot rule out rater expectation, ordering artifacts, or small-sample noise. GPT-4o's margins are smaller, so the 'human design preference' reading depends entirely on the human numbers being valid. Second, Table 1 does not corroborate content consistency: PreGenie is below KCTV on Rough-L (21.95 vs 25.67) and Coverage (27.70 vs 33.82), and the no-page-review ablation is numerically higher on both metrics (22.16 and 28.08) than the full pipeline. The page-review loop improves CLIP/LongCLIP scores, but those measure text-image relevance, not text fidelity. Third, the AutoPresent qualitative comparison in §4.2 is not a like-for-like task: AutoPresent is prompted with a GPT-4o-written description of PreGenie's rendered slide rather than the source document, so the comparison measures prompt-following on a target layout more than independent presentation design. The framework is clearly specified and the code-review ablation is suggestive, but the abstract's broad superiority claim is stronger than the evidence actually reported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces PreGenie, an agentic framework for generating multi-slide presentations from multimodal documents using a Slidev Markdown intermediate representation. The pipeline has two stages: an analysis/initial-generation stage with text summarization and image captioning, and an iterative review stage that alternates code review (LLM) with rendered-page review (VLM) until slides pass. The authors report qualitative comparisons with PPTAgent, KCTV, and AutoPresent; quantitative metrics including Rough-L, Coverage, CLIP/LongClip, success rate, and figure proportion; and a 20-rater human plus GPT-4o evaluation on 10 slide sets. The central claim is that PreGenie outperforms existing models in aesthetics and content consistency and aligns more closely with human design preferences.","tokens_in":14684,"tokens_out":5108,"duration_ms":48346,"significance":"If the empirical claims were fully supported, PreGenie would be a useful practical contribution: it demonstrates a modular way to close the gap between intermediate code and rendered slides via visual page review, and it reports the prompts used and a computational cost breakdown. The framework itself is not mathematically novel, but the combination of code review and page review for presentation generation is a reasonable engineering contribution. However, the current evaluation is not strong enough to establish the headline claims: the human study is small and statistically unsecured, the quantitative table shows mixed signs on text fidelity, and one baseline comparison is not like-for-like. With a strengthened evaluation, the contribution could be significant for applied LLM-agent systems.","major_comments":[{"comment":"The human-preference result is not statistically secured: 20 raters scored 10 slide sets, and the paper reports neither blinding of raters to method, nor confidence intervals, nor any significance test. The observed advantages in Page Design (7.1 vs 6.2) and Page Consistency (8.4 vs 6.6) could plausibly arise from ordering effects or small-sample noise, so they cannot by themselves support the abstract's claim of 'aligning more closely with human design preferences.' Please report per-rater statistics, confidence intervals or significance tests, and a description of the evaluation protocol (blinding, randomization, slide ordering), or substantially increase the evaluation scale.","section":"§4.3, Fig. 7"},{"comment":"The claim that PreGenie excels in 'content consistency' is not corroborated by the text-similarity columns of Table 1: PreGenie scores below KCTV on Rough-L (21.95 vs 25.67) and Coverage (27.70 vs 33.82), and the no-page-review ablation is numerically higher on both metrics (22.16 and 28.08) than the full pipeline. The page-review loop improves CLIP/LongClip scores, but those measure text-image relevance, not text fidelity. Please either soften the content-consistency claim to text-image relevance, or provide additional metrics (e.g., semantic fidelity against the source) that show the full pipeline improves content consistency.","section":"§4.3, Table 1"},{"comment":"The comparison with AutoPresent is not a like-for-like evaluation of presentation generation. AutoPresent is prompted with a GPT-4o-written description of PreGenie's rendered slide rather than with the original source document, so the exercise measures how well AutoPresent follows a target layout description, not how well it independently designs a slide from the source content. Please redesign this baseline to receive the same document input, or reframe Figure 5 as an illustration of layout-transfer capability rather than as evidence of overall superiority.","section":"§4.2, Fig. 5"},{"comment":"The evaluation corpus is a hand-picked subset of 200 DOC2PPT documents 'rich in images, tables, and other content,' and the human study uses a further 10 sets drawn from that subset. No sampling criteria or inclusion/exclusion rules are reported, and 10 slide sets is very small for the generality of the claimed conclusions. Please document the sampling procedure, report the number of slides per set, and ideally validate on a random or stratified sample to support generalization claims.","section":"§4.1 and §4.3"}],"minor_comments":[{"comment":"The sentence 'We compare our method with For both our approach and PPTAgent, we employed...' is grammatically incomplete; it should be split into a clear statement of which methods are compared and how each baseline was run.","section":"§4.1"},{"comment":"The word 'metircs' is a typo for 'metrics.'","section":"§4.3"},{"comment":"Figure 7 is difficult to read because scores are only shown as bars without numerical labels or error bars; a table with means, standard deviations, and sample sizes would be more informative.","section":"Fig. 7"},{"comment":"The relationship between the human evaluation and the GPT-4o evaluation is unclear: it is not stated whether the same question template and scoring rubric were validated against each other, or whether GPT-4o judgments were calibrated to human judgments.","section":"§4.3"},{"comment":"The paper does not state whether the code and the evaluation data will be released; providing the framework prompts (already in Appendix A) alongside code would materially improve reproducibility.","section":"§4.3"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a reasonable systems paper, but the headline claims are substantially stronger than the evidence. I would encourage a major revision with a statistically solid human study, corrected baselines, and a more precise statement of what the page-review loop improves. The paper does not appear to misrepresent prior work or conceal negative results; the limitations section is candid. The main risk is the gap between the abstract's 'comprehensive experiments' and the actual evaluation scale."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a well-specified engineering paper with a genuinely useful idea — adding a VLM page-review loop that inspects rendered slides and feeds layout feedback back into the code generator. The prompts in the appendix are detailed enough to reproduce the pipeline, and the ablation shows the page reviewer adds value on text-image relevance metrics (CLIP/LongCLIP) and the code reviewer improves success rate. That part is credible and the clearest contribution.\n\nThe soft spot is the evaluation. The abstract claims superiority in aesthetics and content consistency, but the automatic metrics do not support the content-consistency half: PreGenie is below KCTV on Rough-L (21.95 vs 25.67) and Coverage (27.70 vs 33.82), and below its own no-page-review variant on those same numbers (22.16 and 28.08). The human study is 20 raters on 10 slide sets, with no reported blinding, no confidence intervals, and no significance tests. The point differences in Fig. 7, which favor PreGenie on Page Design and Page Consistency, could easily be noise. The AutoPresent comparison in §4.2 is not like-for-like: AutoPresent is prompted with a GPT-4o description of PreGenie's rendered slide, so it measures prompt-following on a target layout rather than independent design. The paper's own limitations section concedes MLLM hallucination and weak chart understanding, which narrows the practical claim.\n\nThe framework is plausible, but the evidence for outperforming existing models is not there yet. I would want a larger, blinded human evaluation with significance testing, a more honest framing of the automatic metrics, and a release of code or data for reproducibility.\n\nBottom line: this is worth engaging with. The page-review idea is transferable to other visual generation agents, and the implementation is clearly described. It deserves a serious referee, but the superiority claim needs to be dialed back or supported with stronger evidence.","headline":"PreGenie is a clearly specified agentic pipeline adding a VLM page-review loop to code-generated slides; the framework is plausible and the ablation is suggestive, but the headline 'outperforms' claim rests on a small, under-powered human study and mixed automatic metrics.","tokens_in":15355,"tokens_out":1859,"would_cite":true,"duration_ms":19786,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"PreGenie's two-stage pipeline—text/image analysis plus an iterative code-and-page review loop—generates presentations that outperform existing systems in aesthetic layout and content consistency.","keywords":["visual presentation generation","multimodal large language models","agentic framework","Slidev","page review","layout aesthetics","content consistency","document-to-slide"],"falsifier":"A blind, pre-registered study in which 50 designers rate decks from PreGenie and PPTAgent generated from the same 50 documents, with identities hidden and significance tests, would settle the claim; if the two systems tie on Page Design and Page Consistency under blinding, the visual page-review loop would not have demonstrated a real advantage.","tokens_in":14186,"feed_emoji":"🎨","tokens_out":6845,"duration_ms":63716,"temperature":0.7,"pith_summary":"The paper tries to establish that a single framework can turn a text-and-image document into a polished, multi-page slide deck without losing content accuracy or visual quality. Previous code-based generators often produce slides whose layouts break during rendering, and previous image-generating systems cannot handle complex documents. PreGenie's answer is to generate slides as Slidev Markdown, then run two review loops: an LLM checks the code, and a vision-language model inspects each rendered page and sends layout fixes back to the generator. The paper reports that this visual page-review step closes the gap between code and final appearance, yielding better Page Design and Page Consistency scores than PPTAgent, KCTV, and AutoPresent in human and GPT-4o evaluations. If true, this would make automated presentation generation practical for formal business and research contexts.","feed_headline":"Visual self-review lifts AI slide quality past PPTAgent and KCTV","feed_subtitle":"PreGenie's page-review loop catches overflow and layout errors that code-only systems miss, with better human and GPT-4o ratings.","key_machinery":"The load-bearing mechanism is the two-stage agent loop built around the Slidev Markdown presentation format. Five MLLM agents share a common context: a Text Summarizer, an Image Captioner, a Code Generator, a Code Reviewer, and a Page Reviewer. Stage 1 analyzes the input and produces initial Slidev code; Stage 2 iterates between code review (syntax, content alignment) and page review (rendered-slide inspection for overflow, imbalance, font inconsistency), regenerating slides that fail either check. The Page Reviewer is the novel component: because Slidev renders Markdown deterministically, the VLM can compare the intended layout against the actual pixels and issue concrete, content-preserving corrections such as resizing an overflowing image, switching a column layout, or reformatting crowded text.","core_discovery":"On its own terms, PreGenie's central claim is that the missing ingredient in automatic slide generation is visual self-inspection: an LLM writes Slidev Markdown that renders to slides, an LLM code reviewer catches syntax and content errors, and a VLM page reviewer looks at the rendered pages to catch overflow, misalignment, crowding, and inconsistent fonts that code inspection cannot see. The feedback from both reviewers is fed back into the code generator, which regenerates the problematic slides. The paper argues that this division of labor—multimodal summarization and captioning up front, code-level review, then pixel-level review—is what lets a single framework handle multimodal documents while staying faithful to the source text and images. The reported evaluations on 200 DOC2PPT samples show PreGenie matching or exceeding PPTAgent on text similarity, exceeding it on text-image relevance, and clearly winning on Page Design and Page Consistency in both human and GPT-4o scores.","pith_inferences":["A natural extension is to replace the fixed iteration count with an automatic pass/fail threshold based on page-review confidence, which would cut the generation time that currently comes from repeated VLM calls.","The page-review step could be turned into a training signal: rendered-slide mistakes found by the VLM could be used to fine-tune the code generator, turning a run-time loop into a learned one-shot improvement.","The evaluation's reliance on 10 slide sets and unblinded human raters means the claimed margin over PPTAgent is plausible but not yet pinned down; a larger, pre-registered human study with significance testing would be the obvious next check.","The same visual-review mechanism could also be used to evaluate user-supplied documents for image–text mismatches before generation begins, catching source-side problems rather than only slide-side ones."],"forward_implications":["If PreGenie works as reported, document-to-slide automation becomes reliable enough for formal use, since the page-review loop catches layout failures that code-only pipelines miss.","The framework's modularity means swapping in stronger LLMs and VLMs should translate directly into better summarization, captioning, and visual judgment without changing the pipeline.","The same code-plus-visual-review recipe could extend to other structured multimodal outputs, such as posters, infographics, or web pages, where the gap between intermediate code and final rendering is the hard part.","Because Slidev output is editable Markdown, users can post-edit the generated deck by hand, a capability that prior direct-image-generation methods do not support."],"supporting_citations":[{"why":"Supplies PPTAgent, the main multi-page text-image baseline that PreGenie extends with a visual review loop.","marker":"(Zheng et al., 2025)"},{"why":"Supplies KCTV, the template-based plain-text baseline that PreGenie compares against on multi-page generation.","marker":"(Cachola et al., 2024)"},{"why":"Supplies AutoPresent, the single-page structured-visual baseline used for design-quality comparison.","marker":"(Ge et al., 2025)"},{"why":"Provides the DOC2PPT dataset from which the 200 evaluation samples were selected.","marker":"(Fu et al., 2022)"},{"why":"Provides Qwen2.5-VL, the VLM backbone powering the Image Captioner and Page Reviewer.","marker":"(Bai et al., 2025)"},{"why":"Provides GPT-4o, used both for slide-description prompts and as the model-based evaluator in the user study.","marker":"(Hurst et al., 2024)"},{"why":"Provides CLIP, one of the text-image relevance metrics used in the quantitative comparison.","marker":"(Radford et al., 2021)"},{"why":"Provides Long-CLIP, the long-text text-image relevance metric where PreGenie reports its largest advantage.","marker":"(Zhang et al., 2024a)"}],"fun_headline_variants":["PreGenie's visual self-review beats PPTAgent and KCTV on slide quality","AI slides get better when a visual reviewer checks the render","PreGenie: visual page review loop cuts overflow and layout errors","Two-stage AI framework uses visual feedback to outdo slide generators"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The superiority claim rests on a small, unblinded human and GPT-4o evaluation over 10 slide decks, so if those ratings do not reliably reflect real-world design preferences, the headline advantage could disappear even though the framework works.","fun_headline_variants_meta":{"raw":{"variants":["PreGenie's visual self-review beats PPTAgent and KCTV on slide quality","AI slides get better when a visual reviewer checks the render","PreGenie: visual page review loop cuts overflow and layout errors","Two-stage AI framework uses visual feedback to outdo slide generators"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000388,"raw_usage":{"total_tokens":2036,"prompt_tokens":926,"completion_tokens":1110,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":542,"completion_tokens_details":{"reasoning_tokens":1035}},"tokens_in":542,"tokens_out":1110,"duration_ms":9193,"temperature":1.0,"reasoning_tokens":1035,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:25:15.261267+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A blind, pre-registered study in which 50 designers rate decks from PreGenie and PPTAgent generated from the same 50 documents, with identities hidden and significance tests, would settle the claim; if the two systems tie on Page Design and Page Consistency under blinding, the visual page-review loop would not have demonstrated a real advantage.","supporting_citations":[],"review_version":1}