{"id":"3ea5a93b-1bf3-4ab2-8661-07bfc61c6440","arxiv_id":"2501.06497","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"PASS is an LLM-powered pipeline that creates slides and audio from documents, and its GPT-4o version tops baseline methods on LLM-judged slide quality.","lead":"PASS is a software pipeline that generates presentation slides and an AI-voiced narration from a document. The paper reports that PASS slides score higher than existing methods on an LLM-based quality assessment, though the audio delivery itself was not evaluated.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim's delivery half is unsupported: §2.2's audio/presentation module is never evaluated, and §6 concedes human validation is still essential. This makes 'first to fully automate oral delivery' a stated capability, not a demonstrated one.","rationale":"I read the paper in good faith: the slide-generation pipeline is clearly described, the code is released, and the LLM-judge results give some evidence for the slide-quality sub-claim. However, the headline claim has two halves—content generation and oral delivery—and only the first is evaluated. The reader's weakest_assumption focuses on LLM-judge validity, which is a real but secondary concern: it affects the magnitude of the reported improvements, not the presence of the delivery module. The delivery module is the more load-bearing gap because the paper's claimed novelty is full automation of both. Crucially, Section 6 explicitly concedes that human evaluations are still essential to validate the slide presentation module, including audio delivery. That in-paper limitation directly undermines the central claim's second half. My proposed check is a concrete, feasible validation: run the released TTS pipeline on a sample, measure ASR intelligibility and slide-audio sync, and get human listener ratings. If that test passes, the concern is resolved and the claim becomes credible. If it fails, the verdict should move to REJECT for the central claim. Since the reader already issued CONDITIONAL, my concern reinforces that conditionality without changing the verdict—hence UNCHANGED. I do not see an internal inconsistency in the architecture itself, and I am not disputing that prior work may lack this combined pipeline; the issue is purely that 'fully automates delivery' has not been shown in the manuscript.","tokens_in":7253,"tokens_out":4691,"duration_ms":45199,"concrete_test":"Use the released code to run the full slide-presentation module on a representative subset of the SciDuet test set (e.g., 10 papers). Synthesize audio for each generated presenter script via the provided SpeechBrain/Tacotron-2 pipeline. Then (a) compute the word error rate of the synthesized speech against the input script using a standard ASR system, (b) measure the temporal alignment between slide boundaries and the corresponding audio segments, and (c) collect ratings from at least five human listeners on intelligibility and whether the audio could serve as a standalone oral delivery. If WER is high, sync fails, or listeners judge the audio unusable, the 'fully automates oral delivery' claim collapses; if the module passes these checks, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central contribution—'no prior work fully automates both the content generation and the delivery of a presentation using AI-generated voice' (Section 1)—requires the slide presentation module to actually deliver. However, the evaluation in Sections 3–4 covers only slide content quality (coherence, redundancy, relevance) scored by Llama-3-70B-Instruct on the SciDuet test set. No metric, ablation, or human rating is reported for either the Presenter Script Generator or the Audio Generation/TTS step. The paper itself flags this gap: Section 6 ('Future Work') states that 'conducting human evaluations would be essential to validate the effectiveness of the slide presentation module, particularly in generating high-quality audio delivery for the slides.' That is an in-paper admission that the delivery component—half of the claimed fully automated contribution—has not been validated. Even if the LLM-judge question were resolved, the central claim would remain unsupported because the oral-delivery module's output is never shown to be intelligible, correctly synchronized, or usable as a presentation. The issue is not an internal contradiction; it is a mismatch between the breadth of the claim and the scope of the evidence.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents PASS, a two-module pipeline that converts a document into a slide deck and an AI-generated voiceover. The slide-generation module extracts text and images, generates titles and content, summarizes the extracted content, and optionally maps images to slides; the slide-presentation module generates a presenter script and synthesizes speech using a Tacotron-2/SpeechBrain TTS system. On the SciDuet test set of 81 ICML/NeurIPS papers, the authors evaluate slide content with Llama-3-70B-Instruct as an LLM judge on coherence, redundancy, and relevance, reporting that GPT-PASS achieves the best average score of 8.96 ± 0.04, ahead of Qwen-PASS and several GPT-based baselines. The paper claims that PASS is the first system to fully automate both slide generation and oral delivery, and it presents the LLM-based evaluation metric as one of its contributions.","tokens_in":7574,"tokens_out":7020,"duration_ms":62777,"significance":"The intended contribution is practically useful: presentation preparation is a real bottleneck, and a modular pipeline with public code and evaluation against existing baselines is a reasonable starting point. The paper's strengths are its clear architecture, the flexible 8-10 slide design aimed at reducing redundancy, and the inclusion of comparisons against D2S and three GPT-based prompting baselines. If the delivery module and the LLM judge were validated, this would be a meaningful systems contribution. At present, however, the significance is conditional: the headline novelty claim rests on an unmeasured component, and the quantitative comparison rests on an unvalidated judge. The public GitHub repository is a plus for reproducibility.","major_comments":[{"comment":"The paper's central novelty claim in §1—'no prior work fully automates both the content generation and the delivery of a presentation using AI-generated voice'—is not supported by the evaluation that follows. Sections 3–4 evaluate only slide-text quality (coherence, redundancy, relevance) on SciDuet; the Presenter Script Generator and Audio Generation sub-modules described in §2.2 receive no evaluation. There is no measure of script quality, TTS intelligibility or naturalness, audio-slide synchronization, timing, or end-user assessment. The paper itself acknowledges this in §6: 'conducting human evaluations would be essential to validate the effectiveness of the slide presentation module, particularly in generating high-quality audio delivery for the slides.' As it stands, the delivery half of the central claim is a capability statement rather than a demonstrated result. Please either add an evaluation of the presentation module or narrow the claim to slide generation plus an implemented-but-unevaluated delivery architecture.","section":"§2.2, §3–§4, §6"},{"comment":"All quantitative comparisons depend on Llama-3-70B-Instruct as the evaluator, and the paper presents the LLM-based evaluation metric itself as contribution 4. The only justification offered is a citation to G-Eval (Liu et al., 2023) for 'very high correlation with human evaluations'; no human ratings are collected for slide decks, and no correlation, agreement, or calibration statistics are reported in the slide domain. Because both the produced slides and the judge are LLM outputs, the reported improvements could partly reflect judge preferences for LLM-style text. Please validate the judge on a human-annotated sample of slides (e.g., Spearman correlation and item-level agreement), or explicitly discuss this as an unvalidated proxy and soften the corresponding claims.","section":"§3, contribution 4"},{"comment":"The abstract and introduction repeatedly frame PASS as handling 'general Word documents,' but the only evaluation is on SciDuet, a dataset of 81 ICML/NeurIPS research papers. No non-academic or non-paper document is used, so the claim of generality beyond research papers is untested. The authors should either evaluate on additional document types (e.g., business reports, technical memos) or restrict the claim to research-style documents.","section":"Abstract, §1, §3"},{"comment":"Contribution 2 is a 'novel image mapping module,' but the experiments do not evaluate image mapping at all. Table 1 measures only text-level coherence, redundancy, and relevance of slide content; there is no accuracy, relevance, or usability metric for the mapped images, and the appendix does not describe an image-mapping evaluation. If image mapping is claimed as a contribution, it needs at least a basic evaluation; otherwise it should be presented as a designed component rather than a demonstrated contribution.","section":"§2.1, §3–§4, Table 1"},{"comment":"The conclusion states that PASS 'significantly outperforms' existing methods, but no statistical significance test is reported. The reported means differ by roughly 0.2–0.7 points with small standard errors, yet no pairwise test, confidence interval, or multiple-comparison correction is provided. Please report appropriate significance tests or replace 'significantly' with 'numerically higher' in the conclusion.","section":"§4.4, §5"}],"minor_comments":[{"comment":"There are several copyediting issues: 'comprises of' should be 'comprises' or 'consists of,' 'upto' should be 'up to,' and 'multimodel' is used where 'multimodal' is meant.","section":"§2"},{"comment":"The sentence 'as used by (Liu et al., 2023) which has a very high correlation with human evaluations' is grammatically awkward; consider rephrasing to make clear that the cited high correlation comes from G-Eval's general NLG experiments, not from slide evaluation.","section":"§3"},{"comment":"The evaluation section does not state how many slide decks were generated per document or whether scores are averaged over slides or over decks; please clarify the unit of analysis.","section":"§3"},{"comment":"The Qwen et al. reference contains a stray ':' among the author names; please fix the formatting.","section":"References"},{"comment":"Table 2 presents a single-slide content comparison without explaining how that example was selected; a sentence on the selection criterion would improve transparency.","section":"Appendix, Table 2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within scope for a cs.CL systems paper, and the core architecture is reasonable. The main weakness is a mismatch between the breadth of the claims and the evidence: the delivery module is unevaluated, the LLM judge is unvalidated for slides, and the generalization to non-research documents is untested. These issues are fixable within the manuscript's scope by adding targeted evaluations or by scoping the claims more carefully. I would be willing to review a revised version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The takeaway: this is a competent but incremental pipeline paper. The slide-generation half is plausible and reasonably evaluated; the oral-delivery half of the central claim is essentially unevaluated, and the whole evaluation rests on an LLM judge with no human validation. So treat the headline claim as a stated capability, not a demonstrated one.\n\nWhat's new: the specific integration of LLM slide generation with Tacotron-2 TTS, plus an image-mapping step for multimodal models. The code and data are released, the pipeline is described clearly, and the comparisons against D2S and GPT-prompting baselines are a fair starting point. The flexible 8-10 slide count is a sensible fix for the redundancy problem noted in prior work. Credit where due: the paper is honest enough to say in Section 6 that human evaluation 'would be essential' for the presentation module.\n\nThe soft spots are real. The audio module gets no evaluation at all: no intelligibility check, no sync check, no listening test. The paper's own future-work sentence admits that. Second, the LLM judge (Llama-3-70B) is not validated for slide quality; citing G-Eval's general NLG correlation is not sufficient, and the judge is the same family of model as the generator, so there's a residual circularity. Third, the evaluation is on SciDuet (research papers), so the claim of handling general Word documents is untested. These are not fatal to the slide-generation sub-claim, but they are load-bearing for the 'fully automates delivery' claim, and that claim currently rests on a module that is described but never shown working.\n\nWho this is for: anyone building document-to-slide systems or doing applied LLM evaluation will find the prompts and relative numbers useful. A serious referee should engage with it and ask for human eval, audio eval, and a validation of the judge. It's not a desk reject; it's a revise-and-resubmit.","headline":"Competent slide-generation pipeline, but the oral-delivery half of the central claim is unevaluated and the LLM-based metric lacks human validation.","tokens_in":7984,"tokens_out":2119,"would_cite":false,"duration_ms":20045,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"PASS is the first pipeline that both generates presentation slides from Word documents and delivers them through an AI-generated voice, scoring higher than prior methods on an LLM-judged quality metric.","keywords":["slide generation","text-to-speech","LLM evaluation","presentation automation","document-to-slide","speech synthesis","multimodal models","SciDuet"],"falsifier":"Run a human evaluation on a sample of SciDuet outputs from GPT-PASS and the baselines, asking raters to score the same three dimensions; if human rankings differ from Llama-3-70B's or the absolute scores diverge sharply, the paper's central performance claim loses its support.","tokens_in":7030,"feed_emoji":"🎤","tokens_out":9545,"duration_ms":71330,"temperature":0.7,"pith_summary":"PASS is a pipeline that takes ordinary Word documents and produces a finished presentation: slide titles, bullet content, and an AI-generated spoken narration synchronized with the slides. The paper claims this is the first system to fully automate both content generation and oral delivery, going beyond earlier work that focused on research papers or fixed slide counts. On the SciDuet benchmark, slides made with PASS using GPT-4o scored 8.96 ± 0.04 on an LLM-based evaluation of coherence, relevance, and redundancy, ahead of the D2S, GPT-Flat, GPT-COT, and GPT-Cons baselines. The point of the claim is that presentation creation can become a single automated step, saving preparation and rehearsal time.","feed_headline":"PASS builds slides and a voiceover, outscoring prior slide systems","feed_subtitle":"Word documents become complete slide decks with AI voiceover, rated higher on coherence, relevance, and redundancy.","key_machinery":"The load-bearing object is the two-module PASS architecture. The generation module chains five sub-modules — image/text extraction, title generation, content extraction, summarization, and image mapping — with prompts that allow 8-10 slides instead of a fixed number, which the paper argues is what prevents redundant content. The presentation module adds a presenter script generator and a Tacotron-2, SpeechBrain-based audio generator. Evaluation rests on an LLM judge (Llama-3-70B-Instruct) that scores slides on coherence, redundancy, and relevance, following the G-Eval approach.","core_discovery":"The central claim is that PASS is the first pipeline to automate both slide generation and presentation delivery. The slide generation module extracts text and images from a document, generates up to 8-10 slide titles, pulls relevant content for each title, summarizes that content into concise points, and maps document images to matching slides when a multimodal model is used. The slide presentation module then writes a presenter script for each slide and converts it to speech with a Tacotron-2 based text-to-speech system. On the SciDuet test set of 81 ICML and NeurIPS papers, GPT-PASS (PASS with GPT-4o) achieved an average score of 8.96 ± 0.04 across coherence, redundancy, and relevance, the highest among the compared methods, with Qwen-PASS close behind. The paper also proposes an LLM-based evaluation metric using Llama-3-70B-Instruct to judge these three quality dimensions.","pith_inferences":["A natural testable extension would be replacing the LLM judge with human raters on the same SciDuet samples to check whether the 8.96 average reflects perceived quality.","The same pipeline design could be applied to other input formats, such as PDFs or web pages, since the title-generation and content-extraction steps are not paper-specific.","The claimed firstness depends on how 'delivery' is scoped; a broader search including commercial slide tools would be needed to confirm that no prior system combines slide generation with an AI voiceover.","Integrating retrieval-augmented generation, as the paper lists for future work, could turn the one-way voiceover into an interactive presenter that answers audience questions."],"forward_implications":["If the claim holds, preparing a presentation from a Word document becomes a single automated step, with no separate slide design or narration recording.","General documents, not just structured research papers, can be converted into presentations.","Flexible slide counts (8-10) reduce the content repetition that fixed-slide systems suffer from.","Adding a spoken delivery module means the output is a complete, viewable presentation rather than just a deck.","The LLM-based evaluation metric could be reused to compare future slide-generation systems on the same three dimensions."],"supporting_citations":[{"why":"Supplies the SciDuet test dataset of 81 research papers and the D2S baseline model used for comparison.","marker":"(Sun et al., 2021)"},{"why":"G-Eval method motivating the LLM-based evaluation with claimed high correlation with human judgments.","marker":"(Liu et al., 2023)"},{"why":"Llama 3, the model family from which the LLM judge Llama-3-70B-Instruct is drawn.","marker":"(Grattafiori et al., 2024)"},{"why":"GPT-4o, the multimodal model used in GPT-PASS and in the GPT baselines for slide generation.","marker":"(OpenAI et al., 2024)"},{"why":"Qwen2.5-32B-Instruct, the open-source LLM used in the Qwen-PASS variant.","marker":"(Qwen et al., 2025)"},{"why":"Tacotron-2, the neural text-to-speech architecture used for audio synthesis.","marker":"(Shen et al., 2018)"},{"why":"SpeechBrain, the toolkit providing the Tacotron-2 implementation used in the presentation module.","marker":"(Ravanelli et al., 2021)"},{"why":"Multi-stage LLM approach whose experimental setup and baselines the paper follows.","marker":"(Bandyopadhyay et al., 2024)"},{"why":"Earlier LLM slide generation work whose content-overlap challenge motivates PASS's flexible 8-10 slide design.","marker":"(Mondal et al., 2024)"}],"fun_headline_variants":["PASS auto-builds slides and AI voiceover from any Word doc","PASS creates slides and narrates them, beating prior systems","PASS: one pipeline from Word doc to delivered presentation","PASS auto-generates slide decks and speaks them aloud","PASS: slides and speech from documents, topping quality metrics"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole quality comparison assumes the AI judge's scores match what people would think of the slides, but the paper never tests that on slides.","fun_headline_variants_meta":{"raw":{"variants":["PASS auto-builds slides and AI voiceover from any Word doc","PASS creates slides and narrates them, beating prior systems","PASS: one pipeline from Word doc to delivered presentation","PASS auto-generates slide decks and speaks them aloud","PASS: slides and speech from documents, topping quality metrics"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001017,"raw_usage":{"total_tokens":4297,"prompt_tokens":953,"completion_tokens":3344,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":569,"completion_tokens_details":{"reasoning_tokens":3258}},"tokens_in":569,"tokens_out":3344,"duration_ms":22690,"temperature":1.0,"reasoning_tokens":3258,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:58:18.661097+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a human evaluation on a sample of SciDuet outputs from GPT-PASS and the baselines, asking raters to score the same three dimensions; if human rankings differ from Llama-3-70B's or the absolute scores diverge sharply, the paper's central performance claim loses its support.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Earlier LLM slide generation work whose content-overlap challenge motivates PASS's flexible 8-10 slide design."}],"review_version":1}