{"id":"2a6a1257-fec5-4506-862a-c19ccce7498e","arxiv_id":"2411.10687","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"EDBook combines structured dialogic narratives with open-ended LLM queries to make programming tutorials more engaging and interactive.","lead":"The paper introduces EDBook, a programming tutorial platform that combines pre-written dialog trees with context-aware LLM queries. A 20-participant study found readers spent more time on the material, attempted more coding exercises, and reported higher engagement compared with a conventional webpage plus ChatGPT, though test scores did not improve.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Evaluation confound: EDBook vs Web+ChatGPT varies dialogic structure plus integrated editor, local-context prompting, and reveal-as-you-go interaction, so observed engagement and quiz-attempt gains cannot be cleanly attributed to the claimed dialogic+LLM integration.","rationale":"The reader's weakest assumption identifies the same load-bearing issue: the two experimental conditions differ along multiple dimensions beyond the dialogic/LLM integration. This is the most important soft spot because the paper's positive empirical claims—longer time on materials, more quiz attempts, higher engagement—are all susceptible to the all-in-one interface and local-context confounds. The paper is otherwise a well-described systems paper with a transparent implementation section, a clear data representation, and candid limitations (Sections 4.6 and 6). The authors do not overclaim learning gains, and the qualitative findings are plausibly useful for future design. However, the central assertion that the dialogic+LLM integration enables the observed benefits is not cleanly established by the current study. The verdict of CONDITIONAL remains appropriate: the system is a plausible contribution, but acceptance should require a stronger control or reanalysis that isolates the dialogic structure from the integrated environment. I therefore leave the reader's verdict unchanged rather than escalating to REJECT or UNVERDICTED, because the design contribution is concrete and the weaknesses are in the evaluation's attribution, not in the internal coherence of the system.","tokens_in":20472,"tokens_out":7910,"duration_ms":87758,"concrete_test":"Run an additional control condition using the same EDBook content rendered as a linear webpage but with an equivalent integrated code editor, quiz widgets, and local-context chat, or a 2x2 factorial crossing dialogic structure (yes/no) with integrated environment (yes/no). The key comparison is EDBook versus the linear-but-integrated condition: if engagement and quiz-attempt increases persist when the dialogic structure is removed, the effects cannot be attributed to the dialogic+LLM integration claimed as the central contribution. This would settle whether the headline findings are due to the novel integration or to the all-in-one interface.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical support is underdetermined by the design of the comparative evaluation (Section 4.3, Figure 6). The two conditions are intended to isolate dialogic content plus context-informed LLM interaction, but they also differ in at least four ways: (1) EDBook has an integrated code editor, quiz widgets, and diff views, whereas the Web+ChatGPT condition requires switching between a webpage and separate tools; (2) EDBook automatically supplies local context from the current dialog node and ancestors to the LLM, whereas Web+ChatGPT uses only a global page-level context; (3) EDBook reveals content through interaction, whereas a webpage is scrollable without interaction; and (4) EDBook is an unfamiliar new tool, which may itself inflate time and engagement measures. The significant differences in time on materials, quiz/code attempts, and self-reported engagement (Section 4.4, Table 2; Section 4.5.6) are consistent with any of these factors. The paper partially acknowledges this in Section 4.5.1, noting that EDBooks require user interaction to reveal content and that the platform integrates a code authoring and testing environment, but it does not provide evidence that the dialogic+LLM integration is the operative factor. Since the central claim is that this integration enables the observed benefits, the evaluation leaves the attribution unresolved. The absence of significant learning gains and the uncorrected multiple comparisons further weaken the positive findings.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces EDBook, a Visual Studio Code-based platform for interactive programming tutorials that combines author-authored dialog trees with context-aware large language model (LLM) interactions. The design aims to give learners the pedagogical structure and verified content of traditional e-books while allowing open-ended personalized questions to an LLM, with additional features such as incremental code diffs, deictic code pointers, and embedded multiple-choice and code-writing assessments. The authors report a within-subjects user study with 20 participants, two learning tasks (Scheme basics and Python web-request caching), and two conditions: EDBook and a comparable webpage supplemented with a context-informed ChatGPT. Their quantitative results show significantly more time spent on learning materials and more attempts at quiz/code exercises in the EDBook condition, as well as higher self-reported engagement, while assessment scores were not significantly different. Interview data suggest that participants valued the local context provided to the LLM and the integrated coding environment, though some found the dialogic content 'fluffy' and preferred the webpage for skimming and searching. The paper's central claim is that EDBook is the first platform to integrate pre-written dialogic narratives with open-ended context-informed LLMs, enabling dialogic learning that is open-ended yet goal-aligned.","tokens_in":20718,"tokens_out":4522,"duration_ms":43963,"significance":"If the design claim is taken at face value, EDBook makes a useful contribution to programming education by addressing a real limitation of open-ended LLM tutoring: novices often do not know what to ask, and LLM responses may drift from pedagogical goals. The system's representation of dialogs as trees with a target node, its separation of user state from page content, and its integration of code execution and quizzes are thoughtful design choices that could influence future interactive textbook platforms. The study also has strengths: a within-subjects Latin square design, two varied learning tasks, manual coding of LLM questions using Bloom's taxonomy, and honest discussion of limitations in Sections 4.6 and 6. The source code and a deployment link are provided, which supports reproducibility. However, the empirical support for the central attribution claim is limited because the two experimental conditions differ in multiple dimensions beyond the dialogic/LLM integration, and the survey results are reported without full descriptive statistics.","major_comments":[{"comment":"The comparison between C_EDBook and C_Web+GPT does not isolate the dialogic-narrative-plus-LLM integration that the paper claims is the central innovation. The conditions vary in at least four ways: (1) EDBook integrates a code editor, quiz widgets, and diff views into a single environment, while the webpage condition requires switching between tools; (2) EDBook automatically provides local dialog context to the LLM, while the webpage uses only a global page-level context; (3) EDBook reveals content through interaction, while the webpage is scrollable without interaction; and (4) EDBook is a novel, unfamiliar tool whose novelty may itself inflate time and engagement measures. The significant differences in time on materials, quiz/code attempts, and self-reported engagement (Table 2, Section 4.4.3, Section 4.5.6) are consistent with any of these factors. The authors partially acknowledge this in Sections 4.5.1 and 4.5.2, but they do not provide evidence that the dialogic/LLM integration is the operative factor. Since the paper's central claim is that this integration enables open-ended yet goal-aligned learning, the evaluation leaves the attribution unresolved.","section":"§4.3, §4.4, §4.5.1, §4.5.2"},{"comment":"The statistical reporting for self-report measures is incomplete. Table 4 lists questionnaire statements but does not present means, standard deviations, test statistics, or corrected p-values for any item, despite the text stating that two items reached p < 0.05. With 20 participants and a large number of Likert items examined by ANOVA, uncorrected multiple comparisons make these positive results fragile. Please report full descriptive statistics and either apply a multiple-comparison correction or explicitly label the questionnaire analyses as exploratory.","section":"§4.4.3, Table 4"},{"comment":"The novelty claim that EDBook is 'the first platform to integrate pre-written dialogic narratives with open-ended context-informed LLMs' is repeatedly asserted but not established by a systematic comparison. The related work discusses adjacent systems such as Graphologue, Sensecape, chat.codes, Callisto, Colaroid, and Torii, and the authors state 'As far as we are aware' in Section 3.3, but the abstract and the contribution bullet in Section 1 use unqualified 'first' language. Please either soften these claims to match the evidence or provide a more comprehensive literature search to justify the uniqueness claim.","section":"Abstract, §1, §3.3"},{"comment":"The Bloom's Taxonomy coding of LLM questions is presented as evidence that EDBook led to more 'active' learning, but the counts in Table 3 are not accompanied by any inferential statistics and appear to be based on small numbers of observations. The qualitative difference in question types is interesting, but the current presentation does not support a claim that EDBook changed the nature of learners' questions in a statistically reliable way. Please either add appropriate quantitative analysis or present this strictly as an observational, qualitative finding.","section":"§4.4.2, Table 3"}],"minor_comments":[{"comment":"There is a typo in the Callisto paragraph: 'with Like EDBook, Callisto’s references' should read 'Like EDBook, Callisto’s references...'.","section":"§2.5"},{"comment":"The sentence 'some participants found EDBooks to be more engaging that ChatGPT' should be 'more engaging than ChatGPT'.","section":"§4.5.6"},{"comment":"The figure uses both '≤ 30 minutes' and '≤ 10m' without a consistent time-unit convention; please standardize (e.g., all minutes).","section":"§4.3, Figure 7"},{"comment":"Table 4 would be easier to interpret if it included the actual Likert scale used and columns for means and standard deviations in each condition, rather than only listing the statements.","section":"§4.4.3, Table 4"},{"comment":"The caption of Figure 3 refers to 'icons' and 'red' highlighting but these are not visible in the grayscale reproduction; please ensure the figure is legible in print.","section":"§3.1.2, Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The paper has a useful design contribution and an honest, well-described study, but the central attribution claim is underdetermined by the current comparison. A revision that reframes the study as a feasibility/design exploration, adds clearly labeled exploratory analyses, or adds a follow-up experiment that controls for the integrated-editor and reveal-as-you-go confounds would make the paper substantially stronger. The 'first platform' claim should also be moderated unless the authors provide a more thorough prior-art search."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this one. The EDBook paper is a solid systems-plus-HCI contribution that deserves a serious referee, but do not read the user study as separate evidence for the headline claim. The platform itself—pre-written dialog trees plus context-aware LLM queries, with authorable branch points and code-embedded exercises—is a plausible and novel combination for programming education, and the implementation details are well presented. The paper earns credit for a clean within-subjects design, Latin-square counterbalancing, sensible task selection (novel Lisp syntax and an unfamiliar Python caching example), and careful reporting of qualitative data. It also explicitly admits the big limitation: no measured learning gains.\n\nThe soft spot is the one the stress-test note names. The comparison is EDBook (integrated environment with reveal-as-you-go dialog, local-context LLM prompts, code editor and quizzes in one pane) versus a webpage plus a separate ChatGPT window with global page context. The observed increases in time spent, quiz attempts, and self-reported engagement could be driven by any of those differences, not specifically by the dialog-plus-LLM integration. The paper acknowledges this partially in Section 4.5.1 but does not control for it. Uncorrected multiple comparisons on the survey items add a little noise; the reported p-values are raw and would not all survive a Bonferroni correction.\n\nAnother small gripe: the \"source code\" link is an anonymized placeholder, and no artifacts are shipped, so a reproducibility-minded reviewer cannot inspect the platform or content. For a systems paper that claims authorability and open-ended interactions, that is a real gap.\n\nIs the central argument undermined? Not fatally. The paper does not claim learning gains; it claims engagement and a new platform. The system design is interesting on its own terms, and the qualitative findings about local context (P13's comments) support the value of the local-context prompting even if not isolated. So this is a conditional accept for CHI rather than a reject. The right next step is a follow-up study that separates the interface-integration effect from the dialogic-structure effect, or at least adds a third condition.\n\nWho is this for? HCI and learning-at-scale researchers working on LLM-based interactive textbooks. I would send it to peer review. My own verdict: the platform is worth citing; the evaluation needs strengthening.","headline":"EDBooks is a genuinely new integration of dialog trees with context-aware LLM queries for programming instruction, but its user study confounds the platform's interface with the dialogic structure, so treat the engagement claims as suggestive rather than proven.","tokens_in":21277,"tokens_out":2616,"would_cite":true,"duration_ms":25724,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that EDBook, a platform combining author-written dialog trees with a context-aware LLM, is the first to make dialogic learning open-ended yet goal-aligned for programming education.","keywords":["EDBook","dialogic learning","large language models in education","programming education","dialog trees","goal-aligned LLM interactions","interactive textbooks","self-directed learning"],"falsifier":"Run the same two learning tasks in a third condition that uses the identical EDBook interface but replaces the dialog-tree narrative with a plain linear text layout, keeping the integrated editor, quizzes, and diff view; if the engagement and quiz-attempt gains disappear, the dialogic structure rather than the all-in-one environment is responsible.","tokens_in":20261,"feed_emoji":"🎓","tokens_out":5553,"duration_ms":53271,"temperature":0.7,"pith_summary":"The paper proposes EDBook, a programming-education platform that wraps an author-written, goal-directed dialog tree around a context-aware large language model. The aim is to give learners the structure and accuracy of a curated textbook alongside the flexibility of asking their own questions, so that open-ended interaction stays aligned with a defined learning goal. The authors argue this is the first such integration for programming instruction and report a 20-participant study in which learners using EDBook spent more time with the material, attempted more practice quizzes, and rated engagement higher than with equivalent webpage content plus ChatGPT. If the finding holds, it offers a concrete way to get personalized LLM tutoring without losing pedagogical control.","feed_headline":"EDBook's dialog-plus-LLM format lifts engagement and quiz attempts","feed_subtitle":"In a 20-learner study, the interactive narrative beat a webpage+ChatGPT setup on engagement, though test scores did not differ.","key_machinery":"The central mechanism is a dialog tree rendered as a flattened, readable conversation: each node is a message, children are possible responses, and a designated 'target' leaf marks the learning goal. Every reader query to the LLM is grounded by passing the current node and all ancestors as context, so a terse question like 'what is url' is answered relative to the code sample in view rather than as a dictionary definition. Around this core, EDBook adds several supporting mechanisms: deictic pointers that draw curved lines from narrative cells to code regions, incremental code diffs that build up a larger sample over the story, and interactive multiple-choice and code-writing assessments embedded as directives in the message cells. The tree representation keeps the author's verification burden low, since only the path from root to target must be checked for accuracy, while the branch-on-LLM mechanism preserves open-ended exploration.","core_discovery":"In the paper's terms, EDBook's core discovery is that pre-written dialog trees can act as 'rails' while open-ended LLM queries let students 'go off-road,' and that this combination produces dialogic learning that is open-ended yet goal-aligned. The dialog tree is authored or verified by an instructor, with a single target leaf node representing completion of the learning goals; whenever a reader queries the LLM, the current node and its ancestors are passed as context, and the response becomes a new branch labeled as unverified AI-generated content. In a within-subjects study with 20 programming learners comparing EDBook with equivalent content rendered as a webpage plus context-informed ChatGPT, the EDBook condition showed significantly more time spent learning, significantly more attempts at quiz and example-code questions, and significantly higher self-rated engagement and perceived usefulness of code-writing questions. Assessment scores were numerically higher but not significantly different. The authors interpret the result as evidence that structured dialogic narratives can keep learners active and focused while preserving their agency to pursue personally relevant questions.","pith_inferences":["The all-in-one integration (editor, quizzes, diff view in one place) may explain part of the engagement gains; a component-control study would separate this from the dialogic narrative itself.","Dialog trees could double as curriculum-alignment instrumentation: each branch's divergence from the target gives instructors data on where learners stray, enabling automated curriculum repair.","The approach could be tested in other skill domains where 'building' is central, such as data analysis workflows or hardware description, by swapping the code panel for another editor.","If authors adopt LLM-generated branches at scale, the warning-label mechanism for unverified AI content becomes a reusable design pattern for any adaptive educational material that mixes human and synthetic text."],"forward_implications":["Programming novices can ask questions in an unfamiliar domain without knowing what to ask, because the dialog tree supplies a path and contextual suggestions.","LLM answers stay goal-aligned because the model receives the current conversational branch; off-topic or inaccurate tangents are flagged as unverified AI content.","Authors can guarantee accuracy of core material by writing or verifying only the nodes from root to target, while still allowing free-form exploration.","Learners in the dialogic condition spend more time with the material and attempt more practice questions, a pattern the paper connects to prior findings that engagement improves learning outcomes.","The platform extends beyond programming to any domain where guided practice and open-ended questioning both matter."],"supporting_citations":[{"why":"Supplies the prior example of programming taught through a pre-written learner-instructor dialog, which EDBook extends into tree-structured, LLM-augmented form.","marker":"[13]"},{"why":"Defines curated conversation-driven explanations for technical concepts, the non-adaptive baseline EDBook improves on.","marker":"[57]"},{"why":"Introduces deictic message-to-code references in chat.codes, whose design EDBook adapts and simplifies.","marker":"[45]"},{"why":"Presents Callisto's cell-level deictic references, compared with EDBook's curved-line pointer design.","marker":"[59]"},{"why":"Shows incremental, step-by-step code tutorial construction (Torii), the mechanism EDBook uses for building code samples.","marker":"[20]"},{"why":"Describes Colaroid, a VS Code tutorial authoring tool without dialogic or assessment features, used to position EDBook's uniqueness.","marker":"[58]"},{"why":"Proposes target-guided topic shifting for open-ended conversations, an alternative goal-alignment technique EDBook contrasts with tree 'rails.'","marker":"[54]"},{"why":"Runestone, an interactive programming textbook platform with quizzes and exercises, representing the traditional interactive-book baseline.","marker":"[11]"},{"why":"Bloom's taxonomy, the coding scheme used to categorize the kinds of questions participants asked the LLM in each condition.","marker":"[4]"}],"fun_headline_variants":["EDBook's LLM rails boost engagement and quiz tries in programming ed","EDBook's dialog rails keep learners active while LLM answers go off road","EDBook's interactive narrative with LLM boosts engagement, not test scores","EDBook's LLM-powered dialog trees lift engagement and quiz attempts","EDBook's LLM rails boost learning activity without raising test scores"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The study compares EDBook against a plain webpage plus ChatGPT, so any benefit might come from having editor, quizzes, and content in one place rather than from the dialogic narrative itself.","fun_headline_variants_meta":{"raw":{"variants":["EDBook's LLM rails boost engagement and quiz tries in programming ed","EDBook's dialog rails keep learners active while LLM answers go off road","EDBook's interactive narrative with LLM boosts engagement, not test scores","EDBook's LLM-powered dialog trees lift engagement and quiz attempts","EDBook's LLM rails boost learning activity without raising test scores"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001084,"raw_usage":{"total_tokens":4513,"prompt_tokens":906,"completion_tokens":3607,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":522,"completion_tokens_details":{"reasoning_tokens":3512}},"tokens_in":522,"tokens_out":3607,"duration_ms":24402,"temperature":1.0,"reasoning_tokens":3512,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T19:25:16.534798+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same two learning tasks in a third condition that uses the identical EDBook interface but replaces the dialog-tree narrative with a plain linear text layout, keeping the integrated editor, quizzes, and diff view; if the engagement and quiz-attempt gains disappear, the dialogic structure rather than the all-in-one environment is responsible.","supporting_citations":[{"cited_title":"P., and Felleisen, M","cited_arxiv_id":null,"evidence_quote":"Supplies the prior example of programming taught through a pre-written learner-instructor dialog, which EDBook extends into tree-structured, LLM-augmented form."},{"cited_title":"Y., and Chilana, P","cited_arxiv_id":null,"evidence_quote":"Defines curated conversation-driven explanations for technical concepts, the non-adaptive baseline EDBook improves on."},{"cited_title":"Creating guided code explanations with chat","cited_arxiv_id":null,"evidence_quote":"Introduces deictic message-to-code references in chat.codes, whose design EDBook adapts and simplifies."},{"cited_title":"Y., Wu, Z., Brooks, C., and Oney, S","cited_arxiv_id":null,"evidence_quote":"Presents Callisto's cell-level deictic references, compared with EDBook's curved-line pointer design."},{"cited_title":"A., and Hartmann, B","cited_arxiv_id":null,"evidence_quote":"Shows incremental, step-by-step code tutorial construction (Torii), the mechanism EDBook uses for building code samples."},{"cited_title":"Y., Head, A., Zhang, A","cited_arxiv_id":null,"evidence_quote":"Describes Colaroid, a VS Code tutorial authoring tool without dialogic or assessment features, used to position EDBook's uniqueness."},{"cited_title":"J., and Miller, B","cited_arxiv_id":null,"evidence_quote":"Runestone, an interactive programming textbook platform with quizzes and exercises, representing the traditional interactive-book baseline."},{"cited_title":"Bloom’s taxonomy, 1956","cited_arxiv_id":null,"evidence_quote":"Bloom's taxonomy, the coding scheme used to categorize the kinds of questions participants asked the LLM in each condition."}],"review_version":1}