{"id":"5f4ab6fe-d7d8-41d1-8dad-b6615271c260","arxiv_id":"2505.10069","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"In CourseMapper, building educational knowledge graphs slide-by-slide (bottom-up) gave slightly more accurate concept extraction than whole-document construction, but the advantage was not statistically significant.","lead":"This paper compares two ways of automatically building knowledge graphs for online course materials: starting from the whole document (top-down) or from individual slides (bottom-up). A small user study and expert checks found the bottom-up method slightly better, though the difference was not statistically significant.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's own significance tests contradict its central claim: no statistically significant difference was found, yet the abstract and conclusion assert bottom-up superiority.","rationale":"The reader's verdict is REJECT with high correctness risk, and I agree that the paper should be rejected for its unsupported central claim. However, the reader's weakest_assumption focuses on annotator ground-truth validity, whereas I see a more fundamental and directly checkable problem: the paper's own statistical tests show no significant difference, so the categorical superiority claim is internally contradicted. The annotation-quality issue is real and would further weaken confidence, but it is secondary to the significance problem. My proposed concrete test recomputes the comparison from the reported numbers; it would settle whether the point-estimate gap is meaningful. Since the published numbers already imply non-significance, the conclusion should at minimum be downgraded to a directional finding. Therefore the verdict remains REJECT, and the reader's concern is only partially aligned with mine because the decisive flaw is the statistical overclaim rather than the ground-truth assumption.","tokens_in":6654,"tokens_out":1864,"duration_ms":20487,"concrete_test":"Re-analyze the SRS accuracy comparison from Section 5.2 using the raw sample judgments (roughly 383 Top-down and 380 Bottom-up triples). Compute the two-proportion z-test and 95% confidence interval for the difference: with p_TD=0.38 and p_BU=0.40, the standard error of the difference is about 0.035, giving z≈0.57 and a 95% CI for the difference of roughly [−0.049, 0.089]. If this CI includes zero, as it does, the claim that bottom-up is more accurate is statistically unsupported. The same check should be applied to the user-study metrics, ideally with a paired test on the 19 participants. If the authors can provide raw judgments, recompute; if not, the superiority assertion cannot be verified.","verdict_should_be":"REJECT","load_bearing_attack":"The central claim—that the Bottom-up approach is the most effective and accurate method—rests on Table 1, but the text reports that a T-test found no significant differences in both the user study and the SRS accuracy evaluation. The observed differences are small: P@15 is 0.812 vs 0.807, and SRS accuracy is 0.40 vs 0.38. The reported normal approximation intervals (±0.048 and ±0.049) overlap heavily; a two-proportion test on the SRS judgments would give a z-score around 0.57 and a p-value around 0.57, far from significance. Thus the data support at most a directional, non-significant tendency, not a claim that bottom-up 'emerged as the most effective and accurate method.' This is an internal inconsistency, not a matter of interpretation: the paper explicitly acknowledges non-significance in Sections 5.1 and 5.2, then makes a categorical superiority claim in the abstract, Section 6, and Section 7. The annotator-ground-truth concern raised by the reader compounds this: without inter-annotator agreement or moderator validation, even the point estimates are weakly anchored. But the primary load-bearing weakness is that the reported statistics themselves cannot support the conclusion; the claim would fail even if the ground truth were perfect.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper compares two pipelines for automatic construction of Educational Knowledge Graphs (EduKGs) in the CourseMapper MOOC platform: a Top-down approach that extracts keyphrases from an entire PDF learning material at once, and a Bottom-up approach that extracts keyphrases slide-by-slide and then aggregates upward. Both pipelines use SIFRank SqueezeBERT for keyphrase extraction, DBpedia Spotlight for entity linking, and an SBERT-based weighting strategy. The authors report a user study with 19 participants evaluating Precision@15, MRR, and MAP, and a Simple Random Sampling (SRS) expert annotation study estimating triple accuracy. They also propose a Human-in-the-Loop refinement step for course moderators. The abstract and conclusion claim that the Bottom-up approach outperforms the Top-down approach and is the most effective and accurate method.","tokens_in":6842,"tokens_out":3887,"duration_ms":37156,"significance":"If the claimed superiority of Bottom-up over Top-down EduKG construction were well supported, the paper would offer a practical, evidence-based choice for MOOC platform developers: construct EduKGs incrementally per slide, with moderator review, rather than from whole documents. The human-in-the-loop component is a reasonable proposal for improving low accuracy, though it is not evaluated. The paper reuses components from the authors' prior work (SIFRank SqueezeBERT, SBERT weighting, DBpedia Spotlight), so the principal novelty is the pipeline comparison and the HITL integration. However, the empirical support for the central claim is the main contribution, and that support is lacking: the paper's own statistics show no significant differences, and the SRS ground truth is not validated by inter-annotator agreement or moderator comparison. No code or data are provided for reproducibility.","major_comments":[{"comment":"The paper's central claim that the Bottom-up approach 'outperforms' or 'is the most effective and accurate method' is directly contradicted by its own reported statistics. Section 5.1 states that 'A T-test revealed no significant differences between the models,' and Section 5.2 states 'a T-test found no statistically significant difference.' The observed differences in Table 1 are tiny: P@15 = 0.812 vs. 0.807, MRR identical at 0.941, MAP = 0.812 vs. 0.807, and SRS accuracy = 0.40 vs. 0.38 with overlapping normal-approximation intervals (±0.049 and ±0.048). A two-proportion test on the SRS judgments would yield a z-score around 0.57 and a p-value around 0.57, far from significance. The data support at most a non-significant directional tendency, not the categorical superiority claimed in the abstract, Section 6, and Section 7. This is an internal inconsistency in the manuscript, not a matter of interpretation.","section":"Abstract, §5.1, §5.2, Table 1, §7"},{"comment":"The SRS accuracy evaluation lacks the basic safeguards needed to treat the annotators' judgments as ground truth. The paper reports that two annotators reviewed different random samples (200 vs. 183/180 samples) but reports no inter-annotator agreement measure (e.g., Cohen's kappa), no procedure for resolving disagreements, and no validation against the course moderators who later refine the EduKG in the HITL process. Since each triple receives a single judgment, the measured accuracy gap of 0.40 vs. 0.38 could be annotation noise rather than a genuine pipeline difference. This is load-bearing because the accuracy comparison rests entirely on this unvalidated ground truth.","section":"§5.2"},{"comment":"The comparison depends on several free parameters chosen without reported justification or sensitivity analysis: the keyphrase count n = 15 per slide (Section 4.2), the top-down multiplier n = 15 × (number of slides) (Section 4.1), the top-k = 15 used in the user study, and the SRS sample sizes. The text says these values 'proved to cover all the possible keyphrases based on experiment' and 'more than 95% of slides contain fewer than 15 keyphrases,' but no experiments or data are shown to support these thresholds. If the conclusions are sensitive to these parameters, the observed non-significant differences could be an artifact of the chosen settings. The authors should provide an ablation or at least report the distribution of keyphrase counts per slide.","section":"§5.1, §4.1, §4.2"}],"minor_comments":[{"comment":"The pipeline figures are referenced as 'Figure??a' and 'Figure??b' in Section 4.1 and 4.2; these placeholders must be resolved before publication.","section":"§4.1, Figure placeholders"},{"comment":"There is a typo: 'disucssed' should be 'discussed' (Section 2, description of EduKGs).","section":"§2"},{"comment":"The formula for mean accuracy µs is rendered as a fraction '1/ns Σ f(ti)' without clear delimiter; please use a clearer notation such as (1/ns) Σᵢ f(tᵢ).","section":"§5.2, Equation for µs"},{"comment":"The user study reports no confidence intervals or effect sizes for P@15, MRR, and MAP, and does not describe whether the T-test was paired or unpaired, nor how the 19 participants and 34 learning materials were allocated between pipelines. The claim that 'the bottom-up EduKG was rated more favorably' in user experience is not backed by any reported statistic.","section":"§5.1"},{"comment":"The Human-in-the-Loop approach is proposed but not empirically evaluated. Phrases such as 'This process guarantees accurate EduKGs' are too strong given the lack of validation; the authors should temper the claim or provide evidence.","section":"§6"},{"comment":"No data or code availability statement is provided, which limits reproducibility of the experiments.","section":"General"}],"recommendation":"reject","confidential_remarks":"The paper's central claim is contradicted by its own significance tests, and the SRS ground truth lacks inter-annotator agreement and moderator validation. These are load-bearing issues that require new experiments and a substantial reframing of the contribution, not just local edits. I would advise rejection in this form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Thanks for the memo. I read the paper and I think the reader and the stress-test are right about the main weakness: the abstract and conclusion assert bottom-up superiority while the reported statistics contradict it. Table 1 shows tiny gaps (P@15 0.812 vs 0.807; SRS accuracy 0.40 vs 0.38) with overlapping normal-approximation intervals, and Section 5 explicitly states that a T-test found no significant differences in both evaluations. The data support at most a directional tendency, not a claim that bottom-up \"emerged as the most effective and accurate method.\" That is an internal inconsistency, not a matter of interpretation.\n\nCredit where it is due: this is a legitimate new comparison for a real MOOC platform. The paper describes two concrete construction pipelines, evaluates them with a user study (19 participants, 34 materials) and SRS-based accuracy evaluation by two annotators, and adds a human-in-the-loop review UI for moderators. The pipelines reuse components from earlier work, but the comparison itself is new for this setting. The writing is mostly clear, and the authors do report the null result in Section 5 even though they fail to respect it in the conclusions.\n\nBeyond the core overclaim, the accuracy evaluation is weakly anchored. No inter-annotator agreement is reported, there is no disagreement resolution process, and the annotators are not compared against the course moderators who later refine the EduKG. With a 0.40 vs 0.38 difference, that could be annotation noise. The fixed keyphrase counts (15 per slide, the 15x multiplier in top-down) are free parameters chosen from the authors' prior experiments; they are not fitted to this comparison, so the circularity burden is modest, but the choice deserves more justification. No data, code, or questionnaires are released, so reproducibility is limited. There are also minor presentation issues like broken figure references and typos.\n\nWho is this for? Researchers building EduKG pipelines in MOOCs will find the pipeline descriptions and evaluation protocol useful, and the SRS method is a sensible way to estimate graph accuracy. The paper deserves a serious referee rather than a desk reject, but the referee should demand a rewritten conclusion that matches the statistics, plus release of the evaluation materials and annotator agreement data. My own verdict: the current claims are not sound, but the underlying comparison is worth a major-revision round, not a burial.","headline":"The paper overclaims bottom-up superiority when its own stats show no significant difference; the comparison is new and the pipeline description is useful, but the central conclusion is not supported by the reported evidence.","tokens_in":7411,"tokens_out":1572,"would_cite":false,"duration_ms":15782,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that building educational knowledge graphs slide-by-slide and then aggregating upward is more accurate than extracting from the whole document at once.","keywords":["educational knowledge graphs","MOOC","top-down vs bottom-up","knowledge graph construction","human-in-the-loop","simple random sampling","CourseMapper","SBERT weighting"],"falsifier":"Re-run the SRS evaluation with the two annotators' judgments coded independently, compute inter-annotator agreement (e.g., Cohen's kappa), and compare the 0.40 versus 0.38 accuracy gap against the annotation disagreement rate; if the gap falls within the noise floor, the claimed superiority of the Bottom-up pipeline would not be supported.","tokens_in":6410,"feed_emoji":"🎓","tokens_out":4848,"duration_ms":41443,"temperature":0.7,"pith_summary":"This paper sets out to settle a practical question: when building an educational knowledge graph (EduKG) from MOOC lecture materials, should the pipeline extract concepts from the whole document at once, or from each slide and then merge? The authors claim the slide-by-slide 'Bottom-up' pipeline is more accurate and more useful than the 'Top-down' whole-document pipeline, based on a 19-participant user study and an expert annotation study on the CourseMapper platform. If the claim holds, course platforms can produce more reliable knowledge graphs at little extra cost. The paper also adds a Human-in-the-Loop step in which course moderators review and correct the automatically built graph before it reaches learners.","feed_headline":"Slide-by-slide knowledge graphs beat whole-document","feed_subtitle":"In CourseMapper tests, per-slide extraction scored 0.40 vs 0.38 accuracy, with moderator review as final check.","key_machinery":"The load-bearing mechanism is the two-pipeline comparison inside CourseMapper. Both pipelines use the same building blocks — PDFMiner text extraction, SIFRank SqueezeBERT keyphrase extraction, DBpedia Spotlight entity linking, and SBERT-based concept weighting — and differ only in the order of aggregation: Top-down builds one LM-EduKG from the whole material and then checks slide concepts against it, while Bottom-up builds a Slide-EduKG per slide, computes a per-slide importance score wSlide plus a material-level weight wLM, and aggregates the slide graphs into the LM-EduKG. The Bottom-up pipeline also lets partial slide-level graphs be published before the full graph is ready.","core_discovery":"The paper's central claim is that the Bottom-up pipeline — extract keyphrases from each slide, link them to DBpedia concepts, weight each concept by its similarity to both the slide and the whole learning material, then aggregate the slide-level graphs into a learning-material-level graph — yields a more accurate EduKG than the Top-down pipeline, which extracts from the entire PDF first and then assigns concepts to slides. In the Simple Random Sampling accuracy evaluation, Bottom-up scored a mean accuracy of 0.40 against Top-down's 0.38; the user study likewise rated Bottom-up slightly higher on precision and user experience. The authors interpret this as evidence that per-slide construction better captures which concepts actually belong to which part of the material, and they add a Human-in-the-Loop step so course moderators can review and correct the graph before learners see it.","pith_inferences":["A stronger test of the 0.40 versus 0.38 accuracy gap would measure inter-annotator agreement; if the gap falls within annotation disagreement noise, the ordering of the pipelines could reverse.","The Human-in-the-Loop step could be made adaptive by having moderators review only slides where the two pipelines disagree, concentrating effort where the pipelines are most uncertain.","The per-slide mechanism is generic and could transfer to non-educational domains that segment documents into pages or sections, such as legal or medical document parsing.","One testable extension is whether bottom-up superiority persists when document segmentation is not provided by the source but must be inferred from layout or content."],"forward_implications":["If the paper is right, MOOC platforms should construct EduKGs by extracting keyphrases per slide and aggregating upward, rather than from the whole document at once.","Course moderators can use the proposed Human-in-the-Loop review step to lift accuracy further before the published EduKG reaches learners.","The same pipeline comparison can be reused for other segmented learning materials, such as video lectures with chapter markers, where per-segment extraction is feasible.","The evaluation protocol, which combines a user study with Simple Random Sampling accuracy checks, offers a template for comparing knowledge-graph construction pipelines in educational settings.","The finding implies that text segmentation granularity is a design choice that directly affects which concepts survive in the final graph, not merely a computational convenience."],"supporting_citations":[{"why":"Supplies the Simple Random Sampling method used to compare the accuracy of the two pipelines in Section 5.2.","marker":"[7]"},{"why":"Provides the knowledge-graph-based core concept identification and weighting approach that both pipelines build on.","marker":"[13]"},{"why":"Supplies the SIFRank SqueezeBERT keyphrase extraction algorithm and the prior word-embedding-based EduKG construction approach.","marker":"[2]"},{"why":"Provides DBpedia Spotlight, the entity-linking tool used to map keyphrases to DBpedia concepts.","marker":"[16]"},{"why":"Provides Sentence-BERT, used to generate the embeddings for concept and material similarity weighting.","marker":"[17]"}],"fun_headline_variants":["Per-slide graphs edge out whole-document in MOOC tests","Bottom-up knowledge graph construction wins in CourseMapper","Slide-level extraction improves MOOC knowledge graph accuracy","Per-slide beats whole-document for EduKG accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The accuracy comparison rests on the assumption that the two annotators' Simple Random Sampling judgments of graph triples are a reliable ground truth, despite no reported inter-annotator agreement, disagreement resolution, or comparison against the course moderators who later refine the graph.","fun_headline_variants_meta":{"raw":{"variants":["Per-slide graphs edge out whole-document in MOOC tests","Bottom-up knowledge graph construction wins in CourseMapper","Slide-level extraction improves MOOC knowledge graph accuracy","Per-slide beats whole-document for EduKG accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000733,"raw_usage":{"total_tokens":3240,"prompt_tokens":869,"completion_tokens":2371,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":485,"completion_tokens_details":{"reasoning_tokens":2308}},"tokens_in":485,"tokens_out":2371,"duration_ms":17487,"temperature":1.0,"reasoning_tokens":2308,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:16:45.565656+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the SRS evaluation with the two annotators' judgments coded independently, compute inter-annotator agreement (e.g., Cohen's kappa), and compare the 0.40 versus 0.38 accuracy gap against the annotation disagreement rate; if the gap falls within the noise floor, the claimed superiority of the Bottom-up pipeline would not be supported.","supporting_citations":[{"cited_title":"Information 14(10) (2023)","cited_arxiv_id":null,"evidence_quote":"Supplies the SIFRank SqueezeBERT keyphrase extraction algorithm and the prior word-embedding-based EduKG construction approach."}],"review_version":1}