{"id":"f97c096a-d3a9-40b9-92ea-235fb3cdc791","arxiv_id":"2509.05392","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"Optimizing an EduKG pipeline with local Wikipedia dumps, better text extraction, disambiguation, and pruning lifts accuracy from 0.40 to 0.47 and cuts processing time by 10-100x.","lead":"The authors built an automated pipeline that turns PDF course slides into knowledge graphs, then optimized it for speed and accuracy. The optimized pipeline runs about ten times faster and raises expert-judged graph accuracy from 40% to 47%, a useful but modest gain for educational AI tools.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Precision-only accuracy gain may be a pruning artifact: no recall or KG-size comparison is reported, so the 0.07 improvement could reflect a smaller, more conservative graph.","rationale":"The reader's weakest assumption is sampling noise, but the reported data actually support a borderline-significant difference: n=380 vs n=403 gives SE_diff≈0.035, z≈1.98, p≈0.05. The identical σ=0.049 is the normal-approximation margin z*SE, not a standard error, so it is not an internal inconsistency. The more serious gap is that the accuracy metric is precision-only and pruning is central to the optimization. This concern is load-bearing because the central claim is specifically '17.5% improvement in accuracy'; if the improvement comes from deleting difficult triples, it does not establish a better EduKG. The efficiency improvements (10x generation, 100x expansion) are credible and well-supported, so the paper still has value; the missing recall/size analysis is a concrete, addressable condition. Hence the conditional verdict is unchanged.","tokens_in":11810,"tokens_out":15638,"duration_ms":172592,"concrete_test":"On the same evaluation PDF, report (a) total triples and unique concepts in the initial and optimized EduKGs, and (b) recall: have the two experts (or a domain-expert annotation) mark whether each of K≈100 concepts sampled from the source PDF appears in each KG. If the optimized KG is substantially smaller (e.g., >25% fewer triples) or recall drops by more than the precision gain, the 17.5% accuracy improvement is a precision/coverage tradeoff rather than a genuine quality gain.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline '17.5% improvement in accuracy' is a precision-only claim, and the optimization path makes the missing recall/size information load-bearing. In §3, accuracy is measured by SRS expert judgments on sampled triples—a precision estimate. In §4.4, the optimized pipeline includes KG pruning that deletes every concept with weight below 0.192 (threshold selected on the CCI dataset), and concept disambiguation that can replace/remove annotations. Pruning mechanically removes low-weight triples; whether those triples are wrong or correct is not reported. §5 gives only µs±σ and runtimes, with no count of triples/nodes in the initial vs optimized EduKG and no recall/coverage check against the source PDF or expert-identified concepts. If the optimized KG is smaller and drops many correct triples, the 0.07 accuracy gain is a precision/coverage tradeoff, not evidence that the EduKG is better for learning. This is not addressed by a significance test. (The reader's sampling-noise concern is less decisive than it appears: with n=380 and n=403, a two-proportion z-test gives z≈1.98, p≈0.05, and the reported 0.049 matches z_{0.975}·SE, i.e., a 95% margin rather than a standard error.)","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an end-to-end pipeline for automatic construction of Educational Knowledge Graphs (EduKGs) from PDF learning materials, implemented in the CourseMapper MOOC platform. The pipeline extracts text, identifies keyphrases, links them to DBpedia concepts, expands with related concepts and categories, and merges slide-level graphs into a learning-material-level EduKG. An initial evaluation on a single PDF reports accuracy 0.40±0.049. The authors then introduce several optimizations: a worker-based architecture, offline preprocessing of a Wikipedia dump, enhanced text extraction, concept disambiguation, knowledge-graph pruning, and a new embedding model. The optimized pipeline is reported to achieve accuracy 0.47±0.049 (a 17.5% relative improvement) and large efficiency gains (about 10x per-slide generation, >100x per-concept expansion). The paper claims this demonstrates a state-of-the-art, domain-independent, scalable solution.","tokens_in":12130,"tokens_out":5402,"duration_ms":47455,"significance":"If the reported results are correct, this is a useful applied contribution: a fully implemented, deployed EduKG pipeline in a real MOOC platform, with offline preprocessing to remove Wikipedia API latency, and explicit comparison of multiple text-extraction and keyphrase-extraction variants. The authors are also honest about the initial low accuracy and list limitations. However, the central 17.5% accuracy claim rests on a single-material, precision-style evaluation whose statistical reporting is ambiguous and likely mislabeled; Table 1 contains implausible duplicated numbers; and no recall or KG-size analysis is provided. These issues must be resolved before the headline claims can be accepted.","major_comments":[{"comment":"The reported 'µs ± σ' values are 0.4±0.049 and 0.47±0.049. The text calls 0.049 the standard deviation, but 0.049 is exactly the 95% half-width for a binomial proportion with n≈380 (1.96·sqrt(0.4·0.6/380)=0.049). If it is a margin of error, the two confidence intervals overlap (0.351–0.449 vs. 0.421–0.519); if it is a standard deviation, the difference is not significant under a two-sample z-test. No significance test or confidence interval is provided. Since the 17.5% improvement is the paper's central claim, this ambiguity is load-bearing and must be corrected with proper statistical reporting.","section":"§5.2, Table 7"},{"comment":"The SRS evaluation is a triple-correctness (precision-like) measure; it does not assess recall. The optimized pipeline includes a pruning step that removes every concept with weight below 0.192, yet no node/edge counts, recall, or coverage comparison between the initial and optimized EduKGs is reported. The 0.07 accuracy gain could therefore reflect a smaller, more conservative graph rather than a genuinely better representation of the learning material. Please report KG sizes, recall against expert-identified concepts or the source text, or a precision-recall tradeoff.","section":"§3, §4.4, §5.2"},{"comment":"For every keyphrase-extraction method and every n, the Inspec and SemEval precision and recall values are numerically identical (e.g., zero-shot LLM: 72.95/75.17 for both; SIFRank: 65.21/67.49 for both). It is implausible that two different datasets produce identical results across all rows. This indicates a data handling error and invalidates the keyphrase-extraction comparison that motivates subsequent pipeline choices. The table must be corrected or removed.","section":"Table 1"},{"comment":"The pruning threshold 0.192 and the decision to include disambiguation are tuned on the CCI dataset without held-out validation, and the final accuracy evaluation is on one different PDF. The threshold is a free parameter of the evaluated pipeline; no sensitivity analysis or validation split is provided. The authors' own limitation statement (Section 6) acknowledges that the CCI dataset is 'limited in coverage and outdated,' which further weakens this tuning evidence. At minimum, report the optimized accuracy across a range of thresholds or use a validation split.","section":"§4.4, §6"}],"minor_comments":[{"comment":"The text first says 'A total of 24 PDF materials were selected' but later says 'the final dataset contained 643 annotated text parts from 24 slides across 7 different learning materials.' This inconsistency must be clarified: was the evaluation on 24 materials or 24 slides from 7 materials?","section":"§4.3"},{"comment":"The symbol σ is described as the standard deviation, but the reported value matches a 95% margin of error for a binomial proportion. The terminology should be made consistent and explicit.","section":"§3, §5.2"},{"comment":"The abstract and conclusion state a 'tenfold increase in processing efficiency,' but Table 7 reports more than 10x for generation and more than 100x for expansion. Please use a single consistent phrasing, or specify that the tenfold figure refers to the overall generation time.","section":"§5.1, Table 7"},{"comment":"Table 2 reports runtime for 'LLM-based methods' as a single row (1:40), but the text discusses multiple LLM variants. Clarify whether this is an average and over what input size.","section":"§2.2"}],"recommendation":"major_revision","confidential_remarks":"The paper describes a substantial engineering effort and reports a plausible direction, but the evaluation evidence is not yet reliable. The duplicated values in Table 1 and the ambiguous σ in Table 7 suggest that the authors should be asked to provide raw data and scripts. The missing recall/KG-size analysis is the most serious conceptual gap: without it, the 'improvement' may be a precision-coverage tradeoff. I would not reject the paper, but it needs a statistically sound and more complete evaluation before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know: this is an incremental engineering paper, not a research breakthrough. The real contribution is a set of optimizations to an existing EduKG pipeline—local Wikipedia dump with precomputed embeddings, worker queues, text-extraction heuristics, concept disambiguation, KG pruning—and the 10–100x speedups in generation and expansion are credible and well-supported by runtime tables. The 17.5% accuracy improvement (0.40 to 0.47) is the headline, and it is plausible but not statistically demonstrated.\n\nWhere the paper is genuinely good: the optimization story is coherent, the ablations on embedding models and disambiguation (Tables 5 and 6) are reasonable, and the limitations section is honest about the CCI dataset being outdated and about remaining text-extraction issues. The efficiency evaluation uses a varied dataset and a sensible commodity-server setup. That part deserves credit.\n\nNow the soft spots, in proportion. Table 1 is the biggest red flag: for every keyphrase method, precision and recall are identical across Inspec and SemEval, with F1 differing. That is not plausible unless the two datasets produced identical raw counts, which would require the same underlying data. This looks like a copy/paste or data-handling error, and it undercuts confidence in the table's numbers.\n\nThe accuracy claim has a more subtle problem. The SRS evaluation on triples is essentially a precision estimate. The optimized pipeline prunes low-weight concepts (threshold 0.192, chosen on CCI) and adds disambiguation, which can remove or replace nodes. The paper never reports the number of triples or nodes in the initial vs. optimized KG, nor recall or coverage against the source PDF. So the 0.07 gain could partly be a precision/coverage tradeoff—a smaller, more conservative graph looks more accurate per triple without necessarily being better for learning. The stress-test note is right about this; it is not addressed by a significance test.\n\nOn the sampling-noise question: the reported sigma=0.049 with n=380/403 makes the difference borderline significant (roughly z≈2, p≈0.05), so the reader's concern is valid but not the knockout punch. The missing recall/size comparison is more important.\n\nWho this is for: practitioners building automated KG pipelines for educational content, especially in MOOC platforms. It is not a methods paper. I would not cite it in my own work, but I'd send it to a workshop or applied track.\n\nRecommendation: send to peer review, but with a clear brief: fix Table 1, report recall and KG size for both pipelines, give confidence intervals, and, if possible, release code/data. Without those, the accuracy claim remains a maybe.","headline":"A pragmatic engineering paper with credible efficiency gains; the headline accuracy improvement is plausible but not yet pinned down, and a data table error needs fixing before I'd trust the numbers.","tokens_in":12598,"tokens_out":1617,"would_cite":false,"duration_ms":19521,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An automated pipeline built from slide-level graphs can turn PDF course materials into educational knowledge graphs, and targeted optimizations raise expert-judged triple accuracy from 0.40 to 0.47 while making generation about ten times fa","keywords":["educational knowledge graphs","knowledge modeling","keyphrase extraction","entity linking","knowledge graph construction","PDF text extraction","concept disambiguation","MOOC learning materials"],"falsifier":"A concrete falsifier would be to run the initial and optimized pipelines on the same set of several PDFs with the same expert judges, then compute a confidence interval on the accuracy difference; if the interval includes zero, the 17.5% improvement claim is not supported. The same experiment, with the pruning threshold fixed beforehand, would also show whether the threshold generalizes beyond the dataset it was tuned on.","tokens_in":11728,"feed_emoji":"📚","tokens_out":8157,"duration_ms":80936,"temperature":0.7,"pith_summary":"This paper argues that automatic construction of educational knowledge graphs (EduKGs) from PDF learning materials can be made substantially more reliable and faster by composing a modular pipeline and optimizing each stage. The pipeline extracts text per slide, identifies keyphrases, links them to knowledge-base concepts, expands the concepts, and merges slide-level graphs into one graph for the whole material. On the tested course material, the initial pipeline judged about 40% of triples correct; after optimizations, about 47% were judged correct, a 17.5% relative improvement, while average time per slide fell from 24.35 seconds to 2.3 seconds and per-concept expansion from 222 seconds to 1.89 seconds. The authors attribute the gains to cleaner text extraction, context-aware concept disambiguation, pruning of low-weight concepts, and precomputed embeddings from a local knowledge-base snapshot. If the reported gains hold, automated EduKG construction becomes more practical for real courses, although the authors note that the remaining accuracy ceiling is tied to the shared external knowledge base.","feed_headline":"PDF-to-knowledge-graph pipeline: 17.5% more accurate, 10x faster","feed_subtitle":"Targeted fixes to text extraction, concept disambiguation, and data caching make automated course graphs practical.","key_machinery":"The load-bearing mechanism is a two-level weighting and pruning loop. For each slide, candidate concepts are annotated by linking extracted keyphrases to entries in an external knowledge base; each candidate is then scored by the sum of two cosine similarities: between the whole material's sentence embedding and the concept's article embedding, and between the slide's text embedding and the concept's abstract embedding. This combined weight is used to keep or drop concepts—anything below the chosen threshold is pruned—and to disambiguate concepts that land on a disambiguation page, replacing them with the highest-weighted alternative. A local snapshot of the knowledge base with precomputed e","core_discovery":"The paper's central claim is that a carefully staged pipeline—slide-level knowledge graph construction followed by merging—can generate educational knowledge graphs from raw PDFs, and that accuracy and speed can be improved simultaneously by targeted engineering rather than by swapping in a larger model. The authors report that the optimized pipeline raises expert-judged triple accuracy from 0.40±0.049 to 0.47±0.049, a 17.5% relative improvement, and cuts mean generation time per slide by a factor of about ten and concept expansion time by a factor of over one hundred. They attribute the improvement to four changes: layout-aware text extraction that removes footers, page numbers, and fragmen","pith_inferences":["A natural follow-up is to ablate the optimizations one at a time—text extraction only, disambiguation only, pruning only—on the same PDF set to attribute the 0.07 absolute accuracy gain to specific stages.","Because the authors ascribe the accuracy ceiling to the shared external knowledge base, an extension would be to run a second, independent concept annotator over the extracted keyphrases and merge the two concept sets, keeping only concepts both annotators support.","The pruning threshold is derived from the weight distribution of one dataset; an extension is to treat it as a per-material hyperparameter and select it on a validation set by maximizing agreement with expert annotations.","The efficiency gains come from replacing live network lookups with a locally precomputed knowledge-base snapshot, which suggests the same pipeline could run fully offline in classrooms or low-connectivity settings."],"forward_implications":["Educational materials in PDF form can be processed into a concept-level knowledge graph at roughly two seconds per slide on commodity hardware, making per-course graph construction practical.","Because the pipeline does not require a predefined ontology or expert-annotated training data, it can be pointed at a new course's slides and produce a graph without manual schema design.","Learners can inspect the graph as it is being built, since slide-level graphs are stored incrementally rather than only after the whole material is processed.","The accuracy gain is real but still leaves nearly half of triples judged incorrect, so human-in-the-loop refinement or a better concept source is needed for high-stakes educational use.","The modular stage boundaries mean future improvements in text extraction, keyphrase extraction, or entity linking can be dropped into the pipeline without reworking the rest."],"supporting_citations":[{"why":"Supplies the concept annotation, weighting, and expansion scheme and the CCI dataset used to set the pruning threshold and test disambiguation.","marker":"[17]"},{"why":"Provides the entity-linking service that maps extracted keyphrases to knowledge-base concepts in both pipeline versions.","marker":"[19]"},{"why":"Provides the layout-aware PDF text extraction approach that the initial pipeline uses and the optimized module refines.","marker":"[21]"},{"why":"Defines the sentence-embedding method used to compute concept weights, slide-material similarity, and expansion candidate weights.","marker":"[22]"},{"why":"Supplies the pretrained embedding models compared in the optimization, including the model selected for final embeddings.","marker":"[24]"},{"why":"Supplies the simple random sampling accuracy evaluation method used to produce the 0.40 and 0.47 triple-accuracy estimates.","marker":"[12]"},{"why":"Supplies the error-category evaluation criteria used to compare the initial and optimized text extraction modules.","marker":"[7]"},{"why":"Supplies the open-weights language model used in the keyphrase extraction comparison that motivates the choice of keyphrase extractor.","marker":"[25]"}],"fun_headline_variants":["17.5% smarter EduKG: 10x faster from PDFs","PDFs to knowledge graphs: 17.5% more accurate, 10x faster","EduKG pipeline: 17.5% accuracy boost, 10x speed gain","Optimized EduKG: 17.5% better, 10x quicker from PDFs","Automatic EduKG: 17.5% accuracy lift, 10x faster"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The central claim rests on the assumption that the 7-point accuracy gain between the two pipelines is a real effect rather than sampling noise, since the paper compares two mean estimates with the same reported spread and gives no significance test; the pruning threshold is likewise tuned on one dataset and assumed to work on the evaluation material.","fun_headline_variants_meta":{"raw":{"variants":["17.5% smarter EduKG: 10x faster from PDFs","PDFs to knowledge graphs: 17.5% more accurate, 10x faster","EduKG pipeline: 17.5% accuracy boost, 10x speed gain","Optimized EduKG: 17.5% better, 10x quicker from PDFs","Automatic EduKG: 17.5% accuracy lift, 10x faster"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00023,"raw_usage":{"total_tokens":1315,"prompt_tokens":736,"completion_tokens":579,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":480,"completion_tokens_details":{"reasoning_tokens":466}},"tokens_in":480,"tokens_out":579,"duration_ms":5818,"temperature":1.0,"reasoning_tokens":466,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T05:41:49.548169+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete falsifier would be to run the initial and optimized pipelines on the same set of several PDFs with the same expert judges, then compute a confidence interval on the accuracy difference; if the interval includes zero, the 17.5% improvement claim is not supported. The same experiment, with the pruning threshold fixed beforehand, would also show whether the threshold generalizes beyond the dataset it was tuned on.","supporting_citations":[{"cited_title":"In: Joint International Semantic Technology Conference","cited_arxiv_id":null,"evidence_quote":"Supplies the concept annotation, weighting, and expansion scheme and the CCI dataset used to set the pruning threshold and test disambiguation."},{"cited_title":"In: Proceedings of the 7th international conference on semantic systems","cited_arxiv_id":null,"evidence_quote":"Provides the entity-linking service that maps extracted keyphrases to knowledge-base concepts in both pipeline versions."},{"cited_title":"Source code for biology and medicine (2012)","cited_arxiv_id":null,"evidence_quote":"Provides the layout-aware PDF text extraction approach that the initial pipeline uses and the optimized module refines."},{"cited_title":"sbert.net/docs/sentence_transformer/pretrained_models.html","cited_arxiv_id":null,"evidence_quote":"Supplies the pretrained embedding models compared in the optimization, including the model selected for final embeddings."},{"cited_title":"https://doi.org/10.1109/JCDL.2017.7991564","cited_arxiv_id":null,"evidence_quote":"Supplies the error-category evaluation criteria used to compare the initial and optimized text extraction modules."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the open-weights language model used in the keyphrase extraction comparison that motivates the choice of keyphrase extractor."}],"review_version":1}