Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

Top-Down vs. Bottom-Up Approaches for Automatic Educational Knowledge Graph Construction in CourseMapper

T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Pith's one-line read The paper claims that building educational knowledge graphs slide-by-slide and then aggregating upward is more accurate than extracting from the whole document at once.

desk verdict The paper overclaims bottom-up superiority when its own stats show no significant difference; the comparison is new and the pipeline description is useful, but the central conclusion is not supported by the reported evidence. read the letter →

arxiv 2505.10069 v1 pith:GFKZ5Y2H submitted 2025-05-15 cs.CY

classification cs.CY
keywords educationalknowledgegraphsMOOCtop-downvsbottom-upgraphconstructionhuman-in-the-loopsimplerandomsamplingCourseMapperSBERTweighting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to settle a practical question: when building an educational knowledge graph (EduKG) from MOOC lecture materials, should the pipeline extract concepts from the whole document at once, or from each slide and then merge? The authors claim the slide-by-slide 'Bottom-up' pipeline is more accurate and more useful than the 'Top-down' whole-document pipeline, based on a 19-participant user study and an expert annotation study on the CourseMapper platform. If the claim holds, course platforms can produce more reliable knowledge graphs at little extra cost. The paper also adds a Human-in-the-Loop step in which course moderators review and correct the automatically built graph before it reaches learners.

What carries the argument

The load-bearing mechanism is the two-pipeline comparison inside CourseMapper. Both pipelines use the same building blocks — PDFMiner text extraction, SIFRank SqueezeBERT keyphrase extraction, DBpedia Spotlight entity linking, and SBERT-based concept weighting — and differ only in the order of aggregation: Top-down builds one LM-EduKG from the whole material and then checks slide concepts against it, while Bottom-up builds a Slide-EduKG per slide, computes a per-slide importance score wSlide plus a material-level weight wLM, and aggregates the slide graphs into the LM-EduKG. The Bottom-up pipeline also lets partial slide-level graphs be published before the full graph is ready.

What would settle it

Re-run the SRS evaluation with the two annotators' judgments coded independently, compute inter-annotator agreement (e.g., Cohen's kappa), and compare the 0.40 versus 0.38 accuracy gap against the annotation disagreement rate; if the gap falls within the noise floor, the claimed superiority of the Bottom-up pipeline would not be supported.

Watch

Extended reading notes

Core claim

The paper's central claim is that the Bottom-up pipeline — extract keyphrases from each slide, link them to DBpedia concepts, weight each concept by its similarity to both the slide and the whole learning material, then aggregate the slide-level graphs into a learning-material-level graph — yields a more accurate EduKG than the Top-down pipeline, which extracts from the entire PDF first and then assigns concepts to slides. In the Simple Random Sampling accuracy evaluation, Bottom-up scored a mean accuracy of 0.40 against Top-down's 0.38; the user study likewise rated Bottom-up slightly higher on precision and user experience. The authors interpret this as evidence that per-slide construction better captures which concepts actually belong to which part of the material, and they add a Human-in-the-Loop step so course moderators can review and correct the graph before learners see it.

Load-bearing premise

The accuracy comparison rests on the assumption that the two annotators' Simple Random Sampling judgments of graph triples are a reliable ground truth, despite no reported inter-annotator agreement, disagreement resolution, or comparison against the course moderators who later refine the graph.

Editorial extensions

If this is right

  • If the paper is right, MOOC platforms should construct EduKGs by extracting keyphrases per slide and aggregating upward, rather than from the whole document at once.
  • Course moderators can use the proposed Human-in-the-Loop review step to lift accuracy further before the published EduKG reaches learners.
  • The same pipeline comparison can be reused for other segmented learning materials, such as video lectures with chapter markers, where per-segment extraction is feasible.
  • The evaluation protocol, which combines a user study with Simple Random Sampling accuracy checks, offers a template for comparing knowledge-graph construction pipelines in educational settings.
  • The finding implies that text segmentation granularity is a design choice that directly affects which concepts survive in the final graph, not merely a computational convenience.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A stronger test of the 0.40 versus 0.38 accuracy gap would measure inter-annotator agreement; if the gap falls within annotation disagreement noise, the ordering of the pipelines could reverse.
  • The Human-in-the-Loop step could be made adaptive by having moderators review only slides where the two pipelines disagree, concentrating effort where the pipelines are most uncertain.
  • The per-slide mechanism is generic and could transfer to non-educational domains that segment documents into pages or sections, such as legal or medical document parsing.
  • One testable extension is whether bottom-up superiority persists when document segmentation is not provided by the source but must be inferred from layout or content.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper compares two pipelines for automatic construction of Educational Knowledge Graphs (EduKGs) in the CourseMapper MOOC platform: a Top-down approach that extracts keyphrases from an entire PDF learning material at once, and a Bottom-up approach that extracts keyphrases slide-by-slide and then aggregates upward. Both pipelines use SIFRank SqueezeBERT for keyphrase extraction, DBpedia Spotlight for entity linking, and an SBERT-based weighting strategy. The authors report a user study with 19 participants evaluating Precision@15, MRR, and MAP, and a Simple Random Sampling (SRS) expert annotation study estimating triple accuracy. They also propose a Human-in-the-Loop refinement step for course moderators. The abstract and conclusion claim that the Bottom-up approach outperforms the Top-down approach and is the most effective and accurate method.

Significance. If the claimed superiority of Bottom-up over Top-down EduKG construction were well supported, the paper would offer a practical, evidence-based choice for MOOC platform developers: construct EduKGs incrementally per slide, with moderator review, rather than from whole documents. The human-in-the-loop component is a reasonable proposal for improving low accuracy, though it is not evaluated. The paper reuses components from the authors' prior work (SIFRank SqueezeBERT, SBERT weighting, DBpedia Spotlight), so the principal novelty is the pipeline comparison and the HITL integration. However, the empirical support for the central claim is the main contribution, and that support is lacking: the paper's own statistics show no significant differences, and the SRS ground truth is not validated by inter-annotator agreement or moderator comparison. No code or data are provided for reproducibility.

major comments (3)
  1. [Abstract, §5.1, §5.2, Table 1, §7] The paper's central claim that the Bottom-up approach 'outperforms' or 'is the most effective and accurate method' is directly contradicted by its own reported statistics. Section 5.1 states that 'A T-test revealed no significant differences between the models,' and Section 5.2 states 'a T-test found no statistically significant difference.' The observed differences in Table 1 are tiny: P@15 = 0.812 vs. 0.807, MRR identical at 0.941, MAP = 0.812 vs. 0.807, and SRS accuracy = 0.40 vs. 0.38 with overlapping normal-approximation intervals (±0.049 and ±0.048). A two-proportion test on the SRS judgments would yield a z-score around 0.57 and a p-value around 0.57, far from significance. The data support at most a non-significant directional tendency, not the categorical superiority claimed in the abstract, Section 6, and Section 7. This is an internal inconsistency in the manuscript, not a matter of interpretation.
  2. [§5.2] The SRS accuracy evaluation lacks the basic safeguards needed to treat the annotators' judgments as ground truth. The paper reports that two annotators reviewed different random samples (200 vs. 183/180 samples) but reports no inter-annotator agreement measure (e.g., Cohen's kappa), no procedure for resolving disagreements, and no validation against the course moderators who later refine the EduKG in the HITL process. Since each triple receives a single judgment, the measured accuracy gap of 0.40 vs. 0.38 could be annotation noise rather than a genuine pipeline difference. This is load-bearing because the accuracy comparison rests entirely on this unvalidated ground truth.
  3. [§5.1, §4.1, §4.2] The comparison depends on several free parameters chosen without reported justification or sensitivity analysis: the keyphrase count n = 15 per slide (Section 4.2), the top-down multiplier n = 15 × (number of slides) (Section 4.1), the top-k = 15 used in the user study, and the SRS sample sizes. The text says these values 'proved to cover all the possible keyphrases based on experiment' and 'more than 95% of slides contain fewer than 15 keyphrases,' but no experiments or data are shown to support these thresholds. If the conclusions are sensitive to these parameters, the observed non-significant differences could be an artifact of the chosen settings. The authors should provide an ablation or at least report the distribution of keyphrase counts per slide.
minor comments (6)
  1. [§4.1, Figure placeholders] The pipeline figures are referenced as 'Figure??a' and 'Figure??b' in Section 4.1 and 4.2; these placeholders must be resolved before publication.
  2. [§2] There is a typo: 'disucssed' should be 'discussed' (Section 2, description of EduKGs).
  3. [§5.2, Equation for µs] The formula for mean accuracy µs is rendered as a fraction '1/ns Σ f(ti)' without clear delimiter; please use a clearer notation such as (1/ns) Σᵢ f(tᵢ).
  4. [§5.1] The user study reports no confidence intervals or effect sizes for P@15, MRR, and MAP, and does not describe whether the T-test was paired or unpaired, nor how the 19 participants and 34 learning materials were allocated between pipelines. The claim that 'the bottom-up EduKG was rated more favorably' in user experience is not backed by any reported statistic.
  5. [§6] The Human-in-the-Loop approach is proposed but not empirically evaluated. Phrases such as 'This process guarantees accurate EduKGs' are too strong given the lack of validation; the authors should temper the claim or provide evidence.
  6. [General] No data or code availability statement is provided, which limits reproducibility of the experiments.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central comparison is empirical and self-contained, with only a minor self-citation that is not load-bearing.

full rationale

The paper's central claim is that the Bottom-up pipeline outperforms the Top-down pipeline for automatic EduKG construction. This claim is supported by two empirical evaluations: a user study and an SRS-based human annotation study. Neither evaluation is derived by construction from the pipeline equations or from fitted parameters. The pipelines use fixed heuristic choices, such as extracting 15 keyphrases per slide and n=15 times the number of slides for the top-down approach; these are stated as empirical observations, not as parameters fitted to the reported accuracy labels. The use of SIFRank SqueezeBERT is justified by a self-citation, but that component is held constant across both pipelines and does not by itself determine the comparative outcome, so it is not load-bearing for the central comparison. The significant weakness in the paper is that the authors themselves report no statistically significant difference between the pipelines, while the abstract and conclusion assert bottom-up superiority; that is an internal support/validity problem, not circularity. No step in the paper reduces a claimed prediction or derived result to its own inputs by definition or by a self-citation chain. Therefore, the circularity score is 0.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central comparison relies on a chain of domain assumptions rather than formal theory. Most parameters, such as keyphrase caps, top-k, and sample sizes, are chosen by hand or by unpublished experiments. The external knowledge base and embedding similarity are taken as valid, which is standard for this line of work but untested here.

free parameters (4)
  • keyphrases per slide = 15
    Set to 15 because more than 95% of slides were said to contain fewer than 15 keyphrases; this cap shapes both pipelines and is not derived from an external criterion.
  • top-down keyphrase multiplier = 15 x number of slides
    Claimed to cover all possible keyphrases based on an unspecified experiment; directly controls which concepts enter the top-down EduKG.
  • top-k for evaluation = 15
    Participants were shown the Top-15 main concepts, which can affect relevance and ranking judgments.
  • SRS sample sizes = 200, 183, 180
    Chosen by stopping when the authors judged that margin-of-error criteria were met; the per-model sample sizes differ, which complicates direct comparison.
assumptions (5)
  • domain assumption DBpedia and Wikipedia are adequate concept spaces for MOOC course materials.
    Section 3 uses DBpedia Spotlight for concept identification; if domain concepts are absent from DBpedia, both pipelines inherit the same coverage limitation.
  • domain assumption SBERT cosine similarity captures semantic relevance between learning materials and concepts.
    Section 3 uses wSBERT embeddings to weight concepts; no validation is provided that embedding similarity aligns with educational relevance.
  • domain assumption Self-reported familiarity is a valid proxy for expertise in judging concept relevance.
    Section 5.1 asks participants to choose a material they were most familiar with and then rate concept relevance; familiarity is not measured against an external standard.
  • domain assumption Two annotators' SRS judgments are a reliable ground truth without agreement metrics.
    Section 5.2 computes accuracy from single judgments per sampled triple; no inter-annotator agreement, such as Cohen's kappa, is reported.
  • domain assumption Simple random sampling of triples is representative of the whole EduKG.
    Section 5.2 applies SRS following Gao et al.; the paper does not assess whether the sampled triples cover the diversity of concepts and relation types in the EduKG.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Top-Down vs. Bottom-Up Approaches for Automatic Educational Knowledge Graph Construction in CourseMapper." pith.science (2026). https://pith.science/paper/GFKZ5Y2H

@misc{pith2026250510069,
  author       = {Pith},
  title        = {Pith review of: Top-Down vs. Bottom-Up Approaches for Automatic Educational Knowledge Graph Construction in CourseMapper},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GFKZ5Y2H}},
  note         = {Machine review of arXiv:2505.10069}
}
read the original abstract

The automatic construction of Educational Knowledge Graphs (EduKGs) is crucial for modeling domain knowledge in digital learning environments, particularly in Massive Open Online Courses (MOOCs). However, identifying the most effective approach for constructing accurate EduKGs remains a challenge. This study compares Top-down and Bottom-up approaches for automatic EduKG construction, evaluating their effectiveness in capturing and structuring knowledge concepts from learning materials in our MOOC platform CourseMapper. Through a user study and expert validation using Simple Random Sampling (SRS), results indicate that the Bottom-up approach outperforms the Top-down approach in accurately identifying and mapping key knowledge concepts. To further enhance EduKG accuracy, we integrate a Human-in-the-Loop approach, allowing course moderators to review and refine the EduKG before publication. This structured comparison provides a scalable framework for improving knowledge representation in MOOCs, ultimately supporting more personalized and adaptive learning experiences.

Figures

Figures reproduced from arXiv: 2505.10069 by the authors.

Figure 1
Figure 1. An overview of UI of CourseMapper demonstrating different features [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. UI of CourseMapper to Preview and Edit the EduKG [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. An Optimized Pipeline for Automatic Educational Knowledge Graph Construction

    cs.CY 2025-09 conditional novelty 4.0 of 10

    Optimizing an EduKG pipeline with local Wikipedia dumps, better text extraction, disambiguation, and pruning lifts accuracy from 0.40 to 0.47 and cuts processing time by 10-100x.

Reference graph

Works this paper leans on

19 extracted references · 13 canonical work pages · cited by 1 Pith paper

  1. [1]

    Heliyon (2024)

    Abu-Salih, B., Alotaibi, S.: A systematic literature review of knowledge graph construction and application in education. Heliyon (2024)

  2. [2]

    Information 14(10) (2023)

    Ain, Q.U., Chatti, M.A., Bakar, K.G.C., Joarder, S., Alatrash, R.: Automatic construction of educational knowledge graphs: A word embedding-based approach. Information 14(10) (2023)

  3. [3]

    In: Proceedings of the 14th International Conference on Education Technology and Computers

    Ain, Q.U., Chatti, M.A., Joarder, S., Nassif, I., Wobiwo Teda, B.S., Guesmi, M., Alatrash, R.: Learning channels to support interaction and collaboration in coursemapper. In: Proceedings of the 14th International Conference on Education Technology and Computers. pp. 252–260 (2022) 10 Ain et al

  4. [4]

    Ain, Q.U., Chatti, M.A., Meteng Kamdem, P.A., Alatrash, R., Joarder, S., Siep- mann, C.: Learner modeling and recommendation of learning resources using per- sonalknowledgegraphs.In:Proceedingsofthe14thLearningAnalyticsandKnowl- edge Conference. p. 273–283. LAK ’24 (2024)

  5. [5]

    Computers and Education: Artificial Intelligence 6, 100193 (2024)

    Alatrash, R., Chatti, M.A., Ain, Q.U., Fang, Y., Joarder, S., Siepmann, C.: Con- ceptgcn: Knowledge concept recommendation in moocs based on knowledge graph convolutional networks and sbert. Computers and Education: Artificial Intelligence 6, 100193 (2024)

  6. [6]

    In: IEEE 9th International Conference on Electronics Information and Emergency Communication (ICEIEC)

    Dang, F., Tang, J., Li, S.: Mooc-kg: a mooc knowledge graph for cross-platform online learning resources. In: IEEE 9th International Conference on Electronics Information and Emergency Communication (ICEIEC). IEEE (2019)

  7. [7]

    arXiv preprint arXiv:1907.09657 (2019)

    Gao, J., Li, X., Xu, Y.E., Sisman, B., Dong, X.L., Yang, J.: Efficient knowledge graph accuracy evaluation. arXiv preprint arXiv:1907.09657 (2019)

  8. [8]

    Grévisse, C., Manrique, R., Mariño, O., Rothkugel, S.: Knowledge graph-based teachersupportforlearningmaterialauthoring.In:ColombianConferenceonCom- puting. pp. 177–191. Springer (2018)

Show all 19 references
  1. [9]

    In: International Conference on Intelligent Tutoring Systems

    Jhajj, G., Zhang, X., Gustafson, J.R., Lin, F., Lin, M.P.C.: Educational knowl- edge graph creation and augmentation via llms. In: International Conference on Intelligent Tutoring Systems. pp. 292–304. Springer (2024)

  2. [10]

    Jiang, L., Liu, K., Wang, Y., Wang, D., Wang, P., Fu, Y., Yin, M.: Reinforced explainableknowledgeconceptrecommendationinmoocs.ACMTrans.Intell.Syst. Technol. 14(3) (Apr 2023)

  3. [11]

    In: Companion Proceedings of the 14th International Learning Analytics and Knowledge Conference (2024)

    Joarder, S., Chatti, M.A., Sun, A.: A no-code environment for implementing human-centered learning analytics indicators. In: Companion Proceedings of the 14th International Learning Analytics and Knowledge Conference (2024)

  4. [12]

    In: International Conference on Elec- tronic Information Engineering and Data Processing (2023)

    Liu, T., Chen, Y., Chang, L., Zhu, C.: Knowledge graph-assisted collaborative fil- tering for course recommendation in Mooc. In: International Conference on Elec- tronic Information Engineering and Data Processing (2023)

  5. [13]

    In: Joint International Semantic Technology Conference

    Manrique, R., Grévisse, C., Mariño, O., Rothkugel, S.: Knowledge graph-based core concept identification in learning resources. In: Joint International Semantic Technology Conference. pp. 36–51. Springer (2018)

  6. [14]

    In: Colombian Conference on Computing (2017)

    Manrique, R., Herazo, O., Mariño, O.: Exploring the use of linked open data for user research interest modeling. In: Colombian Conference on Computing (2017)

  7. [15]

    In: KaRS@ RecSys

    Manrique, R., Marino, O.: Knowledge graph-based weighting strategies for a schol- arly paper recommendation scenario. In: KaRS@ RecSys. pp. 5–8 (2018)

  8. [16]

    In: Proceedings of the 7th international conference on semantic systems

    Mendes, P.N., Jakob, M., García-Silva, A., Bizer, C.: Dbpedia spotlight: shedding light on the web of documents. In: Proceedings of the 7th international conference on semantic systems. pp. 1–8 (2011)

  9. [17]

    arXiv preprint arXiv:1908.10084 (2019)

    Reimers, N., Gurevych, I.: Sentence-bert: Sentence embeddings using siamese bert- networks. arXiv preprint arXiv:1908.10084 (2019)

  10. [18]

    GitHub https://github

    Shinyama, Y.: Pdfminer-python pdf parser. GitHub https://github. com/pdfminer/pdfminer. six (2007)

  11. [19]

    Technology, Pedagogy and Education (2025)

    Xia, X., Qi, W.: Learning behaviour prediction and multi-task recommendation based on a knowledge graph in moocs. Technology, Pedagogy and Education (2025)

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.