REVIEW 4 major objections 5 minor 36 references
KNOWPLAN: Knowledge-Driven AI Agents for Smart Degree Pathway Planning
T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read KnowPlan separates curriculum extraction from degree planning, claiming a certified 99.5% planning coverage with a 0.015 utility gap to a gold graph.
desk verdict The extraction-first pipeline and masked-source protocol are genuinely good; the personalization headline, however, rests on a self-defined utility metric and no released artifacts, so the planning claims need external validation. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The paper's load-bearing mechanism is the acquisition–planning interface, realized by two coupled components. In CatalogBrowse, a finite set of atomic catalog obligations drives an agent that selects actions by lower-confidence expected marginal obligation gain per unit of source cost and terminates on a closure certificate, emitting three provenance-linked JSON documents. In DegreeMap, those documents compile into a typed requirement hypergraph and are solved lexicographically with CP-SAT over hard feasibility, completion horizon, load and risk, personalized utility, and option value, with each stage narrowing the search space while preserving certifiability.
What would settle it
Show a set of real students a blinded pair of plans—one from DegreeMap and one from the strongest baseline planner—each certified feasible and matched for the same transcript and preferences, and ask which they would take. If students' choices do not predict the higher Eq. (3) utility better than chance across a sample of 100 such comparisons, the personalized-utility claim is falsified.
Extended reading notes
Core claim
The central claim is that a profile-independent web agent which explores a frozen official-source snapshot by maximizing a lower-confidence estimate of expected marginal gain over a finite set of atomic catalog obligations, and which stops only on a closure certificate covering index, schema, provenance, and reference completeness, can reconstruct an institution's curriculum well enough to support exact personalized planning. The paper further claims that a lexicographic CP-SAT planner over a typed requirement hypergraph, with each objective's proven optimum fixed before the next, certifies plans within a 60-second budget, keeping hard feasibility and providing a measurable utility gap to a gold graph.
Load-bearing premise
The paper's headline utility numbers depend on a hand-designed scoring function (Eq. 3) that serves simultaneously as the planner's objective and as the evaluation metric, and no external data is provided showing that this function matches what students actually value; if it does not, the +0.066 gain and 0.015 gap measure the optimizer's fit to its own score rather than real student satisfaction.
Editorial extensions
If this is right
- Degree planning for a new institution reduces to running a profile-free crawler once; the emitted JSON documents can serve any number of student requests without exposing the planner to the web.
- Plans produced from extracted graphs are checked against a gold graph rather than the extracted graph, so a certified plan is feasible with respect to the institution's true requirements, not merely the extracted approximation.
- The closure certificate is snapshot-relative and recomputable from a hash-chained trace, making the extraction result auditable by a third party without rerunning the agent.
- Masked-source recovery introduces a training signal that does not reward task completion but rewards completeness; better adapters or source discovery should raise recall without sacrificing efficiency.
- The reported 0.015 utility gap calibrates the student-facing cost of acquisition, and the paper's Table 4 shows that weaker extractors widen that gap to 0.117, so improving extraction quality has a direct, measurable effect on plan quality.
Reading between the lines
- A natural next experiment is to validate the Eq. (3) utility function behaviorally: present real students with blinded certified plans from DegreeMap and from the strongest baseline, and test whether the +0.066 utility advantage corresponds to actual preference. The paper does not report such a study, but the architecture would remain unchanged if the weights were replaced by a validated objective
- The masked-source protocol could be generalized into a benchmark for any domain where an agent must reconstruct a world model before an objective is known, such as tax codes, building codes, or grant guidelines; the closure certificate would play the same role as a stopping guarantee.
- Reading the ablations together, the largest single lever on end-to-end quality is the extraction stage: removing adapters costs 11.6 points of inventory F1, whereas planner-side utility shifts are around 0.06. An institution adopting this pipeline should therefore invest in catalog adapters before planner tuning.
- The 0.015 gap to the gold graph is likely a lower bound on real-world loss, because the gold graph is also derived from the same frozen snapshot and the same annotators' interpretations; a fully independent ground-truth source could widen the gap.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes KnowPlan, a two-stage system for degree pathway planning from heterogeneous university web sources. CatalogBrowse performs profile-independent acquisition of course, program, and general-education information using obligation-driven exploration with a closure certificate, emitting three provenance-linked JSON documents. DegreeMap consumes only these documents, compiles them into a typed requirement hypergraph, and solves lexicographic CP-SAT optimization over hard feasibility, completion horizon, load/risk, personalized utility, and option value. The authors evaluate on a 100-university broad track and a six-school dense track, reporting 96.2% inventory recall, 88.7% masked-source recovery, 100.0% hard feasibility, +0.066 utility gain over the strongest baseline, and 99.5% certified coverage with a 0.015 utility gap to a privileged gold graph. The central contribution is the extraction-first boundary and the explicit measurement of the acquisition-to-planning interface, with masks and paired gold-versus-extracted graphs used to make acquisition loss visible.
Significance. If the empirical results held, KnowPlan would be a meaningful step for data-driven degree planning: it directly targets acquisition-planning circularity, introduces a recomputable closure certificate (Proposition 1), evaluates with masked-source and paired gold-versus-extracted tracks, reports cluster-based statistical inference, and provides detailed ablations. The extraction-side numbers are externally grounded through adjudicated gold annotations, and the paper is commendably explicit about what the closure certificate does and does not guarantee. However, the planning-side utility claims currently rest on a hand-set scoring function that is also the optimizer's objective, with no external preference validation; the unresolved annotation pool can inflate extraction metrics; and the empirical foundation is a private dataset with a supplement that is not available. These issues limit the strength of the quantitative conclusions as they stand.
major comments (4)
- [Eq. (3), Tables 3–5] The personalized-utility headline is computed with the same hand-set scoring function S(c) that DegreeMap optimizes lexicographically. The weights w0, wD, wU, wM, wG, wO, wW, wR, wA and the diffusion edge weights are fixed without external preference data, and no human study, behavioral choice data, or held-out preference set is provided. Because the LLMAP-style baseline does not optimize the diffusion and unlock terms, the +0.066 gain partly reflects the planner being measured on a metric it explicitly targets. Moreover, Table 5 shows the weighted-sum variant reaches utility 0.873, above the full system's 0.868, so the full system is not even the best optimizer of this metric. The 0.015 gold-versus-extracted gap is therefore uninterpretable as a student-facing benefit unless S(c) is validated externally or the claims are restricted to 'utility under the proposed scoring rule.'
- [Table 5, Weighted sum row] The weighted-sum ablation attains 0.873 utility with 96.0% certified feasibility, and the text states that the 40 uncertified requests are emitted without proof. It is not stated whether the 0.873 utility is averaged over all 1,000 requests or only over the 960 requests that were certified. If the former, the comparison mixes uncertified plans whose feasibility is unverified; if the latter, the utility values are not comparable because they are averages over different denominators. Please clarify the denominator and, ideally, report utility for both systems over the same certified subset.
- [Experiments, Annotation] The annotation protocol excludes from the 'corresponding correctness denominator' any case whose source text does not determine a unique label. This can inflate inventory recall, typed F1, and AST semantic-equivalence numbers if the unresolved pool is large or systematically different from the resolved pool. The manuscript does not report the size of the unresolved pool or provide a sensitivity analysis such as worst-case scoring of unresolved items as incorrect. Please quantify the pool and show that the headline extraction results and the comparisons in Tables 1 and 2 survive alternative treatments of unresolved cases.
- [Experiments, Benchmark and Protocol] All central empirical results depend on private frozen snapshots, adjudicated gold graphs, and a referenced supplement that is not available in the arXiv version. Without releasing the dataset, annotation instructions, agreement statistics, and the exact schemas of the three JSON documents, the extraction and planning claims cannot be independently reproduced or checked. Please provide a public data/code release or a detailed supplementary artifact as part of the revision, or explicitly discuss why release is not possible.
minor comments (5)
- [Abstract/Experiments] The sentence 'A stratified sample of 3,000 courses, approximately 5.7types and five source families...' appears garbled; please state the intended sample fraction (e.g., 5.7%) and complete the sentence.
- [Table 4] The column label 'Cover.' is not defined in the caption or in the metrics paragraph; please spell out the term and define it precisely.
- [Algorithm 1, line 7] The scoring expression uses 'bp(o|a,G,V)' without defining the probability estimator or how its lower confidence bound is constructed; please add the definition or a reference to the method text.
- [Figure 3 caption] The caption should list the lexicographic rungs in the same order as the method text, since 'legality is settled first' is not enough to identify the full objective ladder.
- [Related Work] The phrase 'We inherit the separation of formalization from optimization' uses first-person plural in a way that is inconsistent with the otherwise impersonal style; please rephrase.
Circularity Check
Utility headline is measured on the same hand-set S(c) that DegreeMap optimizes; extraction and feasibility results are externally grounded.
-
self definitional
[Eq. (3) in 'DegreeMap: Personalized Degree Planning'; Tables 3-5 in 'Results']
"For course c, DegreeMap separates direct semantic interest I0(c), typed graph diffusion ID(c), prerequisite unlock value U(c), major and GE gains, future option value, workload, risk, and aversion: S(c) = w0 I0(c) + wD ID(c) + wU U(c) + wM M(c) + wG G(c) + wO O(c) - wW W(c) - wR R(c) - wA A(c). Objectives are solved lexicographically over hard feasibility, minimum completion horizon, load and risk, personalized utility, option value, and a deterministic tie-break."
The same hand-set S(c) is the objective DegreeMap maximizes inside its lexicographic ladder and the metric reported as 'Utility' in Tables 3-5. The paper provides no external preference data or human validation anchoring S(c) to student satisfaction; the weights and diffusion edge weights are chosen rather than fitted to held-out preferences. Hence the headline +0.066 utility gain is the optimizer outperforming baselines that do not optimize S(c) on its own scoring rule, and the 0.015 gold-graph gap is a gap in the same self-defined score. This is partly structural.
full rationale
CatalogBrowse's inventory recall, masked-source recovery, and DegreeMap's hard feasibility are checked against an adjudicated gold inventory/graph and an independent verifier, so those claims are not circular. The only self-referential element is personalized utility: Eq. (3) defines S(c) with hand-set weights, DegreeMap maximizes S(c), and Tables 3-5 report S(c) as the Utility column. The +0.066 gain over planners that do not optimize S(c) is therefore partly a consequence of optimizing the evaluation metric, and the 0.015 gap to the privileged gold graph is also measured on that same self-defined score. The paper's weighted-sum ablation (0.873 vs. 0.868) shows the result is not fully forced, but without external preference data or a human study the utility numbers do not establish real student-facing benefit. This is a partial self-definitional circularity in the personalization claim; it does not undermine the extraction or hard-feasibility results.
Assumptions & free parameters
free parameters (5)
- Utility weights w0, wD, wU, wM, wG, wO, wW, wR, wA in Eq. (3) =
not reported
- Diffusion edge weights on typed adjacency =
not reported
- Unlock discount factor and division rule for U(c) =
not reported
- Obligation weights v_o in completion potential F(psi), Eq. (1) =
not reported
- LCB prior and confidence level in Eq. (2) =
not reported
assumptions (5)
- ad hoc to paper The utility function S(c) in Eq. (3) is a valid representation of student preferences for personalized planning.
- domain assumption The attested inventory used as the recall denominator is complete and correct for each of the 100 universities.
- domain assumption Gold curriculum graphs and annotations derived from official source text are ground truth, with unique labels determined by the source.
- domain assumption Every planning-relevant policy is expressible in the typed requirement hypergraph; unsupported natural-language exceptions only reduce certified coverage.
- ad hoc to paper Ambiguous official texts whose unique label cannot be determined contribute no information and can be safely excluded from correctness denominators.
invented entities (2)
-
Atomic catalog obligations (obligation set O)
-
Closure certificate over index, schema, provenance, and reference completeness
Cite this review
Pith. "Pith review of KNOWPLAN: Knowledge-Driven AI Agents for Smart Degree Pathway Planning." pith.science (2026). https://pith.science/paper/GHK5K7VM
@misc{pith2026260806530,
author = {Pith},
title = {Pith review of: KNOWPLAN: Knowledge-Driven AI Agents for Smart Degree Pathway Planning},
year = {2026},
howpublished = {\url{https://pith.science/paper/GHK5K7VM}},
note = {Machine review of arXiv:2608.06530}
}
read the original abstract
Planning a degree from official university sources requires solving two problems in order. The institution's curriculum must first be reconstructed from catalogs, departmental pages, JSON endpoints, and PDFs that share no schema, and only then can a student-specific path be optimized under prerequisite logic and overlapping requirement constraints. Coupling the two lets each failure mode hide the other, because a planner that drives its own crawling never learns facts its current plan does not need. We present KnowPlan, which enforces an extraction-first boundary and measures the interface between the stages rather than assuming it. CatalogBrowse explores with no access to any user profile. It scores legal actions by lower-confidence expected marginal gain over a finite set of atomic catalog obligations per unit of source access, parses deterministically through platform adapters with a span-constrained clause-to-AST model fallback, and terminates on a closure certificate over index, schema, provenance, and reference completeness instead of a reward threshold. Its output contract is three provenance-linked JSON documents. DegreeMap consumes only those documents. It compiles them into a typed requirement hypergraph and optimizes lexicographically with CP-SAT over hard feasibility, completion horizon, load and risk, personalized utility, and option value, so that each stage optimizes inside the previous stage's proven optimum and stays certifiable within the solver budget. Across a 100-university broad track and a six-school dense track, CatalogBrowse reaches 96.2% inventory recall and 88.7% masked-source recovery at 47% less source access than an exhaustive crawler, DegreeMap holds 100.0% hard feasibility while improving personalized utility by +0.066 over the strongest baseline, and the full pipeline certifies 99.5% of requests with a utility gap to the privileged gold graph of 0.015.
Figures
Reference graph
Works this paper leans on
-
[1]
arXiv preprint arXiv:2506.03533 , year=
Go-Browse: Training Web Agents with Structured Exploration , author=. arXiv preprint arXiv:2506.03533 , year=
-
[2]
International Conference on Learning Representations , year=
ReAct: Synergizing Reasoning and Acting in Language Models , author=. International Conference on Learning Representations , year=
-
[3]
arXiv preprint arXiv:2506.21506 , year=
Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge , author=. arXiv preprint arXiv:2506.21506 , year=
-
[4]
arXiv preprint arXiv:2412.05467 , year=
The BrowserGym Ecosystem for Web Agent Research , author=. arXiv preprint arXiv:2412.05467 , year=
-
[5]
arXiv preprint arXiv:2410.16464 , year=
Beyond Browsing: API-Based Web Agents , author=. arXiv preprint arXiv:2410.16464 , year=
-
[6]
International Conference on Machine Learning , year=
TravelPlanner: A Benchmark for Real-World Planning with Language Agents , author=. International Conference on Machine Learning , year=
-
[7]
arXiv preprint arXiv:2405.20625 , year=
Robust Planning with LLM-Modulo Framework: Case Study in Travel Planning , author=. arXiv preprint arXiv:2405.20625 , year=
-
[8]
ACM Transactions on Information Systems , volume=
Recommendation Systems with Complex Constraints: A Course Recommendation Perspective , author=. ACM Transactions on Information Systems , volume=
Show all 36 references
-
[9]
Proceedings of the AAAI Conference on Artificial Intelligence , volume=
Recovering Concept Prerequisite Relations from University Course Dependencies , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=
-
[10]
Proceedings of the AAAI Conference on Artificial Intelligence , volume=
What Should I Learn First: Introducing LectureBank for NLP Education and Prerequisite Chain Learning , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=
-
[11]
Proceedings of the Annual Meeting of the Association for Computational Linguistics , year=
TutorialBank: A Manually-Collected Corpus for Prerequisite Chains, Survey Extraction and Resource Recommendation , author=. Proceedings of the Annual Meeting of the Association for Computational Linguistics , year=
-
[12]
Journal of Artificial Intelligence Research , volume=
Adaptive Submodularity: Theory and Applications in Active Learning and Stochastic Optimization , author=. Journal of Artificial Intelligence Research , volume=
-
[13]
arXiv preprint arXiv:2505.09388 , year=
Qwen3 Technical Report , author=. arXiv preprint arXiv:2505.09388 , year=
-
[14]
and Venetis, Petros and Garcia-Molina, Hector , title =
Parameswaran, Aditya G. and Venetis, Petros and Garcia-Molina, Hector , title =. ACM Transactions on Information Systems , volume =. 2011 , doi =
2011
-
[15]
Personalized Course Sequence Recommendations , journal =
Xu, Jie and Xing, Tianwei and. Personalized Course Sequence Recommendations , journal =. 2016 , doi =
2016
-
[16]
Decision Support Systems , volume =
Mohamed, Abdallah , title =. Decision Support Systems , volume =. 2015 , doi =
2015
-
[17]
Computer Applications in Engineering Education , volume =
Shakhsi-Niaei, Majid and Abuei-Mehrizi, Hossein , title =. Computer Applications in Engineering Education , volume =. 2020 , doi =
2020
-
[18]
Alan , title =
Bowman, R. Alan , title =. INFORMS Journal on Applied Analytics , volume =. 2021 , doi =
2021
-
[19]
Khamechian, Mohammad and Petering, Matthew E. H. , title =. Computers & Industrial Engineering , volume =. 2022 , doi =
2022
-
[20]
and Vagianou, Evgenia and Vardoulias, George , title =
Christou, Ioannis T. and Vagianou, Evgenia and Vardoulias, George , title =. INFORMS Journal on Applied Analytics , volume =. 2024 , doi =
2024
-
[21]
arXiv preprint arXiv:2412.19312 , year =
From Interests to Insights: An. arXiv preprint arXiv:2412.19312 , year =. 2412.19312 , archivePrefix =
-
[22]
Spahic-Bogdanovic, Maja and Witschel, Hans Friedrich and Porumboiu, Daniele and Rosati, Piermichele and Hierro Canchari, Piero Jean Pier and Kostic, Milan , title =. Selected Papers of the 3rd International Workshop on Hybrid Artificial Intelligence and Enterprise Modelling fo...
2025
-
[23]
Proceedings of the Eleventh Italian Conference on Computational Linguistics (CLiC-it 2025) , pages =
Mazzei, Samuele and Zambotto, Lorenzo and Tealdo, Gabriele and Macagno, Alberto and Palmero Aprosio, Alessio , title =. Proceedings of the Eleventh Italian Conference on Computational Linguistics (CLiC-it 2025) , pages =. 2025 , url =
2025
-
[24]
Nguyen, Long S. T. and Vo, Quynh T. N. and Luu, Hung C. and Quan, Tho T. , title =. Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics , pages...
2025
-
[25]
Knowledge Graphs , journal =
Hogan, Aidan and Blomqvist, Eva and Cochez, Michael and d'Amato, Claudia and. Knowledge Graphs , journal =. 2021 , doi =
2021
-
[26]
Heliyon , volume =
Abu-Salih, Bilal and Alotaibi, Salihah , title =. Heliyon , volume =. 2024 , doi =
2024
-
[27]
Qu, Kechen and Li, Kam Cheong and Wong, Billy T. M. and Wu, Manfred M. F. and Liu, Mengjin , title =. Electronics , volume =. 2024 , doi =
2024
-
[28]
Advances in Neural Information Processing Systems , volume =
Deng, Xiang and Gu, Yu and Zheng, Boyuan and Chen, Shijie and Stevens, Sam and Wang, Boshi and Sun, Huan and Su, Yu , title =. Advances in Neural Information Processing Systems , volume =. 2023 , doi =
2023
-
[29]
and Zhu, Hao and Zhou, Xuhui and Lo, Robert and Sridhar, Abishek and Cheng, Xianyi and Ou, Tianyue and Bisk, Yonatan and Fried, Daniel and Alon, Uri and Neubig, Graham , title =
Zhou, Shuyan and Xu, Frank F. and Zhu, Hao and Zhou, Xuhui and Lo, Robert and Sridhar, Abishek and Cheng, Xianyi and Ou, Tianyue and Bisk, Yonatan and Fried, Daniel and Alon, Uri and Neubig, Graham , title =. International Conference on Learning Representations , year =
-
[30]
Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages =
He, Hongliang and Yao, Wenlin and Ma, Kaixin and Yu, Wenhao and Dai, Yong and Zhang, Hongming and Lan, Zhenzhong and Yu, Dong , title =. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages =. 2024 , doi =
2024
-
[31]
arXiv preprint arXiv:2304.11477 , year =
Liu, Bo and Jiang, Yuqian and Zhang, Xiaohan and Liu, Qiang and Zhang, Shiqi and Biswas, Joydeep and Stone, Peter , title =. arXiv preprint arXiv:2304.11477 , year =. 2304.11477 , archivePrefix =
-
[32]
Findings of the Association for Computational Linguistics: EMNLP 2025 , pages =
Yuan, Liangqi and Han, Dong-Jun and Brinton, Christopher and Brunswicker, Sabine , title =. Findings of the Association for Computational Linguistics: EMNLP 2025 , pages =. 2025 , doi =
2025
-
[33]
arXiv preprint arXiv:2509.00997 , year=
Supporting our AI overlords: Redesigning data systems to be agent-first , author=. arXiv preprint arXiv:2509.00997 , year=
-
[34]
arXiv preprint arXiv:2603.20576 , year=
Can AI Agents Answer Your Data Questions? A Benchmark for Data Agents , author=. arXiv preprint arXiv:2603.20576 , year=
-
[35]
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=
CapNav: Benchmarking Vision Language Models on Capability-conditioned Indoor Navigation , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=
-
[36]
Proceedings of the AAAI Conference on Artificial Intelligence (AAAI) , year=
Affordance-R1: Reinforcement Learning for Generalizable Affordance Reasoning in Multimodal Large Language Model , author=. Proceedings of the AAAI Conference on Artificial Intelligence (AAAI) , year=
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.