{"id":"19fbf68e-136f-4809-9be5-ecbaa2b6e3cc","arxiv_id":"2608.06530","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"An extraction-first pipeline, CatalogBrowse plus DegreeMap, reconstructs university curricula from heterogeneous public sources and certifiably optimizes feasible personalized degree plans, though no code or data is released.","lead":"KnowPlan is a two-stage AI system that first reads public university catalogs and builds a structured map of courses and degree rules, then computes personalized degree plans with an exact constraint solver. Its reported gains are plausible, but the private dataset and code are not released, so independent verification is not yet possible.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The +0.066 utility gain and 0.015 gap are computed with the same hand-set scoring function DegreeMap optimizes; absent external validation, these numbers do not establish real personalization benefit.","rationale":"The reader identified the same load-bearing concern: S(c) is both the planner's objective and the evaluation metric, with no external validation. This is the most fragile assumption because the headline planning numbers (utility gain, gap to gold) are expressed entirely in units of that metric. The paper's internal consistency is good, and the extraction-first separation is a sound design idea, but the utility evaluation is circular in the sense that the metric is hand-authored by the authors and never anchored to real student preferences. The unresolved-pool issue is real but secondary, affecting typed F1 and AST equivalence rather than the core recall and feasibility claims. A concrete external-validation test, or even a re-evaluation where the strongest baseline is given the same objective, would determine whether the utility advantage is substantive. Given the reader already marked this as the weakest assumption and conditionally accepted, no verdict change is warranted.","tokens_in":11636,"tokens_out":4868,"duration_ms":50437,"concrete_test":"Obtain a holdout set of real student elective choices or satisfaction ratings for courses at one or more Dense-6 institutions. Compute the Spearman correlation between Eq. (3) course-level scores and actual student choices or ratings; alternatively, run a blinded preference study where students compare DegreeMap's top certified plan against the strongest baseline's top plan. If correlation or preference is not significantly positive, the utility metric lacks external validity and the +0.066 / 0.015 numbers should be reframed as improvements on a self-defined scoring rule.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central planning claim rests on the personalized utility metric S(c) in Eq. (3), whose weights (w0, wD, wU, wM, wG, wO, wW, wR, wA) and diffusion edge weights are hand-set. This same S(c) serves both as the objective DegreeMap optimizes inside its lexicographic ladder and as the evaluation metric for Tables 3–5. Any planner that is not explicitly optimizing the novel components of S(c) (diffusion and unlock) is thus measured on a metric it never targeted, so the +0.066 improvement over LLMAP-style MSGS is partly structural. The paper's own ablation confirms this: a weighted-sum variant using the same S(c) achieves utility 0.873, above DegreeMap's 0.868, so the full system is not even the best optimizer of this metric. Without a human study, behavioral choice data, or a held-out preference set, the 0.015 gap to the gold graph and the +0.066 gain are uninterpretable as student-facing claims. This concern is distinct from the secondary issue that ambiguous annotation cases are excluded from F1 denominators, which could inflate extraction metrics but does not directly affect the utility headline.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes KnowPlan, a two-stage system for degree pathway planning from heterogeneous university web sources. CatalogBrowse performs profile-independent acquisition of course, program, and general-education information using obligation-driven exploration with a closure certificate, emitting three provenance-linked JSON documents. DegreeMap consumes only these documents, compiles them into a typed requirement hypergraph, and solves lexicographic CP-SAT optimization over hard feasibility, completion horizon, load/risk, personalized utility, and option value. The authors evaluate on a 100-university broad track and a six-school dense track, reporting 96.2% inventory recall, 88.7% masked-source recovery, 100.0% hard feasibility, +0.066 utility gain over the strongest baseline, and 99.5% certified coverage with a 0.015 utility gap to a privileged gold graph. The central contribution is the extraction-first boundary and the explicit measurement of the acquisition-to-planning interface, with masks and paired gold-versus-extracted graphs used to make acquisition loss visible.","tokens_in":11999,"tokens_out":5735,"duration_ms":53802,"significance":"If the empirical results held, KnowPlan would be a meaningful step for data-driven degree planning: it directly targets acquisition-planning circularity, introduces a recomputable closure certificate (Proposition 1), evaluates with masked-source and paired gold-versus-extracted tracks, reports cluster-based statistical inference, and provides detailed ablations. The extraction-side numbers are externally grounded through adjudicated gold annotations, and the paper is commendably explicit about what the closure certificate does and does not guarantee. However, the planning-side utility claims currently rest on a hand-set scoring function that is also the optimizer's objective, with no external preference validation; the unresolved annotation pool can inflate extraction metrics; and the empirical foundation is a private dataset with a supplement that is not available. These issues limit the strength of the quantitative conclusions as they stand.","major_comments":[{"comment":"The personalized-utility headline is computed with the same hand-set scoring function S(c) that DegreeMap optimizes lexicographically. The weights w0, wD, wU, wM, wG, wO, wW, wR, wA and the diffusion edge weights are fixed without external preference data, and no human study, behavioral choice data, or held-out preference set is provided. Because the LLMAP-style baseline does not optimize the diffusion and unlock terms, the +0.066 gain partly reflects the planner being measured on a metric it explicitly targets. Moreover, Table 5 shows the weighted-sum variant reaches utility 0.873, above the full system's 0.868, so the full system is not even the best optimizer of this metric. The 0.015 gold-versus-extracted gap is therefore uninterpretable as a student-facing benefit unless S(c) is validated externally or the claims are restricted to 'utility under the proposed scoring rule.'","section":"Eq. (3), Tables 3–5"},{"comment":"The weighted-sum ablation attains 0.873 utility with 96.0% certified feasibility, and the text states that the 40 uncertified requests are emitted without proof. It is not stated whether the 0.873 utility is averaged over all 1,000 requests or only over the 960 requests that were certified. If the former, the comparison mixes uncertified plans whose feasibility is unverified; if the latter, the utility values are not comparable because they are averages over different denominators. Please clarify the denominator and, ideally, report utility for both systems over the same certified subset.","section":"Table 5, Weighted sum row"},{"comment":"The annotation protocol excludes from the 'corresponding correctness denominator' any case whose source text does not determine a unique label. This can inflate inventory recall, typed F1, and AST semantic-equivalence numbers if the unresolved pool is large or systematically different from the resolved pool. The manuscript does not report the size of the unresolved pool or provide a sensitivity analysis such as worst-case scoring of unresolved items as incorrect. Please quantify the pool and show that the headline extraction results and the comparisons in Tables 1 and 2 survive alternative treatments of unresolved cases.","section":"Experiments, Annotation"},{"comment":"All central empirical results depend on private frozen snapshots, adjudicated gold graphs, and a referenced supplement that is not available in the arXiv version. Without releasing the dataset, annotation instructions, agreement statistics, and the exact schemas of the three JSON documents, the extraction and planning claims cannot be independently reproduced or checked. Please provide a public data/code release or a detailed supplementary artifact as part of the revision, or explicitly discuss why release is not possible.","section":"Experiments, Benchmark and Protocol"}],"minor_comments":[{"comment":"The sentence 'A stratified sample of 3,000 courses, approximately 5.7types and five source families...' appears garbled; please state the intended sample fraction (e.g., 5.7%) and complete the sentence.","section":"Abstract/Experiments"},{"comment":"The column label 'Cover.' is not defined in the caption or in the metrics paragraph; please spell out the term and define it precisely.","section":"Table 4"},{"comment":"The scoring expression uses 'bp(o|a,G,V)' without defining the probability estimator or how its lower confidence bound is constructed; please add the definition or a reference to the method text.","section":"Algorithm 1, line 7"},{"comment":"The caption should list the lexicographic rungs in the same order as the method text, since 'legality is settled first' is not enough to identify the full objective ladder.","section":"Figure 3 caption"},{"comment":"The phrase 'We inherit the separation of formalization from optimization' uses first-person plural in a way that is inconsistent with the otherwise impersonal style; please rephrase.","section":"Related Work"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the scope of an AI journal well. The extraction contribution is solid and thoughtfully evaluated, but the planning-side utility claims need external validation or a careful reframing before I would recommend acceptance. The weighted-sum ablation and the unresolved-pool exclusion should be handled transparently. I also note that the absence of any public artifact is a significant limitation for an empirical paper of this scale; a release or detailed supplement would substantially strengthen the archival value."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know first: the extraction half of this paper is the real contribution, and it is better than the surrounding LLM-agent noise. CatalogBrowse is a profile-independent crawler that precommits to actions, scores them by lower-confidence expected obligation gain, and stops on a closure certificate over index, schema, provenance, and reference completeness. That is a genuinely different design from task-completion web agents, and the masked-source recovery test—removing index pages and charging agents for what they never discovered—is a clean way to make acquisition loss visible. On top of that, the authors check hard feasibility against the adjudicated gold graph rather than the extracted graph, which is exactly the right way to avoid rewarding an extractor that drops constraints it failed to parse. The extraction results, with 96.2% inventory recall and 88.7% masked recovery at 47% less source access, are internally consistent and the ablations are informative. If I needed a system that reconstructs a curriculum from heterogeneous public sources, I would build on this.\n\nThe soft spots are concentrated in the planning evaluation. The +0.066 utility gain and the 0.015 gap to the gold graph are computed with Eq. (3), the same hand-set scoring function DegreeMap optimizes lexicographically. There is no external preference data, no human study, no held-out preference set. The paper's own ablation makes the circularity concrete: a weighted-sum variant using the same S(c) reaches 0.873 utility, above DegreeMap's 0.868, so DegreeMap is not even the best optimizer of its own metric. That means the headline utility numbers measure how well the planner fits a self-defined scoring rule, not how much students would benefit. The unresolved-pool exclusion is a secondary concern—it can inflate F1, but it does not drive the utility claim. And the paper releases no code or data, which weakens the reproducibility promise that the closure certificate is supposed to deliver; a third party cannot actually recompute the certificate without the frozen snapshots and tooling.\n\nWho is this for? Researchers working on web agents for structured domain acquisition will get real value from the extraction-first boundary and the masked-source evaluation. Degree-planning tool builders should read it too, but they should treat the personalization numbers as hypotheses to be validated, not results. I would send this to serious peer review: the extraction contribution alone is worth referee time. But I would push hard for artifact release and for an external preference validation before the planning claims are presented as student-facing. As it stands, I would not cite the utility numbers in my own work, but I would cite the extraction methodology.","headline":"The extraction-first pipeline and masked-source protocol are genuinely good; the personalization headline, however, rests on a self-defined utility metric and no released artifacts, so the planning claims need external validation.","tokens_in":767,"tokens_out":842,"would_cite":false,"duration_ms":21551,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"KnowPlan separates curriculum extraction from degree planning, claiming a certified 99.5% planning coverage with a 0.015 utility gap to a gold graph.","keywords":["degree pathway planning","curriculum extraction","web exploration agent","closure certificate","typed requirement hypergraph","lexicographic CP-SAT","personalized utility","masked-source recovery"],"falsifier":"Show a set of real students a blinded pair of plans—one from DegreeMap and one from the strongest baseline planner—each certified feasible and matched for the same transcript and preferences, and ask which they would take. If students' choices do not predict the higher Eq. (3) utility better than chance across a sample of 100 such comparisons, the personalized-utility claim is falsified.","tokens_in":11440,"feed_emoji":"🎓","tokens_out":7772,"duration_ms":70041,"temperature":0.7,"pith_summary":"The paper argues that degree planning from public university sources fails when the planner drives its own crawling, because pages not needed by the current plan are never acquired and missing rules stay invisible. To break this circularity, KnowPlan enforces an extraction-first boundary: CatalogBrowse reconstructs the curriculum from heterogeneous catalogs, JSON endpoints, and PDFs before any user profile is seen, and DegreeMap then plans over the extracted graph. The authors report 96.2% inventory recall and 88.7% masked-source recovery with 47% fewer source accesses than exhaustive crawling, and a planner that holds 100% hard feasibility while improving personalized utility by +0.066 over the strongest baseline. They frame the 0.015 utility gap versus a privileged gold graph as the measurable cost of acquisition loss.","feed_headline":"Extraction-first AI builds certified degree plans from raw catalogs","feed_subtitle":"A two-stage pipeline hits 96.2% catalog recall and 99.5% certified planning, within 0.015 of a gold standard.","key_machinery":"The paper's load-bearing mechanism is the acquisition–planning interface, realized by two coupled components. In CatalogBrowse, a finite set of atomic catalog obligations drives an agent that selects actions by lower-confidence expected marginal obligation gain per unit of source cost and terminates on a closure certificate, emitting three provenance-linked JSON documents. In DegreeMap, those documents compile into a typed requirement hypergraph and are solved lexicographically with CP-SAT over hard feasibility, completion horizon, load and risk, personalized utility, and option value, with each stage narrowing the search space while preserving certifiability.","core_discovery":"The central claim is that a profile-independent web agent which explores a frozen official-source snapshot by maximizing a lower-confidence estimate of expected marginal gain over a finite set of atomic catalog obligations, and which stops only on a closure certificate covering index, schema, provenance, and reference completeness, can reconstruct an institution's curriculum well enough to support exact personalized planning. The paper further claims that a lexicographic CP-SAT planner over a typed requirement hypergraph, with each objective's proven optimum fixed before the next, certifies plans within a 60-second budget, keeping hard feasibility and providing a measurable utility gap to a gold graph.","pith_inferences":["A natural next experiment is to validate the Eq. (3) utility function behaviorally: present real students with blinded certified plans from DegreeMap and from the strongest baseline, and test whether the +0.066 utility advantage corresponds to actual preference. The paper does not report such a study, but the architecture would remain unchanged if the weights were replaced by a validated objective","The masked-source protocol could be generalized into a benchmark for any domain where an agent must reconstruct a world model before an objective is known, such as tax codes, building codes, or grant guidelines; the closure certificate would play the same role as a stopping guarantee.","Reading the ablations together, the largest single lever on end-to-end quality is the extraction stage: removing adapters costs 11.6 points of inventory F1, whereas planner-side utility shifts are around 0.06. An institution adopting this pipeline should therefore invest in catalog adapters before planner tuning.","The 0.015 gap to the gold graph is likely a lower bound on real-world loss, because the gold graph is also derived from the same frozen snapshot and the same annotators' interpretations; a fully independent ground-truth source could widen the gap."],"forward_implications":["Degree planning for a new institution reduces to running a profile-free crawler once; the emitted JSON documents can serve any number of student requests without exposing the planner to the web.","Plans produced from extracted graphs are checked against a gold graph rather than the extracted graph, so a certified plan is feasible with respect to the institution's true requirements, not merely the extracted approximation.","The closure certificate is snapshot-relative and recomputable from a hash-chained trace, making the extraction result auditable by a third party without rerunning the agent.","Masked-source recovery introduces a training signal that does not reward task completion but rewards completeness; better adapters or source discovery should raise recall without sacrificing efficiency.","The reported 0.015 utility gap calibrates the student-facing cost of acquisition, and the paper's Table 4 shows that weaker extractors widen that gap to 0.117, so improving extraction quality has a direct, measurable effect on plan quality."],"supporting_citations":[{"why":"Supplies the ReAct loop used by the ReAct-Qwen baseline that CatalogBrowse must beat.","marker":"(Yao et al. 2023)"},{"why":"Go-Browse is the closest structural web-exploration baseline, treated as a graph, against which the masked-source recovery gain is measured.","marker":"(Gandhi and Neubig 2025)"},{"why":"LLM+P is a planner baseline that separates formalization from optimization; DegreeMap inherits that separation but the domain is not given.","marker":"(Liu et al. 2023)"},{"why":"LLMAP-style multi-objective planning is the strongest planner baseline that DegreeMap outperforms on personalized utility.","marker":"(Yuan et al. 2025)"},{"why":"Defines recommendation with complex constraints, the line of work that assumes the requirement structure is already encoded; this paper removes that assumption.","marker":"(Parameswaran, Venetis, and Garcia-Molina 2011a)"},{"why":"Provides the adaptive submodularity theory that the paper uses to explain why no optimality guarantee is claimed for CatalogBrowse, motivating the closure-certificate guarantee.","marker":"(Golovin and Krause 2011)"}],"fun_headline_variants":["Certified degree plans from raw catalogs at 99.5% success","Profile-blind crawler plus CP-SAT certifies 99.5% degree plans","KnowPlan: From PDFs to provable plans with 0.015 utility gap","Extraction-first AI certifies degree plans, 96% recall, 99.5% success","Crawl first, plan later: KnowPlan's two-stage certification guarantee"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's headline utility numbers depend on a hand-designed scoring function (Eq. 3) that serves simultaneously as the planner's objective and as the evaluation metric, and no external data is provided showing that this function matches what students actually value; if it does not, the +0.066 gain and 0.015 gap measure the optimizer's fit to its own score rather than real student satisfaction.","fun_headline_variants_meta":{"raw":{"variants":["Certified degree plans from raw catalogs at 99.5% success","Profile-blind crawler plus CP-SAT certifies 99.5% degree plans","KnowPlan: From PDFs to provable plans with 0.015 utility gap","Extraction-first AI certifies degree plans, 96% recall, 99.5% success","Crawl first, plan later: KnowPlan's two-stage certification guarantee"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00069,"raw_usage":{"total_tokens":3157,"prompt_tokens":1011,"completion_tokens":2146,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":627,"completion_tokens_details":{"reasoning_tokens":2037}},"tokens_in":627,"tokens_out":2146,"duration_ms":16223,"temperature":1.0,"reasoning_tokens":2037,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T04:17:18.079304+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Show a set of real students a blinded pair of plans—one from DegreeMap and one from the strongest baseline planner—each certified feasible and matched for the same transcript and preferences, and ask which they would take. If students' choices do not predict the higher Eq. (3) utility better than chance across a sample of 100 such comparisons, the personalized-utility claim is falsified.","supporting_citations":[{"cited_title":"Findings of the Association for Computational Linguistics: EMNLP 2025 , pages =","cited_arxiv_id":null,"evidence_quote":"LLMAP-style multi-objective planning is the strongest planner baseline that DegreeMap outperforms on personalized utility."}],"review_version":1}