{"id":"32b3d0c7-3069-4fa1-9389-17d3fab9fca2","arxiv_id":"2507.18668","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"DGAKT, a subgraph-based dual-attention GNN, claims to improve knowledge tracing accuracy, AUC, and resource efficiency by capturing high-order paths in integrated student-exercise-KC graphs.","lead":"A graph neural network for knowledge tracing, DGAKT, builds integrated student-exercise-KC subgraphs and uses dual attention to predict quiz performance. The paper claims top accuracy and AUC on three datasets plus better resource efficiency than existing KT models.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The subgraph construction in §2.2 admits future and test-set edges; in the unseen-exercise experiment only test students can have interacted with the target exercise, so DGAKT's reported gains reflect label leakage rather than learning.","rationale":"The reader's weakest assumption is the same load-bearing concern identified here: subgraph construction in §2.2 lacks temporal filtering, and the unseen-exercise split by exercise type forces the model to use test-set edges. My independent review finds no way around that conclusion from the manuscript text. The paper would need to show that 'students who have interacted with the target exercise' means only students whose interactions preceded the target timestamp; no such statement appears. The unseen-exercise experiment is structurally incompatible with a causal interpretation unless the target exercise appears in training, which the protocol explicitly prevents. This is not a disagreement about modeling philosophy or consensus; it is a correctness risk in the central empirical claim. The efficiency claim is also weak, since Figure 5 reports parameter counts but no measured runtime or memory, and Table 7 lists DGAKT complexity as O(n·d^2 + n^2·d), which is quadratic in the subsequence length; however, the leakage concern alone is sufficient to reject the paper's central claim. I therefore agree with the reader's REJECT verdict and recommend no change in the verdict, while noting that a causal reimplementation with strict timestamp filtering would be the decisive follow-up.","tokens_in":12591,"tokens_out":3794,"duration_ms":43461,"concrete_test":"Re-run DGAKT with a strict causal subgraph filter: for each target interaction at time t, include an edge (student, exercise, timestamp) in the subgraph only if timestamp < t. Repeat the main Table 2 experiments on ASSIST2017 and the unseen-exercise splits in Table 5 using this filter, dropping or reporting separately any target exercise for which no other student has a prior interaction. If the reported AUC drops materially, or if the unseen-exercise tables cannot be reproduced without using future edges, the leakage concern is confirmed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central performance claim rests on Table 2 and the unseen-case results in Tables 5–6, and it requires that every input to the model be available before the target response is observed. Section 2.2 constructs the subgraph from 'the students who have interacted with the target exercise' with no temporal constraint, and then states that 'all relationships between the selected nodes are included except the target interaction.' This means edges with timestamps later than the target interaction, including edges from the test set, can enter the subgraph. The unseen-exercise protocol makes the problem acute: because the test exercises are chosen from exercise types not seen in training, no training interaction with those exercises exists, so the only 'students who have interacted with the target exercise' are students in the test set. DGAKT is therefore given other students' actual responses to the same exercise as input features when predicting the target student's response. Such leakage is exactly the kind of signal that would inflate the reported improvements, e.g., ASSIST2017 AUC rising from 0.7985 (IGMC-KC) to 0.8994 (DGAKT) in Table 2 and from 0.8442 to 0.9480 in Table 5. Because the paper never demonstrates a causal protocol that excludes future edges, the headline performance advantage is not supported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DGAKT, a graph neural network for knowledge tracing that constructs local subgraphs from an integrated student-exercise-KC graph and applies dual (local and global) graph attention layers to predict a student's correctness. The authors claim state-of-the-art accuracy and AUC on EdNet, ASSIST2017, and Junyi, plus superior resource efficiency with only 60k parameters. The experimental sections include overall comparisons, ablations, unseen-exercise tests, complexity analysis, and an interpretability case study.","tokens_in":12876,"tokens_out":3603,"duration_ms":39815,"significance":"If the empirical results were valid, the subgraph-based dual-attention design would be a useful contribution to large-scale knowledge tracing, particularly for inductive settings involving new exercises or KCs, and the small parameter count would be an attractive practical property. However, the evaluation as written appears to admit future and test-set interactions into the subgraphs, so the central performance claims are not supported. The efficiency claim is also not backed by actual timing or memory measurements. The paper's significance therefore hinges entirely on whether the leakage can be removed and the experiments redone.","major_comments":[{"comment":"The subgraph construction described in Section 2.2 is not temporally causal. It selects 'the students who have interacted with the target exercise' and then includes 'all relationships between the selected nodes ... except the target interaction.' For a prediction at time t, this permits edges with timestamps after t, including edges from the validation and test portions of the chronological split. In the unseen-exercise experiments (Section 3.4, Tables 5-6), the test exercises are types not seen in training, so no training interaction with those exercises exists; the only 'students who have interacted with the target exercise' are test-set students. Because the local-attention edge features include the response (Section 2.3), DGAKT receives other students' actual responses to the same target exercise as input. The paper never states that only past interactions are used. This label leakage could fully explain the reported improvements (e.g., ASSIST2017 AUC 0.8994 vs 0.7985 in Table 2, and 0.9480 vs 0.8442 in Table 5), so the central performance claim is not supported.","section":"Section 2.2, Tables 2, 5, and 6"},{"comment":"The resource-efficiency claim is not backed by a meaningful comparison. In Table 7, DGAKT's time complexity is given as O(n·d^2 + n^2·d), but no definition of n for the subgraph setting is supplied, and it is not comparable to the sequence-length n used for DKT, AKT, and SAINT. The paper presents no wall-clock training time or peak-memory measurements; Figure 5 reports parameter counts only, which is not the same as computational cost or memory footprint. The abstract's 'new standard in resource efficiency' is therefore overstated as presented.","section":"Section 4.1, Table 7"},{"comment":"The evaluation protocol is underspecified in a way that is load-bearing. The paper says splits are chronological but does not state that subgraphs for a test interaction are built exclusively from training-time edges, nor how the subsequence length interacts with the split. Without that statement, the leakage concern in the first major comment applies by default to every experiment in Table 2.","section":"Section 3.3"}],"minor_comments":[{"comment":"The t-tests are reported only as p<0.01; with five repetitions, the absence of standard deviations or confidence intervals makes this hard to verify.","section":"Section 3.4"},{"comment":"The paper refers to a 'public repository' but gives no URL; the code and dataset split details are essential for verifying the temporal protocol.","section":"Section 3.3"},{"comment":"The min-max normalization is not specified clearly: it is unclear whether the minimum and maximum are computed per subgraph, per student sequence, or globally over the dataset.","section":"Equation (2)"},{"comment":"The subgraph visualization labels are too small to read; the case study would benefit from larger fonts and an explicit description of the color scale and node types.","section":"Figure 7"}],"recommendation":"reject","confidential_remarks":"The central issue is subgraph leakage: as written, the model can consume test-set labels as part of its input features, especially in the unseen-exercise protocol. This is not a presentation issue but a fundamental flaw in the evaluation. A revision would require rebuilding the experimental protocol with a causal temporal filter, rerunning all experiments, and at minimum reporting code and split details. The efficiency claim also needs actual timing and memory comparisons before it can be assessed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: the model idea is worth a look, but the numbers are not trustworthy. Section 2.2 constructs the subgraph from \"the students who have interacted with the target exercise\" and keeps \"all relationships between the selected nodes except the target interaction\" with no temporal filter. In the unseen-exercise protocol, the test exercises were never seen in training, so the only students who have interacted with them are test students. That means DGAKT is receiving other students' actual test responses to the same exercise as input when predicting the target response. This is label leakage, and it likely explains the large jumps over IGMC-KC (ASSIST2017 AUC 0.7985 to 0.8994 in Table 2, 0.8442 to 0.9480 in Table 5). Even in the chronological split, future edges can enter the subgraph. The stress-test note holds up against the paper text.\n\nWhat is genuinely new: integrating student-exercise and exercise-KC relations into one subgraph, the dual local/global attention with a virtual node, and the consistency loss. The ablations in Figures 3 and 4 are clean and show each component earns its place. The interpretability case study is a nice extra. The efficiency motivation is real, and using subgraphs to avoid full-graph cost is sensible.\n\nSoft spots besides the leakage: the efficiency claim is mostly accounting. Table 7 uses an ambiguous n (sequence length? number of nodes?) and reports no wall-clock time or memory measurements. Figure 5 is only parameter counts. The paper says the code is in a public repository but gives no link and I could not find it. These are secondary but should be fixed.\n\nWho is this for? Someone working on scalable graph-based KT could mine the architecture, but only after the leakage is addressed. As is, I would not cite the empirical results, and the \"new standard in resource efficiency\" phrasing in the abstract is not supported by measured evidence.\n\nRecommendation: this is not a desk-reject in the sense of being unserious. The model is coherent and the flaw is identifiable and fixable. I would send it back for major revision, requiring (1) a causal subgraph construction that only uses edges with timestamp at or before the target, (2) reruns of all experiments, (3) actual runtime and memory benchmarks, and (4) a working code link. If the performance collapses after temporal filtering, the remaining contribution is an incremental architecture with an honest ablation, which might still be publishable but not at the current claimed level.","headline":"The architecture is a reasonable synthesis, but the reported gains likely come from temporal leakage in subgraph construction, so the paper's central claims are unsupported until that is fixed.","tokens_in":13367,"tokens_out":4241,"would_cite":false,"duration_ms":39927,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DGAKT builds a small integrated subgraph for each student-exercise interaction and uses dual graph attention to achieve state-of-the-art knowledge tracing accuracy with about 60,000 parameters.","keywords":["knowledge tracing","graph neural network","graph attention network","subgraph learning","higher-order information","educational data mining","resource efficiency","inductive learning"],"falsifier":"Rebuild subgraphs using only interactions timestamped strictly before the target interaction, and exclude test-set edges from subgraph construction, then re-run the ASSIST2017 evaluation; if the reported AUC of 0.8994 falls toward the IGMC-KC baseline of 0.7985, the gains are leakage-driven.","tokens_in":12423,"feed_emoji":"🧠","tokens_out":5133,"duration_ms":49978,"temperature":0.7,"pith_summary":"DGAKT claims that knowledge tracing can be made both more accurate and far cheaper by replacing the full student-exercise-KC graph with a small per-interaction subgraph. For each target student–exercise pair it pulls in the target student's recent exercises, the other students who attempted that exercise, and the exercise's knowledge concepts, then reads the merged graph with two attention layers. The paper reports the highest accuracy and AUC on three datasets — including AUC 0.8994 on ASSIST2017 versus 0.7985 for the best baseline — using only about 60,000 parameters. The result would matter because real online-learning platforms serve hundreds of thousands of students, and existing graph KT models scale quadratically with the number of exercises.","feed_headline":"Subgraph attention model tops knowledge tracing with 60k parameters","feed_subtitle":"Per-interaction subgraphs merge student, exercise, and knowledge-concept links, beating full-graph baselines on three datasets.","key_machinery":"The load-bearing machinery is the dual graph attention layer: a local edge-featured GAT layer that message-passes over subgraph edges, and a global attention layer that aggregates all node embeddings into a virtual subgraph node via one-hot node-type edges. Stacked twice, their outputs are concatenated into a subgraph embedding and a target node-pair embedding, and the final prediction mixes the two views with a weight parameter. The subgraph itself is built around a labeling trick that encodes roles as one-hot features.","core_discovery":"On its own terms, the paper's central discovery is that an integrated subgraph plus dual attention recovers higher-order paths that separate student-exercise and exercise-KC graphs miss. It introduces a labeling trick that assigns role-based one-hot features, stacks local edge-featured GAT layers over edges carrying timestamp, interaction-count and correctness features, and a global attention layer that pools through a virtual subgraph node; the two views are trained jointly with BCE losses plus an RMSE consistency term. The reported result is state-of-the-art ACC/AUC across EdNet, ASSIST2017 and Junyi, with ablations showing both attention mechanisms and all three edge features contribute, and type-split experiments suggesting robustness to unseen exercises.","pith_inferences":["The reported gains may be partly inflated by an untested temporal leak: subgraphs include all students who interacted with the target exercise regardless of whether those interactions occurred after the target timestamp, so a chronological re-split is the natural check.","The global attention layer over co-attempting students is effectively a collaborative-filtering signal; wiring a sequence model into the student node could recover the ordering the current subgraph discards.","The labeling trick fixes node roles as one-hot features, so a genuinely inductive deployment must decide how to assign labels when an unseen exercise brings previously unseen knowledge concepts into the subgraph."],"forward_implications":["Graph-based knowledge tracing can scale to datasets with over 100,000 students without materializing a full exercise-exercise adjacency matrix, since each prediction only needs its own subgraph.","Real-time inference becomes feasible on large platforms because every target interaction is an independent, parallelizable subgraph computation.","New exercises and knowledge concepts can be handled inductively through their roles in the subgraph, which the type-split experiments probe.","The local and global attention views provide a built-in interpretability signal: high-attention exercises and KCs in a subgraph highlight which prior knowledge matters for the target exercise."],"supporting_citations":[{"why":"Supplies the graph attention mechanism that both local and global layers are built from.","marker":"[Veličković et al., 2018]"},{"why":"Provides the edge-featured GAT (EGAT) that lets the local layer consume timestamp, interaction-count, and correctness features on edges.","marker":"[Wang et al., 2021]"},{"why":"Contributes the labeling trick and the inductive subgraph setting, including the unseen-interaction evaluation design.","marker":"[Zhang and Chen, 2020]"},{"why":"Supplies the practice of turning each hyperedge into a virtual node, used here to represent knowledge concepts.","marker":"[Zhu et al., 2022]"},{"why":"Provides the EdNet dataset, one of the three evaluation corpora.","marker":"[Choi et al., 2020b]"},{"why":"Provides the ASSIST2017 dataset, the second evaluation corpus.","marker":"[Cully and Demiris, 2020]"},{"why":"Supplies DGEKT, the strongest graph-based baseline whose quadratic adjacency cost DGAKT is contrasted against.","marker":"[Cui et al., 2024]"}],"fun_headline_variants":["Subgraph dual attention makes knowledge tracing efficient","Higher-order subgraph links cut knowledge tracing costs","DGAKT: subgraph attention for cheaper knowledge tracing","Efficient knowledge tracing with subgraph higher-order links"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"For each prediction, the model builds the subgraph from all students who attempted the target exercise, without restricting to responses that happened before the prediction time, so it assumes that future or test responses are legitimate input signals rather than leaked labels.","fun_headline_variants_meta":{"raw":{"variants":["Subgraph dual attention makes knowledge tracing efficient","Higher-order subgraph links cut knowledge tracing costs","DGAKT: subgraph attention for cheaper knowledge tracing","Efficient knowledge tracing with subgraph higher-order links"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000191,"raw_usage":{"total_tokens":1281,"prompt_tokens":820,"completion_tokens":461,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":436,"completion_tokens_details":{"reasoning_tokens":400}},"tokens_in":436,"tokens_out":461,"duration_ms":5356,"temperature":1.0,"reasoning_tokens":400,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T14:38:15.314634+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rebuild subgraphs using only interactions timestamped strictly before the target interaction, and exclude test-set edges from subgraph construction, then re-run the ASSIST2017 evaluation; if the reported AUC of 0.8994 falls toward the IGMC-KC baseline of 0.7985, the gains are leakage-driven.","supporting_citations":[{"cited_title":"Egat: Edge-featured graph attention network","cited_arxiv_id":null,"evidence_quote":"Provides the edge-featured GAT (EGAT) that lets the local layer consume timestamp, interaction-count, and correctness features on edges."},{"cited_title":"In- ductive matrix completion based on graph neural net- works","cited_arxiv_id":null,"evidence_quote":"Contributes the labeling trick and the inductive subgraph setting, including the unseen-interaction evaluation design."},{"cited_title":"Inhomogeneous Social Rec- ommendation with Hypergraph Convolutional Networks","cited_arxiv_id":null,"evidence_quote":"Supplies the practice of turning each hyperedge into a virtual node, used here to represent knowledge concepts."},{"cited_title":"Online knowledge level tracking with data- driven student models and collaborative filtering","cited_arxiv_id":null,"evidence_quote":"Provides the ASSIST2017 dataset, the second evaluation corpus."}],"review_version":1}