Knowledge Tracing in Programming Education Integrating Students' Questions

Doyoun Kim; Suin Kim; Yojan Jo

REVIEW 4 major objections 5 minor 21 references

Knowledge Tracing in Programming Education Integrating Students' Questions

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Student questions, combined with automatically extracted skill labels, predict a student's success on the next programming problem more accurately than code submissions alone.

desk verdict A promising but compromised first attempt at using student questions for knowledge tracing; the label leakage needs to be fixed before the headline numbers can be trusted. read the letter →

arxiv 2502.10408 v1 pith:VXVC7UN6 submitted 2025-01-22 cs.CY cs.AIcs.SE

Doyoun Kim , Suin Kim , Yojan Jo This is my paper · ORCID

classification cs.CYcs.AIcs.SE

keywords knowledgetracingprogrammingeducationstudentquestionsskillextractioncodeembeddingstransformerAUCadaptivelearning

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that in programming courses, the questions students ask between submissions reveal knowledge gaps that code alone does not capture, and that a knowledge tracing model can exploit those questions to predict whether a student will succeed on the next problem. It introduces SQKT, which embeds student questions, code submissions, problem statements, and skill labels extracted from questions and reference solutions, then feeds the fused history through attention layers. The authors report that this question-aware model beats code-only baselines by large margins and generalizes across courses with different difficulty levels and limited data. If correct, the result would let adaptive learning systems in computer science use naturally occurring Q&A forum activity, instead of manually tagged skills, to decide what to teach next.

What carries the argument

The carrying component is the question-embedding channel, built on CodeT5 and fine-tuned to predict educator responses, combined with the GPT-generated skill extractor and a triplet-loss fusion layer. The skill extractor is a rule-based script produced by GPT-4o that maps noisy Korean-and-code question text to 36 Python concepts and 19 error types; the same script runs on reference solutions to obtain the skills each target problem requires. The fusion layer projects question, code, problem, and skill embeddings into a shared space using triplet loss, so the attention layers can compare a student's historical questions and submissions with the next problem's requirements.

What would settle it

Re-run the in-domain experiments with the success threshold computed only from training students, or set to a fixed per-problem value, and compare AUC across models; if SQKT's 33.1-point gain over the baseline shrinks dramatically, the reported improvement is partly an artifact of using test data in the threshold.

Watch

Extended reading notes

Core claim

SQKT is, the authors claim, the first knowledge tracing model for programming that treats student questions as a first-class input. The model encodes each question with CodeT5, an encoder-decoder trained on code and natural language, and strengthens that embedding by training the encoder to generate the educator's response as an auxiliary task. A GPT-4o-derived rule-based skill extractor labels both the student's question and the target problem's reference solution with a fixed set of 36 Python concepts and 19 error types; these skill labels are embedded with BERT and combined with CodeBERT code embeddings and BERT problem embeddings. A triplet-loss fusion layer projects all embeddings into one 512-dimensional space, and six multi-head self-attention layers trace the student's history to predict success on the target problem. In in-domain tests SQKT reaches AUC 87.1-93.4, beating the strongest baseline by 12.6-20.8 percentage points and the OKT baseline by up to 33.1 points on one course; in cross-domain tests question input yields large gains, including an 11.4-point gain over in-domain training on a data-scarce algorithm course.

Load-bearing premise

A student is labeled successful only if they reach a perfect score within the average number of submissions taken by all students in the course, and that average includes the students whose performance is being predicted.

Editorial extensions

If this is right

Knowledge tracing for programming can work without expert-created knowledge-component tags: skills are extracted automatically from questions and reference solutions.
Student questions are a usable behavioral signal even in noisy real classroom data, since the model did not preprocess the questions.
The approach can transfer across courses with different content structures and to advanced courses with too little data to train in-domain models.
The auxiliary task of predicting educator responses improves question embeddings, turning Q&A forum data into a training signal rather than just an input.
SQKT's question embeddings can be appended to existing KT models, such as KTMFF+ and OKT+, and improve their accuracy too.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial inference: the 33.1-point headline gain is measured against OKT, which has low AUC on these data; against the stronger KTMFF+ baseline the gain is 12.6-20.8 points, so the practical improvement over a strong code-based model is large but smaller than the headline number suggests.
Editorial inference: because success labels depend on a course-wide average submission count that includes test students, re-computing with a training-only threshold could change all reported gains; the model's relative ranking might survive, but the magnitude is unverified.
Editorial inference: the rule-based, GPT-generated skill extractor was validated on 100 questions from one course, so its generalization to other languages, frameworks, or question styles is untested, and a machine-learned extractor might improve scalability.
Editorial inference: the error analysis indicates that complexity from mixed Korean-English code is the most common failure mode, so multilingual tokenization or normalization is a natural next step that could push accuracy further.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper is a solid first cut at something that's genuinely missing: KT for programming that uses student questions as a signal. The architecture is assembled from known pieces, but the combination is new, and the ablations give me confidence that the questions themselves carry predictive information beyond code submissions. The skill extraction via a GPT-4o-generated rule-based script is a pragmatic choice, and the validation, though thin, is at least an attempt at grounding it.

The main problem is the success label. Section 4.2 defines success as scoring 100 within a threshold equal to the average number of submissions across all students in the course. Since the split is by student, the test students' submission counts feed into that average, so the test labels are not independent of the training distribution. The model sees submission counts in its input history, which makes it plausible that some of the large AUC gains come from learning the label rule rather than from the question and skill signals. The ablations still show a similar pattern, so I don't think the qualitative conclusion collapses, but the headline numbers are not trustworthy as reported. They need to recompute the threshold using only training students and report variance across seeds.

There are smaller issues: no error bars anywhere; the cross-domain sample counts in Section 4.2 (5,858 and 1,674) don't match Table 1 or Table 7; the OKT+ row in Table 2 repeats identical numbers for two different courses, which looks like a copy-paste error. The skill extractor was validated on 100 questions annotated by a co-author; that's a weak anchor, though the kappa is high. Code and data are not released.

Overall, the paper deserves a serious referee because the idea is timely and the experiments, once cleaned up, could support a real contribution. I'd want to see the label issue fixed and the numbers reconciled before believing the 33-point claim, but the direction is worth engaging with.

Referee Report

4 major / 5 minor

Summary. The paper introduces SQKT, a knowledge tracing model for programming education that augments code submissions and problem descriptions with student questions, educator responses, and GPT-extracted skill labels. The model is evaluated on four Python courses from a Korean online learning platform using in-domain and cross-domain settings, with ablations and error analyses. The authors report large improvements in AUC, accuracy, and F1 over existing code-based KT baselines, and claim that student questions and automatically extracted skills provide predictive signals beyond code submissions.

Significance. If the reported results are reliable, this is a useful contribution: it demonstrates that student-educator question interactions carry information for knowledge tracing that is not available in code submissions alone, and it proposes a practical way to obtain skill labels without full manual tagging. The paper includes a new real-world dataset, a plausible model architecture, and a range of experiments including cross-domain transfer to data-scarce courses. The main limitations are methodological: the success-label construction depends on test-set statistics, the experiments lack error bars, and the reported dataset sizes are internally inconsistent. These issues directly affect the credibility of the central quantitative claims.

major comments (4)

[Section 4.2 (Evaluation Metrics)] The success threshold is defined as the average number of submissions across all students in each course, and this average is computed over the full course data, including students in the test split. Because test students contribute to the statistic that determines their own success labels, the test labels are not independent of the training data. This violates the student-level split intended by the 8:1:1 partitioning and can let the model exploit features correlated with the label-construction rule (e.g., the number of submissions encoded in the Ui tensors). Please recompute all success labels using a threshold estimated from the training students only (or from a nested cross-validation over students), report the resulting metrics for all models, and include a sensitivity analysis to the threshold choice.
[Section 5 (Results, Tables 2-4, Figure 4)] No error bars, confidence intervals, or repeated-seed results are reported anywhere in the paper. All conclusions rest on single point estimates from one run per configuration, and the margins in some ablations are very small (e.g., Table 4 shows AUC differences of 0.8-1.4 points). This makes the claimed improvements, including the 33.1% absolute AUC gain in the abstract, statistically unverifiable. Please run at least five seeds per model and report means and standard deviations, and apply a significance test (e.g., paired bootstrap over students) for the key comparisons in Table 2 and the ablation tables.
[Section 4.2 (Cross-Domain Experimental Setup)] The sample sizes for the cross-domain settings do not match the dataset statistics in Tables 1 and 7. The text states that the 'Python Introduction' course has 5,858 samples and 'First Python' has 1,674, but Table 7 lists 4,135 and 11,302 test problems, respectively; the '9,390 samples' for the combined training set also does not correspond to any sum in Tables 1/7. Please clarify what unit is being counted (students, problem attempts, or submissions) and correct the tables and text so that the experimental setup is reproducible.
[Section 5.1, Table 3 (Ablation Study)] The text says the ablation results are on the 'Python Basic' course, but the table caption says the study is on the 'Python Introduction' course. Additionally, the claim that 'the same pattern is observed in other courses' is not supported by any reported results. Please correct the caption-to-text mismatch and provide ablation results for all three courses, either in the main text or in an appendix.

minor comments (5)

[Abstract and Section 5.1] The abstract's '33.1% absolute improvement in AUC compared to baseline models' is ambiguous, because Table 2 shows that this is the gain over OKT (the weakest baseline on Python Introduction), not over the strongest baseline (KTMFF+). Please state explicitly that the gain is relative to a specific baseline or report the comparison to the best-performing baseline.
[Section 3.1 (Skill Extraction)] The validation of the skill extractor uses 100 questions annotated by a co-author, and the reported Cohen's kappa of 0.98 is between that co-author and a graduate student, not against an independent gold standard. This is a weak external anchor for the claim that the extractor is reliable. Please provide more detail on the annotation protocol or temper the reliability claim.
[Table 4 caption] The caption says 'Impact of response and triplet loss functions,' but the text refers to 'question loss.' Use consistent terminology for this auxiliary loss.
[Tables 1 and 7] Table 7 reports 'Python Basic # of problems' as 2,665, 362, 412, while Table 1 lists 48 unique problems with 474 submissions per problem. These numbers are not reconcilable as given; please clarify whether the rows in Table 7 count problem attempts (student-problem pairs) and use a consistent term throughout.
[Section 4.2 (In-Domain)] The sentence 'excluding one due to insufficient data' is vague; please name the excluded course (Algorithm, as implied by Table 2) and state the criterion used for 'insufficient.'

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: SQKT is an empirical model comparison; no prediction or first-principles result reduces to its own inputs by construction.

full rationale

The paper is an empirical machine-learning study rather than a derivation, and none of the enumerated circularity patterns is present. The central claim is that adding student-question and auto-extracted-skill embeddings improves AUC over code-only baselines; this is tested by direct comparison and ablations, not derived from the model's own definitions. The skill extractor is validated against human annotations (though one annotator is a co-author, which is a validity concern but not a circular reduction). The success-label threshold defined in Section 4.2 as 'the average number of submissions across all students in each course' is computed over the full dataset including test students; this is a train/test independence flaw that could inflate reported AUC gains, but it is not circularity because the model neither fits nor predicts the threshold, and the labels are not constructed from the model's outputs. The paper contains no load-bearing self-citations: the baselines (KTMFF, OKT, Code-DKT) and pretrained encoders (CodeT5, CodeBERT, BERT) are external works, and no uniqueness theorem or prior result by the same authors is invoked to force the model choice. The comparison to external baselines makes the central empirical claim independently checkable, so the appropriate circularity score is 0.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claim depends on two hand-defined structures: the success label threshold (a data-dependent choice) and the fixed skill taxonomy with a GPT-built extractor. Neither is externally anchored beyond a 100-sample validation, so they count as free choices in the paper's pipeline.

free parameters (2)

success threshold = mean submissions per course (not stated numerically)
Defines the binary success label for every student-problem pair; computed over all students in the course, including test students (Section 4.2). If this threshold changes, all reported AUC/ACC/F1 change.
triplet loss margin and weight lambda = margin not stated; lambda = 1.0
Tuned on validation set; affects embedding alignment but not the central comparison.

assumptions (3)

domain assumption The 36 Python concepts plus 19 error types are a sufficient skill taxonomy for the four courses.
The skill extractor maps questions to this fixed list (Section 3.1, Appendix A). If important skills are missing, the skill signal is incomplete.
domain assumption The rule-based skill extractor, built with GPT-4o and manually corrected, produces reliable skills across all courses.
Validated on only 100 questions from one course (Python Basic), with F1=0.86; other courses are not separately validated.
ad hoc to paper Successfully solving a problem means achieving score 100 within the average number of submissions for the course.
This operational definition is introduced by the authors and directly determines all labels; it is not a standard benchmark definition.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Knowledge Tracing in Programming Education Integrating Students' Questions." pith.science (2026). https://pith.science/paper/VXVC7UN6

@misc{pith2026250210408,
  author       = {Pith},
  title        = {Pith review of: Knowledge Tracing in Programming Education Integrating Students' Questions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VXVC7UN6}},
  note         = {Machine review of arXiv:2502.10408}
}

read the original abstract

Knowledge tracing (KT) in programming education presents unique challenges due to the complexity of coding tasks and the diverse methods students use to solve problems. Although students' questions often contain valuable signals about their understanding and misconceptions, traditional KT models often neglect to incorporate these questions as inputs to address these challenges. This paper introduces SQKT (Students' Question-based Knowledge Tracing), a knowledge tracing model that leverages students' questions and automatically extracted skill information to enhance the accuracy of predicting students' performance on subsequent problems in programming education. Our method creates semantically rich embeddings that capture not only the surface-level content of the questions but also the student's mastery level and conceptual understanding. Experimental results demonstrate SQKT's superior performance in predicting student completion across various Python programming courses of differing difficulty levels. In in-domain experiments, SQKT achieved a 33.1\% absolute improvement in AUC compared to baseline models. The model also exhibited robust generalization capabilities in cross-domain settings, effectively addressing data scarcity issues in advanced programming courses. SQKT can be used to tailor educational content to individual learning needs and design adaptive learning systems in computer science education.

Figures

Figures reproduced from arXiv: 2502.10408 by the authors.

**Figure 2.** Comprehensive architecture of the SQKT. The model processes problem text, code submissions, and [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** This figure illustrates different types of student [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Cross-domain performance. programming courses. The question loss, derived from the task of predicting educator responses to student questions, impacts performance across the three courses, with slight drops in performance when removed. This loss seems to enrich the em…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

21 extracted references · 15 canonical work pages

[1]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

Amjad Altadmri and Neil CC Brown. 2015. 37 million compilations: Investigating novice programming mistakes in large-scale student data. In Proceedings of the 46th ACM technical symposium on computer science education, pages 522--527

work page 2015
[4]

Amal Asselman, Mohamed Khaldi, and Souhaib Aammou. 2020. Evaluating the impact of prior required scaffolding items on the improvement of student performance prediction. Education and Information Technologies, 25:3227--3249

work page 2020
[5]

Brett A Becker, Paul Denny, Raymond Pettit, Durell Bouchard, Dennis J Bouvier, Brian Harrington, Amir Kamil, Amey Karkare, Chris McDonald, Peter-Michael Osera, et al. 2019. Compiler error messages considered unhelpful: The landscape of text-based programming error message research. Proceedings of the working group reports on innovation and technology in c...

work page 2019
[6]

Albert T Corbett and John R Anderson. 1994. Knowledge tracing: Modeling the acquisition of procedural knowledge. In User modeling and user-adapted interaction, volume 4, pages 253--278. Springer

work page 1994
[7]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805

arXiv 2018
[8]

Allen B Downey and Chris Mayfield, editors. 2019. Think Java: How to think like a computer scientist. O'Reilly Media

work page 2019

Show all 21 references

[9]

Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al. 2020. Codebert: A pre-trained model for programming and natural languages. arXiv preprint arXiv:2002.08155

2020 arXiv
[10]

Matthew Hertz and Sarah Michele Ford. 2013. Investigating factors of student learning in introductory courses. In Proceeding of the 44th ACM technical symposium on Computer science education, pages 195--200

2013
[11]

Alison King. 1994. Guiding knowledge construction in the classroom: Effects of teaching children how to question and how to explain. American educational research journal, 31(2):338--368

1994
[12]

Naiming Liu, Zichao Wang, Richard Baraniuk, and Andrew Lan. 2022. Open-ended knowledge tracing for computer science education. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing

2022
[13]

Chris Piech, Jonathan Bassen, Jonathan Huang, Surya Ganguli, Mehran Sahami, Leonidas J Guibas, and Jascha Sohl-Dickstein. 2015. Deep knowledge tracing. Advances in neural information processing systems, 28

2015
[14]

Yang Shi, Min Chi, Tiffany Barnes, and Thomas Price. 2022. Code-dkt: A code-based knowledge tracing model for programming tasks. arXiv preprint arXiv:2206.03545

2022 arXiv
[15]

Zhuoqing Song, Sirui Huang, and Ya Zhou. 2021. A deep knowledge tracking model integrating difficulty factors. In The 2nd International Conference on Computing and Data Science, pages 1--5

2021
[16]

Dan Sun, Fan Ouyang, Yan Li, and Caifeng Zhu. 2021. Comparing learners’ knowledge, behaviors, and attitudes between two instructional modes of computer programming in secondary education. International Journal of STEM Education, 8:1--15

2021
[17]

Xia Sun, Xu Zhao, Bo Li, Yuan Ma, Richard Sutcliffe, and Jun Feng. 2022. https://doi.org/10.1109/TCYB.2021.3051028 Dynamic key-value memory networks with rich features for knowledge tracing . IEEE Transactions on Cybernetics

2022
[18]

Al Sweigart, editor. 2019. Automate the boring stuff with Python: practical programming for total beginners. No Starch Press

2019
[19]

Yue Wang, Weishi Wang, Shafiq Joty, and Steven CH Hoi. 2021. Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. arXiv preprint arXiv:2109.00859

2021 arXiv
[20]

Yongkang Xiao, Rong Xiao, Ning Huang, Yixin Hu, Huan Li, and Bo Sun. 2023. Knowledge tracing based on multi-feature fusion. Neural Computing and Applications, 35(2):1819--1833

2023
[21]

Liang Yu, Peng Tianhao, Pu Yanjun, and Wu Wenjun. 2022. Help-dkt: an interpretable cognitive model of how students learn programming based on deep knowledge tracing. In Nature Scientific Reports

2022

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

Amjad Altadmri and Neil CC Brown. 2015. 37 million compilations: Investigating novice programming mistakes in large-scale student data. In Proceedings of the 46th ACM technical symposium on computer science education, pages 522--527

work page 2015

[4] [4]

Amal Asselman, Mohamed Khaldi, and Souhaib Aammou. 2020. Evaluating the impact of prior required scaffolding items on the improvement of student performance prediction. Education and Information Technologies, 25:3227--3249

work page 2020

[5] [5]

Brett A Becker, Paul Denny, Raymond Pettit, Durell Bouchard, Dennis J Bouvier, Brian Harrington, Amir Kamil, Amey Karkare, Chris McDonald, Peter-Michael Osera, et al. 2019. Compiler error messages considered unhelpful: The landscape of text-based programming error message research. Proceedings of the working group reports on innovation and technology in c...

work page 2019

[6] [6]

Albert T Corbett and John R Anderson. 1994. Knowledge tracing: Modeling the acquisition of procedural knowledge. In User modeling and user-adapted interaction, volume 4, pages 253--278. Springer

work page 1994

[7] [7]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805

arXiv 2018

[8] [8]

Allen B Downey and Chris Mayfield, editors. 2019. Think Java: How to think like a computer scientist. O'Reilly Media

work page 2019