{"id":"cdd55efb-a083-4e32-b2c5-29c0ca9daa8d","arxiv_id":"2505.09287","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Federated learning combined with pairwise-difference features ranks at-risk students from e-book logs as accurately as centralized training, with gains over non-differential baselines.","lead":"The authors combine federated learning with 'differential features' (pairwise differences between students) to train a model that ranks university students by risk of failing, using e-book activity logs. The model performs about as well as centralized training, while claiming better privacy and earlier detection.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Test set is not composed of unseen courses: every hold-out course (A-2022, B-2020, C-2022-1, D-2022, E-2021) is a later offering of a training course, so the claimed cross-course generalization is untested. A leave-one-course-out experiment would settle this.","rationale":"The reader identified the unvalidated assumption that differential features remove client-specific distribution discrepancies (Section 4.3.5), and I agree that assumption is central. However, there is a more immediately checkable flaw in the evidence: the test split is not composed of unseen courses. Since every test label A-E has a same-topic training counterpart in Table 2, Tables 4-6 measure temporal generalization within known course topics, not generalization to 'other courses' as the abstract and Section 2.1 claim. This directly bears on the central generalizability claim, and it is independently verifiable from the two tables without any additional modeling assumptions. The federated-vs-centralized equivalence claim is less affected, because both arms share the same split, so the conclusion that FL matches centralized performance within this setting remains plausible. The differential-features comparison is also confounded by data augmentation: the proposed method trains on n(n-1) pairwise samples per client while the baseline trains on n absolute samples, so the reported improvement could come entirely from increased data volume rather than relative-value encoding. A same-augmentation absolute-feature control would isolate that mechanism. The public code is a real strength and makes these re-analyses straightforward. I therefore keep the reader's CONDITIONAL verdict: the method is plausible and reproducible, but the headline generalization claim needs a stronger evaluation before it can be accepted as stated.","tokens_in":18657,"tokens_out":6469,"duration_ms":65527,"concrete_test":"Run a leave-one-course-out evaluation: for each target label in {A, B, C, D, E}, remove every offering of that course from training (for example, exclude A-2019, A-2020, A-2021 before testing on A-2022) and, if feasible, also test on F-2021 and G-2021 with those courses excluded from training. Compare Top-n precision, nDCG, and PR-AUC against Tables 4-6. If PR-AUC or nDCG drops materially below the reported values, the cross-course generalization claim is not supported by the current split; if the metrics remain stable, the temporal holdout was a sufficient proxy.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The hold-out evaluation does not test the paper's cross-course generalizability claim. The 12 training clients in Table 2 are A-2019/2020/2021, B-2019, C-2021-1/2, D-2020/2021, E-2020-1/2, F-2021, and G-2021, while the 5 test sets in Table 3 are A-2022, B-2020, C-2022-1, D-2022, and E-2021. Every test course is a later offering of one of the training courses, so the abstract's statement that the model was 'validated on hold-out test data from 5 other courses' overstates what was actually evaluated. F and G never appear in the test set, and each test course has a same-topic training counterpart. The claim in Section 2.1 that the model generalizes 'across courses'—and the stronger claim that differential features remove inter-client distribution discrepancies, which Section 4.3.5 admits has not been validated—therefore rests on temporal reuse of known course content, not on generalization to unseen course content. The federated-vs-centralized comparison in Table 4 can still be meaningful within this setting, but the generalizability and early-prediction claims cannot be separated from the fact that the model has already seen the same topics in prior years.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes a method for at-risk student prediction that combines federated learning (FedAvg) with 'differential features,' i.e., pairwise differences between student feature vectors and between grade targets. The model is a small neural network regressor that produces individual risk scores by aggregating pairwise difference predictions, which are then ranked. Using e-book log data from 1,136 students across 12 course-instances at one university, the authors train 12 federated clients and evaluate on 5 hold-out course-instances. The main reported findings are that federated learning matches centralized learning in Top-n precision, nDCG, and PR-AUC, that differential features improve over non-differential features, and that the model can detect at-risk students using only the first half of the lecture data. The authors provide the modeling code publicly.","tokens_in":18947,"tokens_out":6865,"duration_ms":59077,"significance":"If the claims hold, the paper would offer a useful empirical contribution to privacy-preserving educational data mining: it demonstrates on real data that federated learning can match centralized performance for ranking-based at-risk prediction, and that pairwise difference regression improves ranking metrics. The use of real e-book logs from multiple courses and years, the direct comparison between federated and centralized training, and the release of code are clear strengths. However, the evidence as presented does not yet support the stronger claims of cross-course generalizability or a pre-specified early-prediction capability, because the hold-out courses are later offerings of seen course topics and the early-prediction cutoffs are chosen after inspecting test-set curves. The absence of any measure of variability over the 10 training runs further limits confidence in the reported improvements. These issues are fixable with additional experiments and re-framing, and the core FL-versus-centralized comparison is a solid starting point.","major_comments":[{"comment":"The hold-out test sets in Table 3 are later offerings of the same course topics as the training clients in Table 2: A-2022 follows A-2019/2020/2021, B-2020 follows B-2019, C-2022-1 follows C-2021-1/2, D-2022 follows D-2020/2021, and E-2021 follows E-2020-1/2. Courses F and G appear only in training, and no test course has a topic unseen in training. Consequently, the evaluation supports temporal holdout within known course content, not the cross-course or 'different course contents' generalization claimed in §2.1 and the abstract. Please either add a leave-one-course-out experiment (e.g., withhold all offerings of one course topic) or soften the generalizability claim to temporal generalization within the same course topics.","section":"§4.1.2, Tables 2–3, and §2.1"},{"comment":"The early-prediction cutoffs (4th lecture for A-2022 and B-2020; 8th for C-2022-1, D-2022, and E-2021) are selected after inspecting the PR-AUC curves in Figures 10–14, which are computed on the same test data used to report Table 6. Using test data to choose the evaluation point makes the reported early-prediction numbers in-sample and likely optimistic. Please pre-specify the cutoff (e.g., half of the lectures) before evaluation, or use a validation set to select the cutoff and report test performance only for that fixed choice.","section":"§4.2.3 and Table 6"},{"comment":"The paper reports averages over 10 training runs without any measure of dispersion (standard deviation, confidence interval) or significance test. Several proposed-versus-baseline differences are small (e.g., D-2022 PR-AUC 0.83 vs 0.82; A-2022 nDCG 0.83 vs 0.80; C-2022-1 n=15 Top-n precision 0.85 vs 0.79 for the non-differential baseline), so the claim that the proposed method 'consistently outperformed' the baseline is not supported without knowing run-to-run variability. At minimum, report the standard deviation across the 10 runs and, where possible, a paired test across runs.","section":"§4.1.3, Tables 4 and 5"}],"minor_comments":[{"comment":"The omission of L2 normalization is motivated by preserving action counts, but the effect on the distribution of feature magnitudes across clients is not discussed; this is directly relevant to the differential-features rationale in §3.2.1.","section":"§3.1.4 and Figure 4"},{"comment":"Equation (2) defines Gm as a cumulative proportion (MaxScore times the cumulative count up to grade m), so the regression target is essentially a rank-based transformation of grades; this should be stated explicitly, because calling it a 'score' may mislead readers into thinking it is a standard grade point average.","section":"Equation (2)"},{"comment":"Please correct typos: 'Predction' in Figure 2, 'substaintial' in §3.2.1, 'grads' in §4.1.2, and inconsistent capitalization of 'differential features' in headings.","section":"General"},{"comment":"The paper candidly states that the assumption behind differential features has not been validated for a general setting. To strengthen the paper, add a quantitative check of whether pairwise differences reduce inter-client distribution distance (e.g., MMD or KL divergence) on the feature representations.","section":"§4.3.5"},{"comment":"The statement that the number of no-risk students (264) is 'more than twice' the number of at-risk students (127) is technically true but borderline (ratio 2.08); consider saying 'approximately twice.'","section":"Table 3"},{"comment":"The phrase 'more strongly preserved privacy' in §4.2.1 is appropriate, but the abstract's 'addresses privacy concerns' is stronger than what is demonstrated; FL without differential privacy or secure aggregation does not guarantee privacy against inference attacks. Please adjust the wording.","section":"§4.2.1 and Abstract"},{"comment":"In Section 4.1.2, the sentence 'The threshold of 15 was chosen to ensure that the number of at-risk students is sufficient for evaluating Top-n precision (n = 15)' is clear, but the definition of at-risk based on the 15th-ranked student in each course means the at-risk count varies across courses; consider reporting the sensitivity of the results to this threshold.","section":"§4.1.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for EDM and the core empirical comparison (federated vs. centralized learning on real e-book logs) is a useful contribution. The main issues are fixable with additional experiments or careful re-framing, so I recommend major revision rather than rejection. The claim that no EDM conference paper in 2020–2024 has focused on federated learning is a strong negative claim that is difficult to verify; the authors may want to soften it or provide the search method used."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear X,\n\nQuick take: this is a serious empirical EDM paper with a genuinely new combination—federated learning plus pairwise differential features for ranking at-risk students—and it ships code. If you work on privacy-preserving learning analytics, it's worth reading. But the headline claim that the model generalizes “across courses” is not supported by the experimental design: all five hold-out sets are later offerings of courses that appear in the training data. The stress-test note is right, and it lands on a load-bearing claim.\n\nWhat's actually new: applying differential features (pairwise differences vi-vj and gi-gj) inside federated learning for educational data mining. The building blocks, FedAvg and pairwise difference regression, are established elsewhere, but the combination is new to this field, and the trick of deriving individual risk scores by summing pairwise predictions is clearly explained. The federated-vs-centralized comparison is honest: FL roughly matches centralized on most metrics, and differential features beat the no-differential baseline on most test sets. The writing is careful, the limitations section admits the single-institution data source and the unvalidated assumption in 4.3.5, and the code is public. No invented entities, no circularity.\n\nSoft spots, in rough order of importance. First, the hold-out design. The abstract says validated on “5 other courses,” but those are A-2022, B-2020, C-2022-1, D-2022, E-2021—later offerings of courses A–E in the training set. F and G never appear in test. So the “cross-course generalizability” claim is untested; the experiment shows the model handles temporal reuse of known content, not transfer to a new topic. A leave-one-course-out experiment would settle it. Second, differential features increase training data by n(n-1), so the improvement over the non-differential baseline could be data augmentation rather than relative-value normalization; the paper doesn't disentangle these. Third, no error bars or significance tests. Averaging ten runs is fine, but with differences this small (e.g., nDCG 0.83 vs 0.80), noise is a real concern. Fourth, the early-prediction cutoffs are chosen post hoc from the curves in Figures 10–14; that's data snooping, though probably a minor effect. Finally, “more strongly preserved privacy” overstates what vanilla federated learning provides—no secure aggregation or differential privacy is used.\n\nBottom line: the paper deserves a serious referee, not a desk reject. The core empirical claim is plausible, but the generalization claim needs a proper leave-one-course-out experiment, and the augmentation confound should be addressed before publication. I'd send it to review with a request for major revision.\n\nBest,\n[You]","headline":"Useful FL-for-EDM study with real data and code, but the claimed cross-course generalization is untested because every hold-out course is a later offering of a training course.","tokens_in":19485,"tokens_out":2505,"would_cite":false,"duration_ms":25961,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Federated learning matches centralized at-risk rankings","keywords":["at-risk student prediction","federated learning","differential features","risk ranking","early prediction","privacy preservation","learning analytics","rank-based evaluation"],"falsifier":"Train the same federated pipeline on courses drawn from two institutions with very different overall activity levels, then check whether the cross-institution risk ranking stays stable; if the ranking degrades despite differential features, the core transferability claim fails. A cheaper check is to artificially rescale all feature values in one client by a large constant and see whether the model still ranks that client's students correctly.","tokens_in":18467,"feed_emoji":"🎓","tokens_out":4908,"duration_ms":47224,"temperature":0.7,"pith_summary":"The paper tries to show that a school can train an at-risk student warning system without pooling sensitive learning logs in one place. It combines federated learning, where each course trains locally and only model updates are shared, with differential features, which replace each student's raw behavior vector with pairwise differences between students. On data from 1,136 students across 12 course-offerings, the federated model ranks at-risk students about as accurately as a model trained on all data centrally, and more accurately than the same model trained on raw features. The same model also detects at-risk students from logs covering only the first half of a course nearly as well as from full-course logs. The practical payoff would be that many institutions could jointly build a high-performing early-warning tool without exposing grades or reading behavior.","feed_headline":"Federated learning matches centralized at-risk rankings","feed_subtitle":"Pairwise student differences let courses train jointly without sharing logs, and early warnings stay accurate.","key_machinery":"The load-bearing mechanism is the differential feature transform applied inside each client: every student is represented not by absolute activity counts but by differences to every other student in the same client. This does two jobs at once: it multiplies a client's $n$ samples into $n(n-1)$ training pairs, and it strips out client-level offsets such as overall activity level or course pacing, so the server model sees relative behavior rather than raw distributions. The other named component is FedAvg, the aggregation rule that averages locally trained weights proportionally to each client's sample count; the paper uses it to combine course-level models without moving raw logs. On top of the pairwise scores, a regression readout sums each student's predicted differences to recover an individual risk score for ranking.","core_discovery":"The central claim is that replacing absolute feature vectors with pairwise differences makes federated learning viable for educational early-warning systems. For each client, the method forms $d_{ij}=v_i-v_j$ from student feature representations and $e_{ij}=g_i-g_j$ from scored grades, trains a regression network locally on these pairs, and aggregates weights on the server with FedAvg. At prediction time the network outputs pairwise difference scores $p_{ij}$, which are summed per student as $q_i=\\sum_{j\\ne i}p_{ij}$ and sorted ascending to produce a risk ranking. The authors report that this pipeline matches centralized training on Top-n precision, nDCG, and PR-AUC across five hold-out courses, beats the same federated pipeline without differential features, and keeps most of its accuracy using only half the lecture sessions. They frame the result as a step toward privacy-preserving, generalizable student-performance prediction.","pith_inferences":["If the cross-client invariance of pairwise differences holds, the same trick should transfer to other learning-management platforms and log formats, since constant offsets or scales in feature counts cancel in differences; a testable extension is applying the pipeline to clickstream logs from other systems.","The $n(n-1)$ expansion means small clients benefit most from differential features, so a federated network of small courses might see larger gains than the 12-course setup here.","The ranking readout is scale-free in a useful sense: if the regression's pairwise scores shift uniformly across students, the ranking is unchanged, which is the paper's stated reason early prediction works; this also implies calibration of absolute scores is unnecessary for deployment.","A privacy caveat not explored in the paper: sharing model updates, not raw data, reduces but does not eliminate leakage risk, and pairwise differences do not add formal privacy guarantees such as differential privacy; combining the method with noise would be a natural follow-up."],"forward_implications":["A federated model can be trained across courses or institutions without any party seeing another party's raw learning logs or grades, while keeping ranking accuracy close to centralized training.","Switching features from absolute values to pairwise differences improves at-risk ranking on all five hold-out courses, suggesting the transform is the source of the cross-course generalization.","The trained model can be applied after half the lectures with little loss in ranking quality, so instructors can receive early warnings from the same model rather than retraining for each stage.","Because the output is a ranking rather than a hard at-risk label, instructors can act on the top-n list, and students near the boundary remain visible instead of being classified away."],"supporting_citations":[{"why":"Supplies the FedAvg aggregation rule used to combine locally trained model parameters without centralizing data.","marker":"[34]"},{"why":"Introduces pairwise difference regression, the meta-algorithm that the differential-feature idea is borrowed from.","marker":"[46]"},{"why":"Provides pairwise difference learning for classification, the second source for the differential-feature approach.","marker":"[6]"},{"why":"Provides E2Vec, the method that turns digital-textbook operation logs into 100-dimensional student feature representations.","marker":"[35]"},{"why":"The prior educational study of federated learning that the paper extends by adding differential features to handle client distribution shifts.","marker":"[47]"},{"why":"Defines nDCG, one of the three ranking metrics used to evaluate the risk rankings.","marker":"[20]"}],"fun_headline_variants":["Differential features let federated learning match centralized at-risk ranking","Federated learning with differential features reaches centralized-level ranking","Pairwise differences enable privacy-preserving at-risk student ranking","Federated at-risk prediction: differential features match centralized accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that replacing each student's raw behavior numbers with pairwise differences between students removes enough course-to-course and institution-to-institution variation that a model trained on some courses ranks unseen courses correctly, an assumption the authors admit has not been explicitly validated.","fun_headline_variants_meta":{"raw":{"variants":["Differential features let federated learning match centralized at-risk ranking","Federated learning with differential features reaches centralized-level ranking","Pairwise differences enable privacy-preserving at-risk student ranking","Federated at-risk prediction: differential features match centralized accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000839,"raw_usage":{"total_tokens":3673,"prompt_tokens":980,"completion_tokens":2693,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":596,"completion_tokens_details":{"reasoning_tokens":2624}},"tokens_in":596,"tokens_out":2693,"duration_ms":19808,"temperature":1.0,"reasoning_tokens":2624,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:34:53.670872+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same federated pipeline on courses drawn from two institutions with very different overall activity levels, then check whether the cross-institution risk ranking stays stable; if the ranking degrades despite differential features, the core transferability claim fails. A cheaper check is to artificially rescale all feature values in one client by a large constant and see whether the model still ranks that client's students correctly.","supporting_citations":[{"cited_title":"Knowledge Distillation in RNN-Attention Models for Early Prediction of Student Performance","cited_arxiv_id":"2412.14526","evidence_quote":"Introduces pairwise difference regression, the meta-algorithm that the differential-feature idea is borrowed from."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides pairwise difference learning for classification, the second source for the differential-feature approach."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The prior educational study of federated learning that the paper extends by adding differential features to handle client distribution shifts."}],"review_version":1}