{"id":"d8acf27d-3c2d-438b-9c4f-ecd9a4610fad","arxiv_id":"2607.19209","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Clustering teams by their logged actions in cybersecurity tabletop exercises aligns reasonably with instructor scores, while GPT-4o and GPT-5.2 rubric-based assessments of team communication still deviate substantially from human judgment.","lead":"Researchers tested two automatic ways to grade student teams in cybersecurity 'tabletop' exercises: grouping teams by their logged actions with clustering, and using AI language models to grade team emails. Clustering grouped similar teams quickly and cheaply; the AI models disagreed with human graders often, and only the newest model came close.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Clustering 'validity' is measured against milestone score vectors whose inputs are the same milestone IDs used as cluster features; baseline is random partitioning, so the positive claim is unsupported.","rationale":"The reader identified the same weakest assumption: the clustering evaluation relies on ground-truth milestone scores that overlap with the cluster features, and the baseline is a weak random-partition comparison. This is the single most load-bearing concern because the paper's main positive contribution—cluster-based assessment feedback—rests on the claim that clusters are educationally valid. If the concern holds, the abstract's statement that clustering 'was valid and reliable' is unsupported, and the paper's value shifts to the credible negative LLM result and the open resources. The proposed concrete test (feature ablation) would settle whether the low RMSE is merely a tautological artifact. If the RMSE stays low without milestone IDs as features, then the clustering captures genuine behavioral similarity; if it rises to baseline, the circularity is confirmed. The LLM evaluation, though small, does not suffer from this circularity and is a useful empirical finding. No additional load-bearing concerns were found; reporting gaps (DBSCAN parameters, prompt details) are secondary and do not change the conditional verdict. Therefore, the reader's CONDITIONAL verdict remains appropriate, with the condition that the clustering validity claim be re-evaluated or substantially downweighted.","tokens_in":15980,"tokens_out":4911,"duration_ms":53387,"concrete_test":"Re-run the clustering pipeline of §IV-B1 with feature sets that exclude milestone IDs and timestamps, using only tool usage, action sequences, and email-related features if possible. Compute the within-cluster RMSE on the milestone score vectors as in Table III. If the RMSE rises toward the Bell-partition baseline (EXF 0.53, PHI 1.12, avg 0.83), the reported low RMSE (0.45/0.79/0.62) is an artifact of the feature/label overlap. Also report DBSCAN parameters (eps, min_samples) and compare against a stronger baseline, e.g., clustering by total milestone score or by random features with matched cluster sizes.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central positive claim is that clustering TTX logs 'sufficiently aligns with instructor scores and outperforms the baseline' (§V-A1) and is 'valid and reliable' (abstract). The evaluation for this claim is circular. The ground-truth milestone score vectors (§IV-A4) are produced by a script that assigns 2/1/0 to each milestone based on whether the team reached it; the clustering features (§IV-B1) are 'which milestones teams reached... and when.' Thus the score vector is a deterministic function of a subset of the clustering features (the milestone IDs). Any clustering that groups teams by milestone IDs will naturally produce low within-cluster RMSE on those score vectors. The baseline is the average RMSE across all Bell partitions—i.e., essentially random partitions. Beating this baseline only shows that clusters capture milestone IDs better than chance, which is unsurprising because the features contain those IDs. This does not establish that clusters are educationally valid groupings or that they align with instructor judgment in any meaningful sense. The qualitative cluster interpretations (§V-B1) are post-hoc and do not independently validate the clusters. The LLM negative result (GPT-4o at ~55% disagreement, GPT-5.2 at ~35%) is independent of this circularity and appears credible, but it does not rescue the abstract's positive claim about clustering's validity and reliability.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This research-to-practice paper compares two automated methods for assessing student teams in cybersecurity tabletop exercises (TTXs): unsupervised clustering of activity logs and LLM-based rubric scoring of team communication. Using data from 24 teams (23 after consent) across two countries, the authors evaluate each method against instructor scores. They report that clustering achieves low within-cluster RMSE on milestone score vectors and outperforms a random-partition baseline, while GPT-4o's communication scores disagree with human raters at roughly chance level and GPT-5.2 shows lower but still substantial disagreement. The methods are integrated into the open-source INJECT platform, and datasets, code, and exercise definitions are released.","tokens_in":16363,"tokens_out":7350,"duration_ms":81829,"significance":"The LLM negative result is a credible and useful cautionary data point for using general-purpose LLMs in domain-specific assessment: GPT-4o at 55% disagreement on a 3-point scale is a robust failure, and the paper honestly reports it. The open dataset, code, and platform integration are valuable resources for the community. However, the positive clustering claim—the abstract states the method 'was valid and reliable'—is undermined by a circular evaluation: the ground-truth milestone score vectors are computed from the same milestone IDs used as clustering features. Thus the current paper's main contribution is the negative LLM finding and the released artifacts, not the validated clustering method.","major_comments":[{"comment":"The clustering evaluation is circular. The ground-truth milestone score vectors (§IV-A4) are produced by a script that assigns 2/1/0 based on milestone completion; the clustering features (§IV-B1) are explicitly 'which milestones teams reached (milestone IDs) and when.' The scores are therefore a deterministic function of a subset of the clustering features. Low within-cluster RMSE (Table III) is largely guaranteed for any clustering that groups by milestone IDs, and the baseline—the average RMSE over all Bell partitions—is a weaker null than a feature-dependent null. Beating this baseline only shows that clusters capture milestone IDs better than chance; it does not demonstrate 'valid' alignment with instructor judgment. Please re-run clustering without milestone IDs (e.g., using timestamps, sequence, and tool usage only) and compare against the same score vectors, or use a permutation","section":"IV-B1, V-A1, Table III"},{"comment":"The manuscript calls the benchmark 'instructor-assigned scores based on standardized rubrics,' but the milestone scores are assigned by a script, not by human judgment; the instructors manually scored only the communication rubric. This conflation is misleading and should be corrected throughout (e.g., 'script-derived milestone scores' vs. 'instructor-rated communication scores').","section":"IV-A4, Abstract"},{"comment":"The claim that clustering 'was valid and reliable' is not supported by any reliability measure. The use of Unsupervised Consensus Clustering over 50 iterations is mentioned, but no stability index (e.g., adjusted Rand index across runs) is reported. Please report a quantitative stability metric or remove the reliability claim.","section":"V-A1, IV-B1"},{"comment":"The comparison of the clustering RMSE to the average over all Bell partitions lacks a distribution: the reader cannot assess whether the improvement over the baseline is statistically meaningful for n=13 and n=10. A permutation test or a report of the standard deviation of the baseline RMSE distribution should be added.","section":"V-A1, Table III"}],"minor_comments":[{"comment":"Typo: 'throughtabletop' should be 'through tabletop'.","section":"I.A"},{"comment":"The symbol for 'Partially' appears corrupted ('/adjus◎'); check the rendering of the grading symbols.","section":"Table II"},{"comment":"The statement that GPT-4o's disagreement 'essentially matched chance' should be backed by an explicit chance-level baseline, such as the expected RMSE from random ratings with the observed marginal distribution of human scores.","section":"V-A2"},{"comment":"Provide more detail on the feature encoding (how timestamps, sequences, and tool usage are vectorized) and on the DBSCAN hyperparameters (eps, min_samples) for reproducibility.","section":"IV-B1"},{"comment":"The qualitative cluster interpretations are post-hoc; consider explicitly labeling them as illustrative rather than confirmatory, especially given the circularity concern in the quantitative evaluation.","section":"V-B1"},{"comment":"Reference [11] (LLMs for tabletop exercises) is cited only in the TTX introduction; it should also be discussed in the related work section on LLMs for team assessment.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper's central positive claim relies on a circular evaluation. If the authors cannot supply a non-circular validation of the clustering method, the abstract and the answer to RQ1(a) must be substantially revised, potentially to a more modest descriptive claim. The LLM negative result and the open dataset/tools are valuable and should be preserved. I would be willing to review a revised version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: this paper gives you a credible negative result — GPT-4o essentially fails at rubric-based assessment of TTX team communication (55% disagreement, basically chance), and even GPT-5.2 is at 35% — and that is worth taking seriously. The positive result about clustering is real in a narrow sense but the evaluation doesn't support the abstract's claim of validity and reliability. The authors cluster teams from IXP activity logs and then measure within-cluster RMSE on score vectors that are themselves computed from the same milestone IDs used as features. That is not an independent ground truth: the score vector is a deterministic function of a subset of the cluster features, so low RMSE is largely guaranteed. The baseline (average over all Bell partitions) only shows the clusters capture milestone ID patterns better than chance, which is not surprising. So the 'sufficiently aligns with instructor scores' claim in §V-A1 is overreach. The qualitative cluster descriptions in §V-B1 are plausible and might be useful for instructors, but they are post-hoc and don't validate the method.\n\nWhat's genuinely new: this is the first systematic comparison of clustering and LLM assessment on authentic TTX log data, with a real dataset from 24 teams in two countries, public analysis code, and integration into the INJECT platform. The authors are honest about limitations — they admit small sample size, general LLM reliability issues, and the black-box problem — and the LLM negative result is robust enough to stand despite small n. The paper also shares a full TTX scenario and dataset, which is more than most papers in this space do.\n\nSoft spots beyond the circularity: DBSCAN hyperparameters aren't reported, LLM prompt and decoding settings are only partially documented, and the LLM comparison isn't apples-to-apples since GPT-5.2 is a newer model with unknown training overlap. These are fixable in revision. The circularity is the load-bearing issue for the clustering claim; the LLM half of the paper is independent of it and credible.\n\nWho should read this: people working on automated assessment in cybersecurity education or tabletop exercises, and anyone thinking about using general LLMs for grading in domain-specific contexts. It deserves a serious referee — the empirical comparison is valuable even if the clustering claim needs re-scoping. My recommendation: send it out, but require the authors to (a) soften the validity/reliability claim, (b) report the clustering parameters and LLM settings, and (c) either re-frame the clustering evaluation as descriptive rather than validity-bearing or find a genuinely independent benchmark. The dataset and tools are worth citing either way.","headline":"Useful empirical paper on TTX assessment, but the clustering validation is circular and the abstract overstates it; the LLM negative result is the real takeaway.","tokens_in":16818,"tokens_out":1419,"would_cite":true,"duration_ms":18924,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that clustering activity logs yields instructor-aligned team groupings for tabletop cybersecurity exercises in minutes, whereas GPT-4o's rubric-based communication scores disagree with human raters 55% of the time and GPT-5","keywords":["tabletop exercise","team assessment","clustering","large language models","learning analytics","cybersecurity education","rubric-based grading","incident response"],"falsifier":"A concrete test: have independent experts score team performance from a source that does not include milestone logs—e.g., live observation or oral debriefing—and rerun the clustering comparison. If the within-cluster RMSE drops to the Bell-partition baseline, the clustering's apparent validity is an artifact of shared data.","tokens_in":15939,"feed_emoji":"🛡️","tokens_out":5617,"duration_ms":52725,"temperature":0.7,"pith_summary":"This paper tries to establish that automatic team assessment in tabletop exercises (TTXs) is feasible using learning analytics, and that the two candidate methods are not equally trustworthy. It claims that clustering teams by which milestones they reached, and when, groups teams whose instructor-assigned scores are similar—better than the expected baseline across all possible partitions—and can return results in under two minutes, enabling fast cluster-level feedback. It also claims that a general-purpose LLM prompted with a standard rubric does not reliably grade teams' written communication: GPT-4o disagreed with instructors on 55% of scores (essentially chance), and GPT-5.2 on 35%, which the paper deems insufficiently aligned. The paper argues that instructors should adopt clustering for post-exercise assessment and treat LLM output with caution, and it provides the dataset and tools to let others test both methods.","feed_headline":"Clustering aligns with instructors; GPT-4o scores near chance","feed_subtitle":"Cluster team actions to grade cyber exercises in minutes; don't trust generic LLMs for communication scores.","key_machinery":"The load-bearing object is the milestone score vector: each team gets an m-dimensional vector with 2 for an important milestone completed, 1 for an optional milestone, and 0 for a missed one (m = 17 or 18). Clustering uses these same milestone IDs and timestamps as features through DBSCAN with Unsupervised Consensus Clustering over 50 iterations, then the paper measures cluster validity as the average pairwise RMSE of the milestone score vectors within a cluster, compared against the average RMSE over all Bell partitions of the teams. For the LLM arm, the key mechanism is the rubric prompt built from a cybersecurity competency framework's writing criteria (succinct organization, audience app","core_discovery":"The paper's central claim is that in two authentic cybersecurity tabletop exercises (data exfiltration and phishing scenarios, 23 consenting teams across two countries), clustering team activity logs—specifically which milestones each team completed and when—produces groupings with better internal score cohesion than the average over all possible Bell partitions. RMSE within clusters was 0.45 vs 0.53 baseline in the EXF exercise and 0.79 vs 1.12 in PHI; the authors call this 'sufficiently aligns with instructor scores and outperforms the baseline.' The same analysis ran in under two minutes on a standard laptop. On the LLM side, when the same rubric used by human graders was given to GPT-4o,","pith_inferences":["Editorial inference: The near-chance GPT-4o result suggests the failure is not just model weakness but a mismatch between rubric criteria (grammar, audience, concision) and what short incident-response emails reveal; a rubric focusing on incident-handling decision content might perform differently.","Editorial inference: Because the clustering features and the benchmark scores both come from the same milestone logs, a convincing next experiment would use an independent performance measure—such as expert observation or oral debrief scores—to validate the clusters; the paper does not provide that.","Editorial inference: The two methods could complement each other: clustering could select which teams need what feedback, and a domain-adapted LLM could draft the wording of that feedback; the paper's own future-work section gestures at such an ensemble.","Editorial inference: If clustering is adopted broadly, the interpretations of clusters (e.g., 'missed stakeholder contact') depend on the exercise designer being able to read the logs; generalizing to less structured TTXs may require hand-labeled cluster descriptions."],"forward_implications":["Instructors can identify clusters of teams that approached tasks similarly within minutes of a TTX, then deliver feedback tailored to each cluster instead of waiting days.","Because clustering runs locally and uses only activity logs, it avoids sending student data to external services, addressing privacy concerns that come with LLM-based grading.","Teams that are outliers—in no cluster—can be flagged as having unusual approaches, which the paper suggests may indicate either unique strengths or missed protocol.","General-purpose LLMs, even the newer GPT-5.2, should not be used alone for rubric-based assessment of TTX communication; the paper recommends human review or domain-specific models.","The same assessment pipeline can be adapted to other open-ended team exercises (e.g., Capture the Flag or cyber defense exercises) if activity logs record milestone-like checkpoints."],"fun_headline_variants":["Cluster logs beat GPT-4o for grading team cyber drills","Fast clustering matches instructor scores in team exercises","GPT-4o flubs team communication; clustering scores on target","Cluster team actions to grade cyber drills in minutes","Open-source tool clusters team logs, beats GPT-4o for grading"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The evaluation treats the instructor's milestone score vectors as an independent ground truth even though the same milestone IDs and timestamps feed the clustering features; if that overlap inflates within-cluster similarity, the RMSE comparison does not by itself prove educationally valid groupings.","fun_headline_variants_meta":{"raw":{"variants":["Cluster logs beat GPT-4o for grading team cyber drills","Fast clustering matches instructor scores in team exercises","GPT-4o flubs team communication; clustering scores on target","Cluster team actions to grade cyber drills in minutes","Open-source tool clusters team logs, beats GPT-4o for grading"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000606,"raw_usage":{"total_tokens":2678,"prompt_tokens":777,"completion_tokens":1901,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":521,"completion_tokens_details":{"reasoning_tokens":1833}},"tokens_in":521,"tokens_out":1901,"duration_ms":13992,"temperature":1.0,"reasoning_tokens":1833,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T13:04:58.665435+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete test: have independent experts score team performance from a source that does not include milestone logs—e.g., live observation or oral debriefing—and rerun the clustering comparison. If the within-cluster RMSE drops to the Bell-partition baseline, the clustering's apparent validity is an artifact of shared data.","supporting_citations":[],"review_version":1}