{"id":"f1488aad-f4e0-456e-9a32-119aa3eac3bb","arxiv_id":"2508.10310","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A hidden Markov model with nine latent tactics detects three GenAI-writing strategy groups whose essay scores differ, a distinction not found when clustering observed SRL processes directly.","lead":"The authors propose a new way to detect self-regulated learning strategies by adding a hidden layer of 'tactics' between observed behaviors and overall strategies, and apply it to 139 university students writing essays with ChatGPT. They found three learner strategy groups, and students who integrated GenAI into writing scored higher on essays, a difference their method detected but a comparison method did not.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Cluster-then-test inflation may explain the significant performance differences; no permutation test or cross-validation was run.","rationale":"I read the paper in good faith and found a plausible, clearly described methodology with a potentially interesting finding. However, the most load-bearing concern is not only the missing criterion validity (which the authors acknowledge in Section 5.2) but the internal statistical validity of the performance comparisons. The clusters are derived from the data, so testing performance differences across those clusters is a textbook case of 'cluster-then-test' bias. The pipeline's many degrees of freedom—HMM state count, clustering representation, number of clusters—can inflate the false-positive rate. The paper provides no permutation test, no cross-validation, and no correction for multiple comparisons. This threatens the central claim that hidden tactics enable detection of performance-related SRL strategies that the benchmark misses. The reader's focus on construct validity is important, but the statistical concern is more immediate and can be tested with existing data. If the permutation test fails, the empirical finding is unsupported; if it passes, the claim is strengthened. Until such a test is reported, a conditional verdict is appropriate, requiring this additional analysis and transparent reporting of the full pipeline's stability.","tokens_in":16951,"tokens_out":6120,"duration_ms":74701,"concrete_test":"Run a permutation test that repeats the entire pipeline—HMM fitting with state-number selection, Viterbi decoding, Levenshtein-RBF-KMeans clustering, cluster-number selection, and pairwise Mann-Whitney tests—on 1000 datasets where essay scores are randomly permuted across students. Record the minimum p-value across the three pairwise comparisons (or the p-value for the cluster with the highest mean) for each permutation. If the observed p-values from Table 2 (e.g., 0.039 and 0.006) are not below the 5th percentile of this null distribution, the performance differences are not statistically significant after accounting for the data-dependent clustering.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central empirical claim—that HMM-derived SRL strategy clusters differ significantly in essay score (Section 4.3, Table 2)—is established by Mann-Whitney tests on clusters that were themselves derived from the same data. The full pipeline includes several unsupervised choices: the number of HMM states (selected via AIC/BIC/LL), the HMM parameters, the hidden-state decoding (Viterbi), the distance/similarity representation, and the number of clusters (via elbow). Because cluster labels are optimized to capture data structure, they can be correlated with any outcome purely by chance, especially with imbalanced cluster sizes (n=16 in Cluster 2). No multiple-comparison correction was applied to the three pairwise tests, and the smallest p=0.039 would not survive Bonferroni. The benchmark's null result could reflect its more stable clusters rather than a genuine advantage of hidden tactics. The post-hoc 7-versus-12 student comparison is descriptive, lacks a statistical test, and was selected after seeing the clustering output. Thus the paper's conclusion that hidden tactics 'captured more information' than SRL processes rests on potentially spurious p-values.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a layered model of self-regulated learning (SRL) in which observable SRL processes are generated by hidden tactics—latent states modeled with Hidden Markov Models—that in turn combine into SRL strategies via clustering. Using trace data from 139 higher-education students completing a GenAI-assisted academic writing task in the FLoRA platform, the authors identify nine hidden tactics, three strategy clusters (Conventional Strategic Writers, GenAI-Integrated Writers, Intensive Material Reviewers), and report statistically significant essay-score differences among clusters. A benchmark that clusters observed SRL processes directly without the hidden layer yields no significant performance differences, which the paper interprets as evidence that hidden tactics capture information beyond SRL processes. The paper concludes that HMM-derived hidden tactics better reflect the discontinuous, non-linear, and intertwined nature of SRL and support adaptive learning technologies.","tokens_in":17300,"tokens_out":4125,"duration_ms":47393,"significance":"If the empirical claim is robust, this is a valuable methodological contribution: it introduces a new analytical construct ('hidden tactics') and operationalizes it with a principled sequence model, potentially improving how learning analytics infer SRL strategies from trace data. The paper also engages seriously with a benchmark and openly lists limitations, including the absence of think-aloud data. However, the central validation—performance differences across HMM-derived clusters, and the claimed advantage over the benchmark—rests on statistical comparisons that are not yet adequately controlled, so the significance of the finding is currently conditional on those analyses being redone.","major_comments":[{"comment":"The pairwise Mann-Whitney tests are performed on clusters derived from the same data through an unsupervised pipeline (HMM fitting, Viterbi decoding, Levenshtein/RBF/k-means) with the number of clusters selected by elbow on the same data. This cluster-then-test procedure can inflate apparent group differences even when the outcome is independent of cluster labels, particularly with a small cluster such as Cluster 2 (n=16). No omnibus test (e.g., Kruskal-Wallis) is reported, and no multiple-comparison correction is applied; the Cluster 1 vs. Cluster 2 comparison (p=0.039) would not survive a Bonferroni correction for three tests. The paper should report a permutation test that re-assigns cluster labels or re-runs the clustering on permuted outcomes, and should treat p=0.039 as non-significant after correction unless an omnibus test supports an overall group effect.","section":"Section 4.3, Table 2"},{"comment":"The benchmark comparison is confounded by different cluster counts: the proposed method selects three clusters (Figure 5) while the benchmark selects four (Figure 8), so the benchmark's null result could reflect the different cluster solution rather than the absence of hidden tactics. The 7-versus-12 student case analysis is descriptive, was selected post hoc after inspecting the clustering output, and lacks any statistical test of whether the hidden-tactic distributions differ significantly from the SRL-process distributions. The claim that hidden tactics 'captured more information' than SRL processes is therefore not yet supported. A formal comparison is needed—for example, testing whether cluster membership from each method predicts essay score while controlling for cluster count, or using a permutation test on the 7/12 split.","section":"Section 3.2.6, Tables 3-4 and Figures 10-11"},{"comment":"The nine hidden tactics are interpreted as purposeful psychological states (e.g., 'a tactic students used to write while engaging with GenAI') based solely on the HMM emission matrices. The paper explicitly acknowledges in Section 5.2 that no think-aloud or other criterion-related validation data were collected. Without such evidence, the hidden states could be purely statistical latent states of the HMM, and the psychological reality of the proposed 'hidden tactics' construct is not established. This limits the conceptual contribution and weakens the claim that the method models genuine SRL processes. At minimum, the interpretations should be reframed as exploratory and the paper should discuss concrete ways to establish criterion validity (e.g., think-aloud protocols, comparison with self-report measures) before claiming that hidden tactics are a valid analytical layer.","section":"Section 4.1 and Section 5.2"}],"minor_comments":[{"comment":"Several typographical errors: 'T actic' appears in section headings and figure labels instead of 'Tactic'; Section 5.1 contains 'stduents' for 'students'.","section":"Throughout"},{"comment":"The text says 'A HMM'—should be 'An HMM' because HMM is pronounced 'aitch-em-em'.","section":"Section 3.2.3"},{"comment":"The Winne and Hadwin reference is listed as 'Winne, P. H. and Hadwin, A. F. () Studying as self-regulated learning' with a missing year and incomplete bibliographic details; it should be completed.","section":"References"},{"comment":"The outlier removal step is described as removing sequences 'too long or too short' based on z-scores within three standard deviations, but it is not stated whether the z-score is computed across all students' sequence lengths and how the threshold is applied to both tails. Please clarify.","section":"Section 3.2.2"},{"comment":"The hidden-tactic distribution and proportion plots are dense and the labels are small; adding a legend and larger font would improve readability.","section":"Figures 6 and 7"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the scope of learning analytics and human-computer interaction. The methodological idea is promising and the data appear genuine, but the validation section needs substantial additional work before the central claims can be accepted. I recommend major revision rather than rejection, because the issues are addressable with appropriate statistical procedures and more careful framing of construct validity."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper does something genuinely useful: it applies HMMs to theoretically labeled SRL process sequences rather than raw trace data, and interprets the latent states as an intermediate 'hidden tactics' layer. That is a sensible conceptual move, and the authors are admirably transparent about the whole pipeline—state selection via AIC/BIC/LL, emission/transition interpretation, clustering choices, and the benchmark comparison. The nine tactics they extract are plausible and the qualitative accounts of the three strategy clusters read like real behavioral distinctions. The limitations section is honest: no think-aloud data, single dataset, coarse CHATGPT labeling. That earns credit.\n\nBut the central empirical claim—that hidden tactics distinguish performance groups where direct process clustering cannot—is not as solid as stated. The clusters are derived from the same data used to test performance differences, and no permutation test or cross-validation is run. That is a classic cluster-then-test inflation source. On top of that, of the two significant pairwise comparisons (p=0.039 and p=0.006), only the second survives a Bonferroni correction for the three tests. So the evidence is one small cluster (n=16) differing from one larger cluster, after correction. The benchmark comparison is also confounded: it uses four clusters to the proposed method's three, and the cluster sizes are very different, so the benchmark's null result could reflect different granularity rather than a genuine advantage of hidden tactics. The 7-vs-12 student subgroup analysis is post hoc and descriptive, with no statistical test, and was selected after seeing the clustering output.\n\nNone of this means the approach is wrong. It could be confirmed with a pre-registered replication, permuted null tests, matched cluster counts, and think-aloud or other criterion-validity evidence. But as it stands, the abstract overstates what the data show. The paper is written and executed carefully, and it clearly identifies its own limitations, so the authors are not hiding the soft spots. I just would not treat the performance-difference finding as established.\n\nThis is a paper for learning analytics researchers working on SRL process modeling. It deserves a serious referee—not a desk reject—because the method is clearly described, the field needs more transparent examples of latent-state modeling, and the identified flaws are addressable. I would request permutation tests, multiple-comparison correction, and a more balanced benchmark before acceptance. I probably would not cite the performance claim as evidence, but I might cite the pipeline as a worked example of HMM-based tactic modeling.","headline":"A transparent, carefully described HMM pipeline for SRL tactics, but the performance claim is thinner than the abstract suggests: one of two pairwise differences survives correction, and the cluster-then-test design could inflate it.","tokens_in":17750,"tokens_out":2249,"would_cite":false,"duration_ms":24227,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that unobservable 'hidden tactics'—latent states in a Hidden Markov Model—sit between SRL processes and strategies in GenAI-assisted writing, and that grouping students by these tactics yields significant essay-score differ","keywords":["self-regulated learning","hidden tactics","Hidden Markov Model","learning analytics","generative AI","academic writing","learning strategies","trace data"],"falsifier":"Run the same analysis on a new GenAI-writing cohort with think-aloud protocols; if students' stated intentions do not match the tactic labels assigned by the HMM's latent states—for example, a student labeled as using the 'exclusive GenAI interaction' tactic reports careful reading—then the hidden-tactic interpretation is unsupported.","tokens_in":16909,"feed_emoji":"✍️","tokens_out":6720,"duration_ms":62165,"temperature":0.7,"pith_summary":"The paper tries to show that self-regulated learning in GenAI-assisted academic writing cannot be adequately captured by clustering observable learning processes directly. It proposes an intermediate layer of 'hidden tactics'—short, purposeful action states inferred as latent variables—and models them with a Hidden Markov Model. From trace data of 139 students writing with ChatGPT help, the model finds nine hidden tactics and three strategy groups (conventional strategic writers, GenAI-integrated writers, and intensive material reviewers). The GenAI-integrated group scored significantly higher on essays than the other two groups, whereas a benchmark that clusters the raw SRL process sequences found no significant group differences. A sympathetic reader would take this as evidence that the hidden-tactic layer carries information about learning strategies that observable processes obscure.","feed_headline":"Nine hidden tactics sort GenAI writers by essay score","feed_subtitle":"HMM-inferred tactics, not raw click processes, separate top-scoring ChatGPT-integrated writers from planners and re-readers.","key_machinery":"The central object is the Hidden Markov Model's latent state, reinterpreted as a 'hidden tactic.' The model's emission matrix assigns each hidden tactic a probability distribution over the seven SRL process codes plus a CHATGPT code, so a single tactic can represent intertwined processes—the mechanism that lets the model capture non-linear, overlapping regulation. The transition matrix captures how students move between tactics over time. The tactic sequences are then clustered via Levenshtein distance, radial basis transformation, and k-means, with the number of tactics and clusters chosen by information criteria and elbow/silhouette methods.","core_discovery":"The paper's central claim is that between observable self-regulated learning (SRL) processes—the coded click and keystroke events—and broader SRL strategies there is an unobserved layer called hidden tactics, and that modeling this layer is necessary for detecting meaningful strategy differences. Using a Hidden Markov Model on SRL process sequences from 139 higher-education students doing a GenAI-assisted writing task, the study reports nine hidden tactics, each characterized by a probability distribution over SRL processes; some are pure (e.g., initial reading, focused writing, exclusive ChatGPT interaction) while others mix processes (e.g., writing while consulting requirements). Clusterin","pith_inferences":["The two-layer HMM-plus-clustering recipe is portable: any task with noisy, overlapping trace labels—programming, inquiry learning, collaborative problem solving—could gain from an intermediate latent-tactic layer before strategy clustering.","A stronger test of the hidden-tactic construct would be whether the clusters predict performance on a later, unaided transfer task; the paper only measures essay quality within the GenAI-assisted session.","Because the hidden tactics are inferred statistically, the names assigned to them (e.g., 'writing while engaging with GenAI') are interpretations; think-aloud or screen-review data would be needed to confirm that students actually experience these as distinct purposeful tactics."],"forward_implications":["Learning analytics can identify strategy groups in GenAI-assisted writing without assuming SRL processes unfold as clean, non-overlapping linear sequences.","Students who integrate ChatGPT throughout writing tended to produce higher-scoring essays, while students who spent long phases re-reading materials scored lowest in this dataset.","A benchmark method that clusters observable SRL processes directly found no significant performance differences, suggesting the hidden-tactic layer adds discriminatory power.","Adaptive writing tools could in principle monitor hidden-tactic sequences in real time to flag patterns such as persistent re-reading or heavy GenAI use without deep cognitive engagement.","The finding that the highest-scoring group used GenAI heavily but engaged less in complex metacognitive activity supports the concern that high performance during GenAI-assisted tasks may not equal deeper learning."],"supporting_citations":[{"why":"Supplies the trace-data-to-SRL-process labeling pipeline and process library that the paper builds on.","marker":"Fan et al. (2022)"},{"why":"Defines the benchmark method—First-Order Markov Model and Expectation-Maximization clustering—that identifies strategies directly from SRL processes without hidden tactics.","marker":"Srivastava et al. (2022)"},{"why":"Describes the online writing platform used to collect the 241-student dataset and its ChatGPT-integrated trace data.","marker":"Li et al. (2024b)"},{"why":"Provides the cognitive and metacognitive process categories (orientation, planning, monitoring, evaluation, reading, elaboration) used to label learning actions.","marker":"Bannert (2007)"},{"why":"Earlier use of Hidden Markov Models on trace data for learning strategies, which the paper adapts by applying HMM to processed SRL processes instead of raw traces.","marker":"Fincham et al. (2018)"},{"why":"Prior study on GenAI-assisted writing linking active higher-order cognitive engagement to essay quality, used to contextualize the performance differences.","marker":"Yang et al. (2025)"},{"why":"Experimental evidence on 'metacognitive laziness' and GenAI effects on motivation, processes, and performance, which the discussion invokes to interpret the high-performance, low-deep-engagement pattern.","marker":"Fan et al. (2025)"},{"why":"Empirical support that SRL patterns are non-linear and multi-factorial, the motivation for introducing the hidden-tactic layer.","marker":"Li et al. (2020)"}],"fun_headline_variants":["HMM finds 9 hidden tactics in GenAI writing","Nine tactics separate top AI-writing students","Hidden tactics revealed in ChatGPT-assisted essays","Why GenAI writers differ: 9 tactics discovered","Unobserved tactics link to GenAI essay scores"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The load-bearing premise is that the HMM's nine latent states correspond to genuine, purposeful tactics in students' minds, even though the study never validates that mapping against students' own accounts or other external criteria.","fun_headline_variants_meta":{"raw":{"variants":["HMM finds 9 hidden tactics in GenAI writing","Nine tactics separate top AI-writing students","Hidden tactics revealed in ChatGPT-assisted essays","Why GenAI writers differ: 9 tactics discovered","Unobserved tactics link to GenAI essay scores"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000187,"raw_usage":{"total_tokens":1168,"prompt_tokens":748,"completion_tokens":420,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":492,"completion_tokens_details":{"reasoning_tokens":350}},"tokens_in":492,"tokens_out":420,"duration_ms":5273,"temperature":1.0,"reasoning_tokens":350,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T20:31:27.426404+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same analysis on a new GenAI-writing cohort with think-aloud protocols; if students' stated intentions do not match the tactic labels assigned by the HMM's latent states—for example, a student labeled as using the 'exclusive GenAI interaction' tactic reports careful reading—then the hidden-tactic interpretation is unsupported.","supporting_citations":[{"cited_title":"and Ga s evi \\'c , D","cited_arxiv_id":null,"evidence_quote":"Supplies the trace-data-to-SRL-process labeling pipeline and process library that the paper builds on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the benchmark method—First-Order Markov Model and Expectation-Maximization clustering—that identifies strategies directly from SRL processes without hidden tactics."},{"cited_title":"(2007) Metakognition beim lernen mit hypermedien","cited_arxiv_id":null,"evidence_quote":"Provides the cognitive and metacognitive process categories (orientation, planning, monitoring, evaluation, reading, elaboration) used to label learning actions."},{"cited_title":"and Pardo, A","cited_arxiv_id":null,"evidence_quote":"Earlier use of Hidden Markov Models on trace data for learning strategies, which the paper adapts by applying HMM to processed SRL processes instead of raw traces."},{"cited_title":"and Chen, G","cited_arxiv_id":null,"evidence_quote":"Prior study on GenAI-assisted writing linking active higher-order cognitive engagement to essay quality, used to contextualize the performance differences."},{"cited_title":"and Ga s evi \\'c , D","cited_arxiv_id":null,"evidence_quote":"Experimental evidence on 'metacognitive laziness' and GenAI effects on motivation, processes, and performance, which the discussion invokes to interpret the high-performance, low-deep-engagement pattern."}],"review_version":1}