{"id":"d50b7e52-1456-4250-8e0a-09410313bf5c","arxiv_id":"2501.18468","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A 2D CNN trained on gaze scanpath images recognizes sequential, non-sequential, and skimming reading in a new classroom dataset with 0.80 macro F1, but only across 3 of 6 defined behaviors.","lead":"This paper tests whether eye-tracking can tell apart natural classroom reading behaviors like skimming, deep reading, and scanning, rather than behaviors students are told to perform. The authors built a small CNN that sorts gaze patterns into three of six human-defined behavior types, reaching a macro F1 of 0.80 on a new classroom dataset.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported F1=0.80 rests on human annotation labels whose inter-rater reliability is only moderate (Kappa 0.57-0.65); without independent reproducibility of the labels, the classifier may be learning the annotators' visual heuristics rather than stable reading behaviors.","rationale":"All downstream quantitative results flow from the human annotation: the taxonomy (RQ2), the instructed versus in-the-wild statistical distinctions (RQ1), and the supervised classifier (RQ3). The annotation process is explicitly iterative and consensus-based, and the Kappa values are moderate. The CNN's input is the same visual scanpath representation annotators used, so a high F1 may reflect learning the annotation scheme rather than discovering behavior. I considered whether the more serious issue is the per-window evaluation unit (long 'sequential' segments produce many correlated test windows), but LOPOCV removes participant leakage and macro-F1 mitigates the imbalance. The decisive test is therefore external reproducibility of labels. The reader's version of this concern is essentially the same, so agreement is 'agree.' Because the paper is framed as a proof-of-concept and the issue can be resolved by an additional validation step, the reader's CONDITIONAL verdict remains appropriate; thus the verdict should be unchanged.","tokens_in":18370,"tokens_out":5146,"duration_ms":53665,"concrete_test":"Take a random subset of 10-12 coldread sessions from the 27 analyzed sessions. Have two annotators who were not involved in taxonomy development independently assign labels to the existing segment boundaries using only the final codebook (Section 5.2 and Fig. 10). Compute Cohen's Kappa between each new annotator and the original final labels, and also per-class agreement for the 'sequential' versus 'non-sequential' distinction. If average Kappa falls below 0.7, or if the new annotators systematically confuse the two classes the CNN most confuses, the ground-truth labels are not stable across annotators and the reported macro F1 of 0.80 is not a robust estimate of behavior recognition.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.1 (Human Annotation) is the hinge of the paper. It generates the labels used as ground truth for the CNN in Section 5.4 and as the basis for the statistical comparisons in Section 5.3. The protocol is not an application of a fixed codebook: the three reviewers created and renamed labels iteratively, a single reviewer first segments each video, the second reviewer only labels the given segments, and disagreements are resolved collaboratively in IRR rounds. The reported Cohen's Kappa is 0.57 in the first large round and 0.65 in the final round (Table 2), which is moderate agreement. The paper never checks whether the final consensus labels are reproducible by independent coders, and Section 8's limitation list does not mention annotation reliability. The CNN additionally consumes scanplot images rendered with the same blue-to-red temporal gradient used for the human annotations (Section 4.2.2), so high performance may indicate that the network reproduces the annotators' visual heuristics. If the labels are idiosyncratic, then the F1=0.80, the Hotelling T-squared results, and the taxonomy are relative to the original team's judgment, not objectively stable reading behaviors.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a classroom eye-tracking study with 27 sixth-grade students, combining instructed-reading and in-the-wild reading conditions. The authors develop a six-behavior taxonomy ('static', 'deep', 'sequential', 'non-sequential', 'skimming', 'previewing/mapping') through iterative human annotation, propose a three-dimension framework (velocity, density, sequentiality), report statistical differences between instructed and in-the-wild behaviors and between behavior pairs, and train a 2D CNN on 10-fixation scanplot images to classify three of the six behaviors, reporting a macro F1 of 0.80 under leave-one-participant-out cross-validation. The central claims are that in-the-wild reading behaviors differ measurably from instructed ones and that a lightweight CNN can recognize such behaviors in real time.","tokens_in":18624,"tokens_out":7919,"duration_ms":80137,"significance":"If the reported results hold, the paper would make a useful contribution: the ecological classroom dataset, the explicit behavior taxonomy, and the real-time classification pipeline are all potentially valuable to the reading-behavior and educational-technology communities. The use of leave-one-participant-out cross-validation and the reporting of inference times are strengths, and the planned public release of code and data would support reproducibility. However, the central empirical claim currently rests on human annotations whose reliability is moderate (Cohen's Kappa 0.57–0.65), and several validation steps are circular or incompletely specified. The contribution is therefore best viewed as a proof-of-concept whose scope and limitations must be stated more precisely before the claims can be accepted as stated.","major_comments":[{"comment":"The ground truth for both the statistical comparisons and the CNN is the set of human labels produced in the iterative annotation process described in Section 4.1. Section 4.1.2 reports that the taxonomy was created and renamed during the annotation rounds, and that final Cohen's Kappa values are 0.57 and 0.65 (Table 2), which are moderate. Because the same labels are used to train the CNN and to run the Hotelling T-squared and t-tests, the reported F1 and p-values measure the separability of the annotators' own judgments rather than the separability of objectively defined reading states. The paper should either validate the labels with an independent, codebook-based annotation by new coders or explicitly limit all conclusions to 'human-defined behavior classes' and add annotation reliability to Section 8's limitation list.","section":"Sections 4.1, 5.3, 5.4"},{"comment":"The 2D CNN is trained on scanplot images rendered with the same blue-to-red temporal gradient that the human reviewers used when annotating the gaze scanpath videos (see the caption of Fig. 8 and Section 4.1.1). The macro F1 of 0.80 may therefore reflect the network learning the reviewers' visual heuristics rather than recognizing stable reading states. To support the claim of a 'robust approach for understanding in-the-wild reading', the authors should demonstrate that the model generalizes to labels obtained from a fixed codebook or that its predictions correlate with an independent behavioral outcome.","section":"Sections 4.1.1, 4.2.2, Fig. 8"},{"comment":"The abstract states that the 2D CNN achieved an F1 score of 0.8 'for behavior recognition' and that the framework differentiates reading behaviors, but Section 4.2 explicitly restricts the classifier to three of the six defined behaviors ('sequential', 'non-sequential', and 'skimming') because of class imbalance and limited instances of the other behaviors. All abstract and discussion claims should be qualified to say that the F1 score applies to three behavior classes under the study's narrow conditions, not to the full six-behavior taxonomy.","section":"Abstract and Section 4.2"},{"comment":"The baseline results in Table 14a are internally inconsistent under standard definitions. For a three-class problem with a dominant 'sequential' class, an always-majority classifier has macro F1 at most 1/3 in the usual macro-average convention (for a sequential proportion around 0.87, macro F1 is approximately 0.31), yet the table reports 0.56 for both 'Majority Class' and 'SVC'. Similarly, a random classifier with class-prior probabilities would have macro F1 around 0.3 and accuracy around 0.76, not the reported 0.38 and 0.79. Please specify exactly how 'Random' and 'Majority Class' baselines were constructed and report class-wise precision/recall or confusion matrices for all baselines.","section":"Table 14a, Section 5.4"},{"comment":"The exhaustive grid search over learning rate, weight decay, model design, batch size, and scanpath plot generation is described without stating how hyperparameters were selected relative to the leave-one-participant-out folds. If the grid search used the same test folds that produce the reported F1, the performance estimate is optimistically biased. Please describe the inner validation procedure and report the variance or confidence intervals of the macro F1 across the 27 folds.","section":"Section 4.2.2, Section 5.4"}],"minor_comments":[{"comment":"There is a typo in 'readi recognition' that should read 'reading recognition'.","section":"Section 2.3"},{"comment":"The phrase 'p < 0.348' for the skimming comparison should be 'p = 0.348' or similar, since an inequality with a non-significant p-value is not standard.","section":"Section 5.1"},{"comment":"The cross-reference 'as shown in Fig. 5' should refer to Table 5, which contains the pairwise Hotelling T-squared p-values.","section":"Section 5.3"},{"comment":"The caption 'Table 14a' is used for a table and 'Fig. 14b' for a confusion matrix; the numbering of tables and figures should be separated and consistent with the journal style.","section":"Section 5.4"},{"comment":"There is a grammatical error: 'Gaussian noise was add' should be 'Gaussian noise was added'.","section":"Section 4.2.2"},{"comment":"The text says a fixed 10-fixation window corresponds to 6 seconds, but with typical fixation durations of 200–300 ms this would be 2–3 seconds; please clarify how the 6-second estimate is derived.","section":"Section 6.3"},{"comment":"The limitations section does not mention the moderate inter-rater reliability or the lack of independent annotation validation; adding this would give readers an accurate picture of the strength of the ground-truth labels.","section":"Section 8"}],"recommendation":"major_revision","confidential_remarks":"The paper would be strengthened by addressing the annotation-reliability and circularity concerns before publication; without that, the F1 score and statistical tests are relative to the original team's subjective labels. The abstract overstates the scope of the classifier, and the promised GitHub link is still a placeholder. The contribution is potentially valuable as an exploratory proof-of-concept, but the load-bearing claims need substantiation or explicit qualification."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe short version: this is a genuine proof-of-concept for recognizing naturalistic reading behaviors from classroom eye-tracking, and the strongest part is the data collection and the qualitative-to-quantitative bridge. But the headline F1=0.80 sits on human labels with moderate inter-rater agreement, and the paper never tests whether those labels are reproducible by independent coders. I'd send it to review, but the revision bar is real.\n\nWhat's actually new: a classroom dataset with both instructed and in-the-wild reading from the same students, a taxonomy that names behaviors (non-sequential, previewing/mapping, static) that prior work either conflated or ignored, and a head-to-head comparison showing instructed and naturalistic gaze differ on velocity and density for sequential and deep reading. That comparison is the most useful empirical result. The authors are appropriately cautious: they call it exploratory, limit the classifier to three of six behaviors, and use leave-one-participant-out CV.\n\nThe soft spots are where the reader says they are. The annotation protocol has one reviewer segment first, a second labels the given segments, and disagreements are resolved collaboratively; kappa 0.57–0.65 is moderate, not strong. The CNN consumes scanplot images rendered with the same blue-to-red gradient used for human annotation, so part of the model's success may be reproducing annotators' visual heuristics rather than discovering stable gaze signatures. The paper doesn't acknowledge this as a limitation, which is a miss. The baseline table also looks odd—Majority Class and SVC are identical across all four metrics, which suggests either a copy-paste issue or something worth explaining. There are no error bars on the F1, and the promised GitHub repo is still a placeholder.\n\nNone of this kills the central claim that in-the-wild reading is behaviorally richer than instructed reading, and that a lightweight CNN can do better than chance at classifying three of the behaviors. That claim is plausible and the evidence, while preliminary, supports it. The circularity between the human-derived framework and the subsequent validation is real but standard for this kind of exploratory work; they don't over-claim generalizability in the discussion. The citation pattern is fair—Campbell and Maglio, Busjahn, and Chen et al. are the right touchstones.\n\nWho is it for: eye-tracking and educational-technology researchers who need a reference point for naturalistic reading behavior taxonomies. It deserves a serious referee—send it out, but the reviewers should focus on annotation reliability and an honest treatment of what the CNN is learning.","headline":"Genuine proof-of-concept for in-the-wild reading behavior recognition, but the F1=0.80 rests on moderately reliable human labels and needs an honest limitations section.","tokens_in":19204,"tokens_out":2979,"would_cite":true,"duration_ms":25033,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Real classroom reading differs from instructed reading, and an image-based neural network can spot three natural reading behaviors from ten-fixation gaze scanpaths with a macro F1 of 0.80.","keywords":["reading behavior recognition","eye tracking","in-the-wild reading","classroom study","scanpath images","behavior taxonomy","convolutional neural network","real-time classification"],"falsifier":"Take the same gaze videos and have two fresh, independent annotation teams label them under the same protocol: if their agreement is again only moderate (kappa around 0.57–0.65), and a classifier trained on one team's labels does no better than chance on the other team's labels, then the F1 of 0.80 reflects the specific annotators rather than stable reading behaviors.","tokens_in":18155,"feed_emoji":"👁️","tokens_out":11074,"duration_ms":92604,"temperature":0.7,"pith_summary":"The paper asks whether reading behaviors can be recognized from eye gaze in a real classroom, without instructing students how to read. It reports that natural reading produces gaze patterns that differ measurably from instructed reading, and it organizes those patterns into six behaviors defined by velocity, gaze density, and sequentiality. The central empirical claim is that a lightweight 2D convolutional network, fed with images of ten-fixation gaze scanpaths, can classify three of those behaviors—sequential, non-sequential, and skimming—with a macro F1 of 0.80 under leave-one-participant-out cross-validation. If that claim holds, real-time feedback about how students read is feasible on ordinary classroom hardware, although the study is explicitly a proof of concept rather than a generalizable deployment.","feed_headline":"Eye-tracking model spots real classroom reading behaviors at F1 0.80","feed_subtitle":"Ten-fixation scanpath images let a lightweight CNN tell skimming from sequential and non-sequential reading in real time.","key_machinery":"The central mechanism is the scanpath image: a rendered picture of a ten-fixation gaze window over the text, colored blue-to-red by time, which lets a 2D convolutional network see the shape and direction of reading without hand-engineered features. Supporting this, the paper defines a velocity–density–sequentiality framework: words per minute, inverse fixation dispersion, and forward-versus-backward saccade ratio, which human annotators used to define the behavior taxonomy and which the statistical analyses show separates the classes. The data pipeline converts raw gaze points into page coordinates in real time, so scrolling and zooming do not corrupt the scanpath image. The classifier stacks a pretrained residual-network backbone with three fully connected layers and runs at about 3 ms on a GPU and 14 ms on a CPU, far below the roughly six-second input window.","core_discovery":"The paper's central discovery is that in-the-wild reading behavior in a classroom can be defined and automatically detected from gaze data alone, and that it is not the same as behavior performed on instruction. Human reviewers watching replay videos of gaze scanpaths from 27 sixth-grade students identified six archetypes—static, deep, sequential, non-sequential, skimming, and previewing/mapping—and the paper shows these are distinguishable along three continua: words per minute (velocity), inverse fixation dispersion (density), and the forward-versus-backward saccade ratio (sequentiality). Pairwise multivariate tests separate most behavior pairs, with only skimming versus previewing/mapping not reaching significance. Instructed and in-the-wild gaze distributions differ significantly for sequential and deep reading but not for skimming. A lightweight 2D convolutional network using rendered ten-fixation scanpath images as input reaches a macro F1 of 0.80 (precision 0.82, recall 0.81) for sequential, non-sequential, and skimming under leave-one-participant-out cross-validation, with inference fast enough for real-time use.","pith_inferences":["If velocity, density, and sequentiality are true axes of reading behavior, the same ten-fixation scanpath representation should transfer to other texts, age groups, and languages; a direct transfer test would reveal whether the taxonomy is universal or specific to this passage and grade.","Because the classifier is trained on human labels with only moderate inter-rater agreement, its practical ceiling is the reliability of the annotation protocol; checking predicted labels against an independent outcome, such as correctness on comprehension questions tied to specific text sections, would separate behavior recognition from annotator subjectivity.","The statistical overlap between skimming and previewing/mapping suggests the taxonomy may be treating two variants of one fast behavior as two categories; more instances of previewing/mapping could either sharpen the boundary or merge the labels.","The finding that skimming looks similar under instruction and in the wild hints that skimming is a more stable reading mode, while sequential and deep reading are more context-sensitive; this could make skimming a better target for general reading instruction."],"forward_implications":["Instructed reading tasks are not a safe stand-in for real classroom reading, so behavior models meant for classrooms should be trained on naturalistic gaze data.","A ten-fixation window can capture short behaviors such as skims that longer sliding-window methods (over 30 seconds) would blend into multiple labels.","The pipeline runs faster than real time (about 3 ms inference on a GPU and 14 ms on a CPU), so per-section behavior feedback during a lesson is technically plausible.","Mapping detected behaviors to passage locations could let teachers see which sections a student skimmed, read sequentially, or revisited, and connect those patterns to comprehension questions."],"supporting_citations":[{"why":"It supplies the initial 'regular' and 'skimming' labels and the rule-based reading-detection approach from which the taxonomy starts.","marker":"[5]"},{"why":"It provides the time-window deep-versus-skim classification method and motivates the need for finer-grained windows.","marker":"[8]"},{"why":"It presents the real-time reading detection baseline that the convolutional classifier must outperform.","marker":"[21]"},{"why":"It defines the saccade metrics and linear-order measure used for the sequentiality continuum and feature computation.","marker":"[4]"},{"why":"It shows that convolutional networks can decode eye-movement data, inspiring the 1D and 2D CNN input designs.","marker":"[9]"},{"why":"It supplies the pretrained residual-network backbone used by the lightweight 2D CNN.","marker":"[13]"},{"why":"It supports the claim that reading instructions alter natural reading behavior, motivating the in-the-wild comparison.","marker":"[17]"},{"why":"It supports the claim that task demands affect eye movements and that natural reading differs from standardized reading tests.","marker":"[19]"},{"why":"It provides the bag-of-classifiers baselines against which the CNN result is compared.","marker":"[31]"},{"why":"It provides the implementation of the independent t-test used to confirm that sequential and non-sequential reading differ on the forward-versus-backward saccade ratio.","marker":"[38]"}],"fun_headline_variants":["Eye tracking decodes real classroom reading styles, no tasks needed","Lightweight CNN on gaze hits F1 0.8 for reading styles in class","In-the-wild reading recognized by gaze-only AI at F1 0.8","Gaze patterns reveal six reading archetypes, AI tells three apart"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The result rests on the assumption that the behaviors named by the human annotators are stable, real categories rather than subjective interpretations of noisy gaze videos, because both the statistical separations and the classifier's F1 score are measured against those labels.","fun_headline_variants_meta":{"raw":{"variants":["Eye tracking decodes real classroom reading styles, no tasks needed","Lightweight CNN on gaze hits F1 0.8 for reading styles in class","In-the-wild reading recognized by gaze-only AI at F1 0.8","Gaze patterns reveal six reading archetypes, AI tells three apart"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001351,"raw_usage":{"total_tokens":5488,"prompt_tokens":948,"completion_tokens":4540,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":564,"completion_tokens_details":{"reasoning_tokens":4458}},"tokens_in":564,"tokens_out":4540,"duration_ms":29085,"temperature":1.0,"reasoning_tokens":4458,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T23:22:32.712581+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the same gaze videos and have two fresh, independent annotation teams label them under the same protocol: if their agreement is again only moderate (kappa around 0.57–0.65), and a classifier trained on one team's labels does no better than chance on the other team's labels, then the F1 of 0.80 reflects the specific annotators rather than stable reading behaviors.","supporting_citations":[{"cited_title":"Campbell and Paul P","cited_arxiv_id":null,"evidence_quote":"It supplies the initial 'regular' and 'skimming' labels and the rule-based reading-detection approach from which the taxonomy starts."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It provides the time-window deep-versus-skim classification method and motivates the need for finer-grained windows."},{"cited_title":"Das, Dimitris Samaras, and Gregory Zelinsky","cited_arxiv_id":null,"evidence_quote":"It presents the real-time reading detection baseline that the convolutional classifier must outperform."},{"cited_title":"Cole, Karl M","cited_arxiv_id":null,"evidence_quote":"It shows that convolutional networks can decode eye-movement data, inspiring the 1D and 2D CNN input designs."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It supports the claim that reading instructions alter natural reading behavior, motivating the in-the-wild comparison."},{"cited_title":"Kaakinen and Jukka Hyönä","cited_arxiv_id":null,"evidence_quote":"It supports the claim that task demands affect eye movements and that natural reading differs from standardized reading tests."}],"review_version":1}