{"id":"cd57722c-61ad-49ee-9307-a388df845b14","arxiv_id":"2506.11004","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"The paper's high-accuracy dyslexia classifier actually predicts a synthetic 95th percentile eye-tracking label, with no clinically diagnosed dyslexic participants in the data.","lead":"This paper trains a Random Forest to predict a 'reading difficulty' label created by thresholding eye-tracking features at the 95th percentile in the Provo Corpus, reporting 88.58% accuracy. It also clusters readers by eye movement patterns, but the dataset contains no diagnosed dyslexic readers, so the dyslexia claim is not supported.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 88.58% accuracy is for predicting a synthetic 95th-percentile label built from the same eye-tracking features, not for clinically confirmed dyslexia; no external validation supports the claim.","rationale":"This is the weakest load-bearing premise because every downstream quantitative claim inherits it: if Reading_Difficulties is not a valid dyslexia label, the accuracy, precision, recall, F1, and AUC in Section 4.1 describe only predictability of a synthetic percentile label. The flaw is internal to the paper's own methodology, not a matter of disagreement with consensus. The absence of dyslexic participants in the Provo Corpus is documented in Section 3.1 (84 native English-speaking proficient readers), and the label construction is described in Section 3.3; no external validation is offered anywhere. The paper has no independent support that would offset this: no clinical dataset, no transfer test, and no reproducibility artifact. The stress-test concern matches the Reader's weakest_assumption, and the rejection recommendation stands. If the paper were reframed as predicting extreme reading behavior within a normal population, the experiment might be internally coherent, but that is not the claim made in the abstract or conclusion.","tokens_in":7542,"tokens_out":4629,"duration_ms":43468,"concrete_test":"Apply the published feature-selection and Random Forest pipeline unchanged to an independent eye-tracking corpus that contains clinically confirmed dyslexic readers and matched controls (e.g., the eSeek data used in Raatikainen et al. [10]), scoring at the participant level rather than the word level. If accuracy and AUC on this external clinical sample are near chance, the 88.58% accuracy does not transfer to dyslexia detection and the central claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is undermined by the construction of the target label. In Section 3.3 (\"Thresholds for labeling\"), Reading_Difficulties is defined by assigning label 1 to any word-level eye-tracking entry whose value on metrics such as Ia_Dwell_Time or Ia_Fixation_Count exceeds the 95th percentile; all other entries are labeled 0. The dataset is the Provo Corpus [6], collected from 84 proficient readers with no reported clinical dyslexia diagnosis. Thus the classifier trained in Section 4.1 is not asked to detect a disorder; it is asked to recover a percentile-based thresholding artifact from the same eye-tracking measurement space. Even though the authors exclude the features used directly to build the label from the final model, the remaining predictors (regression counts, fixation indices, saccade durations, skip rates) are strongly correlated with the label-generating metrics because they all describe the same fixation-saccade process. The high accuracy (88.58%) and AUC (0.96) therefore reflect how predictable the 95th-percentile extremeness of a row is from correlated eye-movement features, not how detectable dyslexia is. Section 4.2's cluster interpretation (\"poor readers, potentially including individuals with dyslexia\") is a post-hoc label applied to clusters with no external diagnostic ground truth. The conclusion repeats the 95th-percentile framing while claiming an effective diagnostic tool. Without a clinically confirmed outcome variable, the paper's headline claim does not follow.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript proposes an eye-tracking and machine-learning pipeline for dyslexia detection. The authors use the Provo Corpus, a large eye-tracking dataset of 84 proficient readers, and construct a binary target variable called Reading_Difficulties by thresholding eye-movement metrics at the 95th percentile. A Random Forest classifier, trained on the remaining eye-tracking features, is reported to achieve 88.58% accuracy and an ROC AUC of 0.96 under 9-fold cross-validation. Agglomerative hierarchical clustering is applied to the same feature space, and the resulting clusters are interpreted as representing a spectrum of reading ability from proficient to dyslexic. The paper concludes that the method is a cost-effective, non-invasive diagnostic tool for early dyslexia detection.","tokens_in":7882,"tokens_out":2724,"duration_ms":24816,"significance":"If the reported accuracy reflected detection of clinically confirmed dyslexia, the result would be a meaningful contribution to accessible dyslexia screening. The paper also has some positive aspects: it clearly describes a data preprocessing and feature-engineering pipeline, makes an explicit attempt to avoid direct data leakage by excluding the label-generating features from the classifier, and reports cross-validated performance metrics. However, the central claim is undermined by the construction of the target variable: Reading_Difficulties is a synthetic percentile-based label derived from the very same type of eye-tracking measurements that constitute the input features, and the Provo Corpus contains no participants with a clinical dyslexia diagnosis. Consequently, the reported accuracy measures how well the model recovers a 95th-percentile thresholding artifact, not how well it detects dyslexia. The absence of any external or clinical validation makes the diagnostic claim unsupported by the presented evidence.","major_comments":[{"comment":"The target variable Reading_Difficulties is defined by assigning a label of 1 to any word-level entry whose value on metrics such as Ia_Dwell_Time, Ia_Fixation_Count, or Ia_First_Saccade_Amplitude exceeds the 95th percentile, with all other entries labeled 0. Since these are eye-tracking measures, and the classifier in Section 4.1 is trained on other eye-tracking features from the same reading process, the model is effectively predicting a thresholding artifact of the same measurement modality. The reported 88.58% accuracy and 0.96 AUC therefore do not support the abstract's claim of 'detect[ing] dyslexia'; they support only the conclusion that percentile-based extremeness of one eye-tracking feature is predictable from related eye-tracking features. This circularity is load-bearing because the paper's central claim depends on Reading_Difficulties being a valid proxy for dyslexia, which is not established.","section":"Section 3.3, 'Thresholds for labeling'"},{"comment":"The evaluation uses 9-fold cross-validation at the row level over 230,412 word-level entries originating from only 84 participants. Because each participant contributes many rows, random row-level splitting will place the same participant in both the training and test folds, allowing the model to memorize subject-specific eye-movement patterns and inflating the reported accuracy. A participant-level or text-level cross-validation scheme is required to support any claim about detecting reading difficulties in new readers, but no such analysis is reported.","section":"Section 4.1, Table 1 and Figure 1"},{"comment":"Cluster 2 is characterized as 'poor readers, potentially including individuals with dyslexia' and the clustering results are presented as evidence of 'varying severity levels of dyslexia.' However, no diagnostic information, reading-fluency score, or other external ground truth is available to validate these cluster interpretations. The labels assigned to clusters are post-hoc descriptions of the eye-tracking metrics, not validated subgroups, so the clustering analysis does not provide evidence for dyslexia severity.","section":"Section 4.2, 'Cluster and Analysis'"},{"comment":"The conclusion states that the method is 'a non-invasive, objective, and cost-effective diagnostic tool' and 'supports early identification of students needing intervention.' These claims are not supported by the analysis, which never compares the method against a clinical diagnosis or any established reading-difficulty measure. The conclusion overstates the applicability of a percentile-based label from a corpus of proficient readers.","section":"Section 5, Conclusion"}],"minor_comments":[{"comment":"The sentence 'These labels were recorded in a new column named Reading_Difficulties [10]' cites Raatikainen et al. for the labeling method, but that work uses reading-fluency-based identification of dyslexic individuals, not a 95th-percentile threshold on eye-tracking metrics; the citation does not support the described labeling procedure.","section":"Section 3.3, reference [10]"},{"comment":"The feature list includes 'Ia_First_Run_Fixation_.' which appears to be a truncated or corrupted column name; the intended metric should be stated precisely.","section":"Section 3.3, 'Enhancing Dyslexia Features'"},{"comment":"The text refers to 'Section 4a' when describing the basic dyslexia features; this should be 'Section 3.3(a)' or the appropriate section heading.","section":"Section 3.2, 'Feature transformation'"},{"comment":"The paper states that the dataset was balanced by resampling the majority class but does not specify whether balancing was performed before or after the cross-validation split, nor the resulting resampling ratio. If balancing is done on the full dataset before splitting, it introduces a form of leakage; the procedure should be described precisely.","section":"Section 3.3, 'Enhancing Dyslexia Features' and Section 3.4(a)"},{"comment":"Figure 1(b) is described as displaying 'the average ROC curve,' but the text also references ROC curves for each fold; the figure caption should clarify whether per-fold curves or only the average curve is shown.","section":"Section 4.1, Figure 1"}],"recommendation":"reject","confidential_remarks":"The manuscript's central claim is undermined by a fundamental circularity: the target label is constructed from the same eye-tracking measurement space that the classifier uses as input, and the dataset contains no clinically diagnosed dyslexic participants. This is not a matter of presentation or incremental revision; the study design cannot support the stated conclusions without a new dataset with validated labels. The abstract and conclusion also overstate the findings relative to what the analysis demonstrates. If the authors can obtain a clinically annotated eye-tracking dataset and rerun the pipeline with participant-level validation, a substantially revised manuscript might be suitable for a future submission."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The punchline is simple: the 88.58% accuracy is for predicting a synthetic percentile label, not for detecting dyslexia in anyone. The stress-test note is accurate on every load-bearing point.\n\nWhat the paper does well is modest but real. It uses a public corpus (Provo), documents the preprocessing pipeline, and applies a standard Random Forest with cross-validation, feature selection, and Bayesian hyperparameter tuning. The related work section gives a fair tour of the existing eye-tracking-and-ML dyslexia literature. None of that is enough to carry the central claim.\n\nThe soft spot is structural, not cosmetic. In Section 3.3, the target variable Reading_Difficulties is defined by flagging entries above the 95th percentile of metrics like Ia_Dwell_Time and Ia_Fixation_Count. The Provo Corpus contains 84 proficient readers; no one has a clinical dyslexia diagnosis. So the classifier learns to predict which word-level rows fall in the extreme tail of the same eye-movement feature distribution. Excluding those three features from the model does not remove the leakage, because the remaining predictors—regression counts, fixation indices, saccade durations, skip rates—are all measures of the same fixation-saccade process and are strongly correlated with the label-generating features. The 0.96 AUC is a measure of how recoverable that percentile threshold is, not how detectable dyslexia is. The clustering section does not rescue this: labeling Cluster 2 as 'potentially including individuals with dyslexia' is a post-hoc interpretation with no external diagnostic ground truth.\n\nA courtesy note: the paper is transparent about the threshold construction, so the circularity is visible rather than hidden. That does not make the claim valid, but it means this is an error in study design, not an attempt to deceive.\n\nAs it stands, the paper is not a serious contribution to dyslexia detection. It would need a clinically validated outcome variable, ideally with a sample that includes confirmed dyslexic and control participants beyond a normal adult corpus like Provo. A reader looking for an example of a standard ML pipeline applied to eye-tracking data might find it mildly useful, but the paper's framing and conclusion overstate what was established. I would not cite it as evidence about dyslexia, and I would not send it to peer review in its current form; a serious editor or referee would reject it on the same grounds.","headline":"The 88.58% accuracy is for predicting a self-constructed 95th-percentile label from the same eye-tracking feature space, not for clinically confirmed dyslexia; the paper's central claim does not survive reading.","tokens_in":8402,"tokens_out":2883,"would_cite":false,"duration_ms":25549,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proposes that eye-tracking combined with a Random Forest can screen for dyslexia cheaply and non-invasively, reporting 88.58% accuracy and 0.96 AUC.","keywords":["dyslexia detection","eye tracking","random forest","hierarchical clustering","reading difficulty","feature engineering","Provo Corpus","machine learning"],"falsifier":"Run the same feature pipeline on an eye-tracking dataset where each participant has a clinician-administered dyslexia diagnosis, and measure the classifier's accuracy against those ground-truth labels; if the accuracy is near chance or far below the reported 88.58%, the central screening claim fails.","tokens_in":7349,"feed_emoji":"👁","tokens_out":8985,"duration_ms":73058,"temperature":0.7,"pith_summary":"The study sets out to show that eye-tracking data plus machine learning can serve as a cost-effective, non-invasive early dyslexia screener. Using the Provo Corpus, the authors engineer features such as saccade duration and combine them with measures of fixations, dwell time, and regressions. They label reading difficulty through a 95th-percentile threshold on selected eye-tracking metrics and train a Random Forest that reports 88.58% accuracy, 87.91% precision, 89.49% recall, and 0.96 average ROC AUC in 9-fold cross-validation. Hierarchical clustering on the same features yields three groups that the authors interpret as a reading-ability spectrum from fluent readers to a high-effort group that likely includes dyslexic traits.","feed_headline":"Eye tracking plus machine learning scores 88.58% on dyslexia screening","feed_subtitle":"Non-invasive eye-tracking plus a Random Forest reports 88.58% on dyslexia labels.","key_machinery":"The mechanism is a binary Reading_Difficulties label constructed by flagging eye-tracking entries above the 95th percentile for metrics such as Ia_Dwell_Time and Ia_Fixation_Count, with the Random Forest classifier trained on a balanced, resampled dataset using Bayesian hyperparameter tuning and forward feature selection. For cluster analysis, PCA reduces the enhanced feature set to two components and agglomerative hierarchical clustering with Euclidean distance and Ward linkage groups readers. The governing assumption is that extreme eye-movement values encode reading difficulty, so thresholding plus an ensemble classifier can detect it.","core_discovery":"The authors claim that a small set of eye-tracking features—dwell time, fixation count, regression counts, first-saccade amplitude, and a newly engineered Saccade_Duration—captures enough of the reading-difficulty signal for automatic screening. Entries above the 95th percentile on metrics such as Ia_Dwell_Time and Ia_Fixation_Count are labeled as reading difficulties, and a Random Forest trained on a balanced, feature-selected version of the Provo Corpus distinguishes those labels with 88.58% accuracy and 0.96 ROC AUC under 9-fold cross-validation. Agglomerative hierarchical clustering then splits readers into three groups, with the high-dwell-time, high-fixation, high-saccade-amplitude cluster interpreted as the poor/dyslexic end of a reading-behavior spectrum. The paper concludes that eye-tracking plus machine learning is a non-invasive, objective, and cost-effective complement to traditional dyslexia assessment.","pith_inferences":["Read strictly, the reported 88.58% accuracy measures prediction of a threshold-derived label rather than a clinical diagnosis; the paper's proposal would need external validation against clinician-administered assessments before screening use.","Because the label-defining features are withheld from the classifier, the strong AUC partly reflects how well correlated eye-tracking features reproduce the same extreme scores; this is an internal-consistency result, not evidence about clinical validity.","A testable extension would vary the 95th-percentile cutoff and report accuracy as a function of the threshold, or apply the pipeline to a dataset with reading-fluency scores, to see whether the signal is stable.","If the three clusters were compared with independent dyslexia diagnoses, cluster 2 should contain most verified dyslexic readers; that is a concrete prediction the paper leaves untested."],"forward_implications":["If the reported performance holds, schools and clinics could screen for reading difficulties with a relatively cheap eye-tracker rather than a full psychological assessment.","The top-12 feature subset suggests that a streamlined screening protocol could collect only a small set of eye-movement measurements.","The three clusters, if validated, would allow severity-based triage, with the high-effort cluster targeted for early intervention.","The method is non-invasive and objective, potentially reducing reliance on subjective teacher judgment for referral.","Combining eye-tracking with neuroimaging or behavioral batteries could give a multimodal diagnosis that improves on either alone."],"supporting_citations":[{"why":"Supplies the Provo Corpus, the 230,412-entry eye-tracking dataset from 84 readers that all feature engineering and models are built on.","marker":"[6]"},{"why":"Provides the prior machine-learning detection of developmental dyslexia from eye movements and the labeling idea behind the 95th-percentile Reading_Difficulties target.","marker":"[10]"},{"why":"Documents the eye-movement differences between dyslexic and non-dyslexic readers that motivate which fixation and saccade metrics are selected.","marker":"[4]"},{"why":"Supports the regression-count metrics used as basic dyslexia features.","marker":"[16]"},{"why":"Shows AI classification of dyslexia from eye-movement and demographic data, supporting use of fixation counts as a feature.","marker":"[14]"},{"why":"Supplies the feature-based machine-learning approach and novel gaze metrics that frame the paper's enhanced feature set.","marker":"[15]"},{"why":"Provides the Random Forest classification method and its accuracy-and-reliability baseline for ensemble classifiers.","marker":"[2]"},{"why":"Supplies the PCA dimensionality-reduction method used before hierarchical clustering.","marker":"[5]"},{"why":"Supplies the agglomerative hierarchical clustering with Euclidean distance and Ward linkage used for reader subgroups.","marker":"[8]"}],"fun_headline_variants":["Eye-tracking and ML hit 88.58% accuracy for dyslexia screening","Random Forest + eye tracking: 88.58% dyslexia detection","88.58% dyslexia detection via eye tracking and ML","Non-invasive dyslexia screening: 88.58% with eye tracking"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a 95th-percentile threshold on metrics like dwell time and fixation count is a valid label for reading difficulty or dyslexia; if those labels do not track actual dyslexia, the reported accuracy only measures prediction of the synthetic percentiles.","fun_headline_variants_meta":{"raw":{"variants":["Eye-tracking and ML hit 88.58% accuracy for dyslexia screening","Random Forest + eye tracking: 88.58% dyslexia detection","88.58% dyslexia detection via eye tracking and ML","Non-invasive dyslexia screening: 88.58% with eye tracking"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000158,"raw_usage":{"total_tokens":1199,"prompt_tokens":894,"completion_tokens":305,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":510,"completion_tokens_details":{"reasoning_tokens":230}},"tokens_in":510,"tokens_out":305,"duration_ms":2942,"temperature":1.0,"reasoning_tokens":230,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:35:12.596123+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same feature pipeline on an eye-tracking dataset where each participant has a clinician-administered dyslexia diagnosis, and measure the classifier's accuracy against those ground-truth labels; if the accuracy is near chance or far below the reported 88.58%, the central screening claim fails.","supporting_citations":[{"cited_title":"Scientific Reports11(1) (2021).https://doi.org/ 10.1038/s41598-021-84945-9","cited_arxiv_id":null,"evidence_quote":"Documents the eye-movement differences between dyslexic and non-dyslexic readers that motivate which fixation and saccade metrics are selected."},{"cited_title":"Annals of Dyslexia65, 24–32 (2015)","cited_arxiv_id":null,"evidence_quote":"Supports the regression-count metrics used as basic dyslexia features."},{"cited_title":"PLOS ONE (2023)","cited_arxiv_id":null,"evidence_quote":"Shows AI classification of dyslexia from eye-movement and demographic data, supporting use of fixation counts as a feature."},{"cited_title":"Journal of Eye Movement Research 14(2), 10.16910/jemr.14.2.1 (2021)","cited_arxiv_id":null,"evidence_quote":"Supplies the feature-based machine-learning approach and novel gaze metrics that frame the paper's enhanced feature set."},{"cited_title":"Journal of Building Engineering 48, 103971 (2022)","cited_arxiv_id":null,"evidence_quote":"Supplies the PCA dimensionality-reduction method used before hierarchical clustering."},{"cited_title":"Information Retrieval Journal 24(2), 137–173 (2021).https://doi.org/10.1007/s10791-020-09384-y","cited_arxiv_id":null,"evidence_quote":"Supplies the agglomerative hierarchical clustering with Euclidean distance and Ward linkage used for reader subgroups."}],"review_version":1}