Pith. sign in

REVIEW 4 major objections 5 minor 21 references

Developing a Dyslexia Indicator Using Eye Tracking

T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Pith's one-line read This paper proposes that eye-tracking combined with a Random Forest can screen for dyslexia cheaply and non-invasively, reporting 88.58% accuracy and 0.96 AUC.

desk verdict The 88.58% accuracy is for predicting a self-constructed 95th-percentile label from the same eye-tracking feature space, not for clinically confirmed dyslexia; the paper's central claim does not survive reading. read the letter →

arxiv 2506.11004 v1 pith:25TZSEFN submitted 2025-04-21 cs.LG cs.AIcs.CLcs.CVcs.HC

classification cs.LGcs.AIcs.CLcs.CVcs.HC
keywords dyslexiadetectioneyetrackingrandomforesthierarchicalclusteringreadingdifficultyfeatureengineeringProvoCorpusmachinelearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The study sets out to show that eye-tracking data plus machine learning can serve as a cost-effective, non-invasive early dyslexia screener. Using the Provo Corpus, the authors engineer features such as saccade duration and combine them with measures of fixations, dwell time, and regressions. They label reading difficulty through a 95th-percentile threshold on selected eye-tracking metrics and train a Random Forest that reports 88.58% accuracy, 87.91% precision, 89.49% recall, and 0.96 average ROC AUC in 9-fold cross-validation. Hierarchical clustering on the same features yields three groups that the authors interpret as a reading-ability spectrum from fluent readers to a high-effort group that likely includes dyslexic traits.

What carries the argument

The mechanism is a binary Reading_Difficulties label constructed by flagging eye-tracking entries above the 95th percentile for metrics such as Ia_Dwell_Time and Ia_Fixation_Count, with the Random Forest classifier trained on a balanced, resampled dataset using Bayesian hyperparameter tuning and forward feature selection. For cluster analysis, PCA reduces the enhanced feature set to two components and agglomerative hierarchical clustering with Euclidean distance and Ward linkage groups readers. The governing assumption is that extreme eye-movement values encode reading difficulty, so thresholding plus an ensemble classifier can detect it.

What would settle it

Run the same feature pipeline on an eye-tracking dataset where each participant has a clinician-administered dyslexia diagnosis, and measure the classifier's accuracy against those ground-truth labels; if the accuracy is near chance or far below the reported 88.58%, the central screening claim fails.

Watch

Extended reading notes

Core claim

The authors claim that a small set of eye-tracking features—dwell time, fixation count, regression counts, first-saccade amplitude, and a newly engineered Saccade_Duration—captures enough of the reading-difficulty signal for automatic screening. Entries above the 95th percentile on metrics such as Ia_Dwell_Time and Ia_Fixation_Count are labeled as reading difficulties, and a Random Forest trained on a balanced, feature-selected version of the Provo Corpus distinguishes those labels with 88.58% accuracy and 0.96 ROC AUC under 9-fold cross-validation. Agglomerative hierarchical clustering then splits readers into three groups, with the high-dwell-time, high-fixation, high-saccade-amplitude cluster interpreted as the poor/dyslexic end of a reading-behavior spectrum. The paper concludes that eye-tracking plus machine learning is a non-invasive, objective, and cost-effective complement to traditional dyslexia assessment.

Load-bearing premise

The load-bearing premise is that a 95th-percentile threshold on metrics like dwell time and fixation count is a valid label for reading difficulty or dyslexia; if those labels do not track actual dyslexia, the reported accuracy only measures prediction of the synthetic percentiles.

Editorial extensions

If this is right

  • If the reported performance holds, schools and clinics could screen for reading difficulties with a relatively cheap eye-tracker rather than a full psychological assessment.
  • The top-12 feature subset suggests that a streamlined screening protocol could collect only a small set of eye-movement measurements.
  • The three clusters, if validated, would allow severity-based triage, with the high-effort cluster targeted for early intervention.
  • The method is non-invasive and objective, potentially reducing reliance on subjective teacher judgment for referral.
  • Combining eye-tracking with neuroimaging or behavioral batteries could give a multimodal diagnosis that improves on either alone.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Read strictly, the reported 88.58% accuracy measures prediction of a threshold-derived label rather than a clinical diagnosis; the paper's proposal would need external validation against clinician-administered assessments before screening use.
  • Because the label-defining features are withheld from the classifier, the strong AUC partly reflects how well correlated eye-tracking features reproduce the same extreme scores; this is an internal-consistency result, not evidence about clinical validity.
  • A testable extension would vary the 95th-percentile cutoff and report accuracy as a function of the threshold, or apply the pipeline to a dataset with reading-fluency scores, to see whether the signal is stable.
  • If the three clusters were compared with independent dyslexia diagnoses, cluster 2 should contain most verified dyslexic readers; that is a concrete prediction the paper leaves untested.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This manuscript proposes an eye-tracking and machine-learning pipeline for dyslexia detection. The authors use the Provo Corpus, a large eye-tracking dataset of 84 proficient readers, and construct a binary target variable called Reading_Difficulties by thresholding eye-movement metrics at the 95th percentile. A Random Forest classifier, trained on the remaining eye-tracking features, is reported to achieve 88.58% accuracy and an ROC AUC of 0.96 under 9-fold cross-validation. Agglomerative hierarchical clustering is applied to the same feature space, and the resulting clusters are interpreted as representing a spectrum of reading ability from proficient to dyslexic. The paper concludes that the method is a cost-effective, non-invasive diagnostic tool for early dyslexia detection.

Significance. If the reported accuracy reflected detection of clinically confirmed dyslexia, the result would be a meaningful contribution to accessible dyslexia screening. The paper also has some positive aspects: it clearly describes a data preprocessing and feature-engineering pipeline, makes an explicit attempt to avoid direct data leakage by excluding the label-generating features from the classifier, and reports cross-validated performance metrics. However, the central claim is undermined by the construction of the target variable: Reading_Difficulties is a synthetic percentile-based label derived from the very same type of eye-tracking measurements that constitute the input features, and the Provo Corpus contains no participants with a clinical dyslexia diagnosis. Consequently, the reported accuracy measures how well the model recovers a 95th-percentile thresholding artifact, not how well it detects dyslexia. The absence of any external or clinical validation makes the diagnostic claim unsupported by the presented evidence.

major comments (4)
  1. [Section 3.3, 'Thresholds for labeling'] The target variable Reading_Difficulties is defined by assigning a label of 1 to any word-level entry whose value on metrics such as Ia_Dwell_Time, Ia_Fixation_Count, or Ia_First_Saccade_Amplitude exceeds the 95th percentile, with all other entries labeled 0. Since these are eye-tracking measures, and the classifier in Section 4.1 is trained on other eye-tracking features from the same reading process, the model is effectively predicting a thresholding artifact of the same measurement modality. The reported 88.58% accuracy and 0.96 AUC therefore do not support the abstract's claim of 'detect[ing] dyslexia'; they support only the conclusion that percentile-based extremeness of one eye-tracking feature is predictable from related eye-tracking features. This circularity is load-bearing because the paper's central claim depends on Reading_Difficulties being a valid proxy for dyslexia, which is not established.
  2. [Section 4.1, Table 1 and Figure 1] The evaluation uses 9-fold cross-validation at the row level over 230,412 word-level entries originating from only 84 participants. Because each participant contributes many rows, random row-level splitting will place the same participant in both the training and test folds, allowing the model to memorize subject-specific eye-movement patterns and inflating the reported accuracy. A participant-level or text-level cross-validation scheme is required to support any claim about detecting reading difficulties in new readers, but no such analysis is reported.
  3. [Section 4.2, 'Cluster and Analysis'] Cluster 2 is characterized as 'poor readers, potentially including individuals with dyslexia' and the clustering results are presented as evidence of 'varying severity levels of dyslexia.' However, no diagnostic information, reading-fluency score, or other external ground truth is available to validate these cluster interpretations. The labels assigned to clusters are post-hoc descriptions of the eye-tracking metrics, not validated subgroups, so the clustering analysis does not provide evidence for dyslexia severity.
  4. [Section 5, Conclusion] The conclusion states that the method is 'a non-invasive, objective, and cost-effective diagnostic tool' and 'supports early identification of students needing intervention.' These claims are not supported by the analysis, which never compares the method against a clinical diagnosis or any established reading-difficulty measure. The conclusion overstates the applicability of a percentile-based label from a corpus of proficient readers.
minor comments (5)
  1. [Section 3.3, reference [10]] The sentence 'These labels were recorded in a new column named Reading_Difficulties [10]' cites Raatikainen et al. for the labeling method, but that work uses reading-fluency-based identification of dyslexic individuals, not a 95th-percentile threshold on eye-tracking metrics; the citation does not support the described labeling procedure.
  2. [Section 3.3, 'Enhancing Dyslexia Features'] The feature list includes 'Ia_First_Run_Fixation_.' which appears to be a truncated or corrupted column name; the intended metric should be stated precisely.
  3. [Section 3.2, 'Feature transformation'] The text refers to 'Section 4a' when describing the basic dyslexia features; this should be 'Section 3.3(a)' or the appropriate section heading.
  4. [Section 3.3, 'Enhancing Dyslexia Features' and Section 3.4(a)] The paper states that the dataset was balanced by resampling the majority class but does not specify whether balancing was performed before or after the cross-validation split, nor the resulting resampling ratio. If balancing is done on the full dataset before splitting, it introduces a form of leakage; the procedure should be described precisely.
  5. [Section 4.1, Figure 1] Figure 1(b) is described as displaying 'the average ROC curve,' but the text also references ROC curves for each fold; the figure caption should clarify whether per-fold curves or only the average curve is shown.

Circularity Check

2 steps flagged · score 8.0 of 10

The 88.58% accuracy is for predicting a 95th-percentile label constructed from the same eye-tracking features, so the 'dyslexia detection' claim reduces by definition to recovering a threshold artifact.

  1. self definitional [Section 3.3, 'Thresholds for labeling' (page 6)]
    "The 95th percentile was used as the threshold for each metric to differentiate between levels of reading difficulty. Any data entry above this threshold was assigned a label of 1, indicating a higher level of reading difficulty, while entries below were labeled 0, indicating lower difficulty. These labels were recorded in a new column named Reading_Difficulties [10]."

    The target variable Reading_Difficulties is not a clinical dyslexia diagnosis; it is a binarized label obtained by thresholding the very eye-tracking metrics (Ia_Dwell_Time, Ia_Fixation_Count, etc.) that form the model's input space. In Section 4.1 the Random Forest is evaluated against this label, so the reported 88.58% accuracy and 0.96 AUC measure how well the 95th-percentile extremity of a row can be recovered from correlated eye-movement features, not how well dyslexia can be detected. Since the Provo Corpus (Section 3.1) is described as data from proficient readers and no clinical dyslexia outcome is reported, the stated conclusion that the classifier 'detect[s] dyslexia' is equivalent, by construction, to predicting a percentile cutpoint on the predictor variables.

  2. fitted input called prediction [Section 3.3, 'Enhancing Dyslexia Features' (page 6) and Section 4.1]
    "Features directly contributing to the construction of the Reading_Difficulty target, such as Ia_First_Saccade_Amplitude, Ia_Dwell_Time, and Ia_Fixation_Count, were excluded from the analysis to avoid data leakage and ensure an unbiased assessment of feature importance."

    This passage concedes that the label is constructed from the predictors and attempts to repair the leak by dropping named features from the importance analysis. However, the features retained for the final classifier (Ia_Regression_In_Count, Saccade_Duration, Ia_Skip, fixation indices) measure the same fixation-saccade process and are highly correlated with the thresholded metrics. Thus the 'prediction' remains forced: the model is fitting a percentile-threshold artifact of its own input distribution, and no independent dyslexia outcome is ever introduced. The exclusion of a few named columns does not break the definitional link between the synthetic label and the measurement space.

full rationale

The central derivation is circular by construction. In Section 3.3, the single outcome variable used for all supervised learning, Reading_Difficulties, is created by applying a 95th-percentile threshold to eye-tracking columns such as Ia_Dwell_Time and Ia_Fixation_Count. The same eye-tracking measurement space then supplies the predictor features for the Random Forest in Sections 3.4 and 4.1. The 88.58% accuracy and 0.96 AUC therefore describe how well the classifier recovers a synthetic percentile cutpoint from correlated eye-movement features; they do not describe detection of clinically confirmed dyslexia. The paper does not report a clinical dyslexia label in the Provo Corpus (Section 3.1 describes 84 native English-speaking readers) and does not validate the percentile proxy against any external diagnostic benchmark. Excluding the named direct contributors does not break the link, because the retained features (regression counts, fixation indices, saccade durations, skip rates) measure the same reading process. Section 4.2's assignment of 'poor readers, potentially including individuals with dyslexia' to Cluster 2 is a post-hoc interpretation of a clustering over these same features, not an independent confirmation. No load-bearing self-citation chain is involved; the circularity is definitional, so the score is 8 rather than 6 or 10.

Assumptions & free parameters 5 free parameters · 3 assumptions · 1 invented entities

The classification pipeline depends on the assumption that 95th percentile eye-tracking values correspond to reading difficulty or dyslexia, and on the Provo Corpus being a suitable population. No clinical ground truth is used anywhere. The free parameters are the percentile threshold, the number of selected features, and the hyperparameters of the classifier, none of which are externally validated.

free parameters (5)
  • 95th percentile threshold per metric = 95th percentile (per metric)
    Used to construct the binary Reading_Difficulties label from continuous eye-tracking metrics (Section 3.3); chosen without clinical validation.
  • Number of selected features = 12
    Forward selection chose top 12 features (Section 3.4a); no stability analysis or error bars.
  • RF hyperparameters via Bayesian optimization = not reported
    BayesSearchCV tuning is mentioned but the resulting hyperparameters are not given (Section 3.4a).
  • Class balancing resampling ratio = not reported
    Majority class resampled to balance the dataset; ratio not specified (Section 3.3b).
  • Number of clusters interpreted = 3
    The authors interpret three clusters from the dendrogram (Section 4.2), though AHC does not require a preset count.
assumptions (3)
  • ad hoc to paper Eye-tracking metrics above the 95th percentile indicate reading difficulty or dyslexia
    Introduced in Section 3.3 'Thresholds for labeling'; no reference to a clinically validated cutoff.
  • domain assumption The Provo Corpus readers form a valid population for dyslexia screening
    The corpus contains proficient readers, not a clinical sample; used throughout without justification.
  • standard math Standard ML assumptions: train/test split via 9-fold CV and balanced classes generalize to the target population
    Implicit in Section 4.1; standard but insufficient for a clinical diagnostic claim.
invented entities (1)
  • Reading_Difficulties label
    purpose: Serves as the classification target and is interpreted as a dyslexia indicator.
    Created by thresholding the corpus's eye-tracking features at the 95th percentile (Section 3.3); no external diagnostic validation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Developing a Dyslexia Indicator Using Eye Tracking." pith.science (2026). https://pith.science/paper/25TZSEFN

@misc{pith2026250611004,
  author       = {Pith},
  title        = {Pith review of: Developing a Dyslexia Indicator Using Eye Tracking},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/25TZSEFN}},
  note         = {Machine review of arXiv:2506.11004}
}
read the original abstract

Dyslexia, affecting an estimated 10% to 20% of the global population, significantly impairs learning capabilities, highlighting the need for innovative and accessible diagnostic methods. This paper investigates the effectiveness of eye-tracking technology combined with machine learning algorithms as a cost-effective alternative for early dyslexia detection. By analyzing general eye movement patterns, including prolonged fixation durations and erratic saccades, we proposed an enhanced solution for determining eye-tracking-based dyslexia features. A Random Forest Classifier was then employed to detect dyslexia, achieving an accuracy of 88.58\%. Additionally, hierarchical clustering methods were applied to identify varying severity levels of dyslexia. The analysis incorporates diverse methodologies across various populations and settings, demonstrating the potential of this technology to identify individuals with dyslexia, including those with borderline traits, through non-invasive means. Integrating eye-tracking with machine learning represents a significant advancement in the diagnostic process, offering a highly accurate and accessible method in clinical research.

Figures

Figures reproduced from arXiv: 2506.11004 by the authors.

Figure 1
Figure 1. (b) displays the average Area Under the Curve (AUC) for the ROC curve, which is 0.96, indicating an excellent level of discrimination between the positive and negative classes. The consistency of the ROC curves across the nine folds suggests that the model’s performance is robust and reliable across different data subsets. This high and consistent AUC value highlights the model’s suit￾ability for practical applicati… view at source ↗
Figure 2
Figure 2. Cluster Analysis on Our Enhancing Dyslexia Features 5 Conclusion and Future Work The paper highlights the use of eye-tracking and machine learning to effec￾tively detect dyslexia, leveraging metrics like saccade amplitude, dwell time, and fixation count. Operating at the 95th percentile, the method offers a non￾invasive, objective, and cost-effective diagnostic tool, reducing reliance on sub￾jective teacher judgment… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

21 extracted references · 16 canonical work pages

  1. [1]

    NeuroImage 53(2), 682–693 (2010)

    Bach, S., Brandeis, D., Hofstetter, C., Martin, E., Richardson, U., Brem, S.: Early emergence of deviant frontal fmri activity for phonological processes in poor be- ginning readers. NeuroImage 53(2), 682–693 (2010). https://doi.org/10.1016/ j.neuroimage.2010.06.039

  2. [2]

    Environmental Challenges 14, 100800 (2024)

    Chowdhury,M.S.:Comparisonofaccuracyandreliabilityofrandomforest,support vector machine, artificial neural network and maximum likelihood method in land use/cover classification of urban setting. Environmental Challenges 14, 100800 (2024). https://doi.org/https://doi.org/10.1016/j.envc.2023.100800

  3. [3]

    Ireland Government (2021)

    Department of Education: Guidelines for Primary Schools Supporting Pupils with Special Educational Needs in Mainstream Schools. Ireland Government (2021)

  4. [4]

    Scientific Reports11(1) (2021).https://doi.org/ 10.1038/s41598-021-84945-9

    Franzen, L., Stark, Z., Johnson, A.: Individuals with dyslexia use a different visual sampling strategy to read text. Scientific Reports11(1) (2021).https://doi.org/ 10.1038/s41598-021-84945-9

  5. [5]

    Journal of Building Engineering 48, 103971 (2022)

    Kamari, A., Peter Leslie Schultz, C.: A combined principal component analysis and clustering approach for exploring enormous renovation design spaces. Journal of Building Engineering 48, 103971 (2022). https://doi.org/https://doi.org/ 10.1016/j.jobe.2021.103971

  6. [6]

    Behavior Research Methods50(2), 826–833 (2017)

    Luke, S., Christianson, K.: The provo corpus: A large eye-tracking corpus with predictability norms. Behavior Research Methods50(2), 826–833 (2017). https: //doi.org/10.3758/s13428-017-0908-4

  7. [7]

    Scientific Reports11(1) (2021)

    Nerušil, B., Polec, J., Škunda, J., Kačur, J.: Eye tracking based dyslexia detection using a holistic approach. Scientific Reports11(1) (2021). https://doi.org/10. 1038/s41598-021-95275-1 Developing a Dyslexia Indicator Using Eye Tracking 11

  8. [8]

    Information Retrieval Journal 24(2), 137–173 (2021).https://doi.org/10.1007/s10791-020-09384-y

    Ngo, V.M., Helmer, S., Le-Khac, N.A., Kechadi, M.T.: Structural textile pattern recognition and processing based on hypergraphs. Information Retrieval Journal 24(2), 137–173 (2021).https://doi.org/10.1007/s10791-020-09384-y

Show all 21 references
  1. [9]

    Poornappriya, T., Gopinath, R.: Application of machine learning techniques for improving learning disabilities (2020).https://doi.org/10.17605/OSF.IO/39ZQD

  2. [10]

    Array12, 100087–100087 (2021)

    Raatikainen, P., Hautala, J., Loberg, O., Kärkkäinen, T., Leppänen, P., Niem- inen, P.: Detection of developmental dyslexia with machine learning using eye movement data. Array12, 100087–100087 (2021). https://doi.org/10.1016/j. array.2021.100087

  3. [11]

    Wiley-Blackwell (2016)

    Reid, G.: Dyslexia: A Practitioner’s Handbook, 5th Edition. Wiley-Blackwell (2016)

  4. [12]

    Journal of Neurolinguistics 21(4), 294–304 (2008).https://doi.org/10.1016/j.jneuroling.2007.07.002

    Richards, T., Berninger, V.: Abnormal fmri connectivity in children with dyslexia during a phoneme task: Before but not after treatment. Journal of Neurolinguistics 21(4), 294–304 (2008).https://doi.org/10.1016/j.jneuroling.2007.07.002

  5. [13]

    The Computer Journal 62(3), 394–413 (06 2018)

    Scriney,M.,McCarthy,S.,McCarren,A.,Cappellari,P.,Roantree,M.:Automating data mart construction from semi-structured data sources. The Computer Journal 62(3), 394–413 (06 2018). https://doi.org/10.1093/comjnl/bxy064, https:// doi.org/10.1093/comjnl/bxy064

  6. [14]

    PLOS ONE (2023)

    Shalileh, S., Ignatov, D., Lopukhina, A., Dragoy, O.: Identifying dyslexia in school pupils from eye movement and demographic data using artificial intelligence. PLOS ONE (2023). https://doi.org/10.1371/journal.pone.0292047

  7. [15]

    Journal of Eye Movement Research 14(2), 10.16910/jemr.14.2.1 (2021)

    Smyrnakis, I., Andreadakis, V., Rina, A., Boufachrentin, N., Aslanides, I.: Silent versus reading out loud modes: An eye-tracking study. Journal of Eye Movement Research 14(2), 10.16910/jemr.14.2.1 (2021). https://doi.org/10.16910/jemr. 14.2.1

  8. [16]

    Annals of Dyslexia65, 24–32 (2015)

    Vagge, A., Cavanna, M., Traverso, C., et al.: Evaluation of ocular movements in patients with dyslexia. Annals of Dyslexia65, 24–32 (2015). https://doi.org/ 10.1007/s11881-015-0098-7

  9. [17]

    In: 2022 30th European Signal Processing Conference (EUSIPCO) (2022)

    Vajs, I., Ković, V., Papić, T., Savić, A., Janković, M.: Dyslexia detection in chil- dren using eye tracking data based on vgg16 network. In: 2022 30th European Signal Processing Conference (EUSIPCO) (2022). https://doi.org/10.23919/ eusipco55093.2022.9909817

  10. [18]

    Vajs, I., Ković, V., Papić, T., Savić, A., Janković, M.: Spatiotemporal eye-tracking featuresetforimprovedrecognitionofdyslexicreadingpatternsinchildren.Sensors 22(13), 4900–4900 (2022).https://doi.org/10.3390/s22134900

  11. [19]

    Vajs, I., Savić, A., Ković, V., Janković, M., Papić, T.: Dyslexia detection in chil- dren using eye tracking data based on vgg16 network (nd),https://eurasip.org/ Proceedings/Eusipco/Eusipco2022/pdfs/0001601.pdf

  12. [20]

    Scientific Reports 11(1) (2021)

    Vehlen, A., Spenthof, I., Tönsing, D., Heinrichs, M., Domes, G.: Evaluation of an eye tracking setup for studying visual attention in face-to-face conversations. Scientific Reports 11(1) (2021). https://doi.org/10.1038/s41598-021-81987-x

  13. [21]

    Frontiers in Public Health 10 (2022)

    Wu, Y., Cheng, Y., Yang, X., Yu, W., Wan, Y.: Dyslexia: A bibliometric and visualization analysis. Frontiers in Public Health 10 (2022). https://doi.org/ 10.3389/fpubh.2022.915053

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.