Pith. sign in

REVIEW 3 major objections 6 minor 27 references

High-frequency near-eye ground truth for event-based eye tracking

T0 review · 3 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash

Pith's one-line read The paper claims a semi-automatic pipeline can add 200Hz pupil-center annotations to the Angelopoulos event-based eye-tracking dataset, along with blink and saccade labels.

desk verdict The paper's own Table 1 sinks the headline claim: only 7% of frames carry pupil-center labels, so this is not continuous 200 Hz pupil ground truth, but the saccade/blink annotations may still be a useful artifact. read the letter →

arxiv 2502.03057 v1 pith:XY7FPMOS submitted 2025-02-05 cs.CV

classification cs.CV
keywords smarteyeweareyetrackingevent-basedvisionpupildetectiondatasetannotationsaccadeblinksemi-automatic
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper addresses the scarcity of annotated event-based eye-tracking data by adding high-frequency ground truth to the existing Angelopoulos dataset. The authors present a semi-automatic pipeline that first generates frames from the event stream at 200Hz, then automatically detects saccades, locates pupil centers via template matching and RANSAC ellipse fitting, and finally has human annotators review and correct the labels. The released annotations provide pupil center positions at 200Hz, along with blink and saccade event states, for six users of the original dataset. If the labeling is reliable, it gives the community a denser supervision signal for training and evaluating event-based eye trackers, particularly for fast eye movements.

What carries the argument

The pipeline has three automatic stages followed by a human validation stage. First, events are accumulated into RGB frames at 200Hz (5ms windows). Second, an eye-movement detector flags a saccade when a frame contains more than 150 events, marking saccade start and end. Third, during saccades, the pupil center is found by convolving the polarity frame with 8 directional templates, picking the highest-scoring template, and then running RANSAC with 1000 iterations to fit an ellipse to events inside the resulting region of interest. Human annotators then correct any mislabels, and an iterative quality-check step plots inter-frame pupil displacement to catch anomalies.

What would settle it

Re-annotate a random sample of frames from each of the six users with multiple independent human annotators and compute inter-annotator agreement, or compare the released labels against a high-speed camera-based ground truth; if the agreement is poor or the labels deviate systematically, the claim that these annotations constitute valid ground truth at 200Hz would be refuted.

Watch

Extended reading notes

Core claim

On the paper's terms, the central claim is that a high-frequency (200Hz) pupil-center annotation, together with blink and saccade labels, can be produced for the Angelopoulos event-based eye-tracking dataset using a semi-automatic pipeline that combines automatic detection with human correction. The paper reports that 114,222 frames were reviewed and annotated, yielding 22,305 distinct eye center positions, 2,101 saccades, and 120 blinks across six users. This density of labels, the authors argue, is sufficient to capture all documented eye movements, including microsaccades, and improves on the original dataset's screen-space gaze labels.

Load-bearing premise

The released labels are only as good as the human annotators' corrections, but the paper provides no measure of agreement between annotators or against an external gold standard to confirm those corrections are accurate.

Editorial extensions

If this is right

  • Researchers can train and evaluate event-based pupil detectors against a 200Hz supervision signal rather than the original screen-space gaze labels.
  • The released blink and saccade annotations enable supervised learning for eye-state classification, not just pupil localization.
  • The pipeline itself can be reapplied to other event-based eye datasets or future recordings to produce dense labels.
  • Because 200Hz exceeds the documented duration of microsaccades (10-30ms), the ground truth should capture transient eye movements that slower annotations miss.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not quantify agreement between the automatic annotations and the human-corrected labels, so the actual accuracy gain over fully automatic methods is unknown; a natural next step would be to report such statistics.
  • If the human review step is genuinely reliable, the same semi-automatic approach could be used to leverage larger unlabeled event-camera recordings in a human-in-the-loop labeling effort.
  • A testable extension would be to benchmark downstream eye-tracking models trained on these 200Hz labels versus the original dataset's labels, to see whether the denser supervision improves saccade or blink detection in practice.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This paper presents a semi-automatic annotation pipeline for event-based eye-tracking data. The authors accumulate events from the Angelopoulos dataset into 5 ms frames, detect saccades with a threshold, match motion templates, and fit an ellipse with RANSAC to obtain pupil-center candidates; human annotators then review and correct the automatic labels. The pipeline is applied to the left eye of six users, and the released annotations include pupil-center positions, saccade states, and blink states. The central claim is that these annotations constitute high-frequency ground truth for pupil detection at 200 Hz.

Significance. Pupil-level annotations for event-based eye tracking are scarce, and a public, carefully documented dataset would be a valuable resource for training and validation. The paper offers a public release and a reproducible pipeline, which are strengths. However, as presented, the 200 Hz ground-truth claim is not supported by the reported statistics, and the annotation quality is not quantitatively validated. The actual contribution therefore depends on whether the authors can either provide dense annotations or accurately characterize the sparse coverage and demonstrate label quality.

major comments (3)
  1. [Abstract, §2, Table 1] The abstract and Section 2 state that the authors provide 'annotations for pupil detection at 200Hz' and 'retrieve the center of the pupil ... at a rate of 200 Hz', but Table 1 shows only 22,305 eye-center positions across 315,213 analyzed frames (7.1%). Since Section 3.1 computes the pupil center only when a saccade is detected and Section 3.2 excludes frames with few events, the released pupil positions are sparse and do not constitute a continuous 200 Hz ground truth. The claim should be revised to reflect the actual coverage (e.g., pupil centers only in saccade frames), and the authors should report the temporal coverage of pupil positions per user.
  2. [§3.2] The manual review stage is described as the final validation of the ground truth, but no inter-annotator agreement, comparison to an independent gold standard, or uncertainty measure is reported. Without such statistics, readers cannot assess label accuracy. The authors should add a double-annotation study on a subset (e.g., at least two annotators on a few thousand frames) and report agreement metrics such as per-coordinate error and intraclass correlation, and also compare the automatic labels against the corrected labels to quantify the automatic stage's error.
  3. [§3.1] The pipeline's behavior is governed by several ad-hoc thresholds and hyperparameters: the saccade event-count threshold of 150, the per-user event thresholds in §3.2, the number of templates (8), and the number of RANSAC iterations (1000). No sensitivity analysis is provided. Since the saccade threshold directly determines which frames receive pupil labels, a small change in this threshold could substantially alter the coverage and quality of the released annotations. The authors should justify these values empirically or provide a sensitivity analysis.
minor comments (6)
  1. [§2] In the description of the Ini-30 dataset, the phrase 'with labels dire' appears to be a typo; the intended word is likely 'direct'.
  2. [Fig. 1] The caption does not explain the color coding (green/red polarities) or specify which user and acquisition are shown; please add these details.
  3. [§3.1] The sentence 'we employ RANSAC to fit an ellipse described using the equation of a generic conic with the f parameter fixed to -1' does not give the conic equation; please specify the equation and the ellipse constraint explicitly.
  4. [§4] The text says 'We reviewed and annotated 114222 frames' and then 'the total number of frames reviewed was 315213'; please clarify the relationship between reviewed and annotated frames and use consistent terminology.
  5. [§4] The release covers only six of the 27 users in the original Angelopoulos dataset; please state this prominently and explain the selection criteria for these users.
  6. [§5] The conclusion repeats the overclaim that the annotations are 'improved annotations at 200Hz'; please align this sentence with the corrected claim from Major Comment 1.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the annotation pipeline inherits the authors' prior method but the released annotations stand as an independent dataset contribution.

full rationale

The paper's contribution is a semi-automatic annotation pipeline and a public release of pupil, saccade, and blink labels for the Angelopoulos event-based eye-tracking dataset. There is no predictive derivation whose output reduces to a fitted input. The automatic step uses template matching and RANSAC ellipse fitting "inspired by the method described in [16]" (Section 3.1), and [16] is a prior paper by overlapping authors (Mentasti and Matteucci). This is a self-citation, but it is not load-bearing: the central claim is the availability of the computed annotations, whose value can be checked directly against the raw event data and by downstream users; no uniqueness theorem or forbidden-alternative argument rests on [16]. The manual review stage (Section 3.2) treats annotator-corrected positions as ground truth without inter-annotator agreement or an external gold standard. That is a validity limitation, not circularity, because the labels are the output, not a "prediction" of an independently defined quantity. The paper itself flags sparsity: Section 3.2 excludes frames with few events, and Section 4 reports only 22,305 eye-center positions among 315,213 frames, so the advertised continuous 200Hz pupil-center ground truth is internally inconsistent. This is a correctness/consistency concern, not a circular derivation. No equation or fitted parameter is renamed as a prediction, and no load-bearing claim reduces by construction to its own inputs. Score 2 reflects the one minor, non-load-bearing self-citation to [16].

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

The central contribution is the annotations, whose quality depends on several hand-set thresholds and on human review treated as ground truth. No external validation or error analysis is provided, so the released labels carry an unquantified noise level.

free parameters (5)
  • Saccade event-count threshold = 150 events
    Hand-set threshold in Section 3.1 used to decide whether a frame contains a saccade; no sensitivity analysis.
  • Per-user review event threshold = not disclosed
    Section 3.2 says the threshold varies between users; determines which frames receive human review and final labels.
  • Anomaly displacement threshold = not disclosed
    Section 3.2: threshold used to identify label anomalies between consecutive frames; value not reported.
  • Number of motion templates = 8
    Section 3.1: only 8 templates are used to represent movement directions; choice is not justified beyond improving performance over [16].
  • RANSAC iterations = 1000
    Section 3.1: chosen to prioritize reliability; computational detail that affects fit stability.
assumptions (6)
  • domain assumption 200Hz event accumulation captures all relevant eye movements, including saccades and microsaccades.
    Section 3.1 cites [12] for duration ranges, but no experiment in this paper confirms that 5ms windows preserve enough events for stable pupil fitting.
  • ad hoc to paper A fixed event-count threshold of 150 separates saccade frames from non-saccade frames.
    Section 3.1: 'If the events in the frame are more than 150, there is a saccade.' No derivation or sensitivity analysis is given.
  • ad hoc to paper Frames with fewer than a per-user threshold of events contain no usable eye information and can be excluded from annotation.
    Section 3.2: threshold 'varies between users'; values are not disclosed, and exclusion affects which frames get labels.
  • domain assumption Human annotator corrections provide accurate ground truth.
    Section 3.2 describes manual correction but reports no inter-annotator agreement or comparison with an external reference; the quality of the ground truth is assumed.
  • domain assumption Pupil center displacement between consecutive 5ms frames is bounded by the chosen anomaly threshold.
    Section 3.2: reviewers are told to flag frames where the distance exceeds a threshold; the threshold value and physiological basis are not reported.
  • domain assumption RANSAC ellipse fitting on events inside the detected ROI yields the pupil center.
    Section 3.1: the fit follows the method in [16], but no accuracy of the fit against known pupil positions is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of High-frequency near-eye ground truth for event-based eye tracking." pith.science (2026). https://pith.science/paper/XY7FPMOS

@misc{pith2026250203057,
  author       = {Pith},
  title        = {Pith review of: High-frequency near-eye ground truth for event-based eye tracking},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XY7FPMOS}},
  note         = {Machine review of arXiv:2502.03057}
}
read the original abstract

Event-based eye tracking is a promising solution for efficient and low-power eye tracking in smart eyewear technologies. However, the novelty of event-based sensors has resulted in a limited number of available datasets, particularly those with eye-level annotations, crucial for algorithm validation and deep-learning training. This paper addresses this gap by presenting an improved version of a popular event-based eye-tracking dataset. We introduce a semi-automatic annotation pipeline specifically designed for event-based data annotation. Additionally, we provide the scientific community with the computed annotations for pupil detection at 200Hz.

Figures

Figures reproduced from arXiv: 2502.03057 by the authors.

Figure 1
Figure 1. Examples of frames generated accumulating events at 200Hz ing various eye movements. Unlike the previous datasets, 3ET+ provides the ground truth annotated at 100Hz. Moreover, they provide two different labels: one binary value to indicate the blink status, and the human-labeled pupil cen￾ter coordinates. However, the data were collected without an IR-pass filter, so they contain events of object reflections mixed w… view at source ↗
Figure 2
Figure 2. Schema of the automatic annotation pipeline. The system takes the frame generated by accumulating events as input and first predicts if there has been eye movement. If a saccade is detected, the pupil center is determined using a template matching strategy followed by RANSAC estimation. changing pixels, in a 5ms window are summed to reconstruct an RGB image of the eye. This sampling rate was selected to capture all … view at source ↗
Figure 3
Figure 3. Results from different operations performed during the pupil center localization. As we can see, the RANSAC estimation step increases the accuracy of the estimation made by the match template step. The automatic annotation pipeline is shown in [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Interactive plot for annotation correction. Delta in x,y from an eye position to the next is shown in blue and red. Saccades are marked in violet (rising edge when the saccade starts, falling edge when the saccade ends). Blinks are indicated similarly in orange. At the…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

27 extracted references · 22 canonical work pages

  1. [1]

    Ap- plied Linguistics Papers (25/2), 123–132 (2018)

    Andrychowicz-Trojanowska, A.: Basic terminology of eye-tracking research. Ap- plied Linguistics Papers (25/2), 123–132 (2018)

  2. [2]

    arXiv preprint arXiv:2004.03577 (2020)

    Angelopoulos, A.N., Martel, J.N., Kohli, A.P., Conradt, J., Wetzstein, G.: Event based, near eye gaze tracking beyond 10,000 hz. arXiv preprint arXiv:2004.03577 (2020)

  3. [3]

    In: 2020 International congress on human-computer interaction, opti- mization and robotic applications (HORA)

    Angelov, V., Petkov, E., Shipkovenski, G., Kalushkov, T.: Modern virtual reality headsets. In: 2020 International congress on human-computer interaction, opti- mization and robotic applications (HORA). pp. 1–5. IEEE (2020)

  4. [4]

    Computers 11(2), 28 (2022)

    Arena, F., Collotta, M., Pau, G., Termine, F.: An overview of augmented reality. Computers 11(2), 28 (2022)

  5. [5]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Bonazzi, P., Bian, S., Lippolis, G., Li, Y., Sheik, S., Magno, M.: Retina: Low- power eye tracking with event camera and spiking hardware. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5684– 5692 (2024)

  6. [6]

    https://doi.org/10.1109/MPRV

    Bulling, A., Gellersen, H.W.: Toward mobile eye-based human-computer interac- tion.IEEEPervasiveComputing 9,8–12(2010). https://doi.org/10.1109/MPRV. 2010.86

  7. [7]

    Sensors (Basel, Switzerland) 18 (2018)

    Burger, G., Guna, J., Pogacnik, M.: Suitability of inexpensive eye-tracking device for user experience evaluations. Sensors (Basel, Switzerland) 18 (2018). https: //doi.org/10.3390/s18061822

  8. [8]

    In: Proceedings of the 2024 Symposium on Eye Tracking Research and Applications

    Crafa, D.M., Di Giacomo, S., Natali, D., Fiorini, C.E., Carminati, M.: Towards invisible eye tracking with lens-coupled lateral photodetectors. In: Proceedings of the 2024 Symposium on Eye Tracking Research and Applications. pp. 1–7 (2024)

Show all 27 references
  1. [9]

    Communi- cations of the ACM24(6), 381–395 (1981)

    Fischler, M.A., Bolles, R.C.: Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Communi- cations of the ACM24(6), 381–395 (1981)

  2. [10]

    arXiv preprint arXiv:1601.04902 (2016)

    Fuhl, W., Santini, T., Kasneci, G., Kasneci, E.: Pupilnet: Convolutional neural networks for robust pupil detection. arXiv preprint arXiv:1601.04902 (2016)

  3. [11]

    IEEE transactions on pattern analysis and machine intelligence 44(1), 154–180 (2020)

    Gallego, G., Delbrück, T., Orchard, G., Bartolozzi, C., Taba, B., Censi, A., Leutenegger, S., Davison, A.J., Conradt, J., Daniilidis, K., et al.: Event-based vision: A survey. IEEE transactions on pattern analysis and machine intelligence 44(1), 154–180 (2020)

  4. [12]

    Holmqvist, K., Andersson, R.: Eye-tracking: A comprehensive guide to methods, paradigms and measures (11 2017)

  5. [13]

    Kassner, M., Patera, W., Bulling, A.: Pupil: An open source platform for pervasive eye tracking and mobile gaze-based interaction (April 2014),http://arxiv.org/ abs/1405.0006

  6. [14]

    Scientific Reports 7(1), 16823 (Dec 2017)

    Khazali, M.F., Pomper, J.K., Thier, P.: Blink associated resetting eye movements (barms) are functionally complementary to microsaccades in correcting for fixation errors. Scientific Reports 7(1), 16823 (Dec 2017). https://doi.org/10.1038/ s41598-017-17229-w, https://doi.org/1...

  7. [15]

    PLoS ONE 16 (2021)

    Liu, Z., Yang, Z., Gu, Y., Liu, H., Wang, P.: The effectiveness of eye tracking in the diagnosis of cognitive disorders: A systematic review and meta-analysis. PLoS ONE 16 (2021). https://doi.org/10.1371/journal.pone.0254059

  8. [16]

    In: Proceedings of the 2024 Symposium on Eye Tracking Research and Applications

    Mentasti, S., Lattari, F., Santambrogio, R., Careddu, G., Matteucci, M.: Event- based eye tracking for smart eyewear. In: Proceedings of the 2024 Symposium on Eye Tracking Research and Applications. pp. 1–7 (2024) High-frequency near-eye ground truth for event-based eye tracking 11

  9. [17]

    Neurological Sciences36, 683– 688 (2015)

    Popa, L., Selejan, O., Scott, A., Muresanu, D., Balea, M., Rafila, A.: Reading beyond the glance: eye tracking in neurosciences. Neurological Sciences36, 683– 688 (2015). https://doi.org/10.1007/s10072-015-2076-6

  10. [18]

    Computer Vision and Image Understanding170, 40–50 (2018)

    Santini, T., Fuhl, W., Kasneci, E.: Pure: Robust pupil detection for real-time per- vasive eye tracking. Computer Vision and Image Understanding170, 40–50 (2018)

  11. [19]

    IEEE Reviews in Biomedical Engineering 16, 260–277 (2021)

    Skaramagkas, V., Giannakakis, G., Ktistakis, E., Manousos, D., Karatzanis, I., Tachos, N., Tripoliti, E., Marias, K., Fotiadis, D., Tsiknakis, M.: Review of eye tracking metrics involved in emotional and cognitive processes. IEEE Reviews in Biomedical Engineering 16, 260–277 (...

  12. [20]

    Spil, T.A., Kijl, B., Romijnders, V.: The adoption and diffusion of wearables. In: ICT Unbounded, Social Impact of Bright ICT Adoption: IFIP WG 8.6 Interna- tional Conference on Transfer and Diffusion of IT, TDIT 2019, Accra, Ghana, June 21–22, 2019, Proceedings. pp. 31–47. Sp...

  13. [21]

    Computers & Electrical Engineering93, 107193 (2021)

    Wan, Z.H., Xiong, C.H., Chen, W.B., Zhang, H.Y.: Robust and accurate pupil detection for head-mounted eye tracking. Computers & Electrical Engineering93, 107193 (2021)

  14. [22]

    ais 2024 challenge survey (2024),https://arxiv.org/ abs/2404.11770

    Wang, Z., Gao, C., Wu, Z., Conde, M.V., Timofte, R., Liu, S.C., Chen, Q., jun Zha, Z., Zhai, W., Han, H., Liao, B., Wu, Y., Wan, Z., Wang, Z., Cao, Y., Tan, G., Chen, J., Pei, Y.R., Brüers, S., Crouzet, S., McLelland, D., Coenen, O., Zhang, B., Gao, Y., Li, J., So, H.K.H., Bic...

  15. [23]

    In: 2013 Fifth International Workshop on Quality of Multimedia Experience (QoMEX)

    Winkler, S., Subramanian, R.: Overview of eye tracking datasets. In: 2013 Fifth International Workshop on Quality of Multimedia Experience (QoMEX). pp. 212–

  16. [24]

    Sensors23(18), 7753 (2023)

    Zafar, A., Martin Calderon, C., Yeboah, A.M., Dalton, K., Irving, E., Niechwiej- Szwedo, E.: Investigation of camera-free eye-tracking glasses compared to a video- based system. Sensors23(18), 7753 (2023)

  17. [25]

    Computational Intelligence and Neuroscience2017 (2017)

    Zhang, X., Liu, X., Yuan, S., Lin, S.: Eye tracking based control system for natural human-computer interaction. Computational Intelligence and Neuroscience2017 (2017). https://doi.org/10.1155/2017/5739301

  18. [26]

    Advances in Neural Information Processing Systems36 (2024)

    Zhao, G., Yang, Y., Liu, J., Chen, N., Shen, Y., Wen, H., Lan, G.: Ev-eye: Rethink- ing high-frequency eye tracking through the lenses of event cameras. Advances in Neural Information Processing Systems36 (2024)

  19. [27]

    IEEE Transactions on biomedical engineering54(12), 2246–2260 (2007)

    Zhu, Z., Ji, Q.: Novel eye gaze tracking techniques under natural head movement. IEEE Transactions on biomedical engineering54(12), 2246–2260 (2007)

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.