Pith. sign in

REVIEW 2 major objections 5 minor 89 references

CT-ScanGaze: A Dataset and Baselines for 3D Volumetric Scanpath Modeling

T0 review · 2 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A new public dataset of radiologist gaze on CT scans and a 3D scanpath model that mimics expert visual search.

desk verdict First public CT gaze dataset, but the generalization claim in Sec. 5.4 does not hold; worth a serious referee after major revision. read the letter →

arxiv 2507.12591 v1 pith:BBMXDTBH submitted 2025-07-16 cs.CV

Trong-Thang Pham , Akash Awasthi , Saba Khan , Esteban Duran Marti , Tien-Phat Nguyen , Khoa Vo , Minh Tran , Ngoc Son Nguyen
show 8 more authors
Cuong Tran Van Yuki Ikebe Anh Totti Nguyen Anh Nguyen Zhigang Deng Carol C. Wu Hien Van Nguyen Ngan Le
This is my paper
classification cs.CV
keywords CT-ScanGaze3DscanpathpredictioneyetrackingradiologistcomputedtomographyvolumetricmedicalimagingtransformerexplainableAI
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CT-ScanGaze is the first publicly released eye-tracking dataset of CT reading, recording where two expert radiologists look while interpreting chest and abdomen volumes. The paper's central claim is that its transformer model, CT-Searcher, can generate radiologist-like 3D scanpaths from a CT volume, capturing both where fixations land inside a slice and how gaze moves back and forth across slices. On the authors' benchmark, CT-Searcher beats four 2D scanpath models adapted to 3D on every reported metric, with ScanMatch 0.1466 versus 0.0828 for the best baseline. Every CT in the dataset is paired with a speech-to-text radiology report and extracted findings, so the gaze data can be linked to diagnostic reasoning. The broader point is to give volumetric medical imaging a public testbed for modeling expert visual search, a task that previously had no CT-specific dataset.

What carries the argument

The mechanism that carries the argument is CT-Searcher's transformer architecture with explicit depth encoding. A frozen Swin UNETR encoder converts the CT volume into voxel features, which are projected into tokens along with a learned stop token and given 3D sinusoidal positional encodings so slice position is preserved through the network. A transformer decoder attends to these features using learnable gaze queries and feeds two prediction heads: a spatial head produces a softmax distribution over all voxels plus the stop token, and a duration head regresses fixation time using a reparameterization trick. Before fine-tuning on CT-ScanGaze, the model is pretrained on synthetic 3D gaze built from chest X-ray eye-tracking datasets (EGD and REFLACX) by flipping the x-coordinate, fixing slice position at the middle, and mapping y to depth; this pretraining is what lets a small real dataset support a stable scanpath model.

What would settle it

Record eye gaze from a third, never-seen radiologist on a subset of the same CTs and run CT-Searcher's existing 5-fold checkpoints on them; if its ScanMatch and CC scores collapse to near the baseline range, the cross-radiologist generalization claim is falsified.

Watch

Extended reading notes

Core claim

The central discovery is that a 3D-aware predictor of eye movements can imitate expert volumetric search: it lands fixations on diagnostically relevant regions within each axial slice and reproduces the non-linear, back-and-forth slice navigation radiologists use when they revisit suspicious findings. The paper establishes this by introducing CT-ScanGaze (909 CT volumes, 131,618 slices, about 4,772 minutes of gaze), simplifying the dense raw fixation sequences to an average of 222 fixations with the MultiMatch toolbox, and by benchmarking CT-Searcher against PathGAN, HAT, Gazeformer, and GazeformerISP adapted to 3D inputs. CT-Searcher is the only method whose qualitative temporal slice navigation matches the ground-truth revisiting pattern, and it achieves the best scores on scanpath metrics (ScanMatch, MultiMatch, SED) and spatial metrics (CC, KLDiv, NSS). The authors take the small difference between per-radiologist scores as evidence that the model learned shared search patterns rather than one reader's personal habit.

Load-bearing premise

Everything rests on the two participating radiologists being representative of expert CT readers, and their shared visual style dominating the learned signal.

Editorial extensions

If this is right

  • CT-ScanGaze gives the medical imaging community a public benchmark for 3D scanpath prediction, with both original and simplified gaze sequences released.
  • The paper's pretraining recipe means future volumetric gaze models can be seeded with 2D gaze data instead of requiring a new large eye-tracking study for every 3D domain.
  • Predicted radiologist-like scanpaths can be used as attention priors for explainable classification and report generation on CT, connecting visual search to diagnostic output.
  • The released reports and findings let researchers study which gaze patterns precede which diagnostic statements, not just where eyes land.
  • Because the model reproduces revisits to suspicious regions, its outputs could support radiology training tools that show learners where experienced readers look and look again.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's cross-radiologist check still trains on both readers' data, so it does not yet prove the model would predict an unfamiliar radiologist's scanpath; a clean held-out radiologist test is the natural next experiment.
  • The synthetic pretraining heuristic fixes every fixation at the middle slice, which may bias the model toward a center-slice prior; randomizing slice depth during conversion would test whether that bias matters.
  • With more participating radiologists, this dataset could support models that separate shared expert search strategies from individual reading styles, which would sharpen the radiologist-like claim.
  • The same encoder-and-pretraining design could transfer to other volumetric search tasks, such as industrial CT inspection or geological core reading, where expert gaze is equally hard to collect.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper introduces CT-ScanGaze, a new dataset of eye-tracking data from two experienced radiologists reading 909 chest and abdomen CT volumes, including scanpaths, radiology reports, and extracted findings. The authors also propose CT-Searcher, a transformer-based 3D scanpath prediction model that combines a Swin UNETR feature extractor, 3D positional encoding, and separate spatial and duration prediction heads. A pretraining pipeline converts existing 2D CXR gaze datasets (EGD and REFLACX) into synthetic 3D gaze data. Experiments compare CT-Searcher with 3D-adapted versions of PathGAN, HAT, Gazeformer, and GazeformerISP using scanpath and saliency metrics, and ablation studies assess the contributions of the 3D positional encoding and pretraining. The paper also reports a cross-radiologist evaluation in Sec. 5.4.

Significance. If released as promised, CT-ScanGaze is the first public eye-gaze dataset for CT volumes, and it combines four modalities (CT, gaze, report, findings), filling a clear gap in medical imaging and 3D scanpath prediction. The CT-Searcher baseline is a reasonable first 3D scanpath model and outperforms the adapted 2D baselines on the reported metrics; the ablations support the design choices, and the release of code and data is a concrete reproducibility contribution. However, the broader claim that the model learns general radiologist scanpath patterns is not supported: the cross-radiologist evaluation in Sec. 5.4 includes both radiologists in every training fold, so it only tests held-out volumes for familiar readers. The dataset and benchmark for the two participating radiologists are valuable, but the generalization claim needs either a true leave-one-radiologist-out experiment or explicit re-scoping to the two study readers.

major comments (2)
  1. [Sec. 5.4] The cross-radiologist evaluation cannot support the conclusion that CT-Searcher 'successfully learns general scanpath patterns rather than overfitting to an individual radiologist's style.' The checkpoints come from a 5-fold cross-validation over CT volumes, so both radiologists' gaze data appear in every training fold. The test subsets containing only Radiologist #1 or Radiologist #2 are held-out volumes for those same radiologists, not unseen readers. This conflates generalization across volumes with generalization across radiologists. The claim should be explicitly limited to reproducing the two participating readers, or the authors should run a leave-one-radiologist-out experiment to substantiate generalization to an unseen radiologist.
  2. [Sec. 5.4, Table 4] The statement that 'the differences in scores between the two radiologists are insignificant' is contradicted by the reported 95% confidence intervals for CC. Radiologist #1 has 0.1901 ± 0.017 ([0.173, 0.207]) and Radiologist #2 has 0.1503 ± 0.014 ([0.136, 0.164]); these intervals do not overlap, suggesting a statistically significant difference in how well the model reproduces each reader's spatial saliency. The authors need to either support the insignificance claim with explicit statistical tests (e.g., permutation tests or overlap-corrected intervals) or remove the claim and temper the interpretation.
minor comments (5)
  1. [Sec. 5.1 / Appendix D] The protocol for handling ground truth scanpaths longer than the model's fixed output length N=400 is unspecified. Simplified ground truths have up to 1,507 fixations (Sec. 2.3). The paper should state whether ground truths are truncated to N=400 in training and evaluation, or describe how the metrics accommodate variable-length sequences.
  2. [Sec. 4.6] The 2D-to-3D conversion of gaze data uses ad hoc rules (flipping x, fixing a coordinate to 0.5, and mapping the remaining axis to the slice dimension) that are not validated against real 3D gaze data. Since the pretraining step is shown to improve performance in Table 5, the authors should discuss the rationale for these choices and their potential domain gap.
  3. [Sec. 2.2] The frame-to-DICOM mapping relies on OCR with manual correction, but the extent of manual correction is not quantified. Reporting the percentage of frames that required correction would improve reproducibility and help estimate the accuracy of the gaze-slice alignment.
  4. [Table 1] The Duration column for HAT is reported as '–' with no explanation. Please clarify whether HAT does not predict durations, or whether the metric was not computed, and how this affects the comparison.
  5. [Abstract / Conclusion] The phrase 'radiologist-like 3D fixation sequences' should be qualified as 'like the two participating radiologists' in light of the dataset's two-reader design and the limitations of the cross-radiologist evaluation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: CT-Searcher is trained and evaluated on held-out real CT-ScanGaze data, and the pretraining step is auxiliary rather than definitional.

full rationale

The central derivation chain is self-contained. CT-Searcher is trained on real CT-ScanGaze gaze data using a 5-fold cross-validation split and evaluated on held-out CT volumes with standard scanpath and saliency metrics (Tables 1-2). The pretraining step (Sec. 4.6) uses synthetic 3D gaze converted from 2D CXR datasets, but the final reported numbers are computed against real radiologist fixations, not against the synthetic pretraining distribution. No fitted parameter is renamed as a prediction: the model predicts fixations and durations from CT volumes and is scored by ScanMatch, MultiMatch, SED, CC, KLDiv, and NSS, all of which compare predicted sequences to ground-truth fixations. The loss functions in Sec. 4.5 are standard cross-entropy and L1 regression objectives, not identities with the evaluation metrics. The cross-radiologist evaluation (Sec. 5.4) does raise a validity concern: because the 5-fold split is over CT volumes rather than radiologists, both radiologists' gaze data appear in every training fold, so the 'Radiologist #1'/'Radiologist #2' test subsets measure reproduction of familiar readers on held-out volumes, not generalization to an unseen radiologist. Additionally, Table 4's CC intervals (0.1901 +/- 0.017 and 0.1503 +/- 0.014) do not overlap, which is inconsistent with the text's claim that differences between radiologists are insignificant. These are correctness and interpretation problems, not circularity: the model's performance is not equivalent to its inputs by construction, and no load-bearing argument reduces to a self-citation or to a fitted quantity being reported as a prediction.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claims rest on hand-chosen preprocessing parameters (simplification thresholds, pretraining mapping) and domain assumptions about radiologist representativeness and feature transfer. No new physical entities are introduced.

free parameters (5)
  • MultiMatch simplification angular threshold = 45 degrees
    Default threshold used to merge/suppress fixations when simplifying scanpaths from average 543 to 222 fixations. Chosen from the toolbox defaults, it defines the ground truth used for training and evaluation.
  • MultiMatch simplification amplitude threshold = 10% of volume resolution diagonal
    Default amplitude threshold in the same simplification step; directly affects which navigational movements are preserved in the training targets.
  • Pretraining z position = 0.5 (middle slice)
    All 2D CXR fixations are mapped to the middle slice of the synthetic 3D volume (Sec. 4.6). This removes all depth variation from the pretraining signal and is an ad hoc choice.
  • Pretraining x flip = x -> 1 - x
    Applied to account for a right-to-left reading pattern in CXR, with no evidence that this matches CT viewing.
  • Pretraining Gaussian augmentation sigma = 1 degree of visual angle
    Used to jitter fixation coordinates during pretraining; value is a standard assumption, not fitted to CT data.
assumptions (4)
  • domain assumption MultiMatch simplification preserves essential scanpath characteristics.
    Justifies training on simplified ground truth; if false, the model is not actually modeling the original dense radiologist behavior. Invoked in Sec. 2.3 and Appendix H.
  • ad hoc to paper 2D CXR gaze data can be converted to useful 3D CT gaze by flipping x, setting z=0.5, and mapping y to the slice axis.
    Stated in Sec. 4.6 without validation against real 3D gaze. The z=0.5 constant means slice navigation is not learned from pretraining.
  • domain assumption Frozen Swin UNETR features pretrained on LIDC-IDRI transfer to the scanpath prediction task.
    Used for CT-Searcher and all baselines (Sec. 5.1). If these features are poor for gaze, it limits absolute performance, though comparisons remain relative.
  • domain assumption The two participating radiologists are representative of expert CT readers.
    The dataset and the model's "radiologist-like" claim depend on this. The cross-radiologist analysis in Sec. 5.4 does not test on an unseen radiologist.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CT-ScanGaze: A Dataset and Baselines for 3D Volumetric Scanpath Modeling." pith.science (2026). https://pith.science/paper/BBMXDTBH

@misc{pith2026250712591,
  author       = {Pith},
  title        = {Pith review of: CT-ScanGaze: A Dataset and Baselines for 3D Volumetric Scanpath Modeling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BBMXDTBH}},
  note         = {Machine review of arXiv:2507.12591}
}
read the original abstract

Understanding radiologists' eye movement during Computed Tomography (CT) reading is crucial for developing effective interpretable computer-aided diagnosis systems. However, CT research in this area has been limited by the lack of publicly available eye-tracking datasets and the three-dimensional complexity of CT volumes. To address these challenges, we present the first publicly available eye gaze dataset on CT, called CT-ScanGaze. Then, we introduce CT-Searcher, a novel 3D scanpath predictor designed specifically to process CT volumes and generate radiologist-like 3D fixation sequences, overcoming the limitations of current scanpath predictors that only handle 2D inputs. Since deep learning models benefit from a pretraining step, we develop a pipeline that converts existing 2D gaze datasets into 3D gaze data to pretrain CT-Searcher. Through both qualitative and quantitative evaluations on CT-ScanGaze, we demonstrate the effectiveness of our approach and provide a comprehensive assessment framework for 3D scanpath prediction in medical imaging.

Figures

Figures reproduced from arXiv: 2507.12591 by the authors.

Figure 1
Figure 1. Many research directions in CAD would benefit from [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Illustration of our data collection and processing pipeline. The Data Collection panel shows the setup: a radiologist examines CT [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Examples from our dataset. Three CTs are reviewed and concluded with radiology reports as shown in the [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: CT-Searcher processes CT scans V to predict 3D scanpaths. Initially, a 3D visual encoder within the Feature Extraction module extracts voxel features F. These features with a special ‘stop’ token are then transformed into Z by an MLP and combined with 3D positional enc…
Figure 5
Figure 5. Figure 5: Qualitative comparison of our method and other SOTA methods, HAT and GazeformerISP. The [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

89 extracted references · 74 canonical work pages

  1. [1]

    Deep-bcn: Deep net- works meet biased competition to create a brain-inspired model of attention control

    Hossein Adeli and Gregory Zelinsky. Deep-bcn: Deep net- works meet biased competition to create a brain-inspired model of attention control. In CVPR Workshops, 2018. 8

    2018

  2. [2]

    Bruno, Eliz- abeth A

    Robert Alexander, Stephen Waite, Michael A. Bruno, Eliz- abeth A. Krupinski, Leonard Berlin, Stephen Macknik, and Susana Martinez-Conde. Mandating Limits on Workload, Duty, and Speed in Radiology. Radiology, 304(2), 2022. 2

    2022

  3. [3]

    The lung image database con- sortium (lidc) and image database resource initiative (idri): a completed reference database of lung nodules on ct scans

    Samuel G Armato III, Geoffrey McLennan, Luc Bidaut, Michael F McNitt-Gray, Charles R Meyer, Anthony P Reeves, Binsheng Zhao, Denise R Aberle, Claudia I Hen- schke, Eric A Hoffman, et al. The lung image database con- sortium (lidc) and image database resource initiative (idri): a completed reference database of lung nodules on ct scans. Medical physics, 38...

    2011

  4. [4]

    Pathgan: Visual scanpath prediction with generative adversarial networks

    Marc Assens, Xavier Giro-i Nieto, Kevin McGuinness, and NoelE O’Connor. Pathgan: Visual scanpath prediction with generative adversarial networks. ECCV Workshop on Egocentric Perception, Interaction and Computing (EPIC) ,

  5. [5]

    TempSAL - uncovering temporal information for deep saliency prediction

    Bahar Aydemir, Ludo Hoffstetter, Tong Zhang, Mathieu Salzmann, and Sabine Susstrunk. TempSAL - uncovering temporal information for deep saliency prediction. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023. 8

    2023

  6. [6]

    Reflacx, a dataset of reports and eye-tracking data for local- ization of abnormalities in chest x-rays

    Ricardo Bigolin Lanfredi, Mingyuan Zhang, William F Auf- fermann, Jessica Chan, Phuong-Anh T Duong, Vivek Sriku- mar, Trafton Drew, Joyce D Schroeder, and Tolga Tasdizen. Reflacx, a dataset of reports and eye-tracking data for local- ization of abnormalities in chest x-rays. Scientific data, 9(1): 350, 2022. 1, 2, 4, 5, 8

    2022

  7. [7]

    Spontaneous eye movements during visual imagery reflect the content of the visual scene

    Stephan A Brandt and Lawrence W Stark. Spontaneous eye movements during visual imagery reflect the content of the visual scene. Journal of cognitive neuroscience, 9(1):27–38,

  8. [8]

    End-to- end object detection with transformers

    Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to- end object detection with transformers. In European confer- ence on computer vision, pages 213–229. Springer, 2020. 4

    2020

Show all 89 references
  1. [9]

    Zelinsky, and Dimitris Samaras

    Souradeep Chakraborty, Zijun Wei, Conor Kelton, Seoy- oung Ahn, Aruna Balasubramanian, Gregory J. Zelinsky, and Dimitris Samaras. Predicting visual attention in graphic de- sign documents. IEEE Transactions on Multimedia (TMM),

  2. [10]

    AiR: At- tention with reasoning capability

    Shi Chen, Ming Jiang, Jinhui Yang, and Qi Zhao. AiR: At- tention with reasoning capability. In Proceedings of the Eu- ropean Conference on Computer Vision (ECCV), 2020. 8

    2020

  3. [11]

    Learning from unique per- spectives: User-aware saliency modeling

    Shi Chen, Nachiappan Valliappan, Shaolei Shen, Xinyu Ye, Kai Kohlhoff, and Junfeng He. Learning from unique per- spectives: User-aware saliency modeling. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023. 8

    2023

  4. [12]

    Predicting hu- man scanpaths in visual question answering

    Xianyu Chen, Ming Jiang, and Qi Zhao. Predicting hu- man scanpaths in visual question answering. In Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021. 8, 20

    2021

  5. [13]

    Beyond average: Individualized visual scanpath prediction

    Xianyu Chen, Ming Jiang, and Qi Zhao. Beyond average: Individualized visual scanpath prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 25420–25431, 2024. 2, 6, 7, 8, 20

    2024

  6. [14]

    Speech-to-text v2 api

    Google Cloud. Speech-to-text v2 api. https://cloud. google.com/speech- to- text , 2024. Accessed: 2024-03-23. 3

    2024

  7. [15]

    Predicting human eye fixations via an lstm-based saliency attentive model

    Marcella Cornia, Lorenzo Baraldi, Giuseppe Serra, and Rita Cucchiara. Predicting human eye fixations via an lstm-based saliency attentive model. IEEE Transactions on Image Pro- cessing (IEEE TIP), 2018. 8

    2018

  8. [16]

    Scanmatch: A novel method for comparing fixation sequences

    Filipe Cristino, Sebastiaan Math ˆot, Jan Theeuwes, and Iain D Gilchrist. Scanmatch: A novel method for comparing fixation sequences. Behavior research methods, 42:692–700,

  9. [17]

    Scanpathnet: A recurrent mixture density network for scanpath prediction

    Ryan Anthony Jalova de Belen, Tomasz Bednarz, and Arcot Sowmya. Scanpathnet: A recurrent mixture density network for scanpath prediction. In Proceedings of the IEEE Confer- ence on Computer Vision and Pattern Recognition Workshop (CVPRW), 2022. 8

    2022

  10. [18]

    It de- pends on how you look at it: Scanpath comparison in mul- tiple dimensions with multimatch, a vector-based approach

    Richard Dewhurst, Marcus Nystr ¨om, Halszka Jarodzka, Tom Foulsham, Roger Johansson, and Kenneth Holmqvist. It de- pends on how you look at it: Scanpath comparison in mul- tiple dimensions with multimatch, a vector-based approach. Behavior research methods, 44:1079–1100, 2012....

    2012

  11. [19]

    Tutorial on variational autoencoders

    Carl Doersch. Tutorial on variational autoencoders. arXiv preprint arXiv:1606.05908, 2016. 4

    2016 arXiv

  12. [20]

    Machine-learning-based multiple abnor- mality prediction with large-scale chest computed tomogra- phy volumes

    Rachel Lea Draelos, David Dov, Maciej A Mazurowski, Joseph Y Lo, Ricardo Henao, Geoffrey D Rubin, and Lawrence Carin. Machine-learning-based multiple abnor- mality prediction with large-scale chest computed tomogra- phy volumes. Medical image analysis, 67:101857, 2021. 3, 14

    2021

  13. [21]

    Trafton Drew, Karla Evans, Melissa L. H. V ˜o, Francine L. Jacobson, and Jeremy M. Wolfe. Informatics in Radiology: What Can You See in a Single Glance and How Might This Guide Visual Search in Medical Images? RadioGraphics, 33 (1), 2013. 2

    2013

  14. [22]

    Seltzer, and Jeremy M

    Trafton Drew, Melissa Le-Hoa V o, Alex Olwal, Francine Ja- cobson, Steven E. Seltzer, and Jeremy M. Wolfe. Scanners and drillers: Characterizing expert visual search through vol- umetric images. Journal of Vision, 13(10), 2013. 1, 2, 4

    2013

  15. [23]

    A dataset of eye movements for the children with autism spectrum disorder

    Huiyu Duan, Guangtao Zhai, Xiongkuo Min, Zhaohui Che, Yi Fang, Xiaokang Yang, Jes ´us Guti ´errez, and Patrick Le Callet. A dataset of eye movements for the children with autism spectrum disorder. In ACM Multimedia Systems Con- ference (MMSys), 2019. 8

    2019

  16. [24]

    The role of extra-foveal processing in 3d imaging

    Miguel P Eckstein, Miguel A Lago, and Craig K Abbey. The role of extra-foveal processing in 3d imaging. InProceedings of Spie–the International Society for Optical Engineering . NIH Public Access, 2017. 2

    2017

  17. [25]

    Modelling search for people in 900 scenes: A combined source model of eye guidance.Visual cognition, 17(6-7):945–978, 2009

    Krista A Ehinger, Barbara Hidalgo-Sotelo, Antonio Torralba, and Aude Oliva. Modelling search for people in 900 scenes: A combined source model of eye guidance.Visual cognition, 17(6-7):945–978, 2009. 8

    2009

  18. [26]

    Pre- dicting visual importance across graphic design types

    Camilo Fosco, Vincent Casser, Amish Kumar Bedi, Peter O’Donovan, Aaron Hertzmann, and Zoya Bylinskii. Pre- dicting visual importance across graphic design types. In 9 ACM Symposium on User Interface Software and Technol- ogy, 2020. 8

    2020

  19. [27]

    What can saliency models predict about eye movements? spatial and sequential aspects of fixations during encoding and recognition

    Tom Foulsham and Geoffrey Underwood. What can saliency models predict about eye movements? spatial and sequential aspects of fixations during encoding and recognition. Jour- nal of vision, 8(2):6–6, 2008. 6

    2008

  20. [28]

    Pet: An eye-tracking dataset for animal-centric pascal object classes

    Syed Omer Gilani, Ramanathan Subramanian, Yan Yan, David Melcher, Nicu Sebe, and Stefan Winkler. Pet: An eye-tracking dataset for animal-centric pascal object classes. In 2015 IEEE International Conference on Multimedia and Expo (ICME), pages 1–6. IEEE, 2015. 8

    2015

  21. [29]

    Swin unetr: Swin transformers for semantic segmentation of brain tumors in mri images

    Ali Hatamizadeh, Vishwesh Nath, Yucheng Tang, Dong Yang, Holger R Roth, and Daguang Xu. Swin unetr: Swin transformers for semantic segmentation of brain tumors in mri images. In International MICCAI brainlesion workshop, pages 272–284. Springer, 2021. 6, 16

    2021

  22. [30]

    Salicon: Reducing the semantic gap in saliency prediction by adapting deep neural networks

    Xun Huang, Chengyao Shen, Xavier Boix, and Qi Zhao. Salicon: Reducing the semantic gap in saliency prediction by adapting deep neural networks. In Proceedings of the IEEE international conference on computer vision , pages 262–270, 2015. 6

    2015

  23. [31]

    SALICON: Reducing the semantic gap in saliency predic- tion by adapting deep neural networks

    Xun Huang, Chengyao Shen, Xavier Boix, and Qi Zhao. SALICON: Reducing the semantic gap in saliency predic- tion by adapting deep neural networks. InProceedings of the IEEE International Conference on Computer Vision (ICCV),

  24. [32]

    The use of ma- chine learning in eye tracking studies in medical imaging: A review

    Bulat Ibragimov and Claudia Mello-Thoms. The use of ma- chine learning in eye tracking studies in medical imaging: A review. IEEE journal of biomedical and health informatics ,

  25. [33]

    Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison

    Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Sil- viana Ciurea-Ilcus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, et al. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In Proceedings of the AAA...

    2019

  26. [34]

    A model of saliency-based visual attention for rapid scene analysis

    Laurent Itti, Christof Koch, and Ernst Niebur. A model of saliency-based visual attention for rapid scene analysis. IEEE Transactions on Pattern Analysis and Machine Intelli- gence (IEEE TPAMI), 1998. 8

    1998

  27. [35]

    Sen Jia and Neil D. B. Bruce. EML-NET:an expandable multi-layer network for saliency prediction. Image and Vi- sion Computing, 2020. 8

    2020

  28. [36]

    Salicon: Saliency in context

    Ming Jiang, Shengsheng Huang, Juanyong Duan, and Qi Zhao. Salicon: Saliency in context. In The IEEE Confer- ence on Computer Vision and Pattern Recognition (CVPR) ,

  29. [37]

    Leiva, Hamed R

    Yue Jiang, Luis A. Leiva, Hamed R. Tavakoli, Paul R. B. Houssel, Julia Kylm ¨al¨a, and Antti Oulasvirta. UEyes: Un- derstanding visual saliency across user interface types. In ACM CHI Conference on Human Factors in Computing Sys- tems (CHI), 2023. 8

    2023

  30. [38]

    Creation and validation of a chest x-ray dataset with eye-tracking and report dictation for ai development

    Alexandros Karargyris, Satyananda Kashyap, Ismini Lourentzou, Joy T Wu, Arjun Sharma, Matthew Tong, Shafiq Abedin, David Beymer, Vandana Mukherjee, Eliza- beth A Krupinski, et al. Creation and validation of a chest x-ray dataset with eye-tracking and report dictation for ai de...

    2021

  31. [39]

    A collaborative computer aided diagnosis (c-cad) system with eye-tracking, sparse attentional model, and deep learning

    Naji Khosravan, Haydar Celik, Baris Turkbey, Elizabeth C Jones, Bradford Wood, and Ulas Bagci. A collaborative computer aided diagnosis (c-cad) system with eye-tracking, sparse attentional model, and deep learning. Medical image analysis, 51:101–115, 2019. 4

    2019

  32. [40]

    Perspective projection-based 3d ct recon- struction from biplanar x-rays

    Daeun Kyung, Kyungmin Jo, Jaegul Choo, Joonseok Lee, and Edward Choi. Perspective projection-based 3d ct recon- struction from biplanar x-rays. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2023. 5

    2023

  33. [41]

    Matthias K ¨ummerer, Thomas S. A. Wallis, and Matthias Bethge. DeepGaze II: Reading fixations from deep features trained on object recognition. arXiv preprint arXiv:1610.01563, 2016. 8

    2016 arXiv

  34. [42]

    Matthias K ¨ummerer, Matthias Bethge, and Thomas S. A. Wallis. DeepGaze III: Modeling free-viewing human scan- paths with deep learning. Journal of Vision (JoV), 2022. 8

    2022

  35. [43]

    Methods for compar- ing scanpaths and saliency maps: Strengths and weaknesses

    Olivier Le Meur and Thierry Baccino. Methods for compar- ing scanpaths and saliency maps: Strengths and weaknesses. Behavior Research Methods, 45(1), 2013. 6

    2013

  36. [44]

    Feature pyra- mid networks for object detection

    Tsung-Yi Lin, Piotr Doll ´ar, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyra- mid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 2117–2125, 2017. 18

    2017

  37. [45]

    Roberta: A robustly optimized bert pretraining approach

    Yinhan Liu. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019. 6, 19

    1907 arXiv

  38. [46]

    Decoupled weight decay regularization

    I Loshchilov. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. 6

    2017 arXiv

  39. [47]

    Eye-Gaze-Guided Vision Transformer for Rectifying Short- cut Learning

    Chong Ma, Lin Zhao, Yuzhong Chen, Sheng Wang, Lei Guo, Tuo Zhang, Dinggang Shen, Xi Jiang, and Tianming Liu. Eye-Gaze-Guided Vision Transformer for Rectifying Short- cut Learning. IEEE Transactions on Medical Imaging , 42 (11), 2023. 1

    2023

  40. [48]

    Eye- gaze Guided Multi-modal Alignment for Medical Represen- tation Learning, 2024

    Chong Ma, Hanqi Jiang, Wenting Chen, Yiwei Li, Zihao Wu, Xiaowei Yu, Zhengliang Liu, Lei Guo, Dajiang Zhu, Tuo Zhang, Dinggang Shen, Tianming Liu, and Xiang Li. Eye- gaze Guided Multi-modal Alignment for Medical Represen- tation Learning, 2024. 1

    2024

  41. [49]

    Saccadic model of eye move- ments for free-viewing condition

    Olivier Le Meur and Zhi Liu. Saccadic model of eye move- ments for free-viewing condition. Vision Research (VR) ,

  42. [50]

    Gazeformer: Scalable, effective and fast prediction of goal-directed hu- man attention

    Sounak Mondal, Zhibo Yang, Seoyoung Ahn, Gregory Zelinsky, Dimitris Samaras, and Minh Hoai. Gazeformer: Scalable, effective and fast prediction of goal-directed hu- man attention. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023. 2, 6, 7, 8

    2023

  43. [51]

    Shedding light on ai in radiology: A systematic review and taxonomy of eye gaze-driven interpretability in deep learning

    Jos ´e Neves, Chihcheng Hsieh, Isabel Blanco Nobre, San- dra Costa Sousa, Chun Ouyang, Anderson Maciel, Andrew Duchowski, Joaquim Jorge, and Catarina Moreira. Shedding light on ai in radiology: A systematic review and taxonomy of eye gaze-driven interpretability in deep learni...

    2024

  44. [52]

    Training object class detectors from eye tracking data

    Dim P Papadopoulos, Alasdair DF Clarke, Frank Keller, and Vittorio Ferrari. Training object class detectors from eye tracking data. In European conference on computer vision , pages 361–376. Springer, 2014. 8 10

    2014

  45. [53]

    Lndetector: A flexible gaze characterisation col- laborative platform for pulmonary nodule screening

    Jo ˜ao Pedrosa, Guilherme Aresta, Jo ˜ao Rebelo, Eduardo Negr˜ao, Isabel Ramos, Ant ´onio Cunha, and Aur ´elio Campilho. Lndetector: A flexible gaze characterisation col- laborative platform for pulmonary nodule screening. In XV Mediterranean Conference on Medical and Biologic...

    2019

  46. [54]

    Eye gaze guided cross-modal alignment network for radiology report generation

    Peixi Peng, Wanshu Fan, Yue Shen, Wenfei Liu, Xin Yang, Qiang Zhang, Xiaopeng Wei, and Dongsheng Zhou. Eye gaze guided cross-modal alignment network for radiology report generation. IEEE Journal of Biomedical and Health Informatics, 2024. 1

    2024

  47. [55]

    I-ai: A controllable & interpretable ai system for decoding radiologists’ intense focus for accu- rate cxr diagnoses

    Trong Thang Pham, Jacob Brecheisen, Anh Nguyen, Hien Nguyen, and Ngan Le. I-ai: A controllable & interpretable ai system for decoding radiologists’ intense focus for accu- rate cxr diagnoses. In Proceedings of the IEEE/CVF Win- ter Conference on Applications of Computer Vision...

    2024

  48. [56]

    Wu, Hien Nguyen, and Ngan Le

    Trong Thang Pham, Ngoc-Vuong Ho, Nhat-Tan Bui, Thinh Phan, Patel Brijesh, Donald Adjeroh, Gianfranco Doretto, Anh Nguyen, Carol C. Wu, Hien Nguyen, and Ngan Le. Fg- cxr: A radiologist-aligned gaze dataset for enhancing inter- pretability in chest x-ray report generation. In Pr...

    2024

  49. [57]

    Itpctrl-ai: End- to-end interpretable and controllable artificial intelligence by modeling radiologists’ intentions

    Trong-Thang Pham, Jacob Brecheisen, Carol C Wu, Hien Nguyen, Zhigang Deng, Donald Adjeroh, Gianfranco Doretto, Arabinda Choudhary, and Ngan Le. Itpctrl-ai: End- to-end interpretable and controllable artificial intelligence by modeling radiologists’ intentions. Artificial Intel...

    2025

  50. [58]

    Wu, Hien Nguyen, and Ngan Le

    Trong Thang Pham, Tien-Phat Nguyen, Yuki Ikebe, Akash Awasthi, Zhigang Deng, Carol C. Wu, Hien Nguyen, and Ngan Le. Gazesearch: Radiology findings search bench- mark. In 2025 IEEE/CVF Winter Conference on Applica- tions of Computer Vision (WACV), pages 96–106, 2025. 8

    2025

  51. [59]

    Simulating human visual sys- tem based on vision transformer

    Mengyu Qiu, Yi Guo, Mingguang Zhang, Jingwei Zhang, Tian Lan, and Zhilin Liu. Simulating human visual sys- tem based on vision transformer. In Proceedings of the 2023 ACM Symposium on Spatial User Interaction, 2023. 8

    2023

  52. [60]

    Medical report generation through radiology images: an overview

    Graciela Ramirez-Alonso, Olanda Prieto-Ordaz, Roberto L´opez-Santillan, and Manuel Montes-Y-G ´omez. Medical report generation through radiology images: an overview. IEEE Latin America Transactions, 20(6):986–999, 2022. 1

    2022

  53. [61]

    Human attention in fine-grained classification

    Yao Rong, Wenjia Xu, Zeynep Akata, and Enkelejda Kas- neci. Human attention in fine-grained classification. arXiv preprint arXiv:2111.01628, 2021. 1

    2021 arXiv

  54. [62]

    Ng, and Matthew P

    Akshay Smit, Saahil Jain, Pranav Rajpurkar, Anuj Pareek, Andrew Y . Ng, and Matthew P. Lungren. Chexbert: Com- bining automatic labelers and expert annotations for accurate radiology report labeling using bert, 2020. 16

    2020

  55. [63]

    An overview of the tesseract ocr engine

    Ray Smith. An overview of the tesseract ocr engine. In Ninth international conference on document analysis and recogni- tion (ICDAR 2007), pages 629–633. IEEE, 2007. 3

    2007

  56. [64]

    ScanDMM: A deep markov model of scan- path prediction for 360° images

    Xiangjie Sui, Yuming Fang, Hanwei Zhu, Shiqi Wang, and Zhou Wang. ScanDMM: A deep markov model of scan- path prediction for 360° images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recog- nition (CVPR), 2023. 8

    2023

  57. [65]

    Visual scan- path prediction using IOR-ROI recurrent mixture density network

    Wanjie Sun, Zhenzhong Chen, and Feng Wu. Visual scan- path prediction using IOR-ROI recurrent mixture density network. IEEE Transactions on Pattern Analysis and Ma- chine Intelligence (IEEE TPAMI), 2019. 8

    2019

  58. [66]

    Artificially generated visual scanpath improves multi-label thoracic disease classification in chest x-ray images

    Ashish Verma, Aupendu Kar, Krishnendu Ghosh, Sob- han Kanti Dhara, Debashis Sen, and Prabir Kumar Biswas. Artificially generated visual scanpath improves multi-label thoracic disease classification in chest x-ray images. IEEE Transactions on Instrumentation and Measurement, 2024. 1

    2024

  59. [67]

    Interpretive Error in Radi- ology

    Stephen Waite, Jinel Scott, Brian Gale, Travis Fuchs, Srini- vas Kolla, and Deborah Reede. Interpretive Error in Radi- ology. American Journal of Roentgenology , 208(4), 2017. 2

    2017

  60. [68]

    Gazegnn: A gaze- guided graph neural network for chest x-ray classification

    Bin Wang, Hongyi Pan, Armstrong Aboah, Zheyuan Zhang, Elif Keles, Drew Torigian, Baris Turkbey, Elizabeth Krupin- ski, Jayaram Udupa, and Ulas Bagci. Gazegnn: A gaze- guided graph neural network for chest x-ray classification. In Proceedings of the IEEE/CVF Winter Conference o...

    2024

  61. [69]

    Laugeson, Daniel P

    Shuo Wang, Ming Jiang, Xavier Morin, Duchesne, Eliza- beth A. Laugeson, Daniel P. Kennedy, Ralph Adolphs, and Qi Zhao. Atypical visual saliency in autism spectrum dis- order quantified through model-based eye tracking. Neuron,

  62. [70]

    Follow my eye: Using gaze to supervise computer-aided diagnosis

    Sheng Wang, Xi Ouyang, Tianming Liu, Qian Wang, and Dinggang Shen. Follow my eye: Using gaze to supervise computer-aided diagnosis. IEEE Transactions on Medical Imaging, 41(7):1688–1698, 2022. 4

    2022

  63. [71]

    Simulating human saccadic scanpaths on natural images

    Wei Wang, Cheng Chen, Yizhou Wang, Tingting Jiang, Fang Fang, and Yuan Yao. Simulating human saccadic scanpaths on natural images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2011. 8

    2011

  64. [72]

    Learned region sparsity and diversity also predict visual attention

    Zijun Wei, Hossein Adeli, Minh Hoai, Gregory Zelinsky, and Dimitris Samaras. Learned region sparsity and diversity also predict visual attention. In NeurIPS, 2016. 8

    2016

  65. [73]

    Calden Wloka, Iuliia Kotseruba, and John K. Tsotsos. Ac- tive fixation control to predict saccade sequences. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018. 8

    2018

  66. [74]

    Kankanhalli, and Qi Zhao

    Juan Xu, Ming Jiang, Shuo Wang, Mohan S. Kankanhalli, and Qi Zhao. Predicting human gaze beyond pixels. Journal of Vision (JoV), 2014. 8

    2014

  67. [75]

    Vitae: Vision transformer advanced by exploring intrinsic inductive bias

    Yufei Xu, Qiming Zhang, Jing Zhang, and Dacheng Tao. Vitae: Vision transformer advanced by exploring intrinsic inductive bias. Advances in neural information processing systems, 34:28522–28535, 2021. 8

    2021

  68. [76]

    Predicting goal-directed human attention using inverse reinforcement learning

    Zhibo Yang, Lihan Huang, Yupei Chen, Zijun Wei, Seoy- oung Ahn, Gregory Zelinsky, Dimitris Samaras, and Minh Hoai. Predicting goal-directed human attention using inverse reinforcement learning. In Proceedings of the IEEE Confer- ence on Computer Vision and Pattern Recognition (CVPR) ,

  69. [77]

    Target-absent 11 human attention

    Zhibo Yang, Sounak Mondal, Seoyoung Ahn, Gregory Zelinsky, Minh Hoai, and Dimitris Samaras. Target-absent 11 human attention. In Proceedings of the European Confer- ence on Computer Vision (ECCV), 2022. 8

    2022

  70. [78]

    Predicting hu- man attention using computational attention

    Zhibo Yang, Sounak Mondal, Seoyoung Ahn, Gregory Zelinsky, Minh Hoai, and Dimitris Samaras. Predicting hu- man attention using computational attention. arXiv preprint arXiv:2303.09383, 2023. 8

    2023 arXiv

  71. [79]

    Unify- ing top-down and bottom-up scanpath prediction using trans- formers

    Zhibo Yang, Sounak Mondal, Seoyoung Ahn, Ruoyu Xue, Gregory Zelinsky, Minh Hoai, and Dimitris Samaras. Unify- ing top-down and bottom-up scanpath prediction using trans- formers. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 2, 6, 7, 8, 18

    2024

  72. [80]

    Benchmarking gaze prediction for categori- cal visual search

    Gregory Zelinsky, Zhibo Yang, Lihan Huang, Yupei Chen, Seoyoung Ahn, Zijun Wei, Hossein Adeli, Dimitris Samaras, and Minh Hoai. Benchmarking gaze prediction for categori- cal visual search. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Worksh...

    2019

  73. [81]

    Finding any waldo with zero- shot invariant and efficient visual search

    Mengmi Zhang, Jiashi Feng, Keng Teck Ma, Joo Hwee Lim, Qi Zhao, and Gabriel Kreiman. Finding any waldo with zero- shot invariant and efficient visual search. Nature communi- cations, 9(1):3730, 2018. 8 12 CT-ScanGaze: A Dataset and Baselines for 3D Volumetric Scanpath Modeling...

    2018

  74. [82]

    In the original 2D version, this step divides the image into patches

    Converts gaze sequences into strings: a) Dividing the 3D volume into discrete cells. In the original 2D version, this step divides the image into patches. b) Assigning unique characters to each cell. c) Mapping fixation points to these characters in se- quence

  75. [83]

    In our 3D adapted SED, we change the step 1.a from 2D into 3D, the other steps are left as is

    Compares two sequences using Levenshtein distance by counting minimum number of operations (insertions, deletions, substitutions). In our 3D adapted SED, we change the step 1.a from 2D into 3D, the other steps are left as is. Lower SED scores indicate better matching between s...

  76. [84]

    This step is similar to the first step of SED

    Convert fixation sequences into letter strings. This step is similar to the first step of SED. In addition, when con- sidering duration (ScanMatch w/ Dur.), each character is repeated n times, where n is the duration in millisec- onds. This repetition is not performed when dur...

  77. [85]

    The original score function uses 2D Eu- clidean distance, which we extend to 3D Euclidean dis- tance

    Create a substitution matrix with scores for all possible letter pairs. The original score function uses 2D Eu- clidean distance, which we extend to 3D Euclidean dis- tance

  78. [86]

    Higher SM scores in- dicate better matching between sequences, with an upper bound of 1.0

    Sequence comparison: a) Create comparison matrix: • Columns: letters from first sequence • Rows: letters from second sequence • Cell values: costs from substitution matrix b) Apply Needleman-Wunsch algorithm to find optimal alignment path c) Calculate normalized similarity sco...

  79. [87]

    Collect all pairs(i, j) to create a similarity matrix M (i, j) between elements

    Temporal alignment: a) Calculate how similar each element i in one scanpath is compared to each element j in the other scanpath based on a similarity metric. Collect all pairs(i, j) to create a similarity matrix M (i, j) between elements. b) From M (i, j), build adjacency matr...

  80. [88]

    b) Direction: difference in direction (angle) between fixation vectors

    For every align pair of fixation (i, j), we compute simi- larity across five dimensions: a) Vector (shape): shape difference between fixation vectors (ˆxi, ˆyi, ˆzi, ˆti) − (xj, yj, zj, tj). b) Direction: difference in direction (angle) between fixation vectors. We measure ang...

  81. [89]

    b) Direction is normalized by π

    Score normalization: a) Vector, Length, and Position scores are normalized by volume diagonal. b) Direction is normalized by π. c) Duration is normalized by maximum duration. E. 3D Scanpath Prediction Baselines E.1. PathGAN Similar to original PathGAN [4], our CT-adapted Path-...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.