Pith. sign in

REVIEW 3 major objections 6 minor 100 references

Gaze Target Estimation Anywhere with Concepts

T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read GazeAnywhere makes gaze estimation promptable: a text or point prompt replaces head boxes and pose priors in a single end-to-end model.

desk verdict Genuinely new task and dataset, but the benchmark's prompt construction may leak gaze targets to GazeAnywhere, inflating the SOTA claim. read the letter →

arxiv 2608.11367 v1 pith:DRZOWVQS submitted 2026-08-11 cs.CV cs.AI

Xu Cao , Houze Yang , Vipin Gunda , Zhongyi Zhou , Tianyu Xu , Adarsh Kowdle , Inki Kim , James M. Rehg This is my paper · ORCID
classification cs.CVcs.AI
keywords gazetargetestimationpromptableconcept-basedpromptinggaze-followingopen-vocabularydetectiontransformerdetectorGaze-Codatasetsocialanalysis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to replace brittle, multi-stage gaze-following pipelines with one promptable system. It defines the new task of Promptable Gaze Target Estimation (PGE), where the user identifies the person of interest by a short text phrase or a spatial point, and the model must locate that person and predict where they are looking in a single pass. To make this possible, the authors build a 120K-image dataset of concept descriptions and a model called GazeAnywhere. They report that GazeAnywhere outperforms two-stage baselines that chain open-vocabulary detectors to gaze models, including on an out-of-domain clinical dataset of children. If the finding holds, gaze analysis becomes usable by nonexperts in clinics, homes, and augmented-reality devices: describing a person replaces drawing head boxes.

What carries the argument

The load-bearing mechanism is a cross-modal detector transformer that concatenates projected visual patch tokens, text content tokens, a learnable head token initialized from the text EOS embedding, and a target-presence token initialized from the visual CLS embedding into one token sequence. The head token drives an auxiliary head-box head, the presence token drives an in/out-of-frame focal-loss classifier, and the visual patch tokens are reshaped and upsampled into the gaze heatmap. The joint objective is a BCE heatmap loss, a focal presence loss, and an L1 plus GIoU head-box loss; ablations show the head loss helps gaze estimation while the presence loss mainly supports the in/out output. Equally central is the Gaze-Co data engine: it aligns and filters source data, generates four-field concept phrases (appearance, position, action, pose), and verifies them with an MLLM plus human spot-checks, all without exposing private clinical data to cloud processing.

What would settle it

Take a held-out image set with prompts written independently by different human annotators (not the MLLM that generated Gaze-Co), and compare GazeAnywhere's margin over two-stage open-vocabulary-detector baselines; if the margin shrinks or disappears, the reported advantage depends on prompt wording rather than architectural superiority.

Watch

Extended reading notes

Core claim

The central claim is that gaze target estimation can be reformulated as an end-to-end, concept-conditioned task. Given an image and a prompt such as “the boy in the red shirt” or “person at point [0.52, 0.48]”, a single model can localize the subject’s head, decide whether the gaze target is inside the frame, and produce a gaze heatmap—with no auxiliary head detector, pose estimator, or depth module. The authors define this as the Promptable Gaze Target Estimation task, contribute the Gaze-Co dataset by retrofitting the GazeFollow, VideoAttentionTarget, and ChildPlay splits with concept annotations, and introduce GazeAnywhere, a transformer-based detector that fuses frozen vision and text encoders. They report state-of-the-art results on all public PGE benchmarks and on a private out-of-domain dataset of child social-communication videos, and they demonstrate an augmented-reality agent in which an MLLM calls GazeAnywhere as a tool to answer high-level gaze questions.

Load-bearing premise

The load-bearing premise is that the concept annotations in Gaze-Co are accurate and unbiased enough that the measured advantage of GazeAnywhere over two-stage baselines reflects model design rather than a prompt distribution tailored to its text encoder.

Editorial extensions

If this is right

  • Two-stage gaze pipelines no longer need a separate head detector at inference: subject localization and gaze estimation are solved jointly from the prompt.
  • Users can query a specific person in a crowd by describing appearance or by pointing at a coordinate, making gaze analysis practical for clinicians, parents, and AR interfaces.
  • The Gaze-Co benchmark provides a standardized, human-verified format for evaluating concept-conditioned gaze on GazeFollow, VideoAttentionTarget, and ChildPlay splits.
  • The out-of-domain Child-SC results suggest the promptable formulation can generalize to real-world clinical video where open-vocabulary head detection is unreliable.
  • An AR agent built on GazeAnywhere reduces errors in gaze-shift and eye-contact counts compared with a raw multimodal LLM, indicating a path toward deployable behavioral-analysis tools.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the prompt robustness holds, the same frozen-encoder-plus-detector recipe could be lifted to other fine-grained social cues such as pointing, mutual gaze, or attention to specific objects.
  • A valuable stress test would rephrase each prompt with synonyms and reorder the attribute fields; the paper does not report sensitivity to wording, and large sensitivity would bound how deeply “concept-driven” the model really is.
  • The human-in-the-loop data engine could retrofit other behavioral video datasets with concept annotations, potentially scaling PGE from still images to continuous social interaction.
  • The Gaze-Co concept style—appearance, position, action, and pose—may itself bias downstream models toward stable visual attributes; testing with action- or pose-only prompts would clarify which cues carry the gaze signal.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces Promptable Gaze Target Estimation (PGE), a paradigm in which a gaze-following model is conditioned on a natural-language or visual concept prompt identifying the subject, and proposes GazeAnywhere, an end-to-end transformer-based model that fuses frozen image and text encoders to jointly predict the subject's head box, in/out-of-frame presence, and gaze heatmap. To support training and evaluation, the authors build the Gaze-Co dataset (about 120K samples) by converting GazeFollow, VAT, and ChildPlay into a concept-annotated format, and evaluate on the resulting GazeFollow-Concept, VAT-Concept, ChildPlay-Concept benchmarks plus a private child social-communication (Child-SC) out-of-domain set. The paper reports state-of-the-art results against two-stage OVD-plus-gaze-model baselines and against VLM baselines, and includes ablations on prompting, losses, encoders, and detector dimensions, as well as an AR agent demonstration.

Significance. If the empirical claims hold, this is a meaningful step: GazeAnywhere is the first promptable end-to-end gaze-target model that removes the head-box and pose dependencies of prior pipelines, and the Gaze-Co data engine is a reusable contribution with exact prompts and a human-in-the-loop verification protocol. The paper is also commendable for open-sourcing the code, for pairing multiple OVDs with multiple gaze models rather than a single straw-man baseline, and for including a real-world agent evaluation. The central issue is that the reported state-of-the-art margin is only as credible as the concept-prompt benchmark, and that benchmark currently has a potential information-leak problem that must be resolved before the comparison can be interpreted as a clean test of gaze-target inference.

major comments (3)
  1. [§4.1, Appendix §15.1 (Figs. 10–11)] The benchmark prompts are constructed in a way that can leak the gaze target into the text given to GazeAnywhere but not to the two-stage baselines. The concept-generation prompt in Fig. 10 explicitly instructs the MLLM to add 'target/object/direction when visible' to the action field and to include 'facing direction' in the pose field, and the verification prompt in Fig. 11 only checks that the fields match the designated head box, are present, and are non-conflicting; it never prohibits gaze verbs or target-naming phrases such as 'looking at the ball' or 'facing the clinician.' Because GazeAnywhere concatenates the projected text tokens with the visual tokens in the detector transformer (Eq. 5), such phrases can directly bias the heatmap prediction, whereas in the OVD baselines the same text is used only to emit a head box and the gaze stage never sees the action/pose text. Table 3 is consistent with this concern: action-only and pose-only prompts improve Avg L2 from 0.144 (no prompting) to 0.129 and 0.121 on GazeFollow-Concept, which suggests these fields carry information beyond subject disambiguation. As written, the SOTA comparison does not cleanly isolate gaze-target inference from prompt leakage; the authors should either verify that the generated and human-verified prompts avoid gaze-bearing language, or re-run the comparison on appearance-only and leakage-controlled prompts.
  2. [§5.2, Table 1] The SOTA claim rests on small margins reported without error bars, seed variance, or significance tests. For example, on GazeFollow-Concept Avg L2 GazeAnywhere-DINOv3-L scores 0.099 versus 0.097 for ViTGaze with ground-truth head boxes and 0.099 for Gaze-LLE with ground-truth boxes, and on VAT-Concept L2 it scores 0.123 versus 0.121 for RexSeek; several ablation rows differ by only 0.001–0.005 (Tables 4, 8, and 9). Without repeated runs, confidence intervals, or statistical testing, it is not possible to determine which of these differences are genuine, especially for the claim of state-of-the-art status on the public benchmarks.
  3. [§4.2, §10.3] The out-of-domain evaluation on Child-SC is a central piece of evidence for generalization, but the dataset is private, the manual annotations were produced without the MLLM verification pipeline, and no derived annotations or de-identified features are released. The reported L2 of 0.090 and AP of 0.902 on this set cannot be independently checked or compared by other groups. The authors should release at least a de-identified subset or a reproducible public surrogate with the same protocol, or explicitly label the Child-SC numbers as non-reproducible evidence rather than a benchmark result.
minor comments (6)
  1. [§3.2] The paragraph after Eq. (5) describes the input sequence as having NT + NV + 2 tokens, but the definition of NT is not stated; the paper should distinguish the number of content text tokens from the fixed padded context length LT.
  2. [§4.1] The claimed ≤1% field error rate from human spot-checks would be much more informative with the number of spot-checked samples, the inter-annotator agreement, and a few examples of failed cases.
  3. [Table 3] The 'no prompting' row is ambiguous because it is not clear how the model identifies the subject without a text or visual prompt; the caption should specify the exact input used in that condition.
  4. [Appendix §16] The notation list includes E_head but Eq. (3) defines the head token directly as t'_eos; either use E_head in the formula or remove it from the notation list.
  5. [Appendix §15.1] The verification prompt instructs the MLLM to ignore the <count> field, yet the concept-generation output format includes it; the role of the people-count field in the released dataset and benchmarks should be stated.
  6. [§3.2] There is a typo, 'Transfomrer' for 'Transformer', in the detector-transformer paragraph.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: GazeAnywhere's predictions are trained and evaluated on externally sourced gaze labels, and no claimed result reduces to a fitted parameter or self-citation chain.

full rationale

I walked the paper's derivation chain: the PGE task definition (Sec. 3.1), the GazeAnywhere architecture (Sec. 3.2), the joint loss (Eqs. 6-9), the Gaze-Co data engine and benchmark construction (Secs. 4.1-4.2), and the evaluation protocol (Sec. 5). The predicted gaze heatmap is produced by a learned convolutional decoder from fused visual and text tokens, and the supervisory signal is the externally sourced ground-truth gaze point converted into a Gaussian heatmap; no equation defines the prompt in terms of the predicted heatmap or vice versa. The concept annotations are generated by an external MLLM (Gemini 2.5 Pro) with human verification and are used as inputs, not as fitted constants, and the test predictions are not derived from those annotations by construction. The SOTA comparisons use held-out test splits of existing datasets against external baselines; the self-citation to Gaze-LLE is a baseline system from the same lab, but it is not load-bearing for any theoretical claim or for the architecture's derivation. The reviewer concern that the action/pose prompt fields may encode gaze-relevant information is a benchmark-validity/leakage issue, not a circularity of the paper's derivation, because the model never receives the ground-truth gaze point or target heatmap as input and the prompts do not mathematically force the output. No exhibit-able reduction of a prediction to its own input exists, so the appropriate finding is no significant circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper is an empirical deep learning contribution. It introduces no new physical entities or analytical derivations. The central claims rest on the assumption that the assembled dataset, with MLLM-generated and human-verified concept annotations, is a fair testbed, and that the chosen evaluation metrics reflect task performance. The main free parameters are standard hyperparameters from prior work or ablation choices.

free parameters (3)
  • Heatmap Gaussian sigma (σ=3) = 3
    Standard choice in gaze-following heatmap supervision; not tuned in this paper's ablations.
  • Head loss weights λ_l1 and λ_iou = 5 and 2
    Taken from DETR and OWLViT; not ablated.
  • Detector transformer dimension D and layer count k = 256 and 3
    Selected via ablations in Appendix 13.2 and 13.3; performance plateaus near these values.
assumptions (4)
  • domain assumption Ground-truth gaze and head annotations in GazeFollow, VAT, and ChildPlay are accurate.
    Gaze-Co is built directly on these datasets; any label noise propagates to training and evaluation.
  • domain assumption Gemini 2.5 Pro generated concept phrases that correctly identify the target person, verified by humans at or below 1% error.
    Concept quality determines whether prompts are unambiguous for both the proposed model and OVD baselines.
  • domain assumption The frozen DINOv3 and dino.txt encoders provide features that support gaze target estimation in the way assumed.
    The architecture assumes these pretrained features are adequate; ablations show freezing outperforms fine-tuning.
  • standard math The metrics AUC, L2, and AP are standard and interpretable for gaze target estimation.
    These are the metrics used in GazeFollow and related benchmarks.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Gaze Target Estimation Anywhere with Concepts." pith.science (2026). https://pith.science/paper/DRZOWVQS

@misc{pith2026260811367,
  author       = {Pith},
  title        = {Pith review of: Gaze Target Estimation Anywhere with Concepts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DRZOWVQS}},
  note         = {Machine review of arXiv:2608.11367}
}
read the original abstract

Estimating human gaze targets from images in-the-wild is an important and formidable task. Existing approaches primarily employ brittle, multi-stage pipelines that require explicit inputs, like head bounding boxes and human pose, in order to identify the subject of gaze analysis. As a result, detection errors can cascade and lead to failure. Moreover, these prior works lack the flexibility of specifying the gaze analysis task via natural language prompting, an approach which has been shown to have significant benefits in convenience and scalability for other image analysis tasks. To overcome these limitations, we introduce the Promptable Gaze Target Estimation (PGE) task, a new end-to-end, concept-driven paradigm for gaze analysis. PGE conditions gaze prediction on flexible user text or visual prompts (e.g., "the boy in the red shirt" or "person in point [0.52, 0.48]") to identify a specific subject for gaze analysis. This approach integrates subject localization with gaze estimation, and eliminates the rigid dependency on intermediate analysis stages. We develop a scalable data engine to generate Gaze-Co (Gaze Estimation with Concepts), a dataset and benchmark of 120K high-quality, prompt-annotated image pairs. We also propose GazeAnywhere, the first model designed for PGE. GazeAnywhere uses a transformer-based detector to fuse features from frozen encoders and simultaneously solves subject localization, in/out-of-frame presence, and gaze target heatmap estimation. GazeAnywhere achieves state-of-the-art performance on multiple PGE benchmarks, setting a strong baseline for this new problem even on a difficult out-of-domain, real-world clinical dataset. GazeAnywhere is open-sourced in github.com/IrohXu/GazeAnywhere.

Figures

Figures reproduced from arXiv: 2608.11367 by the authors.

Figure 1
Figure 1. Gaze target estimation in in-the-wild environments. Prior [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. An overview of the GazeAnywhere end-to-end framework for PGE. The model uses frozen visual (DINOv3) and text (dino.txt) [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Overview of the GazeAnywhere data engine. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: Visualization of GazeAnywhere’s gaze target estimation results from several datasets. The input is the text prompt (shown in the [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Workflow of MLLM-powered GazeAnywhere Agent. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Step-by-step explanation of how GazeAnywhere Agent works. [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: Training-set statistics of the Gaze-Co dataset: (a) proportion of each source dataset, (b) distribution of apparent subject categories, [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 8
Figure 8. Figure 8: DigiLens ARGO AR glasses used for video and audio [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: Qualitative comparison of gaze-target localization conditioned on appearance prompts. Each column corresponds to a different [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 10
Figure 10. Figure 10: Concept generation prompt used for generating concept phrases for the target person [PITH_FULL_IMAGE:figures/full_fig_p019_10.png]
Figure 11
Figure 11. Figure 11: Concept verification prompt used to check attribute, position, action, and pose consistency for each subject [PITH_FULL_IMAGE:figures/full_fig_p020_11.png]
Figure 12
Figure 12. Figure 12: Gaze target prediction prompt used for in-frame flagging and point estimation on Gemini-2.5 baseline [PITH_FULL_IMAGE:figures/full_fig_p021_12.png]
Figure 13
Figure 13. Figure 13: Gaze target prediction prompt used for in-frame flagging and point estimation on Qwen3-VL-8B. Gaze Shift Analysis Prompt on Single MLLM Solution This is a short video for human gaze target understanding. Can you give me an analysis of these tasks"? The subject child i…
Figure 14
Figure 14. Figure 14: Gaze shift analysis prompt used for counting gaze shifts and eye contact events on Single MLLM [PITH_FULL_IMAGE:figures/full_fig_p022_14.png]
Figure 15
Figure 15. Figure 15: Gaze shift analysis prompt used for counting gaze shifts and eye contact events on GazeAnywhere agent [PITH_FULL_IMAGE:figures/full_fig_p023_15.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

100 extracted references · 64 canonical work pages

  1. [1]

    Social eye gaze in human-robot interaction: a review.Journal of Human-Robot Interaction, 6(1):25–63, 2017

    Henny Admoni and Brian Scassellati. Social eye gaze in human-robot interaction: a review.Journal of Human-Robot Interaction, 6(1):25–63, 2017. 1

    2017

  2. [2]

    Gaze and mutual gaze.The British Journal of Psychiatry, 165(6): 848–850, 1994

    Michael Argyle, Mark Cook, and Duncan Cramer. Gaze and mutual gaze.The British Journal of Psychiatry, 165(6): 848–850, 1994. 1

    1994

  3. [3]

    Qwen2.5- vl technical report.arXiv preprint arXiv:2502.13923, 2025

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5- vl technical report.a...

    arXiv 2025

  4. [4]

    Escnet: Gaze target detec- tion with the understanding of 3d scenes

    Jun Bao, Buyu Liu, and Jun Yu. Escnet: Gaze target detec- tion with the understanding of 3d scenes. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14126–14135, 2022. 2

    2022

  5. [5]

    Access: Advancing innovation: Nsf’s advanced cyberinfrastructure coordination ecosystem: Services & support

    Timothy J Boerner, Stephen Deems, Thomas R Furlani, Shel- ley L Knuth, and John Towns. Access: Advancing innovation: Nsf’s advanced cyberinfrastructure coordination ecosystem: Services & support. InPractice and experience in advanced research computing 2023: Computing for the common good, pages 173–176. 2023. 9

    2023

  6. [6]

    Perception encoder: The best visual embeddings are not at the output of the net- work.arXiv preprint arXiv:2504.13181, 2025

    Daniel Bolya, Po-Yao Huang, Peize Sun, Jang Hyun Cho, Andrea Madotto, Chen Wei, Tengyu Ma, Jiale Zhi, Jathushan Rajasegaran, Hanoona Rasheed, et al. Perception encoder: The best visual embeddings are not at the output of the net- work.arXiv preprint arXiv:2504.13181, 2025. 2

    arXiv 2025

  7. [7]

    Realtime multi-person 2d pose estimation using part affinity fields

    Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Realtime multi-person 2d pose estimation using part affinity fields. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 7291–7299, 2017. 1

    2017

  8. [8]

    End-to- end object detection with transformers

    Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to- end object detection with transformers. InEuropean confer- ence on computer vision, pages 213–229. Springer, 2020. 2, 5, 1

    2020

Show all 100 references
  1. [9]

    Sam 3: Segment anything with concepts.arXiv preprint arXiv:2511.16719, 2025

    Nicolas Carion, Laura Gustafson, Yuan-Ting Hu, Shoub- hik Debnath, Ronghang Hu, Didac Suris, Chaitanya Ryali, Kalyan Vasudev Alwala, Haitham Khedr, Andrew Huang, et al. Sam 3: Segment anything with concepts.arXiv preprint arXiv:2511.16719, 2025. 2, 7

    2025 arXiv

  2. [10]

    Gaze estimation via the joint modeling of multiple cues.IEEE Transactions on Circuits and Systems for Video Technology, 32(3):1390–1402, 2021

    Wenhe Chen, Hui Xu, Chao Zhu, Xiaoli Liu, Yinghua Lu, Caixia Zheng, and Jun Kong. Gaze estimation via the joint modeling of multiple cues.IEEE Transactions on Circuits and Systems for Video Technology, 32(3):1390–1402, 2021. 1

    2021

  3. [11]

    Gaze estimation using transformer

    Yihua Cheng and Feng Lu. Gaze estimation using transformer. In2022 26th International Conference on Pattern Recognition (ICPR), pages 3341–3347. IEEE, 2022. 2

    2022

  4. [12]

    Appearance-based gaze estimation with deep learning: A review and benchmark.IEEE Transactions on Pattern Analy- sis and Machine Intelligence, 46(12):7509–7528, 2024

    Yihua Cheng, Haofei Wang, Yiwei Bao, and Feng Lu. Appearance-based gaze estimation with deep learning: A review and benchmark.IEEE Transactions on Pattern Analy- sis and Machine Intelligence, 46(12):7509–7528, 2024. 1

    2024

  5. [13]

    Eunji Chong, Katha Chanda, Zhefan Ye, Audrey Souther- land, Nataniel Ruiz, Rebecca M Jones, Agata Rozga, and James M Rehg. Detecting gaze towards eyes in natural social interactions and its use in child assessment.Proceedings of the ACM on Interactive, Mobile, Wearable and Ubi...

    2017

  6. [14]

    Connecting gaze, scene, and attention: Generalized attention estimation via joint mod- eling of gaze and scene saliency

    Eunji Chong, Nataniel Ruiz, Yongxin Wang, Yun Zhang, Agata Rozga, and James M Rehg. Connecting gaze, scene, and attention: Generalized attention estimation via joint mod- eling of gaze and scene saliency. InProceedings of the Euro- pean conference on computer vision (ECCV), pa...

  7. [15]

    Detecting attended visual targets in video

    Eunji Chong, Yongxin Wang, Nataniel Ruiz, and James M Rehg. Detecting attended visual targets in video. InProceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5396–5406, 2020. 2, 1

    2020

  8. [16]

    Meta clip 2: A world- wide scaling recipe.arXiv preprint arXiv:2507.22062, 2025

    Yung-Sung Chuang, Yang Li, Dong Wang, Ching-Feng Yeh, Kehan Lyu, Ramya Raghavendra, James Glass, Lifei Huang, Jason Weston, Luke Zettlemoyer, et al. Meta clip 2: A world- wide scaling recipe.arXiv preprint arXiv:2507.22062, 2025. 2, 7

    2025 arXiv

  9. [17]

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long con- text, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261, 2025

    Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long con- text, and next generation agentic capab...

    2025 arXiv

  10. [18]

    Gazedetr: Gaze detection using disen- tangled head and gaze representations.arXiv preprint arXiv:2508.12966, 2025

    Ryan Anthony Jalova de Belen, Gelareh Mohammadi, and Arcot Sowmya. Gazedetr: Gaze detection using disen- tangled head and gaze representations.arXiv preprint arXiv:2508.12966, 2025. 2, 1

    2025 arXiv

  11. [19]

    Boosting image-based mutual gaze detection using pseudo 3d gaze

    Bardia Doosti, Ching-Hui Chen, Raviteja Vemulapalli, Xuhui Jia, Yukun Zhu, and Bradley Green. Boosting image-based mutual gaze detection using pseudo 3d gaze. InProceedings of the AAAI Conference on Artificial Intelligence, pages 1273– 1281, 2021. 1

    2021

  12. [20]

    An image is worth 16x16 words: Trans- formers for image recognition at scale.arXiv preprint arXiv:2010.11929, 2020

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, et al. An image is worth 16x16 words: Trans- formers for image recognition at scale.arXiv preprint ar...

    2010 arXiv

  13. [21]

    The eyes have it: the neuroethology, function and evolution of social gaze.Neuroscience & biobehavioral reviews, 24(6):581–604, 2000

    Nathan J Emery. The eyes have it: the neuroethology, function and evolution of social gaze.Neuroscience & biobehavioral reviews, 24(6):581–604, 2000. 2

    2000

  14. [22]

    Gaze per- formance in children with autism spectrum disorder when observing communicative actions.Journal of autism and developmental disorders, 42(10):2236–2245, 2012

    Terje Falck-Ytter, Elisabeth Fernell, Åsa Lundholm Hedvall, Claes V on Hofsten, and Christopher Gillberg. Gaze per- formance in children with autism spectrum disorder when observing communicative actions.Journal of autism and developmental disorders, 42(10):2236–2245, 2012. 1

    2012

  15. [23]

    Inferring shared attention in social scene videos

    Lifeng Fan, Yixin Chen, Ping Wei, Wenguan Wang, and Song- Chun Zhu. Inferring shared attention in social scene videos. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 6460–6468, 2018. 1

    2018

  16. [24]

    Dual attention guided gaze target detection in the wild

    Yi Fang, Jiapeng Tang, Wang Shen, Wei Shen, Xiao Gu, Li Song, and Guangtao Zhai. Dual attention guided gaze target detection in the wild. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11390–11399, 2021. 1

    2021

  17. [25]

    Llmdet: Learning strong open-vocabulary object detectors under the supervision of large language models

    Shenghao Fu, Qize Yang, Qijie Mo, Junkai Yan, Xihan Wei, Jingke Meng, Xiaohua Xie, and Wei-Shi Zheng. Llmdet: Learning strong open-vocabulary object detectors under the supervision of large language models. InProceedings of the Computer Vision and Pattern Recognition Conferenc...

    2025

  18. [26]

    Gemini api and model card

    Google DeepMind and Google. Gemini api and model card. https://ai.google.dev/gemini-api, 2025. Models: Gemini 2.5 Pro. 5

    2025

  19. [27]

    A modular multimodal architecture for gaze target prediction: Application to privacy-sensitive settings

    Anshul Gupta, Samy Tafasca, and Jean-Marc Odobez. A modular multimodal architecture for gaze target prediction: Application to privacy-sensitive settings. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5041–5050, 2022. 1

    2022

  20. [28]

    Human gaze control during real-world scene perception.Trends in cognitive sciences, 7(11):498– 504, 2003

    John M Henderson. Human gaze control during real-world scene perception.Trends in cognitive sciences, 7(11):498– 504, 2003. 1

    2003

  21. [29]

    Where are they looking in the 3d space? InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2678–2687,

    Nora Horanyi, Linfang Zheng, Eunji Chong, Aleš Leonardis, and Hyung Jin Chang. Where are they looking in the 3d space? InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2678–2687,

  22. [30]

    Gaze target estimation in- spired by interactive attention.IEEE Transactions on Circuits and Systems for Video Technology, 32(12):8524–8536, 2022

    Zhengxi Hu, Kunxu Zhao, Bohan Zhou, Hang Guo, Shichao Wu, Yuxue Yang, and Jingtai Liu. Gaze target estimation in- spired by interactive attention.IEEE Transactions on Circuits and Systems for Video Technology, 32(12):8524–8536, 2022. 2

    2022

  23. [31]

    Referring to any person

    Qing Jiang, Lin Wu, Zhaoyang Zeng, Tianhe Ren, Yuda Xiong, Yihao Chen, Liu Qin, and Lei Zhang. Referring to any person. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 21667–21678, 2025. 2, 6, 3

    2025

  24. [32]

    Multi-person gaze-following with numerical coordinate regression

    Tianlei Jin, Zheyuan Lin, Shiqiang Zhu, Wen Wang, and Shunda Hu. Multi-person gaze-following with numerical coordinate regression. In2021 16th IEEE International Con- ference on Automatic Face and Gesture Recognition (FG 2021), pages 01–08. IEEE, 2021. 2

    2021

  25. [33]

    Depth-aware gaze-following via auxiliary networks for robotics.Engineering Applications of Artificial Intelligence, 113:104924, 2022

    Tianlei Jin, Qizhi Yu, Shiqiang Zhu, Zheyuan Lin, Jie Ren, Yuanhai Zhou, and Wei Song. Depth-aware gaze-following via auxiliary networks for robotics.Engineering Applications of Artificial Intelligence, 113:104924, 2022. 2

    2022

  26. [34]

    Dinov2 meets text: A unified framework for image-and pixel-level vision- language alignment

    Cijo Jose, Théo Moutakanni, Dahyun Kang, Federico Baldas- sarre, Timothée Darcet, Hu Xu, Daniel Li, Marc Szafraniec, Michaël Ramamonjisoa, Maxime Oquab, et al. Dinov2 meets text: A unified framework for image-and pixel-level vision- language alignment. InProceedings of the Com...

  27. [35]

    Gaze360: Physically un- constrained gaze estimation in the wild

    Petr Kellnhofer, Adria Recasens, Simon Stent, Wojciech Matusik, and Antonio Torralba. Gaze360: Physically un- constrained gaze estimation in the wild. InProceedings of the IEEE/CVF international conference on computer vision, pages 6912–6921, 2019. 1

    2019

  28. [36]

    Lisa: Reasoning segmentation via large language model

    Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9579–9589, 2024. 2

    2024

  29. [37]

    Clipgaze: Zero-shot goal-directed scanpath prediction using clip

    Yantao Lai, Rong Quan, Dong Liang, and Jie Qin. Clipgaze: Zero-shot goal-directed scanpath prediction using clip. In ICASSP 2025-2025 IEEE International Conference on Acous- tics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2025. 2

    2025

  30. [38]

    Tar- gets and cues: Gaze-following in children with autism.The Journal of Child Psychology and Psychiatry and Allied Disci- plines, 39(7):951–962, 1998

    Susan R Leekam, Emma Hunnisett, and Chris Moore. Tar- gets and cues: Gaze-following in children with autism.The Journal of Child Psychology and Psychiatry and Allied Disci- plines, 39(7):951–962, 1998. 1

    1998

  31. [39]

    Believe it or not, we know what you are looking at! InAsian Conference on Computer Vision, pages 35–50

    Dongze Lian, Zehao Yu, and Shenghua Gao. Believe it or not, we know what you are looking at! InAsian Conference on Computer Vision, pages 35–50. Springer, 2018. 1

    2018

  32. [40]

    Focal loss for dense object detection

    Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. InPro- ceedings of the IEEE international conference on computer vision, pages 2980–2988, 2017. 5

    2017

  33. [41]

    Gazehta: End-to-end gaze target detection with head- target association

    Zhi-Yi Lin, Jouh Yeong Chew, Jan van Gemert, and Xucong Zhang. Gazehta: End-to-end gaze target detection with head- target association. In2025 IEEE International Conference on Robotics and Automation (ICRA), pages 9447–9454. IEEE,

  34. [42]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. InEuropean con- ference on computer vision, pages 38–55. Springe...

    2024

  35. [43]

    here’s looking at you, kid

    Manuel J Marín-Jiménez, Andrew Zisserman, and Vittorio Ferrari. " here’s looking at you, kid." detecting people looking at each other in videos. 2011. 1

    2011

  36. [44]

    Laeo-net: revisiting peo- ple looking at each other in videos

    Manuel J Marin-Jimenez, Vicky Kalogeiton, Pablo Medina- Suarez, and Andrew Zisserman. Laeo-net: revisiting peo- ple looking at each other in videos. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3477–3485, 2019. 1

    2019

  37. [45]

    Patch-level gaze distribution prediction for gaze following

    Qiaomu Miao, Minh Hoai, and Dimitris Samaras. Patch-level gaze distribution prediction for gaze following. InProceed- ings of the IEEE/CVF winter conference on applications of computer vision, pages 880–889, 2023. 2

    2023

  38. [46]

    Diffusion-refined vqa annotations for semi-supervised gaze following

    Qiaomu Miao, Alexandros Graikos, Jingwei Zhang, Sounak Mondal, Minh Hoai, and Dimitris Samaras. Diffusion-refined vqa annotations for semi-supervised gaze following. In European Conference on Computer Vision, pages 439–457. Springer, 2024. 2

    2024

  39. [47]

    Multi-view gaze target estimation

    Qiaomu Miao, Vivek Raju Golani, Jingyi Xu, Progga Paromita Dutta, Minh Hoai, and Dimitris Samaras. Multi-view gaze target estimation. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 5371–5381, 2025. 2

    2025

  40. [48]

    Simple open-vocabulary object detection

    Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Arnab, Mostafa Dehghani, Zhuoran Shen, et al. Simple open-vocabulary object detection. In European conference on computer vision, pages 728–755. S...

    2022

  41. [49]

    Scal- ing open-vocabulary object detection.Advances in Neural Information Processing Systems, 36:72983–73007, 2023

    Matthias Minderer, Alexey Gritsenko, and Neil Houlsby. Scal- ing open-vocabulary object detection.Advances in Neural Information Processing Systems, 36:72983–73007, 2023. 2, 6, 3

    2023

  42. [50]

    An overview of text-based person search: Recent advances and future directions.IEEE Transactions on Circuits and Systems for Video Technology, 34(9):7803–7819,

    Kai Niu, Yanyi Liu, Yuzhou Long, Yan Huang, Liang Wang, and Yanning Zhang. An overview of text-based person search: Recent advances and future directions.IEEE Transactions on Circuits and Systems for Video Technology, 34(9):7803–7819,

  43. [51]

    The power of human gaze on infant learning.Cognition, 128(2):127–133, 2013

    Yuko Okumura, Yasuhiro Kanakogi, Takayuki Kanda, Hiroshi Ishiguro, and Shoji Itakura. The power of human gaze on infant learning.Cognition, 128(2):127–133, 2013. 1

    2013

  44. [52]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. InInternational conference on machine learning, p...

    2021

  45. [53]

    Robust speech recog- nition via large-scale weak supervision

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recog- nition via large-scale weak supervision. InInternational con- ference on machine learning, pages 28492–28518. PMLR,

  46. [54]

    Where are they looking?Advances in neural information processing systems, 28, 2015

    Adria Recasens, Aditya Khosla, Carl V ondrick, and Antonio Torralba. Where are they looking?Advances in neural information processing systems, 28, 2015. 1, 2

    2015

  47. [55]

    Following gaze in video

    Adria Recasens, Carl V ondrick, Aditya Khosla, and Antonio Torralba. Following gaze in video. InProceedings of the IEEE International Conference on Computer Vision, pages 1435–1443, 2017. 2, 1

    2017

  48. [56]

    Behavioral imaging and autism.IEEE Pervasive Computing, 13(2):84–87, 2014

    James M Rehg, Agata Rozga, Gregory D Abowd, and Matthew S Goodwin. Behavioral imaging and autism.IEEE Pervasive Computing, 13(2):84–87, 2014. 8

    2014

  49. [57]

    Generalized in- tersection over union: A metric and a loss for bounding box regression

    Hamid Rezatofighi, Nathan Tsoi, JunYoung Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese. Generalized in- tersection over union: A metric and a loss for bounding box regression. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 658–666,

  50. [58]

    Gaze-lle: Gaze target estima- tion via large-scale learned encoders

    Fiona Ryan, Ajay Bati, Sangmin Lee, Daniel Bolya, Judy Hoffman, and James M Rehg. Gaze-lle: Gaze target estima- tion via large-scale learned encoders. 2025. 1, 2, 6, 3

    2025

  51. [59]

    Di- nov3.arXiv preprint arXiv:2508.10104, 2025

    Oriane Siméoni, Huy V V o, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Marc Szafraniec, Seungeun Yi, Michaël Ramamonjisoa, et al. Di- nov3.arXiv preprint arXiv:2508.10104, 2025. 2, 4, 7

    2025 arXiv

  52. [60]

    Vitgaze: gaze following with interaction features in vision transformers.Visual Intelli- gence, 2(1):1–15, 2024

    Yuehao Song, Xinggang Wang, Jingfeng Yao, Wenyu Liu, Jinglin Zhang, and Xiangmin Xu. Vitgaze: gaze following with interaction features in vision transformers.Visual Intelli- gence, 2(1):1–15, 2024. 1, 2, 6, 3

    2024

  53. [61]

    Child- play: A new benchmark for understanding children’s gaze behaviour

    Samy Tafasca, Anshul Gupta, and Jean-Marc Odobez. Child- play: A new benchmark for understanding children’s gaze behaviour. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 20935–20946, 2023. 2, 1

    2023

  54. [62]

    Toward semantic gaze target detection.Advances in Neural Information Processing Systems, 37:121422–121448,

    Samy Tafasca, Anshul Gupta, Victor Bros, and Jean-Marc Odobez. Toward semantic gaze target detection.Advances in Neural Information Processing Systems, 37:121422–121448,

  55. [63]

    Sharingan: A transformer architecture for multi-person gaze following

    Samy Tafasca, Anshul Gupta, and Jean-Marc Odobez. Sharingan: A transformer architecture for multi-person gaze following. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2008–2017,

    2008

  56. [64]

    Goo: A dataset for gaze object prediction in retail environments

    Henri Tomas, Marcus Reyes, Raimarc Dionido, Mark Ty, Jonric Mirando, Joel Casimiro, Rowel Atienza, and Richard Guinto. Goo: A dataset for gaze object prediction in retail environments. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3...

    2021

  57. [65]

    Multi- modal across domains gaze target detection

    Francesco Tonini, Cigdem Beyan, and Elisa Ricci. Multi- modal across domains gaze target detection. InProceedings of the 2022 International Conference on Multimodal Interac- tion, pages 420–431, 2022. 2, 1

    2022

  58. [66]

    Object-aware gaze target detection

    Francesco Tonini, Nicola Dall’Asen, Cigdem Beyan, and Elisa Ricci. Object-aware gaze target detection. InProceed- ings of the IEEE/CVF international conference on computer vision, pages 21860–21869, 2023. 2, 1

    2023

  59. [67]

    Siglip 2: Multilingual vision-language en- coders with improved semantic understanding, localization, and dense features.arXiv preprint arXiv:2502.14786, 2025

    Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muham- mad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: Multilingual vision-language en- coders with improved semantic understanding, localization, ...

    2025 arXiv

  60. [68]

    End-to-end human-gaze-target detection with transformers

    Danyang Tu, Xiongkuo Min, Huiyu Duan, Guodong Guo, Guangtao Zhai, and Wei Shen. End-to-end human-gaze-target detection with transformers. In2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2192–2200. IEEE, 2022. 2, 1

    2022

  61. [69]

    Gazeclip: Towards enhanc- ing gaze estimation via text guidance.arXiv preprint arXiv:2401.00260, 2023

    Jun Wang, Hao Ruan, Mingjie Wang, Chuanghui Zhang, Huachun Li, and Jun Zhou. Gazeclip: Towards enhanc- ing gaze estimation via text guidance.arXiv preprint arXiv:2401.00260, 2023. 2

    2023 arXiv

  62. [70]

    Demystifying clip data.arXiv preprint arXiv:2309.16671, 2023

    Hu Xu, Saining Xie, Xiaoqing Ellen Tan, Po-Yao Huang, Rus- sell Howes, Vasu Sharma, Shang-Wen Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. Demystifying clip data.arXiv preprint arXiv:2309.16671, 2023. 2

    2023 arXiv

  63. [71]

    Contextagent: Context-aware proac- tive llm agents with open-world sensory perceptions.arXiv preprint arXiv:2505.14668, 2025

    Bufang Yang, Lilin Xu, Liekang Zeng, Kaiwei Liu, Siyang Jiang, Wenrui Lu, Hongkai Chen, Xiaofan Jiang, Guoliang Xing, and Zhenyu Yan. Contextagent: Context-aware proac- tive llm agents with open-world sensory perceptions.arXiv preprint arXiv:2505.14668, 2025. 7

    2025

  64. [72]

    Gaze target detection based on head-local-global coordination

    Yaokun Yang and Feng Lu. Gaze target detection based on head-local-global coordination. InEuropean Conference on Computer Vision, pages 305–322. Springer, 2024. 2

    2024

  65. [73]

    Gaze target detection by merging human attention and activity cues

    Yaokun Yang, Yihan Yin, and Feng Lu. Gaze target detection by merging human attention and activity cues. InProceedings of the AAAI Conference on Artificial Intelligence, pages 6585– 6593, 2024. 2

    2024

  66. [74]

    Detclip: Dictionary-enriched visual-concept paralleled pre- training for open-world detection.Advances in Neural Infor- mation Processing Systems, 35:9125–9138, 2022

    Lewei Yao, Jianhua Han, Youpeng Wen, Xiaodan Liang, Dan Xu, Wei Zhang, Zhenguo Li, Chunjing Xu, and Hang Xu. Detclip: Dictionary-enriched visual-concept paralleled pre- training for open-world detection.Advances in Neural Infor- mation Processing Systems, 35:9125–9138, 2022. 2

    2022

  67. [75]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. InProceedings of the IEEE/CVF international conference on computer vision, pages 11975–11986, 2023. 2

    2023

  68. [76]

    Gazeonce: Real-time multi-person gaze estimation

    Mingfang Zhang, Yunfei Liu, and Feng Lu. Gazeonce: Real-time multi-person gaze estimation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4197–4206, 2022. 2

    2022

  69. [77]

    Hu- man gaze assisted artificial intelligence: A review

    Ruohan Zhang, Akanksha Saran, Bo Liu, Yifeng Zhu, Sihang Guo, Scott Niekum, Dana Ballard, and Mary Hayhoe. Hu- man gaze assisted artificial intelligence: A review. InIJCAI: Proceedings of the Conference, page 4951, 2020. 2

    2020

  70. [78]

    Text-based person search in full images via semantic-driven proposal generation

    Shizhou Zhang, De Cheng, Wenlong Luo, Yinghui Xing, Duo Long, Hao Li, Kai Niu, Guoqiang Liang, and Yanning Zhang. Text-based person search in full images via semantic-driven proposal generation. InProceedings of the 4th International Workshop on Human-centric Multimedia Analys...

    2023

  71. [79]

    Tcnet: Gaze estimation based on temporal body-head-eyes correlation in dynamic scenes.IEEE Sensors Journal, 2025

    Wulue Zhang, Jianbin Xiong, Xiangjun Dong, Qi Wang, and Weikun Dai. Tcnet: Gaze estimation based on temporal body-head-eyes correlation in dynamic scenes.IEEE Sensors Journal, 2025. 1

    2025

  72. [80]

    Appearance-based gaze estimation in the wild

    Xucong Zhang, Yusuke Sugano, Mario Fritz, and Andreas Bulling. Appearance-based gaze estimation in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4511–4520, 2015. 2

    2015

  73. [81]

    Sec: Advancing complex video object segmentation via progressive concept construction.arXiv preprint arXiv:2507.15852, 2025

    Zhixiong Zhang, Shuangrui Ding, Xiaoyi Dong, Songxin He, Jianfan Lin, Junsong Tang, Yuhang Zang, Yuhang Cao, Dahua Lin, and Jiaqi Wang. Sec: Advancing complex video object segmentation via progressive concept construction.arXiv preprint arXiv:2507.15852, 2025. 2

    2025

  74. [82]

    Learning to draw sight lines.International Journal of Computer Vision, 128(5):1076–1100, 2020

    Hao Zhao, Ming Lu, Anbang Yao, Yurong Chen, and Li Zhang. Learning to draw sight lines.International Journal of Computer Vision, 128(5):1076–1100, 2020. 1

    2020

  75. [83]

    Detecting twenty-thousand classes using image-level supervision

    Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp Krähen- bühl, and Ishan Misra. Detecting twenty-thousand classes using image-level supervision. InEuropean conference on computer vision, pages 350–368. Springer, 2022. 2 Gaze Target Estimation Anywhere with Concepts Supplemen...

    2022

  76. [84]

    the child in the blue shirt

    Further Discussions & Social Impact 8.1. Toward End-to-end Gaze Target Estimation The evolution of human gaze estimation shows a clear trend: a move away from complex auxiliary features like pose and depth towards streamlined, head box-only inputs [ 10, 14, 19, 23, 24, 27, 35,...

  77. [85]

    Figure 6 illus- trate the workflow of the agent

    GazeAnywhere Agent In this section, we introduce the GazeAnywhere Agent, a vi- sual agentic framework designed to process natural-language gaze estimation and post-analysis requests. Figure 6 illus- trate the workflow of the agent. The system dynamically queries a MLLM to orch...

  78. [86]

    Training Set The Gaze-Co training set contains 119,525 samples in to- tal

    Dataset & Benchmark 10.1. Training Set The Gaze-Co training set contains 119,525 samples in to- tal. Each record includes the target head bounding box, normalized gaze point, an in/out-of-frame label, and a com- pact concept phrase (attribute, position, action, and pose). The ...

  79. [87]

    Call whisper-large-v3 to extract the audio-caption

  80. [88]

    There are four people in the scene, the girl is wearing a leopard print top, and she has long blonde hair with a silver headband

    The user ask us to analyze the gaze shift of the closet girl. There are four people in the scene, the girl is wearing a leopard print top, and she has long blonde hair with a silver headband. She is in the right of the stream images. Call AnyGaze to visualize the target with i...

  81. [89]

    Smart Glasses Screen Gaze shift: 2 whisper Query AnyGaze Agent AnyGaze Agent AnyGaze Agent AnyGaze Agent GazeAnywhere Analysis Figure 6

    Analysis with the visualized video stream. Smart Glasses Screen Gaze shift: 2 whisper Query AnyGaze Agent AnyGaze Agent AnyGaze Agent AnyGaze Agent GazeAnywhere Analysis Figure 6. Step-by-step explanation of how GazeAnywhere Agent works. (a) Source dataset distribution (b) App...

  82. [90]

    Open-Vocabulary Detector (OVD) As baselines, we use the OVD models to locate the target person described by a text prompt

    Baseline Details 11.1. Open-Vocabulary Detector (OVD) As baselines, we use the OVD models to locate the target person described by a text prompt. This step supports our main task: to predict the point of view of the subject. Each OVD model takes an image and a prompt, matches ...

  83. [91]

    AR Device for GazeAnywhere Agent We use DigiLens ARGO in the experiment to capture video data in real-world settings (Fig

    Experimental Protocol 12.1. AR Device for GazeAnywhere Agent We use DigiLens ARGO in the experiment to capture video data in real-world settings (Fig. 8). Its 48 MP camera records high-resolution video with autofocus, optical and electronic Figure 8. DigiLens ARGO AR glasses u...

  84. [92]

    Impact of Frozen Encoder

    More Results 13.1. Impact of Frozen Encoder. A key design choice for GazeAnywhere is to keep the im- age and text encoders frozen. We validate this approach in Table 7, which compares the default frozen model against one where the DINOv3 image encoder or the text encoder are f...

  85. [93]

    Although Gaze-LLE performs well in sparse scenes with only one or two individuals, its performance degrades noticeably as crowd density increases

    Qualitative Analysis In Figure 9, we qualitatively compare GazeAnywhere with the current state-of-the-art model, Gaze-LLE. Although Gaze-LLE performs well in sparse scenes with only one or two individuals, its performance degrades noticeably as crowd density increases. As show...

  86. [94]

    Related Prompts For reproducibility, we include the exact natural-language prompts used to query the MLLM in our pipeline. These prompts support three major components: the concept- generation data engine, the MLLM-only gaze prediction baseline, and the GazeAnywhere Agent for ...

  87. [95]

    what is being done

    Notations We present the description all the notations in our paper in the last two pages. Concept Generation Prompt TASK Return a description for the person with a green bounding box in head: The description is a natural, concise attribute phrase (<30 words in total). STYLE &...

  88. [96]

    Output 1 if the gaze target lies within the image frame

    In-frame gaze flag Indicate whether the subject person is looking at a target inside the image frame. Output 1 if the gaze target lies within the image frame. Output 0 if the subject person is looking outside the image frame

  89. [97]

    in_frame_gaze

    Gaze target point Predict the gaze target of the subject person as a point(x, y)∈[0,1], with exactly three decimal places for both x and y. The values must be normalized by the image width and height. Output format Return only a valid JSON object, with no extra text, in the fo...

  90. [99]

    Total gaze shift count

    "Total gaze shift count": The number of gaze shifts happened. (change the gaze target to another object or out-of-frame) Hint: Gaze shifting is the coordinated movement of the eyes and head to look at a new target. Per gaze shift means changing the gaze target from one object/...

  91. [100]

    Gaze shift to social partner

    "Gaze shift to social partner": The number of gaze shifts to the nearby person happened

  92. [101]

    Total gaze shift count

    "Total gaze shift count": The number of gaze shifts happened. (change the gaze target to another object or out-of-frame) Hint: Gaze shifting is the coordinated movement of the eyes and head to look at a new target. Our green point in the frame can indicate the target location....

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.