{"id":"2e35526c-0c33-4f3f-ac26-27cad12fda82","arxiv_id":"2608.11367","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"The paper introduces Promptable Gaze Target Estimation (PGE), builds the Gaze-Co dataset of 120K concept-annotated images, and proposes GazeAnywhere, a frozen-encoder transformer that estimates gaze heatmaps from text or point prompts.","lead":"A new method lets users identify a person in an image by typing a description, such as \"the girl in the red shirt,\" and then estimates where that person is looking. The authors built a 120K-image dataset and a promptable model that outperforms pipeline-based gaze estimators on several benchmarks, including a private real-world clinical set.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The benchmark prompt itself may leak the gaze target: Section 4.1's concept-generation rule asks for action 'with object or direction' and pose 'facing direction,' so GazeAnywhere can read gaze cues from text while two-stage baselines cannot.","rationale":"The reader's conditional verdict is reasonable, but the weakest assumption can be sharpened. It is not merely that Gemini-style phrases may fit dino.txt better; the phrase-generation instructions may encode the answer to the task. Since the PGE benchmark is self-created and the SOTA claim is the central contribution, this is the most load-bearing point. The paper's own data-engine prompts (Appendix 15.1, Figures 10 and 11) contain no exclusion of gaze-revealing content, and the verification step checks consistency with the head box, not whether the phrase names the gaze target. The architecture's cross-modal attention over text tokens makes such leakage exploitable. The concrete test separates the two hypotheses: if removing action/pose leaves the margin intact, the benchmark is clean and the reader's concern remains a distributional caveat; if the margin collapses, the SOTA claim must be re-scoped. No formal verification exists, but the open-source release and honest ablation design count in the paper's favor; the issue is one of benchmark validity, not internal inconsistency. Both the reader and I arrive at CONDITIONAL; no verdict change is needed, but the condition should explicitly include a leakage audit of the test prompts.","tokens_in":25040,"tokens_out":12384,"duration_ms":117849,"concrete_test":"Take a random sample of 200 human-verified test prompts from GazeFollow-Concept, VAT-Concept, and ChildPlay-Concept and have two independent annotators classify each action and pose field for (a) gaze/attention verbs such as 'look,' 'watch,' or 'stare' whose object is a visible scene element, or (b) directional phrases pointing at a target object or person. If more than a pre-registered threshold (for example, 5%) of prompts contain such target-revealing content, re-run the full Table 1 comparison with the action and pose fields removed, keeping only appearance and position. If GazeAnywhere's margin over the best OVD baseline (which never sees action/pose in its gaze stage) shrinks substantially on Avg L2/AP, the headline SOTA claim depends on leaked gaze cues in the prompt.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing weakness is in the construction of the Gaze-Co test prompts (Section 4.1 and Appendix 15.1). Concept generation explicitly instructs the MLLM to include 'target/object/direction when visible' in the action field and 'facing direction' in the pose field. Nothing in the generation or verification prompts prohibits gaze verbs or target-naming phrases such as 'looking at the ball' or 'facing the clinician.' The human and MLLM verification checks only that the concept matches the designated head box and has four non-conflicting fields, not that the phrase avoids encoding the gaze target. GazeAnywhere's detector transformer concatenates text tokens with visual tokens, so such phrases can directly bias the heatmap. The OVD baselines consume the same text only to emit a head box; their gaze stages never see the action/pose text, so they cannot exploit this information. The reported comparison is therefore not a pure test of gaze-target estimation: part of the SOTA margin may reflect prompt leakage rather than better gaze inference. Table 3 is consistent with this reading: action-only and pose-only prompts improve over no prompting on GazeFollow-Concept Avg L2 (0.129 and 0.121 vs 0.144), which is more than subject-disambiguation alone would predict if those fields were inert. This concern is independent of the model's internal correctness; it targets the benchmark validity on which the SOTA claim rests.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Promptable Gaze Target Estimation (PGE), a paradigm in which a gaze-following model is conditioned on a natural-language or visual concept prompt identifying the subject, and proposes GazeAnywhere, an end-to-end transformer-based model that fuses frozen image and text encoders to jointly predict the subject's head box, in/out-of-frame presence, and gaze heatmap. To support training and evaluation, the authors build the Gaze-Co dataset (about 120K samples) by converting GazeFollow, VAT, and ChildPlay into a concept-annotated format, and evaluate on the resulting GazeFollow-Concept, VAT-Concept, ChildPlay-Concept benchmarks plus a private child social-communication (Child-SC) out-of-domain set. The paper reports state-of-the-art results against two-stage OVD-plus-gaze-model baselines and against VLM baselines, and includes ablations on prompting, losses, encoders, and detector dimensions, as well as an AR agent demonstration.","tokens_in":25368,"tokens_out":5459,"duration_ms":60505,"significance":"If the empirical claims hold, this is a meaningful step: GazeAnywhere is the first promptable end-to-end gaze-target model that removes the head-box and pose dependencies of prior pipelines, and the Gaze-Co data engine is a reusable contribution with exact prompts and a human-in-the-loop verification protocol. The paper is also commendable for open-sourcing the code, for pairing multiple OVDs with multiple gaze models rather than a single straw-man baseline, and for including a real-world agent evaluation. The central issue is that the reported state-of-the-art margin is only as credible as the concept-prompt benchmark, and that benchmark currently has a potential information-leak problem that must be resolved before the comparison can be interpreted as a clean test of gaze-target inference.","major_comments":[{"comment":"The benchmark prompts are constructed in a way that can leak the gaze target into the text given to GazeAnywhere but not to the two-stage baselines. The concept-generation prompt in Fig. 10 explicitly instructs the MLLM to add 'target/object/direction when visible' to the action field and to include 'facing direction' in the pose field, and the verification prompt in Fig. 11 only checks that the fields match the designated head box, are present, and are non-conflicting; it never prohibits gaze verbs or target-naming phrases such as 'looking at the ball' or 'facing the clinician.' Because GazeAnywhere concatenates the projected text tokens with the visual tokens in the detector transformer (Eq. 5), such phrases can directly bias the heatmap prediction, whereas in the OVD baselines the same text is used only to emit a head box and the gaze stage never sees the action/pose text. Table 3 is consistent with this concern: action-only and pose-only prompts improve Avg L2 from 0.144 (no prompting) to 0.129 and 0.121 on GazeFollow-Concept, which suggests these fields carry information beyond subject disambiguation. As written, the SOTA comparison does not cleanly isolate gaze-target inference from prompt leakage; the authors should either verify that the generated and human-verified prompts avoid gaze-bearing language, or re-run the comparison on appearance-only and leakage-controlled prompts.","section":"§4.1, Appendix §15.1 (Figs. 10–11)"},{"comment":"The SOTA claim rests on small margins reported without error bars, seed variance, or significance tests. For example, on GazeFollow-Concept Avg L2 GazeAnywhere-DINOv3-L scores 0.099 versus 0.097 for ViTGaze with ground-truth head boxes and 0.099 for Gaze-LLE with ground-truth boxes, and on VAT-Concept L2 it scores 0.123 versus 0.121 for RexSeek; several ablation rows differ by only 0.001–0.005 (Tables 4, 8, and 9). Without repeated runs, confidence intervals, or statistical testing, it is not possible to determine which of these differences are genuine, especially for the claim of state-of-the-art status on the public benchmarks.","section":"§5.2, Table 1"},{"comment":"The out-of-domain evaluation on Child-SC is a central piece of evidence for generalization, but the dataset is private, the manual annotations were produced without the MLLM verification pipeline, and no derived annotations or de-identified features are released. The reported L2 of 0.090 and AP of 0.902 on this set cannot be independently checked or compared by other groups. The authors should release at least a de-identified subset or a reproducible public surrogate with the same protocol, or explicitly label the Child-SC numbers as non-reproducible evidence rather than a benchmark result.","section":"§4.2, §10.3"}],"minor_comments":[{"comment":"The paragraph after Eq. (5) describes the input sequence as having NT + NV + 2 tokens, but the definition of NT is not stated; the paper should distinguish the number of content text tokens from the fixed padded context length LT.","section":"§3.2"},{"comment":"The claimed ≤1% field error rate from human spot-checks would be much more informative with the number of spot-checked samples, the inter-annotator agreement, and a few examples of failed cases.","section":"§4.1"},{"comment":"The 'no prompting' row is ambiguous because it is not clear how the model identifies the subject without a text or visual prompt; the caption should specify the exact input used in that condition.","section":"Table 3"},{"comment":"The notation list includes E_head but Eq. (3) defines the head token directly as t'_eos; either use E_head in the formula or remove it from the notation list.","section":"Appendix §16"},{"comment":"The verification prompt instructs the MLLM to ignore the <count> field, yet the concept-generation output format includes it; the role of the people-count field in the released dataset and benchmarks should be stated.","section":"Appendix §15.1"},{"comment":"There is a typo, 'Transfomrer' for 'Transformer', in the detector-transformer paragraph.","section":"§3.2"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper defines a new task — promptable gaze target estimation (PGE) — and delivers a large dataset, Gaze-Co, and a working model, GazeAnywhere. The task is genuinely new: prior gaze-following methods need head boxes or pose and cannot be queried by text. The model combines frozen vision and text encoders with a detector transformer to jointly output a head box, in/out-presence, and gaze heatmap from a text or visual prompt. The ablations are informative, and the code and data are open-sourced.\n\nThe main empirical claim has a load-bearing soft spot. The benchmark's concept prompts may leak the gaze target. The generation prompt in Appendix 15.1 tells the MLLM to include 'target/object/direction when visible' in the action field and 'facing direction' in the pose field. The verification check does not prohibit gaze-encoding phrases such as 'looking at the ball.' GazeAnywhere sees the full text in its transformer; the two-stage baselines use the text only to emit a head box and never hand it to the gaze stage. So part of the reported SOTA margin could come from reading the answer in the prompt rather than from better gaze inference. Table 3 fits this reading: action-only and pose-only prompts improve over no prompting (0.129 and 0.121 vs 0.144 Avg L2 on GazeFollow-Concept), more than subject disambiguation alone would predict.\n\nThis does not sink the task definition or the architecture, but it does mean the headline benchmark number is not clean. The authors should rerun the comparison after filtering gaze-related phrases from the prompts, or give baselines the same text in their gaze stage.\n\nOther soft spots are minor: no error bars, the out-of-domain set is private, and the OVD baselines use single best-box selection. None of these are fatal.\n\nThe paper is for anyone in gaze following, promptable perception, or human behavior analysis. The dataset and task are valuable even if the current leaderboard needs a caveat.\n\nSend it to peer review, but condition acceptance on addressing the prompt leakage. The flaw is fixable, and the contribution deserves referee time.","headline":"Genuinely new task and dataset, but the benchmark's prompt construction may leak gaze targets to GazeAnywhere, inflating the SOTA claim.","tokens_in":25885,"tokens_out":3386,"would_cite":true,"duration_ms":27068,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GazeAnywhere makes gaze estimation promptable: a text or point prompt replaces head boxes and pose priors in a single end-to-end model.","keywords":["gaze target estimation","promptable gaze estimation","concept-based prompting","gaze-following","open-vocabulary detection","transformer detector","Gaze-Co dataset","social gaze analysis"],"falsifier":"Take a held-out image set with prompts written independently by different human annotators (not the MLLM that generated Gaze-Co), and compare GazeAnywhere's margin over two-stage open-vocabulary-detector baselines; if the margin shrinks or disappears, the reported advantage depends on prompt wording rather than architectural superiority.","tokens_in":24874,"feed_emoji":"👀","tokens_out":6785,"duration_ms":56194,"temperature":0.7,"pith_summary":"This paper tries to replace brittle, multi-stage gaze-following pipelines with one promptable system. It defines the new task of Promptable Gaze Target Estimation (PGE), where the user identifies the person of interest by a short text phrase or a spatial point, and the model must locate that person and predict where they are looking in a single pass. To make this possible, the authors build a 120K-image dataset of concept descriptions and a model called GazeAnywhere. They report that GazeAnywhere outperforms two-stage baselines that chain open-vocabulary detectors to gaze models, including on an out-of-domain clinical dataset of children. If the finding holds, gaze analysis becomes usable by nonexperts in clinics, homes, and augmented-reality devices: describing a person replaces drawing head boxes.","feed_headline":"Text prompts replace head boxes for gaze tracking","feed_subtitle":"End-to-end model finds a described person and their gaze, beating two-stage pipelines on public and clinical benchmarks.","key_machinery":"The load-bearing mechanism is a cross-modal detector transformer that concatenates projected visual patch tokens, text content tokens, a learnable head token initialized from the text EOS embedding, and a target-presence token initialized from the visual CLS embedding into one token sequence. The head token drives an auxiliary head-box head, the presence token drives an in/out-of-frame focal-loss classifier, and the visual patch tokens are reshaped and upsampled into the gaze heatmap. The joint objective is a BCE heatmap loss, a focal presence loss, and an L1 plus GIoU head-box loss; ablations show the head loss helps gaze estimation while the presence loss mainly supports the in/out output. Equally central is the Gaze-Co data engine: it aligns and filters source data, generates four-field concept phrases (appearance, position, action, pose), and verifies them with an MLLM plus human spot-checks, all without exposing private clinical data to cloud processing.","core_discovery":"The central claim is that gaze target estimation can be reformulated as an end-to-end, concept-conditioned task. Given an image and a prompt such as “the boy in the red shirt” or “person at point [0.52, 0.48]”, a single model can localize the subject’s head, decide whether the gaze target is inside the frame, and produce a gaze heatmap—with no auxiliary head detector, pose estimator, or depth module. The authors define this as the Promptable Gaze Target Estimation task, contribute the Gaze-Co dataset by retrofitting the GazeFollow, VideoAttentionTarget, and ChildPlay splits with concept annotations, and introduce GazeAnywhere, a transformer-based detector that fuses frozen vision and text encoders. They report state-of-the-art results on all public PGE benchmarks and on a private out-of-domain dataset of child social-communication videos, and they demonstrate an augmented-reality agent in which an MLLM calls GazeAnywhere as a tool to answer high-level gaze questions.","pith_inferences":["If the prompt robustness holds, the same frozen-encoder-plus-detector recipe could be lifted to other fine-grained social cues such as pointing, mutual gaze, or attention to specific objects.","A valuable stress test would rephrase each prompt with synonyms and reorder the attribute fields; the paper does not report sensitivity to wording, and large sensitivity would bound how deeply “concept-driven” the model really is.","The human-in-the-loop data engine could retrofit other behavioral video datasets with concept annotations, potentially scaling PGE from still images to continuous social interaction.","The Gaze-Co concept style—appearance, position, action, and pose—may itself bias downstream models toward stable visual attributes; testing with action- or pose-only prompts would clarify which cues carry the gaze signal."],"forward_implications":["Two-stage gaze pipelines no longer need a separate head detector at inference: subject localization and gaze estimation are solved jointly from the prompt.","Users can query a specific person in a crowd by describing appearance or by pointing at a coordinate, making gaze analysis practical for clinicians, parents, and AR interfaces.","The Gaze-Co benchmark provides a standardized, human-verified format for evaluating concept-conditioned gaze on GazeFollow, VideoAttentionTarget, and ChildPlay splits.","The out-of-domain Child-SC results suggest the promptable formulation can generalize to real-world clinical video where open-vocabulary head detection is unreliable.","An AR agent built on GazeAnywhere reduces errors in gaze-shift and eye-contact counts compared with a raw multimodal LLM, indicating a path toward deployable behavioral-analysis tools."],"supporting_citations":[{"why":"Provides the GazeFollow dataset, the largest source of Gaze-Co training samples and the source of multiple gaze annotations used for Avg L2 and Min L2 metrics.","marker":"[54]"},{"why":"Provides the VideoAttentionTarget dataset, contributing video-based gaze samples with in/out-of-frame labels converted into PGE format.","marker":"[15]"},{"why":"Provides the ChildPlay dataset, contributing children's gaze-behavior samples converted into PGE format.","marker":"[61]"},{"why":"Gaze-LLE is one of the state-of-the-art gaze models used in the two-stage baseline comparisons.","marker":"[58]"},{"why":"ViTGaze is one of the state-of-the-art gaze models used in the two-stage baseline comparisons.","marker":"[60]"},{"why":"Sharingan is one of the state-of-the-art gaze models used in the two-stage baseline comparisons.","marker":"[63]"},{"why":"GroundingDINO is an open-vocabulary detector used to produce head boxes for the two-stage baselines.","marker":"[42]"},{"why":"OWLv2 is another open-vocabulary detector used to produce head boxes for the two-stage baselines.","marker":"[49]"},{"why":"RexSeek is the strongest referring-expression baseline used to localize prompted subjects for comparison.","marker":"[31]"},{"why":"DINOv3 supplies the frozen visual encoder used in the best-performing GazeAnywhere configuration.","marker":"[59]"}],"fun_headline_variants":["Say who, it finds their gaze target","Text prompts replace head boxes for gaze","One model, any described person, gaze heatmap","End-to-end gaze tracking from concept prompts","Prompt the subject, then predict the gaze"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the concept annotations in Gaze-Co are accurate and unbiased enough that the measured advantage of GazeAnywhere over two-stage baselines reflects model design rather than a prompt distribution tailored to its text encoder.","fun_headline_variants_meta":{"raw":{"variants":["Say who, it finds their gaze target","Text prompts replace head boxes for gaze","One model, any described person, gaze heatmap","End-to-end gaze tracking from concept prompts","Prompt the subject, then predict the gaze"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000268,"raw_usage":{"total_tokens":1677,"prompt_tokens":1065,"completion_tokens":612,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":681,"completion_tokens_details":{"reasoning_tokens":545}},"tokens_in":681,"tokens_out":612,"duration_ms":5993,"temperature":1.0,"reasoning_tokens":545,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:12:07.958699+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a held-out image set with prompts written independently by different human annotators (not the MLLM that generated Gaze-Co), and compare GazeAnywhere's margin over two-stage open-vocabulary-detector baselines; if the margin shrinks or disappears, the reported advantage depends on prompt wording rather than architectural superiority.","supporting_citations":[{"cited_title":"Where are they looking?Advances in neural information processing systems, 28, 2015","cited_arxiv_id":null,"evidence_quote":"Provides the GazeFollow dataset, the largest source of Gaze-Co training samples and the source of multiple gaze annotations used for Avg L2 and Min L2 metrics."},{"cited_title":"Child- play: A new benchmark for understanding children’s gaze behaviour","cited_arxiv_id":null,"evidence_quote":"Provides the ChildPlay dataset, contributing children's gaze-behavior samples converted into PGE format."},{"cited_title":"Gaze-lle: Gaze target estima- tion via large-scale learned encoders","cited_arxiv_id":null,"evidence_quote":"Gaze-LLE is one of the state-of-the-art gaze models used in the two-stage baseline comparisons."},{"cited_title":"Vitgaze: gaze following with interaction features in vision transformers.Visual Intelli- gence, 2(1):1–15, 2024","cited_arxiv_id":null,"evidence_quote":"ViTGaze is one of the state-of-the-art gaze models used in the two-stage baseline comparisons."},{"cited_title":"Sharingan: A transformer architecture for multi-person gaze following","cited_arxiv_id":null,"evidence_quote":"Sharingan is one of the state-of-the-art gaze models used in the two-stage baseline comparisons."},{"cited_title":"Grounding dino: Marrying dino with grounded pre-training for open-set object detection","cited_arxiv_id":null,"evidence_quote":"GroundingDINO is an open-vocabulary detector used to produce head boxes for the two-stage baselines."},{"cited_title":"Scal- ing open-vocabulary object detection.Advances in Neural Information Processing Systems, 36:72983–73007, 2023","cited_arxiv_id":null,"evidence_quote":"OWLv2 is another open-vocabulary detector used to produce head boxes for the two-stage baselines."},{"cited_title":"Referring to any person","cited_arxiv_id":null,"evidence_quote":"RexSeek is the strongest referring-expression baseline used to localize prompted subjects for comparison."}],"review_version":1}