Pith. sign in

REVIEW 3 major objections 6 minor 121 references

CoMind is a multimodal cooking-collaboration dataset and three Theory-of-Mind vision tasks that show current vision-language models lack social grounding, while fine-tuning on the data substantially closes the gap.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-10 23:18 UTC pith:Q2VH56CY

load-bearing objection Strong multi-agent ego-exo cooking resource with social-cue labels and three hard prediction tasks; the ToM framing is marketing, not a load-bearing flaw. the 3 major comments →

arxiv 2607.06691 v1 pith:Q2VH56CY submitted 2026-07-07 cs.CV

CoMind: Understanding Collaborative Human Activity from Multiple Minds and Views

Alexey Gavryushin , Dingxi Zhang , Zhao Huang , Alexandros Delitzas , Jiaqi Chen , Ben Ellis , Cedric Z\"ollner , Manthan Patel
show 3 more authors
Manuel Kaufmann Marc Pollefeys Xi Wang
This is my paper
classification cs.CV
keywords egocentric visiontheory of mindhuman-human collaborationjoint attentionaction anticipationobject handovermultimodal datasetsocial cues
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that Theory of Mind—the ability to infer others’ intentions from social cues—is essential for real collaboration yet almost unmeasurable in natural settings because prior datasets and benchmarks stay textual, single-agent, or short-horizon. CoMind fills that gap with 41 hours of synchronized egocentric and exocentric video, audio, gaze, hand tracking, and 3D scans of people cooking together, plus dense annotations of shared attention, social cues, and object interactions. Three escalating tasks are defined: estimating joint attention on an object, anticipating a helper’s next socially conditioned action, and predicting a handover before any reach begins. Zero-shot vision-language models score near zero on spatial localization and only modestly on cue and category recognition; fine-tuning open-weight models on CoMind training splits produces large gains, especially on action verbs and initiator identity. The claim is that this resource is now the practical foundation for training and evaluating AI that can read social signals and assist proactively in shared physical work.

Core claim

Current state-of-the-art vision-language models exhibit severe deficiencies on three new social-reasoning tasks grounded in real dual-person cooking collaboration—particularly near-zero accuracy on bounding-box localization of jointly attended or soon-to-be-handed objects—while fine-tuning the same open-weight models on CoMind’s training split yields large, consistent lifts (for example action-verb accuracy rising from roughly 0.09–0.14 to 0.64–0.65), establishing the dataset as a usable training and evaluation foundation for socially aware AI.

What carries the argument

Three interdependent vision tasks that operationalize Theory of Mind for physical collaboration: Joint Attention Estimation (shared object, dual-view boxes, cue type), Socially Conditioned Object Interaction Anticipation (helper’s next verb-noun-box given leader cues), and Collaborative Handover Prediction (time-to-handover, delivery flow, initiator, cue, object box before any reach).

Load-bearing premise

That success on these three hand-crafted vision tasks is a valid stand-in for Theory of Mind in real collaboration, without independent cognitive or behavioral validation that the tasks measure mental-state inference rather than pattern matching of surface cues.

What would settle it

Train models to high accuracy on the three CoMind tasks, then test whether the same models correctly predict a held-out partner’s next need or successful assistance in a new, unscripted kitchen session whose social-cue distribution differs from the training kitchens; failure of transfer would falsify the claim that the tasks capture general collaborative ToM.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Multimodal perception systems can be trained to detect joint attention and social cues from synchronized first- and third-person video plus gaze and speech.
  • Proactive assistive agents can be scored on whether they correctly anticipate a partner’s next object interaction or handover before physical motion begins.
  • Collaborative planning models gain temporally aligned, 3D-grounded training data for long-horizon kitchen tasks that include verbal and gestural intent.
  • Open-weight vision-language models can be domain-adapted for social grounding, turning near-zero spatial scores into competitive ones after fine-tuning on CoMind.
  • Future 3D extensions become feasible by lifting the existing 2D boxes into the provided scene and object scans for embodied spatial reasoning.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because the tasks fix the helper as the agent that must react to the leader, the same protocol can be reused to train robot helpers that must act without explicit commands.
  • The large zero-shot gap on bounding boxes versus moderate category accuracy suggests current VLMs already parse linguistic intent but lack the cross-view geometric binding needed for embodied collaboration.
  • Gaze-following and body-pose pseudo-labels already reconstructible from the dual views and Aria MPS data could bootstrap denser social-cue supervision without new manual annotation.
  • If the three tasks truly track ToM, performance curves on CoMind should correlate with independent ToM battery scores of the same models on classic false-belief or intention-inference tests.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. CoMind presents a multimodal ego–exocentric dataset of unscripted two-person cooking collaboration (80 sessions, ~41 h dual-view / ~81 h single-view, 125 participants) with synchronized Aria egocentric video, dual GoPro exocentric views, gaze and hand tracking, audio/transcripts, camera trajectories, dense kitchen scans, and scanned object meshes. The authors define three vision benchmarks intended to operationalize Theory of Mind in physical collaboration—Joint Attention Estimation, Socially Conditioned Object Interaction Anticipation, and Collaborative Handover Prediction—with manual annotations for shared objects, social cue types, verbs/nouns, delivery flow, initiator, and time-to-handover. They evaluate multiple closed- and open-weight VLMs under a shared prompting protocol with participant-disjoint train/test splits, report near-zero spatial grounding for most proprietary models, and show large gains after LoRA fine-tuning of Qwen3-VL (e.g., action-verb accuracy rising from ~0.09–0.14 to ~0.64–0.65).

Significance. If the resource and benchmarks hold as described, CoMind fills a clear gap relative to prior ego/exo and multi-agent datasets (Table 1): long-horizon goal-directed physical collaboration with gaze, verbal/gestural cue labels, and 3D scene/object grounding. The participant-disjoint evaluation and the documented fine-tuning gains provide concrete evidence that the training split is useful for socially conditioned perception, not only as a zero-shot stress test. The three tasks are well-specified for proactive assistance research even if one disputes the strongest ToM rhetoric. Public release of data and benchmarks is a genuine contribution to multimodal and embodied social AI.

major comments (3)
  1. Abstract, Introduction, and §3.1 frame the three vision tasks as a formalization of Theory of Mind for physical collaboration, yet success is defined solely by matching annotated boxes, verbs/nouns, cue types, flow, initiator, and TTH. No human study, correlation with established ToM instruments (those cited in Related Work), or ablation that isolates mental-state inference from surface multimodal pattern matching is provided. The empirical VLM numbers remain valid as a social-perception benchmark; the stronger claim that CoMind advances ToM-capable AI should be tempered to “social cue–conditioned collaborative perception,” or supported by an explicit validation argument.
  2. §3.4 describes multi-stage human annotation with QA and author review, but the manuscript reports no inter-annotator agreement (e.g., IoU agreement on boxes, Cohen/Fleiss κ on cue type, initiator, delivery flow, or verb/noun labels). For a dataset paper whose central value is the annotations (Tables 2–4, Figs. 13–15, Supp. annotation guides), IAA (or at least double-annotation on a held-out subset) is load-bearing for trusting the reported metrics and the fine-tuning gains.
  3. §4 evaluates only general VLMs (plus random/most-frequent priors). For Joint Attention Estimation and handover timing/localization, the literature already has specialized gaze, mutual-attention, and action-anticipation models (cited in §2). Without at least one non-VLM or modular baseline (e.g., gaze-intersection + object detector, or a standard anticipation transformer), it is hard to separate “VLMs fail at social grounding” from “the tasks are hard for any current method.” Adding such baselines would strengthen the claim of a significant performance deficiency.
minor comments (6)
  1. Abstract vs. body: the abstract lists “Action Anticipation” while §3.1 and Tables use “Socially Conditioned Object Interaction Anticipation”; align naming throughout.
  2. §4: only 5 uniformly sampled frames from the 10 s context are fed to VLMs. An ablation on frame count / video input (beyond the partial Supp. Table S6) would clarify whether spatial failures are partly an input bottleneck.
  3. Table 4 TTH metric uses a tight ±0.25 s window; report also mean absolute error or a coarser bin so temporal performance is easier to interpret.
  4. Object hierarchy (L1–L3) and synonym handling for Cat. (L1) are important for reproducibility; point more explicitly from the main text to Supp. §S5 / Fig. S9.
  5. Fig. 1 / Table 1: “81” hours ego vs. “40h 43m” single-view wording can confuse; state dual-view vs. single-view totals once in a consistent way.
  6. Minor polish: arXiv id and some model version strings (Claude Opus 4.5/4.6, GPT 5.x) will age quickly—cite system cards with access dates as already partly done in references.

Circularity Check

0 steps flagged

No circularity: empirical dataset and VLM benchmarks with held-out evaluation; no fitted parameters re-labeled as predictions and no self-referential derivation.

full rationale

CoMind is a resource paper that collects multimodal cooking collaboration data, defines three annotation-driven vision tasks (Joint Attention Estimation, Socially Conditioned Object Interaction Anticipation, Collaborative Handover Prediction), and reports zero-shot and fine-tuned VLM numbers on a participant-disjoint test split. Performance metrics (IoU@0.5, cue-type accuracy, verb/noun matches, TTH, etc.) are obtained by comparing model outputs against human annotations; fine-tuning gains are measured on the same held-out set. There are no equations that define a quantity in terms of itself, no parameters fitted to data and then re-presented as independent predictions, and no uniqueness theorems or ansatzes imported via self-citation that force the central claims. The framing that the three tasks operationalize Theory of Mind is an interpretive claim about task design, not a circular derivation. The empirical results are therefore self-contained against external models and held-out data.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 2 invented entities

As a dataset paper the central claims rest on the fidelity of capture, the correctness of human annotations, and the claim that the three tasks measure socially conditioned reasoning. No free parameters are fitted to produce a scientific constant; the free choices are design decisions of the benchmark.

free parameters (3)
  • context window length (10 s)
    Fixed by authors for all three tasks; not derived from data but chosen for VLM context limits and annotation practicality.
  • IoU threshold 0.5 and TTH tolerance 0.25 s
    Standard but arbitrary evaluation thresholds that directly affect reported scores.
  • LoRA rank r=64, alpha=128, lr=2e-4, epochs=3–5
    Hyper-parameters of the fine-tuning experiments that produce the largest reported gains.
axioms (3)
  • ad hoc to paper Success on the three vision tasks is a valid proxy for Theory-of-Mind ability in physical collaboration.
    Stated in Introduction and §3.1; no independent cognitive or behavioral validation is supplied.
  • domain assumption Cooking in real kitchens with unscripted pairs yields representative collaborative social cues.
    Underpins the claim of ecological validity; participants are young adults (18–38) in 55 kitchens.
  • domain assumption Human annotations of joint attention, cue type and handover initiator are sufficiently reliable after QA review.
    No quantitative inter-annotator agreement is reported; reliability is asserted via multi-stage review.
invented entities (2)
  • Socially Conditioned Object Interaction Anticipation task no independent evidence
    purpose: Operationalize intent inference from social cues into a structured prediction problem (verb, noun, box, cue type).
    New task definition introduced by the paper; no prior benchmark uses exactly this formulation.
  • Collaborative Handover Prediction task (pre-reach TTH + initiator + flow) no independent evidence
    purpose: Measure proactive physical coordination before any reaching motion.
    Distinct from prior kinematic handover datasets that annotate the transfer itself rather than its cognitive precursors.

pith-pipeline@v1.1.0-grok45 · 38356 in / 2487 out tokens · 31523 ms · 2026-07-10T23:18:23.950643+00:00 · methodology

0 comments
read the original abstract

Human-human collaboration is a fundamental aspect of everyday life, essential to success in a wide range of goal-directed activities from household tasks to professional teamwork. While much research has focused on modeling coordination and task execution, the cognitive processes that support such collaboration, particularly Theory of Mind (the ability to infer the mental states of others), remain difficult to study in natural settings. To address this gap, we introduce a novel egocentric and exocentric video dataset capturing real-world collaboration in cooking scenarios. The dataset integrates multi-perspective video, high-quality audio, gaze tracking, and 3D scene and object scans, with annotations for shared attention to objects, social cues and interactions between agents, as well as agent-object interactions. We establish benchmarks for Joint Attention Estimation, Socially Conditioned Object Interaction Anticipation, and Collaborative Handover Prediction, enabling research on multimodal perception, proactive assistance, and collaborative planning. By providing temporally aligned, richly annotated multimodal data, CoMind facilitates the development and evaluation of AI systems capable of modeling complex social interactions and reasoning about human behaviors in collaborative environments. Our dataset and benchmarks are made available at https://comind.ethz.ch/.

Figures

Figures reproduced from arXiv: 2607.06691 by Alexandros Delitzas, Alexey Gavryushin, Ben Ellis, Cedric Z\"ollner, Dingxi Zhang, Jiaqi Chen, Manthan Patel, Manuel Kaufmann, Marc Pollefeys, Xi Wang, Zhao Huang.

Figure 1
Figure 1. Figure 1: The CoMind dataset captures 41 hours of human-human collaboration across 80 sessions, combining egocentric/exocentric videos, audio, gaze, and high-quality 3D scene and object scans, as well as rich annotations for three novel social reasoning tasks. Abstract. Human-human collaboration is a fundamental aspect of ev￾eryday life, essential to success in a wide range of goal-directed activities from household… view at source ↗
Figure 2
Figure 2. Figure 2: Joint Attention Estimation Task. From the audio or transcribed speech in a context win￾dow, together with a target frame (green), the model predicts the jointly attended object, its bounding box in the left and right views, and the social cue type (blue). "Can you please wash the garlic and onion and peel them?" "Yeah, just cut it like that" [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 5
Figure 5. Figure 5: Example anno￾tations. The prediction frame shows the ground￾truth action verb, noun, ob￾ject bounding box, and cue type(s). of the cue eliciting the joint attention on the object. This task measures the capacity to recognize a collaborative agent’s shared mental state. Task 2: Socially Conditioned Object Interaction Anticipation. We define a socially conditioned action of participant i ∈ {ℓ, h} as any acti… view at source ↗
Figure 6
Figure 6. Figure 6: Given a prediction frame f before any reaching or grasping motion towards a handed-over object, together with the two synchronized egocentric [PITH_FULL_IMAGE:figures/full_fig_p006_6.png] view at source ↗
Figure 6
Figure 6. Figure 6: Collaborative Handover Prediction Task. From context frames (or video) and audio or transcribed speech, together with a prediction frame (green), the model predicts the delivery flow (who hands to whom), the object category and bounding box in the view of the handing participant, the ini￾tiator, and the cue type (blue). "May I ask you to pass me the sugar?" "..." sugar Initiator: leader helper to leader ve… view at source ↗
Figure 9
Figure 9. Figure 9: 3D scans of ob￾jects used in the record￾ings. Participants work with objects that have been 3D-scanned. BLK2GO scanner prior to the session. Lastly, we reuse a set of common cooking utensils (pans and pots) for each session, which are scanned using an Artec 3D Leo [6] scanner to obtain high-fidelity object meshes. We provide a visualization of the sensors and hardware in [PITH_FULL_IMAGE:figures/full_fig_… view at source ↗
Figure 10
Figure 10. Figure 10: Scan Alignment. To transform the 3D scan of the environment into Aria’s world frame, we follow a two-stage procedure. In the first stage, we localize the BLK2GO images within Aria’s semi-dense point cloud using Hierarchical Localization [72, 73]. Each successfully localized image yields an independent estimate of the rigid transformation between the two coordinate frames. We reject outlier estimates and c… view at source ↗
Figure 10
Figure 10. Figure 10: Modalities in the CoMind dataset. The dataset provides synchronized multimodal recordings of collaborative sessions, including egocentric video with eye gaze and hand tracking, audio and IMU signals, exocentric camera views, camera trajectories and spatial mapping, dense 3D scans of the scene, and a set of high-quality 3D scans of commonly used objects. follow instructions and tutorial videos designed to … view at source ↗
Figure 11
Figure 11. Figure 11: Histogram of recording lengths. Our lengthy recordings allow for the study of human-human collaborative work in long-context settings. In total, we record 41 hours of two-person resp. 81 hours of single-view data. 51.2% 48.1% 0.6% Male Female Undisclosed Gender 54.4% 45.6% Western Eastern Ethnicity [PITH_FULL_IMAGE:figures/full_fig_p010_11.png] view at source ↗
Figure 13
Figure 13. Figure 13: Action verb and action noun distribution of the Socially Conditioned [PITH_FULL_IMAGE:figures/full_fig_p010_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Annotation statistics for the Joint Attention Estimation task. [PITH_FULL_IMAGE:figures/full_fig_p011_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Annotation statistics for the Collaborative Handover Prediction task. [PITH_FULL_IMAGE:figures/full_fig_p011_15.png] view at source ↗
Figure 13
Figure 13. Figure 13: We further provide statistics regarding the object categories for the [PITH_FULL_IMAGE:figures/full_fig_p011_13.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

121 extracted references · 121 canonical work pages · 15 internal anchors

  1. [1]

    In: Proceedings of the IEEE conference on computer vision and pattern recognition

    Abu Farha, Y., Richard, A., Gall, J.: When will you do what?-anticipating temporal occurrences of activities. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 5343–5352 (2018) 16 A. Gavryushin et al

    work page 2018

  2. [2]

    Current directions in psy- chological science26(3), 243–248 (2017)

    Adams Jr, R.B., Albohn, D.N., Kveraga, K.: Social vision: Applying a social- functional approach to face and expression perception. Current directions in psy- chological science26(3), 243–248 (2017)

    work page 2017

  3. [3]

    System card (Nov 2025),https://www

    Anthropic: Claude opus 4.5 system card. System card (Nov 2025),https://www. anthropic.com/claude-opus-4-5-system-card, accessed 2026-03-04

    work page 2025

  4. [4]

    Anthropic News (Feb 2026),https://www

    Anthropic: Introducing claude opus 4.6. Anthropic News (Feb 2026),https://www. anthropic.com/news/claude-opus-4-6, accessed 2026-03-04

    work page 2026

  5. [5]

    theory of mind

    Apperly, I.: Mindreaders: the cognitive basis of" theory of mind". Psychology Press (2010)

    work page 2010

  6. [6]

    com/portable-3d-scanners/artec-leo, accessed: 2026-03-04

    Artec 3D: Artec leo: Wireless professional 3d scanner (2024),https://www.artec3d. com/portable-3d-scanners/artec-leo, accessed: 2026-03-04

    work page 2024

  7. [7]

    Qwen3-VL Technical Report

    Bai, S., Cai, Y., Chen, R., Chen, K., Chen, X., Cheng, Z., Deng, L., Ding, W., Gao, C., Ge, C., et al.: Qwen3-vl technical report. arXiv preprint arXiv:2511.21631 (2025)

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  8. [8]

    Bain, M., Huh, J., Han, T., Zisserman, A.: Whisperx: Time-accurate speech tran- scription of long-form audio (2023),https://arxiv.org/abs/2303.00747

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  9. [9]

    HOT3D: Hand and Object Tracking in 3D from Egocentric Multi-View Videos

    Banerjee, P., Shkodrani, S., Moulon, P., Hampali, S., Han, S., Zhang, F., Zhang, L., Fountain, J., Miller, E., Basol, S., et al.: Hot3d: Hand and object tracking in 3d from egocentric multi-view videos. arXiv preprint arXiv:2411.19167 (2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  10. [10]

    In: NeuRIPS Workshop on Gaze Meets ML (2023)

    de Belen, R.A., Mohammadi, G., Sowmya, A.: Temporal understanding of gaze communication with gazetransformer. In: NeuRIPS Workshop on Gaze Meets ML (2023)

    work page 2023

  11. [11]

    EPFL-Smart-Kitchen-30: Densely annotated cooking dataset with 3D kinematics to challenge video and language models

    Bonnetto, A., Qi, H., Leong, F., Tashkovska, M., Rad, M., Shokur, S., Hummel, F., Micera, S., Pollefeys, M., Mathis, A.: Epfl-smart-kitchen-30: Densely annotated cooking dataset with 3d kinematics to challenge video and language models. arXiv preprint arXiv:2506.01608 (2025)

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  12. [12]

    In: 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems

    Cakmak, M., Srinivasa, S.S., Lee, M.K., Forlizzi, J., Kiesler, S.: Human prefer- ences for robot-human hand-over configurations. In: 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems. pp. 1986–1993. IEEE (2011)

    work page 2011

  13. [13]

    PARTNR: A Benchmark for Planning and Reasoning in Embodied Multi-agent Tasks

    Chang, M., Chhablani, G., Clegg, A., Cote, M.D., Desai, R., Hlavac, M., Karashchuk, V., Krantz, J., Mottaghi, R., Parashar, P., et al.: Partnr: A benchmark for planning and reasoning in embodied multi-agent tasks. arXiv preprint arXiv:2411.00081 (2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  14. [14]

    In: Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track (2025)

    Chavan, V., Imgrund, Y., Dao, T., Bai, S., Wang, B., Lu, Z., Heimann, O., Krüger, J.: Indego: A dataset of industrial scenarios and collaborative work for egocentric assistants. In: Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track (2025)

    work page 2025

  15. [15]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Chen, J., Mittal, G., Yu, Y., Kong, Y., Chen, M.: Gatehub: Gated history unit with background suppression for online action detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 19925– 19934 (2022)

    work page 2022

  16. [16]

    In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

    Chen, Z., Wu, J., Zhou, J., Wen, B., Bi, G., Jiang, G., Cao, Y., Hu, M., Lai, Y., Xiong, Z., et al.: Tombench: Benchmarking theory of mind in large language models. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 15959–15983 (2024)

    work page 2024

  17. [17]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    Chong, E., Wang, Y., Ruiz, N., Rehg, J.M.: Detecting attended visual targets in video. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 5396–5406 (2020)

    work page 2020

  18. [18]

    Trends in Cognitive Sciences13(4), 148–153 (2009) CoMind: Understanding Collaborative Human Activity 17

    Csibra, G., Gergely, G.: Natural pedagogy. Trends in Cognitive Sciences13(4), 148–153 (2009) CoMind: Understanding Collaborative Human Activity 17

    work page 2009

  19. [19]

    In: Proceedings of the European conference on computer vision (ECCV)

    Damen, D., Doughty, H., Farinella, G.M., Fidler, S., Furnari, A., Kazakos, E., Moltisanti, D., Munro, J., Perrett, T., Price, W., et al.: Scaling egocentric vision: The epic-kitchens dataset. In: Proceedings of the European conference on computer vision (ECCV). pp. 720–736 (2018)

    work page 2018

  20. [20]

    International Journal of Computer Vision (IJCV)130, 33–55 (2022),https://doi.org/10.1007/s11263- 021-01531-2

    Damen, D., Doughty, H., Farinella, G.M., Furnari, A., Ma, J., Kazakos, E., Molti- santi, D., Munro, J., Perrett, T., Price, W., Wray, M.: Rescaling egocentric vision: Collection, pipeline and challenges for epic-kitchens-100. International Journal of Computer Vision (IJCV)130, 33–55 (2022),https://doi.org/10.1007/s11263- 021-01531-2

    work page doi:10.1007/s11263- 2022

  21. [21]

    https://deepmind.google/models/gemma/gemma- 4/ (2026), accessed: July 2026

    DeepMind, G.: Gemma 4. https://deepmind.google/models/gemma/gemma- 4/ (2026), accessed: July 2026

    work page 2026

  22. [22]

    https://blog.google/products-and-platforms/products/gemini/gemini- 3-flash/(Dec 2025), google Blog

    Doshi, T., the Gemini Team: Gemini 3 flash: Frontier intelligence built for speed. https://blog.google/products-and-platforms/products/gemini/gemini- 3-flash/(Dec 2025), google Blog

    work page 2025

  23. [23]

    In: Proceedings of the 27th ACM International Conference on Multimedia

    Dutta, A., Zisserman, A.: The VIA annotation software for images, audio and video. In: Proceedings of the 27th ACM International Conference on Multimedia. MM ’19, ACM, New York, NY, USA (2019)

    work page 2019

  24. [24]

    Project Aria: A New Tool for Egocentric Multi-Modal AI Research

    Engel, J., Somasundaram, K., Goesele, M., Sun, A., Gamino, A., Turner, A., Talattof, A., Yuan, A., Souti, B., Meredith, B., et al.: Project aria: A new tool for egocentric multi-modal ai research. arXiv preprint arXiv:2308.13561 (2023)

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  25. [25]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)

    Fan, L., Wang, W., Huang, S., Tang, X., Zhu, S.C.: Understanding human gaze com- munication by spatio-temporal graph reasoning. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 5724–5733 (2019)

    work page 2019

  26. [26]

    In: Proceedings of the European Conference on Computer Vision (ECCV) Workshops

    Fathi, A., Li, Y., Rehg, J.M.: Learning to recognize daily actions using gaze. In: Proceedings of the European Conference on Computer Vision (ECCV) Workshops. pp. 314–327. Springer (2012)

    work page 2012

  27. [27]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Fu, R., Zhang, D., Jiang, A., Fu, W., Funk, A., Ritchie, D., Sridhar, S.: Gigahands: A massive annotated dataset of bimanual hand activities. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 17461–17474 (2025)

    work page 2025

  28. [28]

    IEEE transactions on pattern analysis and machine intelligence 43(11), 4021–4036 (2020)

    Furnari, A., Farinella, G.M.: Rolling-unrolling lstms for action anticipation from first-person video. IEEE transactions on pattern analysis and machine intelligence 43(11), 4021–4036 (2020)

    work page 2020

  29. [29]

    In: 2022 26th International Conference on Pattern Recognition (ICPR)

    Furnari, A., Farinella, G.M.: Towards streaming egocentric action anticipation. In: 2022 26th International Conference on Pattern Recognition (ICPR). pp. 1250–1257. IEEE (2022)

    work page 2022

  30. [30]

    RED: Reinforced Encoder-Decoder Networks for Action Anticipation

    Gao, J., Yang, Z., Nevatia, R.: Red: Reinforced encoder-decoder networks for action anticipation. arXiv preprint arXiv:1707.04818 (2017)

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  31. [31]

    In: Proceedings of the IEEE/CVF international conference on computer vision

    Girdhar, R., Grauman, K.: Anticipative video transformer. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 13505–13515 (2021)

    work page 2021

  32. [32]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Grauman, K., Westbury, A., Byrne, E., Chavis, Z., Furnari, A., Girdhar, R., Hamburger, J., Jiang, H., Liu, M., Liu, X., et al.: Ego4d: Around the world in 3,000 hours of egocentric video. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 18995–19012 (2022)

    work page 2022

  33. [33]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Grauman, K., Westbury, A., Torresani, L., Kitani, K., Malik, J., Afouras, T., Ashutosh, K., Baiyya, V., Bansal, S., Boote, B., et al.: Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 19383–19400 (2024)

    work page 2024

  34. [34]

    Neural Computing and Applications35(2), 2007–2024 (2023) 18 A

    Guo, Z., Hou, Y., Wang, P., Gao, Z., Xu, M., Li, W.: Ft-hid: a large-scale rgb-d dataset for first-and third-person human interaction analysis. Neural Computing and Applications35(2), 2007–2024 (2023) 18 A. Gavryushin et al

    work page 2007

  35. [35]

    In: ICLR (2022)

    Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W.: LoRA: Low-rank adaptation of large language models. In: ICLR (2022)

    work page 2022

  36. [36]

    Palm: Predicting Actions through Language Models @ Ego4D Long-Term Action Anticipation Challenge 2023

    Huang, D., Hilliges, O., Van Gool, L., Wang, X.: Palm: Predicting actions through language models@ ego4d long-term action anticipation challenge 2023. arXiv preprint arXiv:2306.16545 (2023)

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  37. [37]

    IEEE Transactions on Human-Machine Systems50(4), 306–316 (2020)

    Huang, Y., Cai, M., Sato, Y.: An ego-vision system for discovering human joint attention. IEEE Transactions on Human-Machine Systems50(4), 306–316 (2020)

    work page 2020

  38. [38]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2024)

    Huang, Y., Chen, G., Xu, J., Zhang, M., Yang, L., Pei, B., Zhang, H., Lu, D., Wang, Y., Wang, L., Qiao, Y.: Egoexolearn: A dataset for bridging asynchronous ego- and exo-centric view of procedural activities in real world. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2024)

    work page 2024

  39. [39]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops

    Jang, Y., Sullivan, B., Ludwig, C., Gilchrist, I., Damen, D., Mayol-Cuevas, W.: Epic-tent: An egocentric video dataset for camping tent assembly. In: Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops. pp. 0–0 (2019)

    work page 2019

  40. [40]

    In: European Conference on Computer Vision

    Jia, B., Chen, Y., Huang, S., Zhu, Y., Zhu, S.C.: Lemma: A multi-view dataset for le arning m ulti-agent m ulti-task a ctivities. In: European Conference on Computer Vision. pp. 767–786. Springer (2020)

    work page 2020

  41. [41]

    EgoHumans: An Egocentric 3D Multi-Human Benchmark

    Khirodkar, R., Bansal, A., Ma, L., Newcombe, R., Vo, M., Kitani, K.: Egohumans: An egocentric 3d multi-human benchmark. arXiv preprint arXiv:2305.16487 (2023)

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  42. [42]

    EgoToM: Benchmarking Theory of Mind Reasoning from Egocentric Videos

    Li, Y., Veerabadran, V., Iuzzolino, M.L., Roads, B.D., Celikyilmaz, A., Ridgeway, K.: Egotom: Benchmarking theory of mind reasoning from egocentric videos. arXiv preprint arXiv:2503.22152 (2025)

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  43. [43]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Liu, S., Tripathi, S., Majumdar, S., Wang, X.: Joint hand motion and interaction hotspots prediction from egocentric videos. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 3282–3292 (2022)

    work page 2022

  44. [44]

    CORE4D: A 4D Human-Object-Human Interaction Dataset for Collaborative Object REarrangement

    Liu, Y., Zhang, C., Xing, R., Tang, B., Yang, B., Yi, L.: Core4d: A 4d human- object-human interaction dataset for collaborative object rearrangement. arXiv preprint arXiv:2406.19353 (2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  45. [45]

    International Journal of Computer Vision (IJCV) 106, 282–296 (2014)

    Marin-Jimenez, M.J., Zisserman, A., Eichner, M., Ferrari, V.: Detecting people looking at each other in videos. International Journal of Computer Vision (IJCV) 106, 282–296 (2014)

    work page 2014

  46. [46]

    In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision

    Mascaró, E.V., Ahn, H., Lee, D.: Intention-conditioned long-term human egocentric action anticipation. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 6048–6057 (2023)

    work page 2023

  47. [47]

    Technical report, arXiv (2025), https://arxiv.org/pdf/2510.16258, arXiv preprint

    McLean, C., Meendering, M., Swartz, T., Gabbay, O., Olsen, A., Jacobs, R., Rosen, N., de Bree, P., Garcia, T., Merrill, G., Sandakly, J., Buffalini, J., Jain, N., Krenn, S., Kumar, M., Markovic, D., Ng, E., Prada, F., Saba, A., Zhang, S., Agrawal, V., Godisart, T., Richard, A., Zollhoefer, M.: Embody 3d: A large- scale multimodal motion and behavior datas...

    work page arXiv 2025

  48. [48]

    IEEE transactions on pattern analysis and machine intelligence (TPAMI)10(2021)

    Medina, M.J.M.K.V.: Suárez p zisserman a laeo-net++: revisiting people looking at each other in videos. IEEE transactions on pattern analysis and machine intelligence (TPAMI)10(2021)

    work page 2021

  49. [49]

    https://facebookresearch.github.io/projectaria_tools/docs/ARK/mps (2024), accessed: 2026-03-04

    Meta Reality Labs Research: Project aria machine perception services (mps). https://facebookresearch.github.io/projectaria_tools/docs/ARK/mps (2024), accessed: 2026-03-04

    work page 2024

  50. [50]

    Advances in Neural Information Processing Systems35, 23765–23779 (2022) CoMind: Understanding Collaborative Human Activity 19

    Mittal, H., Morgado, P., Jain, U., Gupta, A.: Learning state-aware visual repre- sentations from audible interactions. Advances in Neural Information Processing Systems35, 23765–23779 (2022) CoMind: Understanding Collaborative Human Activity 19

    work page 2022

  51. [51]

    Current directions in psychological science16, 269–274 (11 2007)

    Mundy, P., Newell, L.: Attention, joint attention, and social cognition. Current directions in psychological science16, 269–274 (11 2007)

    work page 2007

  52. [52]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Ngo, T.D., Hua, B.S., Nguyen, K.: Isbnet: a 3d point cloud instance segmenta- tion network with instance-aware sampling and box-aware dynamic convolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 13550–13559 (2023)

    work page 2023

  53. [53]

    System card (Aug 2024),https://openai.com/index/ gpt-4o-system-card/, accessed 2026-03-04

    OpenAI: Gpt-4o system card. System card (Aug 2024),https://openai.com/index/ gpt-4o-system-card/, accessed 2026-03-04

    work page 2024

  54. [54]

    System card addendum (Nov 2025),https://openai.com/index/gpt-5-system-card-addendum- gpt-5-1/, accessed 2026-03-04

    OpenAI: Gpt-5.1 instant and gpt-5.1 thinking system card addendum. System card addendum (Nov 2025),https://openai.com/index/gpt-5-system-card-addendum- gpt-5-1/, accessed 2026-03-04

    work page 2025

  55. [55]

    OpenAI Index (Dec 2025),https://openai.com/ index/introducing-gpt-5-2/, accessed 2026-03-04

    OpenAI: Introducing gpt-5.2. OpenAI Index (Dec 2025),https://openai.com/ index/introducing-gpt-5-2/, accessed 2026-03-04

    work page 2025

  56. [56]

    IEEE Transactions on Robotics37(6), 1855–1873 (2021)

    Ortenzi, V., Cosgun, A., Pardi, T., Chan, W.P., Croft, E., Kulić, D.: Object handovers: a review for robotics. IEEE Transactions on Robotics37(6), 1855–1873 (2021)

    work page 2021

  57. [57]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Osman, N., Camporese, G., Coscia, P., Ballan, L.: Slowfast rolling-unrolling lstms for action anticipation in egocentric videos. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 3437–3445 (2021)

    work page 2021

  58. [58]

    In: Conference on Computer Vision and Pattern Recognition 2024 (2024),https://eth-ait.github

    Pasca, R., Gavryushin, A., Hamza, M., Kuo, Y.L., Mo, K., Van Gool, L., Hilliges, O., Wang, X.: Summarize the past to predict the future: Natural language descriptions of context boost multimodal object interaction anticipation. In: Conference on Computer Vision and Pattern Recognition 2024 (2024),https://eth-ait.github. io/transfusion-proj/

    work page 2024

  59. [59]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Perrett, T., Darkhalil, A., Sinha, S., Emara, O., Pollard, S., Parida, K.K., Liu, K., Gatti, P., Bansal, S., Flanagan, K., et al.: Hd-epic: A highly-detailed egocentric video dataset. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 23901–23913 (2025)

    work page 2025

  60. [60]

    Behavioral and Brain Sciences1, 515 – 526 (12 1978)

    Premack, D., Woodruff, G.: Does a chimpanzee have a theory of mind. Behavioral and Brain Sciences1, 515 – 526 (12 1978)

    work page 1978

  61. [61]

    Puig, X., Ra, K., Boben, M., Li, J., Wang, T., Fidler, S., Torralba, A.: Virtualhome: Simulating household activities via programs (2018)

    work page 2018

  62. [62]

    Watch-And-Help: A Challenge for Social Perception and Human-AI Collaboration

    Puig, X., Shu, T., Li, S., Wang, Z., Liao, Y.H., Tenenbaum, J.B., Fidler, S., Torralba, A.: Watch-and-help: A challenge for social perception and human-ai collaboration. arXiv preprint arXiv:2010.09890 (2020)

    work page internal anchor Pith review Pith/arXiv arXiv 2010

  63. [63]

    IEEE Transactions on Pattern Analysis and Machine Intelligence (2021)

    Qi, Z., Wang, S., Su, C., Su, L., Huang, Q., Tian, Q.: Self-regulated learning for egocentric video activity anticipation. IEEE Transactions on Pattern Analysis and Machine Intelligence (2021)

    work page 2021

  64. [64]

    EgoMe: A New Dataset and Challenge for Following Me via Egocentric View in Real World

    Qiu, H., Shi, Z., Wang, L., Xiong, H., Li, X., Li, H.: Egome: A new dataset and challenge for following me via egocentric view in real world. arXiv preprint arXiv:2501.19061 (2025)

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  65. [65]

    In: International conference on machine learning

    Rabinowitz, N., Perbet, F., Song, F., Zhang, C., Eslami, S.A., Botvinick, M.: Machine theory of mind. In: International conference on machine learning. pp. 4218–4227. PMLR (2018)

    work page 2018

  66. [66]

    In: International conference on machine learning

    Radford, A., Kim, J.W., Xu, T., Brockman, G., McLeavey, C., Sutskever, I.: Robust speech recognition via large-scale weak supervision. In: International conference on machine learning. pp. 28492–28518. PMLR (2023)

    work page 2023

  67. [67]

    Recasens, A., Khosla, A., Vondrick, C., Torralba, A.: Where are they looking? In: NeurIPS (2015)

    work page 2015

  68. [68]

    Joint attention: Communication and other minds: Issues in philosophy and psychology p

    Reddy, V.: Understanding attention to self. Joint attention: Communication and other minds: Issues in philosophy and psychology p. 85 (2005) 20 A. Gavryushin et al

    work page 2005

  69. [69]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Rizve, M.N., Mittal, G., Yu, Y., Hall, M., Sajeev, S., Shah, M., Chen, M.: Piv- otal: Prior-driven supervision for weakly-supervised temporal action localization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 22992–23002 (2023)

    work page 2023

  70. [70]

    In: International Conference on Image Analysis and Processing

    Rodin, I., Furnari, A., Mavroeidis, D., Farinella, G.M.: Untrimmed action anticipa- tion. In: International Conference on Image Analysis and Processing. pp. 337–348. Springer (2022)

    work page 2022

  71. [71]

    In: Proceedings third international conference on 3-D digital imaging and modeling

    Rusinkiewicz, S., Levoy, M.: Efficient variants of the icp algorithm. In: Proceedings third international conference on 3-D digital imaging and modeling. pp. 145–152. IEEE (2001)

    work page 2001

  72. [72]

    In: CVPR (2019)

    Sarlin, P.E., Cadena, C., Siegwart, R., Dymczyk, M.: From coarse to fine: Robust hierarchical localization at large scale. In: CVPR (2019)

    work page 2019

  73. [73]

    In: CVPR (2020)

    Sarlin, P.E., DeTone, D., Malisiewicz, T., Rabinovich, A.: SuperGlue: Learning feature matching with graph neural networks. In: CVPR (2020)

    work page 2020

  74. [74]

    Behavioral and Brain Sciences 36(4), 393–414 (2013)

    Schilbach, L., Timmermans, B., Reddy, V., Costall, A., Bente, G., Schlicht, T., Vogeley, K.: Toward a second-person neuroscience. Behavioral and Brain Sciences 36(4), 393–414 (2013)

    work page 2013

  75. [75]

    Trends in Cognitive Sciences10(2), 70–76 (2006)

    Sebanz, N., Bekkering, H., Knoblich, G.: Joint action: bodies and minds moving together. Trends in Cognitive Sciences10(2), 70–76 (2006)

    work page 2006

  76. [76]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Sener, F., Chatterjee, D., Shelepov, D., He, K., Singhania, D., Wang, R., Yao, A.: Assembly101: A large-scale multi-view video dataset for understanding procedural activities. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 21096–21106 (2022)

    work page 2022

  77. [77]

    In: Proceedings of the AAAI Conference on Artificial Intelligence (2025)

    Shi, H., Ye, S., Fang, X., Jin, C., Isik, L., Kuo, Y.L., Shu, T.: Muma-tom: Multi- modal multi-agent theory of mind. In: Proceedings of the AAAI Conference on Artificial Intelligence (2025)

    work page 2025

  78. [78]

    In: Proceedings of the European Conference on Computer Vision (ECCV)

    Shi, Y., Fernando, B., Hartley, R.: Action anticipation with rbf kernelized feature mapping rnn. In: Proceedings of the European Conference on Computer Vision (ECCV). pp. 301–317 (2018)

    work page 2018

  79. [79]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Shridhar, M., Thomason, J., Gordon, D., Bisk, Y., Han, W., Mottaghi, R., Zettle- moyer, L., Fox, D.: Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10740–10749 (2020)

    work page 2020

  80. [80]

    Journal of Multimodal User Interfaces9(3), 223–229 (2015).https: //doi.org/10.1007/s12193-015-0196-1

    Six, J., Leman, M.: Synchronizing Multimodal Recordings Using Audio-To-Audio Alignment. Journal of Multimodal User Interfaces9(3), 223–229 (2015).https: //doi.org/10.1007/s12193-015-0196-1

    work page doi:10.1007/s12193-015-0196-1 2015

Showing first 80 references.