Pith. sign in

REVIEW 4 major objections 5 minor 35 references

The Wilhelm Tell Dataset of Affordance Demonstrations

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Pith's one-line read This paper presents a curated set of 189 dual-perspective videos of everyday household tasks, labelled with the affordances they manifest, so that robots can learn to recognize possibilities for action from observed events rather than…

desk verdict A genuinely new paired first/ and third-person affordance video dataset, but the unvalidated test annotations and missing baselines leave the intended use undemonstrated. read the letter →

arxiv 2507.17401 v1 pith:U3KGUGLI submitted 2025-07-23 cs.RO cs.HC

classification cs.ROcs.HC
keywords affordancedemonstrationsrobotlearningfromobservationvideodatasethouseholdtasksegocentricandthird-personrecognitionsemanticsegmentation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper presents the Wilhelm Tell dataset, a curated collection of 189 videos totalling about seven hours of people performing 20 everyday tasks centered on apples and household objects, recorded simultaneously from a first-person head camera and an external frontal camera. The authors' aim is to provide training and evaluation data for perception systems that recognize affordances — possibilities for action — as they are manifested in real events, rather than from static images of object parts. Each task is labelled with the affordance it demonstrates (drawn from existing shape datasets, plus 'Stab' and 'Pour'), and the test videos come with CSV time-interval annotations describing when object interactions happen, some with hotspot-annotated still frames. The intended use is to train automatic affordance-labeling systems and then semantic segmentation models that detect functional object parts both during and outside moments of use.

What carries the argument

The central object is the dataset itself: 189 MP4 videos recorded with a GoPro headset (ego perspective) and a frontal camera (third-person perspective), grouped into bundles of tasks 1–20 and accompanied by metadata linking each task to an affordance label such as Cut, Contain, Stab, Pour, Stack, Wrap, Filter, or Connect. The 'affordance manifestation' is the mechanism: each video shows an event in which an object's functional part is actually used, and the test split additionally provides CSV time intervals marking when objects interact and hotspot-annotated frames marking where the affordance is located. These annotations make the videos usable as supervision for an automatic affordance-hotspot labeling system, which in turn can train a semantic segmentation model to recognize functional parts both in use and not in use.

What would settle it

Compute inter-annotator agreement on the hotspot still frames and check whether the annotated time intervals align with frames in which the object is actually grasped, lifted, or otherwise used; if agreement is near chance or the intervals frequently miss the interaction, then the dataset cannot support the intended supervision.

Watch

Extended reading notes

Core claim

The central claim is that affordances are better learned through video demonstrations of their manifestation than through static shape annotations, and that a dataset of such demonstrations is a reusable resource for robot training. Concretely, the paper contributes 189 MP4 videos (417 minutes) from 23 participants, with task-level affordance labels, a train/test split of 102/44 videos, test-video CSV files marking time intervals of object interactions, and still frames with affordance hotspots for some test frames. The authors deliberately gave participants only broad instructions, so task performances vary in preparation maneuvers and object placement; this variety is itself part of the contribution, enabling study of how people arrange task space. The paper positions the dataset between ego-centric video collections and static-shape benchmarks by adding an observer perspective and a focus on affordances involving multiple objects interacting.

Load-bearing premise

The test-video CSV time intervals and the hotspot annotations on selected still frames are accurate and consistent enough to serve as ground truth for training and evaluating affordance recognition systems.

Editorial extensions

If this is right

  • Using the 102 training and 44 test videos, a researcher can train an automatic affordance-hotspot labeling system and then use the generated labels to train an image segmentation model that recognizes functional parts during and outside moments of use.
  • Because every task is recorded from both ego and frontal views, methods can learn from either perspective or exploit cross-view consistency between them.
  • The twenty tasks cover affordances that involve multiple objects interacting with each other and the agent, such as Stack, Wrap, Filter, and Connect, going beyond the agent-object interactions covered by earlier video affordance datasets.
  • The deliberate lack of prescribed preparation procedures means the same recordings can support research on preparatory maneuvers and space organization, informing collaborative robots that anticipate human actions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The absence of any reported annotation protocol, inter-annotator agreement, or baseline evaluation in the paper means the dataset should initially be read as a set of candidate labels; establishing benchmark quality would require an external reproducibility study.
  • The a/b task variants with different object sets create a ready-made domain-shift test: a model trained on one variant can be evaluated on the other to check whether it learned the affordance itself rather than features of a particular object.
  • At roughly seven hours of video, the dataset is small by current standards, but its dense annotation of demonstrations could serve as a seed set for bootstrapping larger automatically labeled video corpora for affordance learning.
  • If the preparatory maneuvers observed in the recordings were later annotated systematically, the same videos would double as a resource for studying spatial arrangement and expected action sequences, not only affordance hotspots.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces the Wilhelm Tell Dataset, a collection of 189 videos (417 minutes) of human demonstrations of 20 household tasks, mainly centered on apples, recorded from both ego and third-person perspectives. The dataset includes task-level affordance labels, a train/test split, and for test videos CSV interval annotations and some still frames with affordance hotspot annotations. The authors state the intended use is to train and evaluate visual perception systems for affordance recognition, with an envisioned pipeline that relies on an external automatic hotspot labeling system. The paper describes the study setup, participant recruitment and consent, task lists, and usage notes, and the dataset is released under CC-BY 4.0.

Significance. If validated, the dataset would fill a notable gap by providing a video-based alternative to static-image affordance datasets, with multiple perspectives and an emphasis on affordances manifested through object-object and agent-object interactions. The inclusion of varied task performances and preparatory maneuvers could support research on task-space organization and collaborative robots. The paper also documents ethical procedures, GDPR-compliant storage, participant consent, and a public license, which are strengths. However, the contribution is currently unvalidated: the paper provides no annotation protocol, inter-annotator agreement measures, or baseline experiments, so the central usability claim is not yet supported.

major comments (4)
  1. [VI] The test-video CSV time intervals and hotspot annotations are the only human-verified labels in the dataset, but no annotation protocol is described and no inter-annotator agreement is reported. For example, for the Stab task it is unspecified whether the interaction interval starts at first contact, at full penetration, or at the lift of the fork. Without a precise semantic definition of an 'interaction' interval, annotations may be inconsistent across annotators and across sessions, undermining their validity as ground truth for the proposed evaluation pipeline.
  2. [VI] The paper claims the dataset is 'aimed towards training perception systems to recognize affordance manifestations' and 'can be used for robot training purposes,' yet no experiment or proof-of-concept demonstrates that the dataset actually supports such training. The envisioned use relies on an external automatic labeling system (e.g., [29]) that is not evaluated here. At least a baseline experiment, even a simple one, or quantitative evidence of annotation quality (e.g., coverage statistics, agreement) is needed to substantiate the usability claim.
  3. [V and VI] There is a numerical inconsistency in the dataset description: Section V reports 189 total videos, while Section VI states the videos are split into 102 training and 44 testing videos, which sums to 146. The paper does not explain how the remaining 43 videos, the combined per-perspective videos, and the incomplete participant entries relate to the train/test split. This ambiguity affects reproducibility and the interpretation of the dataset's composition.
  4. [IV and V] The mapping between tasks and affordance labels is not specified in a machine-readable schema. For instance, Task 4 in Table I is associated with two source labels (Contain-ability from AfNet and Contain from AffordanceNet), and it is unclear whether the CSV files contain only task identifiers or also the affordance labels derived from these sources. A clear metadata schema or example annotation file would significantly improve the dataset's usability.
minor comments (5)
  1. [II] The text contains several typos, including 'afordance' instead of 'affordance', 'demonstartion' instead of 'demonstration', and 'developement' instead of 'development'.
  2. [IV] There are typos in the methods section: 'headstreap' should be 'headstrap', 'corrent' should be 'correct', and 'layed out' should be 'laid out'.
  3. [V] The sentence 'The Readme file for the dataset containing information as well as a download link to the video material can be found on GitHub' is awkwardly phrased; consider rewriting for clarity and including the direct repository URL in the text.
  4. [VI] The word 'automtically' is a typo for 'automatically' in the description of the labeling pipeline.
  5. [IV] The explanation of the session ordering (tasks 8-10, then 1-7, then 11-20, with a/b variants) is somewhat confusing; a timeline or flow diagram of the bundles would help readers understand the recording structure.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the dataset paper derives no quantitative claims from its own inputs.

full rationale

This is a dataset introduction paper, not a derivation or modeling paper. It contains no equations, no fitted parameters, and no predicted benchmark numbers that could reduce to its own inputs by construction. The central contribution is the curated collection of 189 videos with task-level affordance labels, and the paper explicitly leaves training and evaluation to future users. The intended use case is stated as 'the training and evaluation of visual perception systems for affordance recognition,' which is a forward-looking goal rather than a result derived from the dataset. The affordance label set is taken from pre-existing external datasets (AfNet and AffordanceNet), with two added affordances chosen by the authors; this is an annotation design choice, not a circular derivation. The only self-citation among the authors, reference [17], appears in background context as a formal affordance model and is not load-bearing for the dataset's construction or claims. The lack of an annotation protocol or inter-annotator agreement for the test-video CSV intervals and hotspot annotations is a real validity and usability risk, but it is a data-quality concern, not circularity: the intervals are not being used to predict a quantity defined in terms of themselves. No self-definitional step, fitted-input-called-prediction step, or self-citation chain that forces the paper's conclusions was found. The honest finding is therefore a circularity score of 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claims depend on domain assumptions about learning affordances from video, the suitability of prior affordance label sets, the adequacy of the two camera views, and the quality of the test-set annotations. No free parameters or invented entities are introduced.

assumptions (4)
  • domain assumption Affordances can be learned by observing video demonstrations of the actions that manifest them.
    This is the design thesis of the paper, stated in Section II: an agent should learn to recognize an affordance by observing events that manifest it.
  • domain assumption The affordance labels in AfNet and AffordanceNet are an appropriate prior set of important affordances for robotics.
    Section III says the considered affordances mostly come from AfNet/AffordanceNet and treats their presence as an indication of importance for robotics; Stab and Pour were added by author judgment.
  • domain assumption The head-mounted and frontal cameras capture enough visual information to recognize the target affordance manifestations.
    Section IV describes the recording setup as sufficient without reporting any validation that the videos contain the necessary evidence.
  • domain assumption The test-set CSV interval annotations and hotspot labels are accurate enough for training and evaluation.
    Section VI relies on these annotations for the intended use case, but no annotation quality measures are reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Wilhelm Tell Dataset of Affordance Demonstrations." pith.science (2026). https://pith.science/paper/U3KGUGLI

@misc{pith2026250717401,
  author       = {Pith},
  title        = {Pith review of: The Wilhelm Tell Dataset of Affordance Demonstrations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/U3KGUGLI}},
  note         = {Machine review of arXiv:2507.17401}
}
read the original abstract

Affordances - i.e. possibilities for action that an environment or objects in it provide - are important for robots operating in human environments to perceive. Existing approaches train such capabilities on annotated static images or shapes. This work presents a novel dataset for affordance learning of common household tasks. Unlike previous approaches, our dataset consists of video sequences demonstrating the tasks from first- and third-person perspectives, along with metadata about the affordances that are manifested in the task, and is aimed towards training perception systems to recognize affordance manifestations. The demonstrations were collected from several participants and in total record about seven hours of human activity. The variety of task performances also allows studying preparatory maneuvers that people may perform for a task, such as how they arrange their task space, which is also relevant for collaborative service robots.

Figures

Figures reproduced from arXiv: 2507.17401 by the authors.

Figure 1
Figure 1. Picture of lab environment showing the table with the objects as well [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

35 extracted references · 32 canonical work pages

  1. [29]

    Multi-label affordance mapping from egocentric vision,

    L. Mur-Labadia, J. J. Guerrero, and R. Martinez-Cantin, “Multi-label affordance mapping from egocentric vision,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 5238–5249

  2. [1]

    An interactive approach for situated task specification through verbal instructions,

    C. Meric ¸li, S. D. Klee, J. Paparian, and M. Veloso, “An interactive approach for situated task specification through verbal instructions,” in Proceedings of the 2014 international conference on Autonomous agents and multi-agent systems , 2014, pp. 1069–1076

  3. [2]

    Recent advances in robot learning from demonstration,

    H. Ravichandar, A. S. Polydoros, S. Chernova, and A. Billard, “Recent advances in robot learning from demonstration,” Annual review of control, robotics, and autonomous systems , vol. 3, no. 1, pp. 297–330, 2020

  4. [3]

    Describing upper-body motions based on labanotation for learning-from-observation robots,

    K. Ikeuchi, Z. Ma, Z. Yan, S. Kudoh, and M. Nakamura, “Describing upper-body motions based on labanotation for learning-from-observation robots,” International Journal of Computer Vision , vol. 126, pp. 1415– 1429, 2018

  5. [4]

    Service robots, customers and service employees: what can we learn from the academic literature and where are the gaps?

    V . N. Lu, J. Wirtz, W. H. Kunz, S. Paluch, T. Gruber, A. Martins, and P. G. Patterson, “Service robots, customers and service employees: what can we learn from the academic literature and where are the gaps?” Journal of Service Theory and Practice , vol. 30, no. 3, pp. 361–391, 2020

  6. [5]

    Droid: A large-scale in-the-wild robot manipulation dataset,

    A. Khazatsky, K. Pertsch, S. Nair, A. Balakrishna, S. Dasari, S. Karam- cheti, S. Nasiriany, M. K. Srirama, L. Y . Chen, K. Ellis et al., “Droid: A large-scale in-the-wild robot manipulation dataset,” arXiv preprint arXiv:2403.12945, 2024

  7. [6]

    Prima-care: Privacy-preserving multi-modal dataset for human ac- tivity recognition in care robots,

    A. Baselizadeh, M. Z. Uddin, W. Khaksar, D. S. Lindblom, and J. Torre- sen, “Prima-care: Privacy-preserving multi-modal dataset for human ac- tivity recognition in care robots,” in Companion of the 2024 ACM/IEEE International Conference on Human-Robot Interaction , 2024, pp. 233– 237

  8. [7]

    Semi-automatic dataset generation for object detection and recognition and its evaluation on domestic service robots,

    Y . Ishida and H. Tamukoh, “Semi-automatic dataset generation for object detection and recognition and its evaluation on domestic service robots,” Journal of Robotics and Mechatronics, vol. 32, no. 1, pp. 245–253, 2020

Show all 35 references
  1. [8]

    Object detection for sweeping robots in home scenes (odsr-ihs): a novel benchmark dataset,

    Y . Lv, Y . Fang, W. Chi, G. Chen, and L. Sun, “Object detection for sweeping robots in home scenes (odsr-ihs): a novel benchmark dataset,” IEEE Access, vol. 9, pp. 17 820–17 828, 2021

  2. [9]

    A multimodal dataset for object model learning from natural human-robot interaction,

    P. Azagra, F. Golemo, Y . Mollard, M. Lopes, J. Civera, and A. C. Murillo, “A multimodal dataset for object model learning from natural human-robot interaction,” in 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2017, pp. 6134–6141

  3. [10]

    Air-act2act: Human–human interaction dataset for teaching non-verbal social behaviors to robots,

    W.-R. Ko, M. Jang, J. Lee, and J. Kim, “Air-act2act: Human–human interaction dataset for teaching non-verbal social behaviors to robots,” The International Journal of Robotics Research , vol. 40, no. 4-5, pp. 691–697, 2021

  4. [11]

    Robocup@ home: Evaluation of voice recognition systems for domestic service robots and introducing latino dataset,

    E. Brinckhaus, G. T. Barnech, M. Etcheverry, and F. Andrade, “Robocup@ home: Evaluation of voice recognition systems for domestic service robots and introducing latino dataset,” in 2021 Latin American Robotics Symposium (LARS), 2021 Brazilian Symposium on Robotics (SBR), and 2...

  5. [12]

    A dataset and system for service robot action interaction based on skeleton action recognition,

    J. Wang, T. Zhang, X. Wu, and L. Zeng, “A dataset and system for service robot action interaction based on skeleton action recognition,” in 2023 8th International Conference on Signal and Image Processing (ICSIP). IEEE, 2023, pp. 41–48

  6. [13]

    Etri-activity3d: A large-scale rgb-d dataset for robots to recognize daily activities of the elderly,

    J. Jang, D. Kim, C. Park, M. Jang, J. Lee, and J. Kim, “Etri-activity3d: A large-scale rgb-d dataset for robots to recognize daily activities of the elderly,” in 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) . IEEE, 2020, pp. 10 990–10 997

  7. [14]

    Socially compliant navigation dataset (scand): A large-scale dataset of demonstrations for social navigation,

    H. Karnan, A. Nair, X. Xiao, G. Warnell, S. Pirk, A. Toshev, J. Hart, J. Biswas, and P. Stone, “Socially compliant navigation dataset (scand): A large-scale dataset of demonstrations for social navigation,” IEEE Robotics and Automation Letters, vol. 7, no. 4, pp. 11 807–11 814, 2022

  8. [15]

    Ssup-hri: Social signaling in urban public human-robot interaction dataset,

    F. Bu and W. Ju, “Ssup-hri: Social signaling in urban public human-robot interaction dataset,” arXiv preprint arXiv:2403.10994 , 2024

  9. [16]

    J. J. Gibson, The Ecological Approach to Visual Perception. Psychology Press Classic Editions, 1979

  10. [17]

    A formal model of affordances for flexible robotic task execution,

    D. Beßler, R. Porzel, M. Pomarlan, M. Beetz, R. Malaka, and J. Bate- man, “A formal model of affordances for flexible robotic task execution,” in Proceedings of the 24th European Conference on Artificial Intelli- gence (ECAI), 09 2020

  11. [18]

    Affordance ontology: towards a unified description of affordances as events,

    L. A. Moralez, “Affordance ontology: towards a unified description of affordances as events,” Res. Cogitans, vol. 7, no. 1, pp. 35–45, 2016

  12. [19]

    Modeling affordances with dispositions,

    F. TOYOSHIMA, “Modeling affordances with dispositions,” in Proceed- ings of the Joint Ontology Workshops , 2018

  13. [20]

    Affordances, motivation, and the design of user interfaces,

    J. Karat, C.-m. Karat, and J. Ukelson, “Affordances, motivation, and the design of user interfaces,” Commun. ACM, vol. 43, pp. 49–51, 08 2000

  14. [21]

    A review of affordances and affordance-based design to address usability,

    N. Masoudi, G. M. Fadel, C. C. Pagano, and M. V . Elena, “A review of affordances and affordance-based design to address usability,” Proceed- ings of the Design Society: International Conference on Engineering Design, vol. 1, no. 1, p. 1353–1362, 2019

  15. [22]

    Visual affordance and function understanding: A survey,

    M. Hassanin, S. Khan, and M. Tahtali, “Visual affordance and function understanding: A survey,” ACM Comput. Surv. , vol. 54, no. 3, Apr

  16. [23]

    Afnet: The affordance network,

    K. M. Varadarajan and M. Vincze, “Afnet: The affordance network,” in Asian Conference on Computer Vision , 2012. [Online]. Available: https://api.semanticscholar.org/CorpusID:39914795

  17. [24]

    3d affordancenet: A benchmark for visual object affordance understanding,

    S. Deng, X. Xu, C. Wu, K. Chen, and K. Jia, “3d affordancenet: A benchmark for visual object affordance understanding,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2021

  18. [25]

    Open-vocabulary affordance detection in 3d point clouds,

    T. Nguyen, M. N. Vu, A. Vuong, D. Nguyen, T. V o, N. Le, and A. Nguyen, “Open-vocabulary affordance detection in 3d point clouds,” in Proceedings of the International Conference on Robotic Systems (IROS), 2023

  19. [26]

    Learning affordance grounding from exocentric images,

    H. Luo, W. Zhai, J. Zhang, Y . Cao, and D. Tao, “Learning affordance grounding from exocentric images,” 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pp. 2242–2251, 2022. [Online]. Available: https://api.semanticscholar.org/CorpusID:247594536

  20. [27]

    Cerberus transformer: Joint semantic, affordance and attribute parsing,

    X. Chen, T. Liu, H. Zhao, G. Zhou, and Y .-Q. Zhang, “Cerberus transformer: Joint semantic, affordance and attribute parsing,” 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pp. 19 617–19 626, 2021. [Online]. Available: https://api.semanticscholar...

  21. [28]

    Openscene: 3d scene understanding with open vocabularies,

    S. Peng, K. Genova, ChiyuMaxJiang, A. Tagliasacchi, M. Pollefeys, and T. A. Funkhouser, “Openscene: 3d scene understanding with open vocabularies,” 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pp. 815–824, 2022. [Online]. Available: https://api....

  22. [30]

    Learning visual affordance grounding from demonstration videos,

    H. Luo, W. Zhai, J. Zhang, Y . Cao, and D. Tao, “Learning visual affordance grounding from demonstration videos,” IEEE Transactions on Neural Networks and Learning Systems , 2023

  23. [31]

    Demo2vec: Reasoning object affordances from online videos,

    K. Fang, T.-L. Wu, D. Yang, S. Savarese, and J. J. Lim, “Demo2vec: Reasoning object affordances from online videos,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2018, pp. 2139–2147

  24. [32]

    The epic- kitchens dataset: Collection, challenges and baselines,

    D. Damen, H. Doughty, G. M. Farinella, S. Fidler, A. Furnari, E. Kaza- kos, D. Moltisanti, J. Munro, T. Perrett, W. Price et al. , “The epic- kitchens dataset: Collection, challenges and baselines,” IEEE Transac- tions on Pattern Analysis and Machine Intelligence , vol. 43, no...

  25. [33]

    Ego4d: Around the world in 3,000 hours of egocentric video,

    K. Grauman, A. Westbury, E. Byrne, Z. Chavis, A. Furnari, R. Girdhar, J. Hamburger, H. Jiang, M. Liu, X. Liu et al., “Ego4d: Around the world in 3,000 hours of egocentric video,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 1...

  26. [34]

    Affordances from human videos as a versatile representation for robotics,

    S. Bahl, R. Mendonca, L. Chen, U. Jain, and D. Pathak, “Affordances from human videos as a versatile representation for robotics,” in Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 13 778–13 790

  27. [2021]

    Available: https://doi.org/10.1145/3446370

    [Online]. Available: https://doi.org/10.1145/3446370

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.