Pith. sign in

REVIEW 3 major objections 5 minor 90 references

Can Vision Language Models Understand Mimed Actions?

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Vision-language models cannot reliably identify mimed actions once salient context is stripped away, even though human adults find the same gestures almost effortless to read.

desk verdict A well-built mime benchmark with a real human–VLM gap, but the headline claim that VLMs lack gesture understanding is only partially supported because the REAL control conflates context removal with synthetic-domain shift. read the letter →

arxiv 2506.21586 v2 pith:Z7O26I4G submitted 2025-06-17 cs.CL cs.AIcs.CV

Hyundong Cho , Spencer Lin , Tejas Srinivasan , Michael Saxon , Deuksin Kwon , Natali T. Chavez , Jonathan May This is my paper
classification cs.CLcs.AIcs.CV
keywords vision-languagemodelsmimerecognitionnonverbalcommunicationvideoquestionansweringactionmotioncapturebenchmarkgestureunderstanding
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Vision-language models can describe actions in ordinary video, but this paper argues that they are not actually reading the body: when the scene around a gesture is emptied of context, the same models lose track of actions that human adults identify almost without effort. The paper introduces MIME, a video question-answering benchmark built from motion-captured actors retargeted to 3D characters, with 86 mimed actions and ten variants that swap backgrounds, characters, and camera angles so the gesture itself is the only reliable cue. Humans score 99.6% on the multiple-choice condition and 89.5% on free-form naming; the strongest VLM scores 52.3% and 19.8%, and in free-form answers under adversarial conditions every evaluated model falls below 10%. Aligned backgrounds sharply raise model scores while misaligned backgrounds drag them down, indicating that current models lean on scene hints rather than on the mimed movement. The paper concludes that dependable gesture understanding is a missing prerequisite for vision-language models to move toward broader nonverbal-communication competence.

What carries the argument

The load-bearing object is MIME, a video-based question-answering benchmark whose construction pipeline makes context removal and recombination systematic: motion-capture takes are retargeted to 3D characters, rendered with transparent backgrounds, and overlaid with blank, aligned, or misaligned background images; the same action is re-rendered with a different human character, an adversarial sci-fi character, and camera rotations of 0, 90, 180, and 270 degrees. Each of the 86 actions therefore appears in ten controlled conditions, which is what lets the paper isolate the contribution of body movement from the contribution of scene. The paired REAL dataset of live-action clips provides the control showing that the same models perform well when salient context is present.

What would settle it

Create a photorealistic benchmark of the same 86 actions with every salient object and scene cue removed, and run the same models on it: if accuracy rises to near-human levels, the MIME gap would be explained by synthetic-domain shift, whereas if it remains near MIME's low levels, the claim that models fail to read the gesture itself survives.

Watch

Extended reading notes

Core claim

The central claim is that contemporary VLMs identify actions through salient context, not through the gesture itself, and therefore cannot reliably recognize mime even though it is a low-variance, universally readable form of nonverbal communication. On MIME's base setting, models that are near-perfect on the REAL live-action control drop sharply when objects, outfits, and settings are absent; in the free-form naming condition the gap is extreme, with the best model at 19.8% versus humans at 89.5%. Perturbations tell the same story in the other direction: an aligned background like a basketball court lifts even an adversarially clothed character's score, while a misaligned background can hijack the answer. The paper's failure-mode analysis attributes the majority of remaining errors to incorrect visual descriptions of the demonstrated movements rather than to failures of language reasoning, and its preliminary interventions (Chain-of-Thought, few-shot prompting, and small-scale fine-tuning) do not close the human-model gap.

Load-bearing premise

The load-bearing assumption is that the rendered 3D characters and synthetic backgrounds behave like real human mime for a vision system, so that low scores on MIME mean a deficit in gesture understanding rather than an artifact of the animated domain.

Editorial extensions

If this is right

  • High accuracy on standard action-recognition video should not be read as evidence of body-motion understanding; context-stripped evaluations like MIME are needed to separate scene cues from gesture understanding.
  • A VLM that matched humans on MIME would clear one explicit prerequisite for interpreting, and eventually generating, the subtler varieties of nonverbal communication.
  • The aligned-background boost shows that scene priors can mask a missing grasp of gestures, so evaluation protocols for action understanding should control background content rather than leave it uncontrolled.
  • Because Chain-of-Thought, few-shot prompting, and small-scale fine-tuning leave free-form accuracy near zero, the paper's results imply the deficit is in visual observation of movement, not in the language-model reasoning layer.
  • The same rendering pipeline can produce fresh permutations of mimed actions on demand, giving the benchmark a defense against models that memorize public test items.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the same perturbation template were applied to photorealistic avatars, the MIME-versus-REAL gap could be separated into a synthetic-domain-shift component and a gesture-understanding component; the current REAL control cannot make that split because every REAL clip contains salient context.
  • The misaligned-background results suggest a testable shortcut detector: regress model predictions on automatically generated scene labels while holding the gesture fixed; if scene labels predict answers better than motion features do, the context-dependence claim is quantified rather than merely demonstrated by accuracy swings.
  • MIME's 86 actions are deliberately high-agreement mimes, so the benchmark should be read as the easy floor for gesture understanding; a model that passes it would still face the far harder, culturally variable remainder of nonverbal communication.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces MIME (Mime Identification Multimodal Evaluation), a video-based question-answering benchmark built from motion-captured mimes retargeted onto 3D characters, with systematic perturbations of character, background, and camera angle. The benchmark contains 86 mimed action samples across 47 action types, each evaluated in multiple-choice and free-form answer formats. The authors measure human performance with 60 participants and evaluate five VLMs (Qwen2.5-VL 3B/7B, InternVL2.5 8B, Phi-3.5, GPT-4o Mini, Gemini 1.5 Flash) in zero-shot, chain-of-thought, few-shot, and fine-tuned settings. They find that humans are nearly perfect and robust to all perturbations, while VLMs perform far worse, especially in free-form format and under adversarial perturbations, and are substantially helped by aligned backgrounds. The paper concludes that current VLMs lack robust understanding of mimed actions and rely heavily on salient context.

Significance. If the central claim survives scrutiny, MIME is a valuable new benchmark: it offers a controlled, replicable pipeline for probing action understanding without salient context, uses motion capture and rendering to permute factors that are hard to ablate in real video, includes human validation with three raters per sample, and ships data and code. The finding that aligned backgrounds sharply boost VLM performance is reproducible and falsifiable, and the paper's attempt to ground MIME in a REAL control is a useful design choice. The main weakness is that the broad conclusion about gesture understanding is partly confounded by the synthetic nature of MIME; the paper acknowledges this in Limitations but does not fully resolve it. On balance, the contribution is potentially significant if the domain-shift concern is addressed or the claims are appropriately narrowed.

major comments (3)
  1. [Section 3.4 and Figure 4] The REAL control does not isolate gesture recognition from the synthetic-to-real domain shift. Every REAL clip contains the full salient context (e.g., a barbell and gym for deadlifting), so the MIME-vs-REAL gap in Figure 5 conflates two variables: removal of context and the switch from Mixamo-rendered characters to photorealistic footage. The Limitations section acknowledges this risk, but the argument that human-level performance on MIME implies models should also succeed is an assumption, not empirical evidence. Please add a control that uses real human mime without salient context (e.g., the Mimetics dataset or a newly collected real-mime subset), or alternatively a within-MIME control that adds context/objects to the same rendered characters, or substantially narrow the conclusion from 'VLMs cannot understand mimed actions' to 'VLMs perform poorly on synthetic out-of-context mime and are highly context-dependent.'
  2. [Section 5.2 and Table 4] The paper states that 'the only method that consistently improves model performance over zero-shot is few-shot in-context learning for API-based black-box models,' but Table 4 contradicts this for the free-form format: GPT-4o Mini's FF accuracy on Base & blank drops from 11.6% to 9.3% with few-shot, Gemini 1.5 Flash's FF accuracy drops from 19.8% to 14.0%, and on Base & misaligned background GPT-4o Mini falls from 3.5% to 0.0%. The claim should be restricted to the multiple-choice condition or otherwise qualified to account for these decreases.
  3. [Section 1 and Table 1] The sentence 'Accuracy is even lower for videos with adversarial perturbations, for which all evaluated models achieve less than 10%' is contradicted by Table 1: for the MC condition with the adversarial character and a blank background, model accuracies range from 25.6% (InternVL2.5 8B) to 44.2% (Phi-3.5 and Gemini 1.5 Flash). If the intended claim refers only to the FF format, it should say so explicitly; as written, it overstates the results.
minor comments (5)
  1. [Section 3.1] The free-form correctness metric uses a heuristically selected cosine-similarity threshold of 0.5. Since this is not fitted to model predictions it is not circular, but the paper should report sensitivity of the main FF conclusions to this threshold, or provide precision/recall numbers, so readers can see that the 19.8% and sub-10% figures are not threshold artifacts.
  2. [Section 3.1] The multiple-choice distractors remove the top 10 most similar action labels by sentence-embedding cosine similarity. This makes MC easier and is acknowledged, but there is no experiment showing how sensitive rankings are to this choice; a random-distractor ablation would make the comparison more informative.
  3. [Section 5.1] The method enumeration lists (i) Chain-of-Thought, (ii) Few-shot, and then (iv) Fine-tuning; the numbering skips (iii). Please renumber.
  4. [Table 6] The symbols #, H#, and the checkmark used to label failure modes in Table 6 are not defined in the table caption; please add a legend or spell out the meanings in the caption.
  5. [Section 1 and Section 4.2] There are typos: 'free-from short answers' should be 'free-form short answers' in Section 1, and 'resuls' should be 'results' in Section 4.2.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the evaluation is externally grounded and no fitted quantity is renamed as a prediction.

full rationale

The paper's central claim—that VLMs perform far below humans on mimed-action recognition—is supported by an externally grounded evaluation, not by a self-referential derivation. Gold labels are fixed by a human-consensus filtering step ('Only the motion capture data for which at least two out of three authors assign the same label to the final rendered output without seeing the action name are included in MIME', Section 2.1), and model accuracy is scored against those fixed labels in multiple-choice and free-form formats. Nothing in the benchmark construction or scoring fits a parameter to the tested models' outputs. The free-form scorer's cosine-similarity threshold of 0.5 is explicitly described as 'heuristically-selected' (Section 3.1) and is applied identically to humans and models, with acknowledged false positives and false negatives; it is therefore an evaluation choice, not a fitted quantity that manufactures the reported gap. The REAL control, background, character, and viewpoint perturbations are ablations of the input video content, and the observed boost from aligned backgrounds is an empirical result about model sensitivity to context, not a quantity derived from the definition of the benchmark. The acknowledged lack of photorealism in the Limitations ('MIME is not photorealistic as it contains animated videos of motion capture data... may introduce a domain shift for VLMs') is a genuine validity threat about whether synthetic-domain performance transfers to real human mime, but a domain-shift confound is not circularity: the paper's scores are measured, not entailed by its assumptions. No self-citation is load-bearing; the only author citations (e.g., Saxon et al. 2024 for the text-image similarity metric) are used to justify a standard evaluation heuristic, not to establish the empirical finding. The derivation chain from data collection and human labeling to measured model accuracy is self-contained, so the appropriate finding is no significant circularity.

Assumptions & free parameters 1 free parameters · 2 assumptions · 0 invented entities

The benchmark rests on two domain assumptions: stable human agreement on mime, and semantic preservation through motion-capture retargeting to 3D avatars. The only hand-set numeric constant in evaluation is the 0.5 cosine threshold for free-form scoring; no fitted model parameters enter the labels.

free parameters (1)
  • free-form correctness threshold = 0.5 (cosine similarity in Sentence-BERT embedding space)
    Heuristically chosen threshold to decide whether a VLM's free-form answer matches the gold label; used for all human and model FF scores, but no principled justification is given beyond the authors' inspection.
assumptions (2)
  • domain assumption Mimed actions have low interpretation variance across humans, making them reliable ground truth.
    Invoked in Section 1 and Section 3.3; human evaluation shows ~99% MC agreement, supporting it, but participants are a convenience sample of 60 from one university.
  • domain assumption Retargeting motion capture data to 3D Mixamo characters preserves the semantic content of each mimed action.
    Used throughout Sections 2.2-2.3; they verify recognizability only through human eval, and acknowledge that characters with divergent body proportions can make an action unrecognizable.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Can Vision Language Models Understand Mimed Actions?." pith.science (2026). https://pith.science/paper/Z7O26I4G

@misc{pith2026250621586,
  author       = {Pith},
  title        = {Pith review of: Can Vision Language Models Understand Mimed Actions?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Z7O26I4G}},
  note         = {Machine review of arXiv:2506.21586}
}
read the original abstract

Nonverbal communication (NVC) plays an integral role in human language, but studying NVC in general is challenging because of its broad scope and high variance in interpretation among individuals and cultures. However, mime -- the theatrical technique of suggesting intent using only gesture, expression, and movement -- is a subset of NVC that consists of explicit and embodied actions with much lower human interpretation variance. We argue that a solid understanding of mimed actions is a crucial prerequisite for vision-language models capable of interpreting and commanding more subtle aspects of NVC. Hence, we propose Mime Identification Multimodal Evaluation (MIME), a novel video-based question answering benchmark comprising of 86 mimed actions. Constructed with motion capture data, MIME consists of variations of each action with perturbations applied to the character, background, and viewpoint for evaluating recognition robustness. We find that both open-weight and API-based vision-language models perform significantly worse than humans on MIME, motivating the need for increased research for instilling more robust understanding of human gestures.

Figures

Figures reproduced from arXiv: 2506.21586 by the authors.

Figure 1
Figure 1. Simplified illustration of a sample in MIME shown with a few frames from a video of a 3D male character miming a basketball shot in a living room. Humans achieve almost perfect accuracy on identifying mimed actions regardless of evaluation format, adver￾sarial perturbations, and the absence of salient context (e.g., basketball, court, basketball outfit), while VLMs struggle without salient context. Neill, 1992; Eave… view at source ↗
Figure 2
Figure 2. An overview of the pipeline for constructing [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Overview of variations of each action in [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: A frame from videos of deadlifting from MIME (left) and REAL (right). In MIME, salient context is missing (e.g., barbell and gym clothing). For open-source models, we evaluate on (i) Qwen 2.5 VL Instruction (Bai et al., 2025), both 3B and 7B versions, (ii) InternVL 2.5…
Figure 5
Figure 5. Figure 5: Performance comparison on the base setting of [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Our interface for human evaluation. The evaluators can only attempt to answer the question after seeing [PITH_FULL_IMAGE:figures/full_fig_p018_6.png]
Figure 7
Figure 7. Figure 7: Snapshots of outputs from various video generation models to generate mimed actions. All models that [PITH_FULL_IMAGE:figures/full_fig_p019_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

90 extracted references · 59 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Qin Cai, Vishrav Chaudhary, Dong Chen, Dongdong Chen, Weizhu Chen, Yen-Chun Chen, Yi-Ling Chen, Hao Cheng, Parul Chopra, Xiyang Dai, Matt...

    arXiv 2024

  4. [4]

    Simon Alexanderson, Carol O'Sullivan, Michael Neff, and Jonas Beskow. 2017. Mimebot—investigating the expressibility of non-verbal communication across agent embodiments. ACM Transactions on Applied Perception, 14(4)

    2017

  5. [5]

    Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. 2017. Localizing moments in video with natural language. In Proceedings of the IEEE international conference on computer vision, pages 5803--5812

    2017

  6. [6]

    Michael Argyle and Peter Trower. 1979. Person to person: ways of communicating. (No Title)

    1979

  7. [7]

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966

    arXiv 2023

  8. [8]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. 2025. Qwen2.5-vl technical rep...

    arXiv 2025

Show all 90 references
  1. [9]

    Kristy Boyer, Joseph Grafsgaard, Eun Young Ha, Robert Phillips, and James Lester. 2011. https://aclanthology.org/P11-1119/ An affect-enriched dialogue act classification model for task-oriented dialogue . In Proceedings of the 49th Annual Meeting of the Association for Computa...

    2011

  2. [10]

    Michael Burke and Joan Lasenby. 2015. https://doi.org/10.1109/TRO.2015.2475956 Pantomimic gestures for human–robot interaction . Trans. Rob., 31(5):1225–1237

    2015

  3. [11]

    Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N Chang, Sungbok Lee, and Shrikanth S Narayanan. 2008. Iemocap: Interactive emotional dyadic motion capture database. Language resources and evaluation, 42:335--359

    2008

  4. [12]

    Narayanan

    Carlos Busso, Zhigang Deng, Michael Grimm, Ulrich Neumann, and Shrikanth S. Narayanan. 2007. https://api.semanticscholar.org/CorpusID:11553244 Rigid head motion in expressive speech animation: Analysis and synthesis . IEEE Transactions on Audio, Speech, and Language Processing...

    2007

  5. [13]

    Cass, Kristina Striegnitz, and Nick Webb

    Aaron G. Cass, Kristina Striegnitz, and Nick Webb. 2018. https://api.semanticscholar.org/CorpusID:198357469 A farewell to arms: Non-verbal communication for non-humanoid robots . In IEEE/ACM International Conference on Human-Robot Interaction

    2018

  6. [14]

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. 2024. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Comput...

    2024

  7. [15]

    Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. 2018. Scaling egocentric vision: The epic-kitchens dataset. In Proceedings of the European conference on ...

    2018

  8. [16]

    Starkey Duncan Jr. 1969. Nonverbal communication. Psychological bulletin, 72(2):118

    1969

  9. [17]

    Michael Eaves and Dale G Leathers. 2015. Successful nonverbal communication: Principles and applications

    2015

  10. [18]

    Howard S Friedman. 1979. Nonverbal communication between patients and medical practitioners. Journal of Social Issues, 35(1):82--99

    1979

  11. [19]

    Gemini. 2024. https://arxiv.org/abs/2403.05530 Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context . Preprint, arXiv:2403.05530

    2024 arXiv

  12. [20]

    Hans Peter Graf, Eric Cosatto, Volker Strom, and Fu Jie Huang. 2002. https://api.semanticscholar.org/CorpusID:14906573 Visual prosody: facial movements accompanying speech . Proceedings of Fifth IEEE International Conference on Automatic Face Gesture Recognition, pages 396--401

    2002

  13. [21]

    Grafsgaard, Christopher Mitchell, Kristy Elizabeth Boyer, and James C

    Eun Young Ha, Joseph F. Grafsgaard, Christopher Mitchell, Kristy Elizabeth Boyer, and James C. Lester. 2012. https://aclanthology.org/W12-1634/ Combining verbal and nonverbal features to overcome the information gap in task-oriented dialogue . In Proceedings of the 13th Annual...

    2012

  14. [22]

    Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. 2015. https://api.semanticscholar.org/CorpusID:1710722 Activitynet: A large-scale video benchmark for human activity understanding . 2015 IEEE Conference on Computer Vision and Pattern Recognition ...

    2015

  15. [23]

    Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, and Noah A Smith. 2023. Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering. In Proceedings of the IEEE/CVF International Conference on Computer Vision, ...

    2023

  16. [24]

    in the wild

    Haroon Idrees, Amir R Zamir, Yu-Gang Jiang, Alex Gorban, Ivan Laptev, Rahul Sukthankar, and Mubarak Shah. 2017. The thumos challenge on action recognition for videos “in the wild”. Computer Vision and Image Understanding, 155:1--23

    2017

  17. [25]

    Ryo Ishii, Ryuichiro Higashinaka, and Junji Tomita. 2018. https://aclanthology.org/L18-1465/ Predicting nods by using dialogue acts in dialogue . In Proceedings of the Eleventh International Conference on Language Resources and Evaluation ( LREC 2018) , Miyazaki, Japan. Europe...

    2018

  18. [26]

    Ryo Ishii, Shiro Kumano, and Kazuhiro Otsuka. 2015. https://api.semanticscholar.org/CorpusID:8344315 Predicting next speaker based on head movement in multi-party meetings . 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2319--2323

    2015

  19. [27]

    Ryo Ishii, Kazuhiro Otsuka, Shiro Kumano, Masafumi Matsuda, and Junji Yamato. 2013. https://api.semanticscholar.org/CorpusID:18666652 Predicting next speaker and timing from gaze transition patterns in multi-party meetings . In International Conference on Multimodal Interaction

    2013

  20. [28]

    Ryo Ishii, Kazuhiro Otsuka, Shiro Kumano, and Junji Yamato. 2014. https://api.semanticscholar.org/CorpusID:18163970 Analysis and modeling of next speaking start timing based on gaze behavior in multi-party meetings . 2014 IEEE International Conference on Acoustics, Speech and ...

    2014

  21. [29]

    Ryo Ishii, Kazuhiro Otsuka, Shiro Kumano, and Junji Yamato. 2016 a . https://api.semanticscholar.org/CorpusID:15381585 Prediction of who will be the next speaker and when using gaze behavior in multiparty meetings . ACM Transactions on Interactive Intelligent Systems (TiiS), 6:1 -- 31

    2016

  22. [30]

    Ryo Ishii, Kazuhiro Otsuka, Shiro Kumano, and Junji Yamato. 2016 b . https://api.semanticscholar.org/CorpusID:10106978 Using respiration to predict who will speak next and when in multiparty meetings . ACM Trans. Interact. Intell. Syst., 6:20:1--20:20

    2016

  23. [31]

    Minhao Jiang, Ken Ziyu Liu, Ming Zhong, Rylan Schaeffer, Siru Ouyang, Jiawei Han, and Sanmi Koyejo. 2024. Investigating data contamination for pre-training language models. arXiv preprint arXiv:2401.06059

    2024 arXiv

  24. [32]

    Kristiina Jokinen. 2010. https://aclanthology.org/L10-1113/ Non-verbal signals for turn-taking and feedback . In Proceedings of the Seventh International Conference on Language Resources and Evaluation ( LREC `10) , Valletta, Malta. European Language Resources Association (ELRA)

    2010

  25. [33]

    Alexander Kapitanov, Karina Kvanchiani, Alexander Nagaev, Roman Kraynov, and Andrei Makhliarchuk. 2024. Hagrid -- hand gesture recognition image dataset. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 4572--4581

    2024

  26. [34]

    Piyali Karmakar and Manjira Sinha. 2024. https://aclanthology.org/2024.icon-1.37/ Aiding non-verbal communication: A bidirectional language agnostic framework for automating text to AAC generation . In Proceedings of the 21st International Conference on Natural Language Proces...

    2024

  27. [35]

    Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Li Fei-Fei. 2014. Large-scale video classification with convolutional neural networks. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 1725--1732

    2014

  28. [36]

    Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. 2017. The kinetics human action video dataset. arXiv preprint arXiv:1705.06950

    2017 arXiv

  29. [37]

    Adam Kendon. 1967. Some functions of gaze-direction in social interaction. Acta psychologica, 26:22--63

    1967

  30. [38]

    Lee Kezar, Jesse Thomason, and Zed Sehyr. 2023. https://doi.org/10.18653/v1/2023.eacl-main.200 Improving sign recognition with phonology . In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2732--2737, Dubrovni...

    2023 doi

  31. [39]

    Sotaro Kita. 2009. Cross-cultural variation of speech-accompanying gesture: A review. Language and cognitive processes, 24(2):145--167

    2009

  32. [40]

    Yu Kong and Yun Fu. 2022. Human action recognition and prediction: A survey. International Journal of Computer Vision, 130(5):1366--1401

    2022

  33. [41]

    Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. 2017. Dense-captioning events in videos. In Proceedings of the IEEE international conference on computer vision, pages 706--715

    2017

  34. [42]

    Hildegard Kuehne, Hueihan Jhuang, Est \' baliz Garrote, Tomaso Poggio, and Thomas Serre. 2011. Hmdb: a large video database for human motion recognition. In 2011 International conference on computer vision, pages 2556--2563. IEEE

    2011

  35. [43]

    Hodgins, Adam W

    Fernando De la Torre, Jessica K. Hodgins, Adam W. Bargteil, Xavier Martin, J. Robert Macey, Alex Tusell Collado, and Pep Beltran. 2008. https://api.semanticscholar.org/CorpusID:16721121 Guide to the carnegie mellon university multimodal activity (cmu-mmac) database

    2008

  36. [44]

    Jina Lee and Stacy Marsella. 2010. https://api.semanticscholar.org/CorpusID:17629956 Predicting speaker head nods and the effects of affective information . IEEE Transactions on Multimedia, 12:552--562

    2010

  37. [45]

    Yuncheng Li, Yale Song, Liangliang Cao, Joel Tetreault, Larry Goldberg, Alejandro Jaimes, and Jiebo Luo. 2016. Tgif: A new dataset and benchmark on animated gif description. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4641--4650

    2016

  38. [46]

    Patrick C Little and Chaz Firestone. 2021. Physically implied surfaces. Psychological Science, 32(5):799--808

    2021

  39. [47]

    Marianne Schmid Mast. 2007. On the importance of nonverbal communication in the physician--patient interaction. Patient education and counseling, 67(3):315--318

    2007

  40. [48]

    David Matsumoto and Hyisung C Hwang. 2013. Cultural similarities and differences in emblematic gestures. Journal of Nonverbal Behavior, 37:1--27

    2013

  41. [49]

    David McNeill. 1992. Hand and mind: What gestures reveal about thought

    1992

  42. [50]

    Albert Mehrabian. 1972. Nonverbal communication

    1972

  43. [51]

    Antoine Miech, Jean-Baptiste Alayrac, Ivan Laptev, Josef Sivic, and Andrew Zisserman. 2020. https://api.semanticscholar.org/CorpusID:220936243 Rareact: A video dataset of unusual interactions . ArXiv, abs/2008.01018

    2020 arXiv

  44. [52]

    Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. 2019. Howto100m: Learning a text-video embedding by watching hundred million narrated video clips. In Proceedings of the IEEE/CVF international conference on computer vision, ...

    2019

  45. [53]

    Anne Watson O'Reilly. 1995. Using representations: Comprehension and production of actions with imagined objects. Child development, 66(4):999--1010

    1995

  46. [54]

    Fran c ois Osiurak, Christophe Jarry, Nicolas Baltenneck, Bertrand Boudin, and Didier Le Gall. 2012. Make a gesture and i will tell you what you are miming. pantomime recognition in healthy subjects. cortex, 48(5):584--592

    2012

  47. [55]

    Ilias Papastratis, Kosmas Dimitropoulos, and Petros Daras. 2021. https://api.semanticscholar.org/CorpusID:233208980 Continuous sign language recognition through a context-aware generative adversarial network . Sensors (Basel, Switzerland), 21

    2021

  48. [56]

    Chanjun Park, Yoonna Jang, Seolhwa Lee, Jaehyung Seo, Kisu Yang, and Heuiseok Lim. 2022. https://doi.org/10.18653/v1/2022.aacl-demo.3 P ic T alky: Augmentative and alternative communication for language developmental disabilities . In Proceedings of the 2nd Conference of the A...

    2022 doi

  49. [57]

    Fernando Poyatos. 1983. Language and nonverbal systems in the structure of face-to-face interaction. Language & Communication, 3(2):129--140

    1983

  50. [58]

    Haoxuan Qu, Yujun Cai, and Jun Liu. 2024. Llms are good action recognizers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18395--18406

    2024

  51. [59]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learni...

    2021

  52. [60]

    Nils Reimers and Iryna Gurevych. 2019. https://doi.org/10.18653/v1/D19-1410 Sentence- BERT : Sentence embeddings using S iamese BERT -networks . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference...

    2019 doi

  53. [61]

    Anna Rohrbach, Marcus Rohrbach, Niket Tandon, and Bernt Schiele. 2015. A dataset for movie description. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3202--3212

    2015

  54. [62]

    Anna Rohrbach, Atousa Torabi, Marcus Rohrbach, Niket Tandon, Christopher Pal, Hugo Larochelle, Aaron Courville, and Bernt Schiele. 2017. Movie description. International Journal of Computer Vision, 123:94--120

    2017

  55. [63]

    Kurima Sakai, Carlos Toshinori Ishi, Takashi Minato, and Hiroshi Ishiguro. 2015. https://api.semanticscholar.org/CorpusID:3027054 Online speech-driven head motion generating system and evaluation on a tele-operated robot . 2015 24th IEEE International Symposium on Robot and Hu...

    2015

  56. [64]

    Ramon Sanabria, Ozan Caglayan, Shruti Palaskar, Desmond Elliott, Lo \" c Barrault, Lucia Specia, and Florian Metze. 2018. How2: A large-scale dataset for multimodal language understanding. In NeurIPS

    2018

  57. [65]

    Michael Saxon, Fatima Jahara, Mahsa Khoshnoodi, Yujie Lu, Aditya Sharma, and William Yang Wang. 2024. Who evaluates the evaluations? objectively scoring text-to-image prompt coherence metrics with t2iscorescore (ts2). arXiv preprint arXiv:2404.04251

    2024 arXiv

  58. [66]

    Zoya Shafique, Haiyan Wang, and Yingli Tian. 2023. Nonverbal communication cue recognition: A pathway to more accessible communication. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5666--5674

    2023

  59. [67]

    Syamimi Shamsuddin, Luthffi Idzhar Ismail, Hanafiah Yussof, Nur Ismarrubie Zahari, Saiful Bahari, Hafizan Hashim, and Ahmed Jaffar. 2011. Humanoid robot nao: Review of control and motion exploration. In 2011 IEEE international conference on Control System, Computing and Engine...

    2011

  60. [68]

    Gunnar A Sigurdsson, G \"u l Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta. 2016. Hollywood in homes: Crowdsourcing data collection for activity understanding. In Computer Vision--ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11--...

    2016

  61. [69]

    Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. 2012. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402

    2012 arXiv

  62. [70]

    Narayanan

    Vivek Kumar Rangarajan Sridhar, Srinivas Bangalore, and Shrikanth S. Narayanan. 2009. https://api.semanticscholar.org/CorpusID:10893126 Combining lexical, syntactic and prosodic cues for improved online dialog act tagging . Comput. Speech Lang., 23:407--422

    2009

  63. [71]

    Theodore Stickley. 2011. From soler to surety for effective non-verbal communication. Nurse education in practice, 11(6):395--398

    2011

  64. [72]

    Xingwu Sun, Yanfeng Chen, and Yiqing Huang. 2024. https://arxiv.org/abs/2411.02265 Hunyuan-large: An open-source moe model with 52 billion activated parameters by tencent . Preprint, arXiv:2411.02265

    2024 arXiv

  65. [73]

    Zehua Sun, Qiuhong Ke, Hossein Rahmani, Mohammed Bennamoun, Gang Wang, and Jun Liu. 2022. Human action recognition from various data modalities: A review. IEEE transactions on pattern analysis and machine intelligence, 45(3):3200--3225

    2022

  66. [74]

    Yunlong Tang, Jing Bi, Siting Xu, Luchuan Song, Susan Liang, Teng Wang, Daoan Zhang, Jie An, Jingyang Lin, Rongyi Zhu, et al. 2025. Video understanding with large language models: A survey. IEEE Transactions on Circuits and Systems for Video Technology

    2025

  67. [75]

    Ole Tange. 2024. https://doi.org/10.5281/zenodo.11247979 Gnu parallel 20240522 ('tbilisi')

    2024 doi

  68. [76]

    Atousa Torabi, Christopher Pal, Hugo Larochelle, and Aaron Courville. 2015. Using descriptive video services to create a large data source for video annotation research. arXiv preprint arXiv:1503.01070

    2015 arXiv

  69. [77]

    Indrit Troshani, Sally Rao Hill, Claire Sherman, and Damien Arthur. 2021. Do we trust in ai? role of anthropomorphism and intelligence. Journal of Computer Information Systems, 61(5):481--491

    2021

  70. [78]

    Karin van Nispen, W Mieke E van de Sandt-Koenderman, and Emiel Krahmer. 2017. Production and comprehension of pantomimes used to depict objects. Frontiers in Psychology, 8:1095

    2017

  71. [79]

    Mengmeng Wang, Jiazheng Xing, Jianbiao Mei, Yong Liu, and Yunliang Jiang. 2023. Actionclip: Adapting language-image pretrained models for video action recognition. IEEE Transactions on Neural Networks and Learning Systems

    2023

  72. [80]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824--24837

    2022

  73. [81]

    Zijian Zhang Weijie Kong, Qi Tian. 2024. https://arxiv.org/abs/2412.03603 Hunyuanvideo: A systematic framework for large video generative models

    2024 arXiv

  74. [82]

    Philippe Weinzaepfel and Gr \'e gory Rogez. 2021. Mimetics: Towards understanding human actions out of context. International Journal of Computer Vision, 129(5):1675--1690

    2021

  75. [83]

    Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.544 V ideo CLIP : Contrastive pre-training for zero-shot video-text understanding . In Proceeding...

    2021 doi

  76. [84]

    Jun Xu, Tao Mei, Ting Yao, and Yong Rui. 2016. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5288--5296

    2016

  77. [85]

    Ruijie Xu, Zengzhi Wang, Run-Ze Fan, and Pengfei Liu. 2024. Benchmarking benchmark leakage in large language models. arXiv preprint arXiv:2404.18824

    2024 arXiv

  78. [86]

    Yang Xu, Yang Cheng, and Riya Bhatia. 2022. https://aclanthology.org/2022.coling-1.12/ Gestures are used rationally: Information theoretic evidence from neural sequential models . In Proceedings of the 29th International Conference on Computational Linguistics, pages 134--140,...

    2022

  79. [87]

    Sitao Zhang, Yimu Pan, and James Z Wang. 2023. Learning emotion representations from verbal and nonverbal communication. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18993--19004

    2023

  80. [88]

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675

    2019 arXiv

  81. [89]

    Kun Zhou, Yutao Zhu, Zhipeng Chen, Wentong Chen, Wayne Xin Zhao, Xu Chen, Yankai Lin, Ji-Rong Wen, and Jiawei Han. 2023. Don't make your llm an evaluation benchmark cheater. arXiv preprint arXiv:2311.01964

    2023 arXiv

  82. [90]

    Luowei Zhou, Chenliang Xu, and Jason Corso. 2018. Towards automatic learning of procedures from web instructional videos. In Proceedings of the AAAI conference on artificial intelligence, volume 32

    2018

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.