Pith. sign in

REVIEW 3 major objections 6 minor 67 references

PAC Bench: Do Foundation Models Understand Prerequisites for Executing Manipulation Policies?

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper reports near-zero accuracy for current vision-language models on the physical constraints that gate robot manipulation, despite moderate scores on simpler object-property questions.

desk verdict Useful benchmark idea, but the headline constraint results are a measurement artifact and the paper needs major fixes before the numbers can be trusted. read the letter →

arxiv 2506.23725 v1 pith:ETAQWJBV submitted 2025-06-30 cs.RO cs.AI

Atharva Gundawar , Som Sagar , Ransalu Senanayake This is my paper · ORCID
classification cs.ROcs.AI
keywords PACBenchrobotmanipulationvision-languagemodelsaffordancesphysicalconstraintsobjectpropertiesreasoningevaluationbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that off-the-shelf vision-language models, despite their growing use in robot planning, failure detection, and policy generation, have not actually learned the granular physical knowledge a manipulator needs. To test this, PAC Bench decomposes manipulation prerequisites into object Properties, action Affordances, and physical Constraints, then probes 17 model variants with over 30,000 annotations drawn from real images, simulated scenes, and a humanoid robot's own camera views. The headline finding is stark: models do moderately on simple property questions and single affordances, but they collapse to near zero when asked to judge whether an action is blocked by geometry, occlusion, stability, or reachability—Claude 3.5 Sonnet averages 0.4% and Llama 3.2 90B Vision averages 0.2% on the constraint tasks. If this is right, end-to-end manipulation policies built on such backbones inherit a blind spot at exactly the preconditions they must verify before touching the world.

What carries the argument

The carrying mechanism is the PAC decomposition: every manipulation prerequisite is classified into one of three queryable components—Properties (12 intrinsic physical attributes such as weight, sealing, and density), Affordances (one to three action possibilities per object class, scored both as "at least one correct" and "all correct"), and Constraints (four simulated task domains plus real robot-view scenarios, scored as feasibility judgments). The dataset combines real web images with bounding-box crops, synthetic single-object renders from multiple viewpoints, and paired first- and third-person humanoid-camera scenes, so each component is tested across visual domains and viewpoints. Automated verification prompts check whether a model's free-text answer contains the required ground-truth content, converting qualitative VLM outputs into strict accuracy numbers.

What would settle it

Run the same affordance and constraint prompts with the image replaced by a blank or scrambled version; if accuracy stays roughly the same, the test is measuring language priors, not visual physical understanding, and the paper's central claim about VLM perception would be unsupported.

Watch

Extended reading notes

Core claim

PAC Bench's central claim is that contemporary vision-language models do not possess the granular, physically grounded understanding of object properties, action affordances, and physical constraints that robot manipulation actually demands. On the benchmark's constraint tasks, averaged over geometric/size mismatch, occlusion, stability, reachability, and real-world humanoid-view scenarios, most models score near zero: Claude 3.5 Sonnet averages 0.4% and Llama 3.2 90B Vision averages 0.2%, with the strongest tested model around 26%. On properties, accuracy is moderate at best and shifts sharply between web images and robot views; on affordances, models often find one correct action for an object but almost never recover the full ground-truth set. The authors read this as evidence that VLM sophistication on general visual question answering does not transfer to the task-executability prerequisites that a reliable manipulation policy must verify before acting.

Load-bearing premise

The whole benchmark rests on the assumption that a question's answer actually has to come from looking at the image: the affordance prompts name the object class, so a model could answer from language knowledge alone, and the simulated constraint scenes must actually show the blocker the label claims.

Editorial extensions

If this is right

  • Constraint reasoning, not object recognition, is the bottleneck: the strongest tested model averaged only about 26% on the constraint suite, and most models stayed below 10%, several below 2%.
  • Simulation scores do not transfer to robot-view reality: performance on simulated constraint scenes diverged sharply from performance on humanoid-camera scenarios, so sim-to-real transfer of constraint understanding remains an open problem.
  • A VLA built on an off-the-shelf VLM may inherit a blind spot for action infeasibility; the paper's diagnostic framing implies that failures traced to the backbone VLM should be fixed at the foundation-model level rather than in policy fine-tuning.
  • Chain-of-thought prompting is not a reliable fix: it helped some property queries for some models but gave only marginal, inconsistent gains on the constraint tasks.
  • Property and affordance accuracy is domain- and viewpoint-sensitive, so a model can look competent on web images while failing from an egocentric robot view.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct next experiment is a text-only control on the affordance and constraint items: if removing the image leaves accuracy roughly unchanged, the scores measure language priors rather than visual physical understanding.
  • The modular PAC decomposition invites a safety application the paper does not build: a refusal filter that blocks any action the VLM cannot certify as feasible, which would make near-zero constraint scores a conservative failure mode rather than a dangerous one.
  • One can test whether constraint reasoning is learnable by fine-tuning a VLM on the benchmark's negative-action examples (for instance, "penstand inverted" or "cube will not balance") and then measuring downstream manipulation success; the released dataset is structured to support such a study.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. PAC Bench (arXiv:2506.23725) introduces a benchmark that decomposes the physical prerequisites of robot manipulation into object Properties, action Affordances, and physical Constraints, and evaluates a wide range of vision-language models on these three components. The dataset combines OpenImages V7 images, RoboCasa synthetic renders, Unitree G1 humanoid captures, and MuJoCo-simulated scenes, with human annotations and automated verification prompts. The paper reports moderate performance on property and single-affordance tasks but near-zero accuracy on constraint tasks, and concludes that off-the-shelf VLMs lack the granular, physically grounded understanding needed for reliable manipulation.

Significance. If valid, PAC Bench would fill a genuine evaluation gap: existing benchmarks mostly target end-to-end policy performance or broad physical QA rather than the granular executability prerequisites that this paper decomposes. The hybrid data acquisition strategy, the use of consensus-based property annotation, the transparency about API costs, and the stated intent to release the data are all positive features. The paper also provides a broad model comparison and a useful diagnostic decomposition that could aid VLA development. However, the headline constraint result is currently not interpretable because the constraint protocol contains only impossible scenarios, and several dataset-count inconsistencies undermine the reported scale. These issues are substantial but appear fixable with additional positive controls, clearer metrics, and a data-accounting pass.

major comments (3)
  1. [§4.4, Table 4; Appendix B.3, C.2] The constraint evaluation cannot support the paper's central claim that VLMs 'profoundly struggle' with physical constraints. Every real-robot constraint answer in Appendix B.3 is negative, and every simulated scenario in Appendix C.2 is procedurally generated so that the proposed action is infeasible. Consequently, each cell of Table 4 is a negative-only accuracy: a degenerate model that always answers 'No, impossible' would score 100% on every constraint task without looking at the image. The near-zero values reported in Table 4 therefore measure an optimism/response prior rather than constraint understanding. In addition, the verification prompts in C.2 check only whether the model text says the task is impossible (or 'out of reach'/'might fall'); they never check whether the stated reason is the true physical cause, so a model that says 'No, impossible' for the wrong reason is scored as correct. The manuscript should add feasible positive scenarios, report balanced accuracy or a related metric, include a no-vision/text-only baseline, and score the explanation content rather than only the binary verdict.
  2. [Appendix C.3, C.4; §4.3] The affordance evaluation reveals the object class name directly in the prompt ('List all the possible affordances of a <object_name>') and uses class-level ground-truth affordances, so a model can answer from lexical priors without using any visual information from the image. This means the reported affordance scores, including the 'moderate' performance on single affordances and the near-zero performance on all-affordances recognition, do not measure visual affordance understanding. The paper should either remove or mask class-name cues, or explicitly present these results as a language-prior evaluation and add image-only or no-caption controls before claiming visual affordance deficits.
  3. [§3.1, Table 1, Appendix B.1, C.4] The dataset size claims are internally inconsistent. The main text reports 27,674 property annotations, but the component statistics in Appendix B.1 sum to only 785 (Real Robo) + 424 (RoboCasa) + 10,506 (OpenImages) = 11,715 property annotations, and C.4 alternately describes the OpenImages split as 10,506 annotations, 8,148 (image, property) pairs, and 12,421 annotation entries. The abstract's 673 real-world images and 100 real-world humanoid scenarios also conflict with the appendix's 679 OpenImages frames and 53 constraint scenarios / 67-68 property-affordance scenario pairs. These discrepancies must be reconciled before the benchmark's scale and reproducibility can be assessed.
minor comments (6)
  1. [§4.2, Table 2] The property index mapping is inconsistent: Section 4.2 defines P1-P12 with P1=Capacity, P2=Color, ..., P12=Weight, while Table 2 states that P1-P6 are Color, Contents, Weight, Density, Sealing, and Hardness. Please harmonize all property labels across the main text, tables, and appendices.
  2. [§3.1, Abstract, Appendix E] The number of property types is stated as 12 in Section 3.1 but as 15 in the abstract and Appendix E, while C.4 lists 12 named properties plus four domain-specific traits. Please make the property-type count consistent throughout.
  3. [Tables 3, 9, 10; Appendix D.3] The affordance category labels are inconsistent: Table 3 uses A1-A14, while Appendix D.3 uses A1-A18 and also refers to C1-C18; Table 10's note says 'C1-C18' but its columns are named A1-A18. Please standardize the category naming.
  4. [Appendix C.3, C.4, Figure 8] The property-evaluation prompt says 'the red bounding box' while Figure 8 shows a yellow bounding box; please align the text with the actual annotation visualization.
  5. [Appendix B.1, C.5] The Real Robo property subset is described as having 785 annotations across 67 scenarios with the schema expecting 12 properties per scenario; please clarify whether the 57 missing property rows (e.g., SEALING in 19 scenarios and other gaps) are intentional or a reporting artifact.
  6. [References] Reference [24] has a malformed author list ('Som Sager' and missing given names for several authors); please correct it, and also check the reference for the same-lab prior work.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: PAC Bench is an empirical evaluation benchmark whose results are measured against independent human annotations; the one self-citation is not load-bearing, and the constraint-set validity concerns are correctness risks, not circular derivation.

full rationale

PAC Bench makes no first-principles derivation or fitted prediction; it reports measured accuracies of VLMs against human-annotated ground truth. The only self-citation is reference [24], from the same lab, used as an example of downstream consumers of generic VLMs; it does not justify any load-bearing premise. The affordance protocol does reveal the object class name in the prompt (Appendix C.3), and the constraint evaluation uses only infeasible scenarios with all-negative ground truth (Appendices B.3 and C.2), so those scores may reflect language priors and negative-response bias rather than visual physical understanding. These are benchmark-validity threats and should be weighed as correctness risks, but they are not circular: the accuracy numbers are not fitted parameters, and no claim reduces by construction to a prior citation or to the benchmark's own definitions. The central empirical claim is therefore independent content, and the circularity burden is low.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

This is a benchmark paper, so the ledger is about the validity of the measurement, not about physical models. The key assumptions are that human labels are correct, that the PAC taxonomy is the right decomposition, that simulated scenes instantiate the intended constraints, and that the QA format measures understanding. No new physical entities or fitted parameters are introduced.

assumptions (4)
  • domain assumption Human annotations (two-annotator consensus for properties, single-annotator for affordances) are assumed to be correct ground truth.
    All accuracy numbers are computed against these labels; if the labels are noisy or biased, the benchmark scores are not meaningful. The paper acknowledges single-annotator affordances in Section 5.
  • domain assumption The PAC decomposition (Properties, Affordances, Constraints) captures the essential prerequisites for executing manipulation policies.
    The entire benchmark is built on this taxonomy, introduced in Section 3. If other dimensions (e.g., force dynamics, kinematic feasibility, task context) are equally important, the benchmark is incomplete as a diagnostic.
  • domain assumption MuJoCo procedural scenarios are valid instantiations of the intended constraint types (geometric mismatch, occlusion, stability, reachability).
    The paper generates 10 random scenes per domain with randomization of positions and sizes (Appendix C.2), but does not verify that a human would agree on the salient constraint in every scene.
  • domain assumption Multiple-choice accuracy and strict-match text evaluation measure VLM physical understanding.
    The evaluation protocol (Appendix C) assumes that a correct answer in this QA format reflects genuine comprehension rather than memorized associations or prompt artifacts. The near-zero constraint scores raise the question of whether the evaluation format itself is causing failures.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PAC Bench: Do Foundation Models Understand Prerequisites for Executing Manipulation Policies?." pith.science (2026). https://pith.science/paper/ETAQWJBV

@misc{pith2026250623725,
  author       = {Pith},
  title        = {Pith review of: PAC Bench: Do Foundation Models Understand Prerequisites for Executing Manipulation Policies?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ETAQWJBV}},
  note         = {Machine review of arXiv:2506.23725}
}
read the original abstract

Vision-Language Models (VLMs) are increasingly pivotal for generalist robot manipulation, enabling tasks such as physical reasoning, policy generation, and failure detection. However, their proficiency in these high-level applications often assumes a deep understanding of low-level physical prerequisites, a capability that remains largely unverified. For robots to perform actions reliably, they must comprehend intrinsic object properties (e.g., material, weight), action affordances (e.g., graspable, stackable), and physical constraints (e.g., stability, reachability, or an object's state, such as being closed). Despite the widespread use of VLMs in manipulation tasks, we argue that off-the-shelf models may lack this granular, physically grounded understanding, as such prerequisites are often overlooked during training. To address this critical gap, we introduce PAC Bench, a comprehensive benchmark designed to systematically evaluate VLMs on their understanding of core Properties, Affordances, and Constraints (PAC) from a task executability perspective. PAC Bench features a diverse dataset with over 30,000 annotations, comprising 673 real-world images (115 object classes, 15 property types, and 1 to 3 affordances defined per class), 100 real-world humanoid-view scenarios, and 120 unique simulated constraint scenarios across four tasks. Our evaluations reveal significant gaps in the ability of current VLMs to grasp fundamental physical concepts, highlighting limitations in their suitability for reliable robot manipulation and pointing to key areas for targeted research. PAC Bench also serves as a standardized benchmark for rigorously evaluating physical reasoning in VLMs and guiding the development of more robust, physically grounded models for robotic applications. Project Page: https://pacbench.github.io/

Figures

Figures reproduced from arXiv: 2506.23725 by the authors.

Figure 1
Figure 1. Evaluating foundation models’ understanding of Properties, Affordances, and Constraints [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Distribution of annotations in PAC Bench across three dimensions: (Left) physical proper [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Examples from PAC Bench. (Left 4 Images) Scenarios designed to evaluate understanding [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: Comparative PAC understanding profiles of selected VLMs. The x-axis indicates nominal [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: All affordance subset heatmap. Full heatmap in Appendix D.3 Overall Property Performance and Domain Sensitivity [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Example scenes corresponding to each constraint domain (left to right): (a) [PITH_FULL_IMAGE:figures/full_fig_p020_6.png]
Figure 7
Figure 7. Figure 7: Samples from robocasa datapoint in PACBench [PITH_FULL_IMAGE:figures/full_fig_p021_7.png]
Figure 8
Figure 8. Figure 8: Example from our Open Images subset: a single object annotated with a red bounding box. [PITH_FULL_IMAGE:figures/full_fig_p025_8.png]
Figure 9
Figure 9. Figure 9: Samples from Unitree G1 humanoid from PacBench [PITH_FULL_IMAGE:figures/full_fig_p025_9.png]
Figure 10
Figure 10. Figure 10: Performance of best models from each family [PITH_FULL_IMAGE:figures/full_fig_p038_10.png]
Figure 11
Figure 11. Figure 11: TkInter single-image property annotator (synthetic objects). [PITH_FULL_IMAGE:figures/full_fig_p041_11.png]
Figure 12
Figure 12. Figure 12: TkInter dual-view affordance annotator [PITH_FULL_IMAGE:figures/full_fig_p042_12.png]
Figure 13
Figure 13. Figure 13: TkInter dual-view property annotator (Real-Robo / RoboCasa). [PITH_FULL_IMAGE:figures/full_fig_p042_13.png]
Figure 14
Figure 14. Figure 14: Label Studio project dashboard with 15 property jobs. [PITH_FULL_IMAGE:figures/full_fig_p043_14.png]
Figure 15
Figure 15. Figure 15: Label Studio image view with bounding box and radio-button options. [PITH_FULL_IMAGE:figures/full_fig_p043_15.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

67 extracted references · 55 canonical work pages

  1. [1]

    Openvla: An open-source vision-language-action model

    Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, Quan Vuong, Thomas Kollar, Benjamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn. Openvla: An open-source vision-language-action model. arXiv preprint arXiv:2406.09246, 2024

    arXiv 2024

  2. [2]

    Octo: An open-source generalist robot policy

    Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, et al. Octo: An open-source generalist robot policy. arXiv preprint arXiv:2405.12213, 2024

    arXiv 2024

  3. [3]

    Rt-2: Vision-language- action models transfer web knowledge to robotic control

    Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choro- manski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision-language- action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818, 2023

    arXiv 2023

  4. [4]

    Lawrence Zitnick, and Devi Parikh

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2015

    work page 2015

  5. [5]

    The Colosseum: A benchmark for evaluating generalization for robotic manipulation, 2024

    Wilbert Pumacay, Ishika Singh, Jiafei Duan, Ranjay Krishna, Jesse Thomason, and Dieter Fox. The Colosseum: A benchmark for evaluating generalization for robotic manipulation, 2024

    work page 2024

  6. [6]

    PHYBench: Holistic evaluation of physical perception and reasoning in large language models, 2025

    Shi Qiu, Chenrui Zhang, Xin Zhao, Kai Wang, Jiazheng Zhang, Ruijie Yang, Junjie Hu, Hongyang Zhang, Yi Zhou, Wenhao Wang, et al. PHYBench: Holistic evaluation of physical perception and reasoning in large language models, 2025

    work page 2025

  7. [7]

    PhysBench: Benchmarking and enhancing vision-language models for physical world understanding, 2024

    Kang Zhu, Yifan Mai, Qian Huang, Ze Chen, Zhi-Yong He, Yue Xu, Wei Liu, Jure Leskovec, Anima Anandkumar, Cihang Xie, and Huaxiu Yao. PhysBench: Benchmarking and enhancing vision-language models for physical world understanding, 2024

    work page 2024

  8. [8]

    Decodingtrust: A comprehensive assessment of trustworthiness in gpt models

    Boxin Wang, Weixin Chen, Hengzhi Pei, Chulin Xie, Mintong Kang, Chenhui Zhang, Chejian Xu, Zidi Xiong, Ritik Dutta, Rylan Schaeffer, et al. Decodingtrust: A comprehensive assessment of trustworthiness in gpt models. In NeurIPS, 2023

    work page 2023

Show all 67 references
  1. [9]

    Air-bench 2024: A safety benchmark based on risk categories from regulations and policies

    Yi Zeng, Yu Yang, Andy Zhou, Jeffrey Ziwei Tan, Yuheng Tu, Yifan Mai, Kevin Klyman, Minzhou Pan, Ruoxi Jia, Dawn Song, et al. Air-bench 2024: A safety benchmark based on risk categories from regulations and policies. arXiv preprint arXiv:2407.17436, 2024

    2024 arXiv

  2. [10]

    Holistic evaluation of language models

    Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110, 2022

    2022 arXiv

  3. [11]

    Vhelm: A holistic evaluation of vision language models

    Tony Lee, Haoqin Tu, Chi Heem Wong, Wenhao Zheng, Yiyang Zhou, Yifan Mai, Jos- selin Somerville Roberts, Michihiro Yasunaga, Huaxiu Yao, Cihang Xie, et al. Vhelm: A holistic evaluation of vision language models. arXiv preprint arXiv:2410.07112, 2024

    arXiv

  4. [12]

    Cares: A comprehensive benchmark of trustworthiness in medical vision language models

    Peng Xia, Ze Chen, Juanxi Tian, Yangrui Gong, Ruibo Hou, Yue Xu, Zhenbang Wu, Zhiyuan Fan, Yiyang Zhou, Kangyu Zhu, et al. Cares: A comprehensive benchmark of trustworthiness in medical vision language models. arXiv preprint arXiv:2406.06007, 2024

    2024 arXiv

  5. [13]

    Mlagentbench: Evaluating language agents on machine learning experimentation

    Qian Huang, Jian V ora, Percy Liang, and Jure Leskovec. Mlagentbench: Evaluating language agents on machine learning experimentation. InForty-first International Conference on Machine Learning, 2024

    2024

  6. [14]

    2handedafforder: Learning precise actionable bimanual affordances from human videos, 2025

    Ananye Mukherjee, Priyanshu Agarwal, Ashish Kumar, Jason Corso, and Ashwin Balakrishna. 2handedafforder: Learning precise actionable bimanual affordances from human videos, 2025

    2025

  7. [15]

    Physvlm: Enabling visual language models to understand robotic physical reachability, 2025

    Yitao Liu, Chenfei Yuan, Shiqi Liu, Renjie Li, Zixuan Wang, Li Yi, and Yang Lv. Physvlm: Enabling visual language models to understand robotic physical reachability, 2025. 11

    2025

  8. [16]

    Physion: Evaluating physical prediction from vision in humans and machines

    Daniel M Bear, Judy Fan, Elias Dyer, T K Marks, Richard Futrell, Joshua B Tenenbaum, Elizabeth S Spelke, and Daniel L K Yamins. Physion: Evaluating physical prediction from vision in humans and machines. In Advances in Neural Information Processing Systems 34 (NeurIPS 2021), 2021

    2021

  9. [17]

    Ma- nipvqa: Injecting robotic affordance and physically grounded information into multi-modal large language models, 2024

    Haoyu Xu, Zhipeng Cao, Zhijun Zhang, Yifei Chen, Yixin Chen, and Zhiyong Wu. Ma- nipvqa: Injecting robotic affordance and physically grounded information into multi-modal large language models, 2024

    2024

  10. [18]

    Uniaff: A unified representation of affordances for tool usage and articulation with vision-language models, 2024

    Yilong Zhou, Zeyi Liu, Zihan Ding, Wei-Chih Hung, Pieter Abbeel, and Huazhe Xu. Uniaff: A unified representation of affordances for tool usage and articulation with vision-language models, 2024

    2024

  11. [19]

    Naturalvlm: Leveraging fine-grained natural language for affordance-guided visual manipulation, 2024

    Guangzheng Chen, Hongtao Wu, Jiafeng Gu, and Qing Li. Naturalvlm: Leveraging fine-grained natural language for affordance-guided visual manipulation, 2024

    2024

  12. [20]

    A brief review of affordance in robotic manipulation research

    Natsuki Yamanobe, Weiwei Wan, Ixchel G Ramirez-Alpizar, Damien Petit, Tokuo Tsuji, Shuichi Akizuki, Manabu Hashimoto, Kazuyuki Nagata, and Kensuke Harada. A brief review of affordance in robotic manipulation research. Advanced Robotics, 31(19-20):1086–1101, 2017

    2017

  13. [21]

    The theory of affordances:(1979)

    James J Gibson. The theory of affordances:(1979). In The people, place, and space reader, pages 56–60. Routledge, 2014

    1979

  14. [22]

    Human affordances for robotic pre-training

    Andy Zeng, Shuran Song, Johnny Lee, Stefan Lee, Alberto Rodriguez, and Silvio Savarese. Human affordances for robotic pre-training. In Proceedings of Robotics: Science and Systems (RSS), 2020

    2020

  15. [23]

    Visual affordance-guided policy optimization

    Corey Lynch and Pierre Sermanet. Visual affordance-guided policy optimization. InConference on Robot Learning (CoRL), 2021

    2021

  16. [24]

    From mystery to mastery: Failure diagnosis for improving manipulation policies

    Sreevishakh Vasudevan Yifan Zhou Heni Ben Amor Dieter Fox Ransalu Senanayake Som Sager, Jiafei Duan. From mystery to mastery: Failure diagnosis for improving manipulation policies. In arXiv:2412.02818, 2025

    2025 arXiv

  17. [25]

    The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale

    Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, Tom Duerig, and Vittorio Ferrari. The open images dataset v4: Unified image classification, object detection, and visu...

    2020

  18. [26]

    Robocasa: Large-scale simulation of everyday tasks for generalist robots

    Soroush Nasiriany, Abhiram Maddukuri, Lance Zhang, Adeet Parikh, Aaron Lo, Abhishek Joshi, Ajay Mandlekar, and Yuke Zhu. Robocasa: Large-scale simulation of everyday tasks for generalist robots. arXiv preprint arXiv:2406.02523, 2024

    2024 arXiv

  19. [27]

    Mujoco: A physics engine for model-based control

    Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pages 5026–5033. IEEE, 2012

    2012

  20. [28]

    Introducing Claude 3.5 Sonnet

    Anthropic. Introducing Claude 3.5 Sonnet. https://www.anthropic.com/news/ claude-3-5-sonnet , June 2024

    2024

  21. [29]

    Claude 3.7 Sonnet (System Prompt Release Notes)

    Anthropic. Claude 3.7 Sonnet (System Prompt Release Notes). https://docs.anthropic. com/en/release-notes/system-prompts , February 2025. System prompt release notes

    2025

  22. [30]

    Introducing GPT-4.1 (mini) in the API

    OpenAI. Introducing GPT-4.1 (mini) in the API. https://openai.com/index/gpt-4-1/ , April 2025

    2025

  23. [31]

    Grok-2 Beta Release

    xAI. Grok-2 Beta Release. https://x.ai/blog/grok-2, August 2024

    2024

  24. [32]

    xAI’s Grok Chatbot Can Now ’See’ the World Around It

    xAI. xAI’s Grok Chatbot Can Now ’See’ the World Around It. TechCrunch, April 2025

    2025

  25. [33]

    Llama 3.2 Vision Instruct Model Card

    Meta. Llama 3.2 Vision Instruct Model Card. https://huggingface.co/meta-llama/ Llama-3.2-11B-Vision-Instruct , September 2024. 12

    2024

  26. [34]

    Meta Releases Llama 4: A New Crop of Flagship AI Models

    TechCrunch. Meta Releases Llama 4: A New Crop of Flagship AI Models. TechCrunch, April 2025

    2025

  27. [35]

    Qwen-VL-Plus Model

    QwenLM. Qwen-VL-Plus Model. https://github.com/QwenLM/Qwen-VL, 2025

    2025

  28. [36]

    Qwen2.5-VL Technical Report

    Shuai Bai, Keqin Chen, Xuejing Liu, and et al. Qwen2.5-VL Technical Report. https: //arxiv.org/abs/2502.13923, February 2025

    2025 arXiv

  29. [37]

    Can we keep the ball inside the penstand?

    Susan Carey and Elizabeth Spelke. Science and core knowledge. Philosophy of science , 63(4):515–533, 1996. 13 A Broader impacts The primary goal of PAC Bench is to catalyze the development of more capable, reliable, and physically grounded VLMs and their fine-tuned variants, o...

    1996

  30. [38]

    Claude 3.7 Sonnet: https://openrouter.ai/anthropic/claude-3.7-sonnet

  31. [39]

    Claude 3.7 Sonnet (T): https://openrouter.ai/anthropic/claude-3.7-sonnet: thinking (This denotes Chain-of-Thought prompting applied to the Claude 3.7 Sonnet model.)

  32. [40]

    Claude 3.5 Sonnet: https://openrouter.ai/anthropic/claude-3.5-sonnet

  33. [41]

    Gemini 2.0 Flash 001: https://openrouter.ai/google/gemini-2.0-flash-001

  34. [42]

    5-flash-preview

    Gemini 2.5 Flash P: https://openrouter.ai/google/gemini-2. 5-flash-preview

  35. [43]

    5-pro-preview-03-25

    Gemini 2.5 Pro P: https://openrouter.ai/google/gemini-2. 5-pro-preview-03-25

  36. [44]

    GPT-4.1: https://openrouter.ai/openai/gpt-4.1

  37. [45]

    o4-mini-high: https://openrouter.ai/openai/o4-mini-high (Note: The "(T)" for this model in some tables also indicates Chain-of-Thought prompting.)

  38. [46]

    GPT-4.1 Mini: https://openrouter.ai/openai/gpt-4.1-mini

  39. [47]

    Llama 4 Maverick: https://openrouter.ai/meta-llama/llama-4-maverick

  40. [48]

    Llama 4 Scout: https://openrouter.ai/meta-llama/llama-4-scout

  41. [49]

    2-90b-vision-instruct (VI denotes Vision Instruct

    Llama 3.2 90B VI: https://openrouter.ai/meta-llama/llama-3. 2-90b-vision-instruct (VI denotes Vision Instruct. Your tables may use Llama 3.2 90B Vision I)

  42. [50]

    Grok 2 Vision: https://openrouter.ai/x-ai/grok-2-vision-1212

  43. [51]

    Grok Vision Beta: https://openrouter.ai/x-ai/grok-vision-beta

  44. [52]

    Qwen2.5 VL: https://openrouter.ai/qwen/qwen2.5-vl-72b-instruct (VL de- notes Vision Language.)

  45. [53]

    Qwen VL Plus: https://openrouter.ai/qwen/qwen-vl-plus

  46. [54]

    I want to place the blue object inside the brown box, are there any constraints that prevent me from doing so or is it possible?

    Qwen 3 (235B): https://openrouter.ai/qwen/qwen3-235b-a22b (This appears as "Qwen 3".) C.2 Simulated Constraint Scenario Generation To generate a diverse and controllable set of scenarios for evaluating VLM understanding of physical constraints, we developed a simulation-based ...

  47. [55]

    All-correct: Does the LLM output contain all ground-truth affordances?

  48. [56]

    "" Light: Featherweight, Lightweight Medium: Moderate, Balanced Heavy: Bulky, Dense Dynamic: Fluctuating, Variable

    At-least-one: Does the LLM output contain at least one ground-truth affordance? Verification prompts were: 21 Given the following ground truth affordances for a <object_name>: <list> And the following LLM response: <llm_response> Does the LLM response contain all the ground tr...

  49. [57]

    Claude 3.7 Sonnet: 108.5$

  50. [58]

    Claude 3.7 Sonnet (T): 167.8$

  51. [59]

    Claude 3.5 Sonnet: 73.9$

  52. [60]

    Gemini 2.0 Flash 001: 2.6$

  53. [61]

    Gemini 2.5 Flash P: 2.9$

  54. [62]

    Gemini 2.5 Pro P: 150.2$

  55. [63]

    Llama 4 Maverick: 40.8$

  56. [64]

    Llama 3.2 90B VI: 16.8$

  57. [65]

    Grok 2 Vision: 66.7$

  58. [66]

    Grok Vision Beta: 22.4$

  59. [67]

    (T)" or

    Qwen 3 (235B): 24.0$ Overall Cost Summary The total estimated cost for running all models across the entire PAC benchmark is $769.30. The cost breakdown by PAC category, aggregated across all models, is as follows: • Properties: $ 695.76 • Affordances: $ 62.31 • Constraints: $...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.