Pith. sign in

REVIEW 3 major objections 5 minor 73 references

GraspMAS: Zero-Shot Language-driven Grasp Detection with Multi-Agent System

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read GraspMAS claims that a zero-shot, training-free multi-agent loop—Planner, Coder, Observer—outperforms baselines for language-driven grasping, with 0.62/0.68 benchmark and 0.80/0.76 real-robot success.

desk verdict GraspMAS is a credible zero-shot grasping system with strong numbers, but the Observer loop that supposedly drives the gains is never ablated, and the paper lacks statistical rigor. read the letter →

arxiv 2506.18448 v2 pith:2ITD3UTP submitted 2025-06-23 cs.RO

Quang Nguyen , Tri Le , Huy Nguyen , Thieu Vo , Tung D. Ta , Baoru Huang , Minh N. Vu , Anh Nguyen This is my paper
classification cs.RO
keywords language-drivengraspingzero-shotgraspdetectionmulti-agentsystemroboticmanipulationfoundationmodelslargelanguagepose
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that language-driven grasp detection can be done zero-shot, with no training or fine-tuning, by composing off-the-shelf vision and language models inside a closed multi-agent loop. The loop has three specialist roles: a Planner that turns an image and a text query into a step-by-step strategy, a Coder that writes and runs Python code calling a small toolset, and an Observer that inspects intermediate crops, grasp rectangles, and error logs and sends textual feedback that refines the plan. On the OCID-VLG and GraspAnything++ benchmarks the authors report success rates of 0.62 and 0.68, beating end-to-end detectors and prior compositional systems such as ViperGPT given the same tools. Simulated and real-robot trials report 0.82/0.72 and 0.80/0.76 success in single-object and cluttered settings. If the results hold, language-conditioned grasping becomes a composition problem rather than a dataset-collection problem.

What carries the argument

The central object is the three-agent closed loop. The Planner (GPT-4) receives the image, the query, and any prior feedback, and emits a step-by-step grasp strategy. The Coder converts that strategy into executable Python that calls nine predefined tool functions, each wrapping a foundation model: open-vocabulary detection (find), part detection (find_part), grasp-pose proposal (grasp_detection), depth estimation (compute_depth), segmentation (masks), and VLM-based verification and grounding (exists, verify_property, best_image_match, llm_query). The Observer (GPT-4o) examines the visualized intermediate results and code error logs and writes textual feedback that starts the next Planner-Coder round. This loop is what distinguishes GraspMAS from a one-shot program generator such as ViperGPT, and it is the component the paper credits for refinement of ambiguous or task-oriented instructions.

What would settle it

Run GraspMAS on the OCID-VLG and GraspAnything++ test splits with the Observer disabled, so only the Planner and Coder operate in a single pass, and compare success rates with the reported 0.62 and 0.68; if the scores are essentially unchanged, the feedback loop is not carrying the claimed improvement, and tracking how often Observer feedback alters the final grasp pose would measure the mechanism directly.

Watch

Extended reading notes

Core claim

The paper's central claim is that GraspMAS, a closed-loop multi-agent system, achieves state-of-the-art zero-shot language-driven grasp detection without any training or fine-tuning. It reports success rates of 0.62 on OCID-VLG and 0.68 on GraspAnything++, surpassing end-to-end baselines (OWLv2, GroundingDINO, and QWEN2-VL, each paired with a grasp detector) and compositional baselines (OWG and ViperGPT, with ViperGPT given the identical toolset). In robot experiments the method reports 0.82 single-object and 0.72 clutter success in ManiSkill simulation, and 0.80 single-object and 0.76 clutter success with a Kinova Gen3 real robot. The authors further report that GraspMAS outperforms supervised grasp-detection models on GraspAnything++, including GraspSAM, without having been trained on grasp data. The mechanism behind the gains is iterative feedback from the Observer, which lets the Planner re-express ambiguous queries (resolving 'Kleenex' into 'blue rectangle box') and choose task-appropriate grasp points (the handle for cutting, the blade for handing a knife over safely).

Load-bearing premise

The load-bearing premise is that the Observer feedback loop is what produces the reported accuracy; the paper presents the closed-loop refinement as central but never tests a version without the Observer, so if a Planner+Coder-only variant scored the same, the system's distinctive mechanism would be unverified.

Editorial extensions

If this is right

  • If the reported numbers hold, supervised training on grasp datasets is no longer necessary for language-conditioned grasping in new settings, because the method composes existing foundation models.
  • Because the toolset is modular, replacing any component (detector, part segmenter, grasp proposer, depth estimator) with a better one should improve overall success without retraining the pipeline.
  • The Observer's ability to reinterpret queries using world knowledge suggests the approach can handle brand names and functional descriptions that closed-vocabulary detectors cannot.
  • The reported inference time of about 2.1 seconds per query, versus fractions of a second for end-to-end baselines, means the approach currently trades speed for reasoning and fits non-time-critical human-robot interaction better than industrial pick-and-place.
  • The failure cases shown for heavily overlapping objects indicate that dense clutter still needs an obstacle-removal or re-arrangement capability before the method can fully solve it.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper never reports a Planner+Coder-only version, so whether the Observer feedback loop is the source of the gains, rather than the specific prompts, tool selection, or model choices, is not established by the experiments.
  • A direct test would record, for each successful query, whether the Observer's feedback changed the final plan; if most successes come from the first Planner-Coder pass, the loop contributes less than the paper's framing suggests.
  • The same Planner-Coder-Observer pattern could transfer to other language-conditioned manipulation outputs, such as placing, tool use, or non-prehensile pushing, since the loop is task-agnostic apart from the toolset.
  • The zero-shot claim currently rests on proprietary GPT-4 and GPT-4o APIs; whether the success rates survive with open-weight language models is untested and affects reproducibility.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes GraspMAS, a training-free multi-agent system for zero-shot language-driven grasp detection. The system has three agents: a Planner (GPT-4) that generates step-by-step grasp strategies, a Coder that turns plans into executable Python using a fixed toolset (object detection, part detection, grasp detection, depth estimation, etc.), and an Observer (GPT-4o) that evaluates intermediate visual results and error logs and feeds textual feedback back to the Planner for refinement (Sec. III). Experiments on OCID-VLG and GraspAnything++ report success rates of 0.62 and 0.68, respectively, outperforming baselines such as ViperGPT (0.44 and 0.57); comparisons with supervised methods (Table III) and simulation/real-robot trials (Tables IV and V) also report large margins. The paper includes qualitative in-the-wild examples, failure cases, and a limitations discussion.

Significance. If the reported results hold, GraspMAS is a meaningful advance: it demonstrates that an orchestrated set of frozen foundation models can handle compositional language commands for grasping without task-specific training, and the margins over ViperGPT with the same toolset are large. The work is an empirical systems comparison against external baselines and ground-truth datasets, so circularity is not a concern. The paper is explicit about its toolset, provides quantitative robot experiments, and acknowledges limitations such as slow inference and failure in dense clutter. However, the central mechanistic claim—that the Observer feedback loop drives the improvements—is not tested by any ablation, and the 'significantly outperforms' claim is not supported by confidence intervals or significance tests, especially given small robot-trial counts. These two omissions currently prevent full confidence in the causal attribution and in the magnitude of the reported gains.

major comments (3)
  1. [III-D and IV] The Observer feedback loop is never ablated. Since ViperGPT, the strongest baseline, is also a Planner+Coder system equipped with the same toolset (Sec. IV-A), the only structural differences are the Planner prompt and the Observer loop. To support the claim that the closed-loop reasoning drives the reported gains (0.62 vs 0.44 on OCID-VLG, 0.68 vs 0.57 on GraspAnything++, and the robot margins in Tables IV and V), please add a GraspMAS variant without the Observer (Planner+Coder only, no refinement) and, if feasible, a variant with a fixed maximum number of refinement loops. Also report the distribution of the number of loops and, on a sample, the per-step success rate so the reader can see how often the Observer's feedback actually changes the final grasp.
  2. [Tables II, IV, and V] No confidence intervals or significance tests are reported for any success rate, although the text repeatedly says 'significantly outperforms' (Secs. IV-A, IV-D, IV-E). For the real robot experiments, each condition uses only 25 trials (Sec. IV-E); for example, 0.80 vs 0.64 in Table V corresponds to 20 vs 16 successes, whose 95% binomial confidence intervals overlap substantially. Please report binomial confidence intervals and appropriate statistical tests (e.g., Fisher's exact test or bootstrap) for at least the simulation and robot results, and state the number of independent runs or seeds used for the dataset evaluations.
  3. [IV-A and IV-B] The metric definitions need clarification. Section IV-A says the GraspAnything++ evaluation uses the 'Harmonic Mean metric,' but Section IV-B says the supervised comparison uses 'success rate as defined in [25]' and Table III reports GraspMAS at 0.68. Please clarify whether the Harmonic Mean and the success rate are the same quantity for this dataset, and if they differ, justify using different metrics for the zero-shot comparison and the supervised comparison. Please also clarify the scope of the 'zero-shot' claim given that the toolset includes RAGT-3/3 (Table I), a trained grasp detector, and whether 'training-free' refers only to language-driven grasp models.
minor comments (5)
  1. [IV-A and Tables] Notation is inconsistent: the text uses 'ViberGPT' (Sec. IV-A) while tables use 'ViperGPT'; 'GroundDINO' (Table IV) appears alongside 'GrDINO' (Table II) and 'GroundingDINO' (Sec. IV-A); and 'QWEN2' (Sec. IV-A) appears as 'QWEN2-VL' in Table II. Please unify the names and citations.
  2. [Fig. 2 and Table I] There are typos in the code examples and tool names: 'grasp_detction' and 'image_pacth' appear in Fig. 2, and Table I lists 'graspdetection' while the code uses 'grasp_detection'. Please fix these inconsistencies so the toolset matches the executable code.
  3. [III-B and III-D] The full system prompts, LLM versions (GPT-4 for Planner and GPT-4o for Observer), sampling temperature, and the loop termination criterion are not specified. Please include the complete prompts and generation settings in an appendix for reproducibility.
  4. [IV-D and IV-E] The simulation section reports 50 samples per scenario and the real-robot section reports 25 repetitions, but the paper does not state whether the same object arrangements and text prompts were used across all methods for the simulation experiments; Sec. IV-E only says 'the same text prompts are used for all methods.' Please describe the evaluation protocol more precisely.
  5. [Abstract] The abstract says 'limiting their generation in real-world applications,' which appears to be a typo for 'generalization.'

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity; the GraspMAS evaluation is a self-contained empirical comparison against external baselines and ground-truth datasets.

full rationale

The paper is an empirical systems paper: GraspMAS composes off-the-shelf tools (GrDINO, VLPart, RAGT, BLIP2, MiDaS, SAM, GPT-4) and is evaluated against external baselines and ground-truth annotations on OCID-VLG, GraspAnything++, simulation, and a real robot. No parameter is fitted to the test data, and the reported success rates are not defined in terms of any internal variable, so no step reduces to its own inputs by construction. The self-citations to prior works from the same group (e.g., GraspAnything++ [8], mask-guided attention [20], Grasp-Anything [25]) are contextual: they provide the benchmark, test split, or baseline method, not a load-bearing theoretical premise. The unablated Observer feedback loop is a real verification gap, because the paper attributes gains to the closed-loop mechanism without an Observer-disabled variant, but that is a causal-support problem, not circularity: no equation or fitted quantity makes the Observer's contribution equivalent to the reported result. The central claim therefore retains independent content and is externally falsifiable.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The agents are software roles, not new physical or mathematical entities; no new particles, fields, or conserved quantities are introduced.

free parameters (2)
  • workspace distance threshold = 0.3 m
    Shown in Fig. 2 as 'distance = 0.3 (m)' during the workspace verification step; chosen by hand, not fitted, but it affects whether a grasp is accepted in the simulated and real robot checks.
  • maximum feedback iterations = not specified
    The system runs 'one or several reasoning loops' with no explicit stopping criterion, which is a free algorithmic parameter and could affect success rate versus inference time.
assumptions (4)
  • domain assumption The nine underlying tools (GroundingDINO, VLPart, RAGT, BLIP-2, MiDaS, SAM, GPT-4) perform accurately enough for object detection, part detection, grasp detection, and verification.
    Table I lists these tools as the system's backbone; if any tool fails on the task or dataset, the pipeline fails regardless of the agents. No per-tool accuracy is reported.
  • domain assumption The success-rate metric from [5] (IoU > 0.25 and angle difference <= 30 degrees) and the Harmonic Mean metric from [8] are valid and applied consistently across methods.
    Section IV-A defines the metric but does not define the Harmonic Mean; the comparison assumes the same protocol was used by all baselines.
  • domain assumption GPT-4 as Planner can generate correct, executable code plans for the given visual reasoning tasks without training.
    Section III-B relies entirely on the LLM's in-context reasoning and code generation; no analysis of code-error rates is provided.
  • domain assumption The Observer's multimodal feedback is reliable enough to guide plan refinement.
    Section III-D states the Observer evaluates grasp poses and sends feedback to the Planner, but there is no ablation or measurement of feedback quality.

how reviews work

0 comments
Cite this review

Pith. "Pith review of GraspMAS: Zero-Shot Language-driven Grasp Detection with Multi-Agent System." pith.science (2026). https://pith.science/paper/2ITD3UTP

@misc{pith2026250618448,
  author       = {Pith},
  title        = {Pith review of: GraspMAS: Zero-Shot Language-driven Grasp Detection with Multi-Agent System},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2ITD3UTP}},
  note         = {Machine review of arXiv:2506.18448}
}
read the original abstract

Language-driven grasp detection has the potential to revolutionize human-robot interaction by allowing robots to understand and execute grasping tasks based on natural language commands. However, existing approaches face two key challenges. First, they often struggle to interpret complex text instructions or operate ineffectively in densely cluttered environments. Second, most methods require a training or finetuning step to adapt to new domains, limiting their generation in real-world applications. In this paper, we introduce GraspMAS, a new multi-agent system framework for language-driven grasp detection. GraspMAS is designed to reason through ambiguities and improve decision-making in real-world scenarios. Our framework consists of three specialized agents: Planner, responsible for strategizing complex queries; Coder, which generates and executes source code; and Observer, which evaluates the outcomes and provides feedback. Intensive experiments on two large-scale datasets demonstrate that our GraspMAS significantly outperforms existing baselines. Additionally, robot experiments conducted in both simulation and real-world settings further validate the effectiveness of our approach. Our project page is available at https://zquang2202.github.io/GraspMAS

Figures

Figures reproduced from arXiv: 2506.18448 by the authors.

Figure 1
Figure 1. Comparison between the traditional methods (a) and our approach [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. (a) An overview of our GraspMAS framework, a multi-agent system for zero-shot language-driven grasp detection. GraspMAS consists of three [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Diagrammatic illustration of the Coder agent. C. Coder Agent The detailed design of the Coder agent is illustrated in [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Qualitative comparison with other methods. [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: In-the-wild results. This figure illustrates the detailed processing steps of our method, including the planning process, code generation, execution [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Robotic demonstration. (a) Simulation environment with Franka [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Failure cases of our method. F. Discussion Practical application. Through intensive experiments, our GraspMAS shows a strong reasoning ability through its compositional design, which allows the robots to interpret complex human commands and deliver precise inferences. …

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

73 extracted references · 60 canonical work pages

  1. [25]

    Grasp-anything: Large-scale grasp dataset from foundation models,

    A. D. Vuong, M. N. Vu, H. Le, B. Huang, B. Huynh, T. V o, A. Kugi, and A. Nguyen, “Grasp-anything: Large-scale grasp dataset from foundation models,” in ICRA, 2024

    work page 2024

  2. [1]

    A survey on learning-based robotic grasping,

    K. Kleeberger, R. Bormann, W. Kraus, and M. F. Huber, “A survey on learning-based robotic grasping,” Current Robotics Reports , 2020

    work page 2020

  3. [2]

    Review of deep learning methods in robotic grasp detection,

    S. Caldera, A. Rassau, and D. Chai, “Review of deep learning methods in robotic grasp detection,” Multimodal Technologies and Interaction , 2018

    work page 2018

  4. [3]

    Real-time grasp detection using convo- lutional neural networks,

    J. Redmon and A. Angelova, “Real-time grasp detection using convo- lutional neural networks,” in ICRA, 2015

    work page 2015

  5. [4]

    Preparatory object reorientation for task-oriented grasping,

    A. Nguyen, D. Kanoulas, D. G. Caldwell, and N. G. Tsagarakis, “Preparatory object reorientation for task-oriented grasping,” in IROS, 2016

    work page 2016

  6. [5]

    Jacquard: A large scale dataset for robotic grasp detection,

    A. Depierre, E. Dellandr ´ea, and L. Chen, “Jacquard: A large scale dataset for robotic grasp detection,” in IROS, 2018

    work page 2018

  7. [6]

    Graspnet-1billion: A large- scale benchmark for general object grasping,

    H.-S. Fang, C. Wang, M. Gou, and C. Lu, “Graspnet-1billion: A large- scale benchmark for general object grasping,” in CVPR, 2020

    work page 2020

  8. [7]

    When transformer meets robotic grasping: Exploits context for efficient grasp detection,

    S. Wang, Z. Zhou, and Z. Kan, “When transformer meets robotic grasping: Exploits context for efficient grasp detection,” IRAL, 2022

    work page 2022

Show all 73 references
  1. [8]

    Language-driven grasp detection,

    A. D. Vuong, M. N. Vu, B. Huang, N. Nguyen, H. Le, T. V o, and A. Nguyen, “Language-driven grasp detection,” in CVPR, 2024

    2024

  2. [9]

    Language-guided robot grasping: Clip-based referring grasp synthesis in clutter,

    G. Tziafas, Y . Xu, A. Goel, M. Kasaei, Z. Li, and H. Kasaei, “Language-guided robot grasping: Clip-based referring grasp synthesis in clutter,” arXiv:2311.05779, 2023

    2023 arXiv

  3. [10]

    Language models are few-shot learners,

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. , “Language models are few-shot learners,” NeuRIPS, 2020

    2020

  4. [11]

    Gpt-4 technical report,

    OpenAI, “Gpt-4 technical report,” 2023

    2023

  5. [12]

    Llama 2: Open foundation and fine-tuned chat models,

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y . Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale,et al., “Llama 2: Open foundation and fine-tuned chat models,” arXiv:2307.09288, 2023

    2023 arXiv

  6. [13]

    Deepseek-v3 technical report,

    A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan, et al. , “Deepseek-v3 technical report,” arXiv:2412.19437, 2024

    2024 arXiv

  7. [14]

    Do as i can, not as i say: Grounding language in robotic affordances,

    M. Ahn, A. Brohan, N. Brown, Y . Chebotar, O. Cortes, B. David, C. Finn, C. Fu, K. Gopalakrishnan, K. Hausman, et al. , “Do as i can, not as i say: Grounding language in robotic affordances,” CoRL, 2023

    2023

  8. [15]

    Progprompt: Generating situated robot task plans using large language models,

    I. Singh, V . Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox, J. Thomason, and A. Garg, “Progprompt: Generating situated robot task plans using large language models,” in ICRA, 2023

    2023

  9. [16]

    Code as policies: Language model programs for embodied control,

    J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng, “Code as policies: Language model programs for embodied control,” in ICRA, 2023

    2023

  10. [17]

    Socratic models: Composing zero-shot multimodal reasoning with language,

    A. Zeng, M. Attarian, B. Ichter, K. Choromanski, A. Wong, S. Welker, F. Tombari, A. Purohit, M. Ryoo, V . Sindhwani, et al. , “Socratic models: Composing zero-shot multimodal reasoning with language,” in ICLR, 2023

    2023

  11. [18]

    L3mvn: Leveraging large language models for visual target navigation,

    B. Yu, H. Kasaei, and M. Cao, “L3mvn: Leveraging large language models for visual target navigation,” in IROS, 2023

    2023

  12. [19]

    Adapt: Vision-language navigation with modality-aligned action prompts,

    B. Lin, Y . Zhu, Z. Chen, X. Liang, J. Liu, and X. Liang, “Adapt: Vision-language navigation with modality-aligned action prompts,” in CVPR, 2022

    2022

  13. [20]

    Language-driven grasp detection with mask-guided attention,

    V . V o, M. Vu, B. Huang, A. Vuong, N. Le, T. V o, and A. Nguyen, “Language-driven grasp detection with mask-guided attention,” in IROS, 2024

    2024

  14. [21]

    Cliport: What and where pathways for robotic manipulation,

    M. Shridhar, L. Manuelli, and D. Fox, “Cliport: What and where pathways for robotic manipulation,” in CoRL, 2022

    2022

  15. [22]

    A joint modeling of vision-language-action for target- oriented grasping in clutter,

    K. Xu, S. Zhao, Z. Zhou, Z. Li, H. Pi, Y . Zhu, Y . Wang, and R. Xiong, “A joint modeling of vision-language-action for target- oriented grasping in clutter,” in ICRA, 2023

    2023

  16. [23]

    Language-driven 6-dof grasp detection using negative prompt guidance,

    T. Nguyen, M. N. Vu, B. Huang, A. Vuong, Q. Vuong, N. Le, T. V o, and A. Nguyen, “Language-driven 6-dof grasp detection using negative prompt guidance,” in ECCV, 2024

    2024

  17. [24]

    Lightweight language-driven grasp detection using con- ditional consistency model,

    N. Nguyen, M. N. Vu, B. Huang, A. Vuong, N. Le, T. V o, and A. Nguyen, “Lightweight language-driven grasp detection using con- ditional consistency model,” in IROS, 2024

    2024

  18. [26]

    Graspgpt: Leveraging semantic knowledge from a large language model for task- oriented grasping,

    C. Tang, D. Huang, W. Ge, W. Liu, and H. Zhang, “Graspgpt: Leveraging semantic knowledge from a large language model for task- oriented grasping,” IRAL, 2023

    2023

  19. [27]

    Towards open-world grasping with large vision-language models,

    G. Tziafas and H. Kasaei, “Towards open-world grasping with large vision-language models,” in CoRL, 2024

    2024

  20. [28]

    Thinkgrasp: A vision-language system for strategic part grasping in clutter,

    Y . Qian, X. Zhu, O. Biza, S. Jiang, L. Zhao, H. Huang, Y . Qi, and R. Platt, “Thinkgrasp: A vision-language system for strategic part grasping in clutter,” in CoRL, 2024

    2024

  21. [29]

    Visual programming: Compositional visual reasoning without training,

    T. Gupta and A. Kembhavi, “Visual programming: Compositional visual reasoning without training,” in CVPR, 2023

    2023

  22. [30]

    Chameleon: Plug-and-play compositional reasoning with large language models,

    P. Lu, B. Peng, H. Cheng, M. Galley, K.-W. Chang, Y . N. Wu, S.-C. Zhu, and J. Gao, “Chameleon: Plug-and-play compositional reasoning with large language models,” NeuRIPS, 2024

    2024

  23. [31]

    Vipergpt: Visual inference via python execution for reasoning,

    D. Sur ´ıs, S. Menon, and C. V ondrick, “Vipergpt: Visual inference via python execution for reasoning,” in CVPR, 2023

    2023

  24. [32]

    Videoagent: A memory-augmented multimodal agent for video understanding,

    Y . Fan, X. Ma, R. Wu, Y . Du, J. Li, Z. Gao, and Q. Li, “Videoagent: A memory-augmented multimodal agent for video understanding,” in ECCV, 2024

    2024

  25. [33]

    Robocoder: Robotic learning from basic skills to general tasks with large language models,

    J. Li, P. Chen, S. Wu, C. Zheng, H. Xu, and J. Jia, “Robocoder: Robotic learning from basic skills to general tasks with large language models,” arXiv:2406.03757, 2024

    2024 arXiv

  26. [34]

    Mp5: A multi-modal open-ended embodied system in minecraft via active perception,

    Y . Qin, E. Zhou, Q. Liu, Z. Yin, L. Sheng, R. Zhang, Y . Qiao, and J. Shao, “Mp5: A multi-modal open-ended embodied system in minecraft via active perception,” in CVPR, 2024

    2024

  27. [35]

    Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents,

    Z. Wang, S. Cai, G. Chen, A. Liu, X. Ma, and Y . Liang, “Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents,” NeuRIPS, 2023

    2023

  28. [36]

    Cloth grasp point detection based on multiple-view geometric cues with application to robotic towel folding,

    J. Maitin-Shepard, M. Cusumano-Towner, J. Lei, and P. Abbeel, “Cloth grasp point detection based on multiple-view geometric cues with application to robotic towel folding,” in ICRA, 2010

    2010

  29. [37]

    Fast graspability evaluation on single depth maps for bin picking with general grippers,

    Y . Domae, H. Okuda, Y . Taguchi, K. Sumi, and T. Hirai, “Fast graspability evaluation on single depth maps for bin picking with general grippers,” in ICRA, 2014

    2014

  30. [38]

    Grasp quality measures: review and performance,

    M. A. Roa and R. Su ´arez, “Grasp quality measures: review and performance,” Autonomous robots, 2015

    2015

  31. [39]

    Deep learning for detecting robotic grasps,

    I. Lenz, H. Lee, and A. Saxena, “Deep learning for detecting robotic grasps,” IJRR, 2015

    2015

  32. [40]

    Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot hours,

    L. Pinto and A. Gupta, “Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot hours,” in ICRA, 2016

    2016

  33. [41]

    Antipodal robotic grasping using generative residual convolutional neural network,

    S. Kumra, S. Joshi, and F. Sahin, “Antipodal robotic grasping using generative residual convolutional neural network,” in IROS, 2020

    2020

  34. [42]

    Vl-grasp: a 6- dof interactive grasp policy for language-oriented objects in cluttered indoor scenes,

    Y . Lu, Y . Fan, B. Deng, F. Liu, Y . Li, and S. Wang, “Vl-grasp: a 6- dof interactive grasp policy for language-oriented objects in cluttered indoor scenes,” in IROS, 2023

    2023

  35. [43]

    Learning 6-dof object poses to grasp category-level objects by language instructions,

    C. Cheang, H. Lin, Y . Fu, and X. Xue, “Learning 6-dof object poses to grasp category-level objects by language instructions,” in ICRA, 2022

    2022

  36. [44]

    A joint network for grasp detection conditioned on natural language commands,

    Y . Chen, R. Xu, Y . Lin, and P. A. Vela, “A joint network for grasp detection conditioned on natural language commands,” in ICRA, 2021

    2021

  37. [45]

    Reasoning grasping via multi- modal large language model,

    S. Jin, J. Xu, Y . Lei, and L. Zhang, “Reasoning grasping via multi- modal large language model,” arXiv:2402.06798, 2024

    2024 arXiv

  38. [46]

    Sparks of artificial general intelligence: Early experiments with gpt-4,

    S. Bubeck, V . Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, et al. , “Sparks of artificial general intelligence: Early experiments with gpt-4,” arXiv:2303.12712, 2023

    2023 arXiv

  39. [47]

    Hydra: A hyper agent for dynamic compositional visual reasoning,

    F. Ke, Z. Cai, S. Jahangard, W. Wang, P. D. Haghighi, and H. Rezatofighi, “Hydra: A hyper agent for dynamic compositional visual reasoning,” in ECCV, 2024

    2024

  40. [48]

    Visual program distillation: Distilling tools and programmatic reasoning into vision-language models,

    Y . Hu, O. Stretcu, C.-T. Lu, K. Viswanathan, K. Hata, E. Luo, R. Krishna, and A. Fuxman, “Visual program distillation: Distilling tools and programmatic reasoning into vision-language models,” in CVPR, 2024

    2024

  41. [49]

    Craft: Customizing llms by creating and retrieving from specialized toolsets,

    L. Yuan, Y . Chen, X. Wang, Y . R. Fung, H. Peng, and H. Ji, “Craft: Customizing llms by creating and retrieving from specialized toolsets,” arXiv preprint arXiv:2309.17428 , 2023

    2023 arXiv

  42. [50]

    Genegpt: Augmenting large language models with domain tools for improved access to biomedical information,

    Q. Jin, Y . Yang, Q. Chen, and Z. Lu, “Genegpt: Augmenting large language models with domain tools for improved access to biomedical information,” Bioinformatics, 2024

    2024

  43. [51]

    Building cooperative embodied agents modularly with large language models,

    H. Zhang, W. Du, J. Shan, Q. Zhou, Y . Du, J. B. Tenenbaum, T. Shu, and C. Gan, “Building cooperative embodied agents modularly with large language models,” in ICLR, 2024

    2024

  44. [52]

    Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face,

    Y . Shen, K. Song, X. Tan, D. Li, W. Lu, and Y . Zhuang, “Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face,” NeuRIPS, 2024

    2024

  45. [53]

    Learning to compose visual relations,

    N. Liu, S. Li, Y . Du, J. Tenenbaum, and A. Torralba, “Learning to compose visual relations,” NeuRIPS, 2021

    2021

  46. [54]

    Cplip: zero-shot learning for histopathology with comprehensive vision-language alignment,

    S. Javed, A. Mahmood, I. I. Ganapathi, F. A. Dharejo, N. Werghi, and M. Bennamoun, “Cplip: zero-shot learning for histopathology with comprehensive vision-language alignment,” in CVPR, 2024

    2024

  47. [55]

    Zero-shot object detection through vision- language embedding alignment,

    J. Xie and S. Zheng, “Zero-shot object detection through vision- language embedding alignment,” in ICDMW, 2022

    2022

  48. [56]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution,

    P. Wang, S. Bai, S. Tan, S. Wang, Z. Fan, J. Bai, K. Chen, X. Liu, J. Wang, W. Ge,et al., “Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution,” arXiv:2409.12191, 2024

    2024 arXiv

  49. [57]

    Malmm: Multi- agent large language models for zero-shot robotics manipulation,

    H. Singh, R. J. Das, M. Han, P. Nakov, and I. Laptev, “Malmm: Multi- agent large language models for zero-shot robotics manipulation,” arXiv:2411.17636, 2024

    2024 arXiv

  50. [58]

    Set-of- mark prompting unleashes extraordinary visual grounding in gpt-4v,

    J. Yang, H. Zhang, F. Li, X. Zou, C. Li, and J. Gao, “Set-of- mark prompting unleashes extraordinary visual grounding in gpt-4v,” arXiv:2310.11441, 2023

    2023 arXiv

  51. [59]

    Shapegrasp: Zero-shot task-oriented grasping with large language models through geometric decomposition,

    S. Li, S. Bhagat, J. Campbell, Y . Xie, W. Kim, K. Sycara, and S. Stepputtis, “Shapegrasp: Zero-shot task-oriented grasping with large language models through geometric decomposition,” in IROS, 2024

    2024

  52. [60]

    Mm-react: Prompting chatgpt for multimodal reasoning and action,

    Z. Yang, L. Li, J. Wang, K. Lin, E. Azarnasab, F. Ahmed, Z. Liu, C. Liu, M. Zeng, and L. Wang, “Mm-react: Prompting chatgpt for multimodal reasoning and action,” arXiv:2303.11381, 2023

    2023 arXiv

  53. [61]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection,

    S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, Q. Jiang, C. Li, J. Yang, H. Su, et al., “Grounding dino: Marrying dino with grounded pre-training for open-set object detection,” in ECCV, 2024

    2024

  54. [62]

    Going denser with open-vocabulary part segmentation,

    P. Sun, S. Chen, C. Zhu, F. Xiao, P. Luo, S. Xie, and Z. Yan, “Going denser with open-vocabulary part segmentation,” in ICCV, 2023

    2023

  55. [63]

    Nbmod: Find it and grasp it in noisy background,

    B. Cao, X. Zhou, C. Guo, B. Zhang, Y . Liu, and Q. Tan, “Nbmod: Find it and grasp it in noisy background,” arXiv:2306.10265, 2023

    2023 arXiv

  56. [64]

    Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models,

    J. Li, D. Li, S. Savarese, and S. Hoi, “Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models,” in ICML, 2023

    2023

  57. [65]

    Towards robust monocular depth estimation: Mixing datasets for zero- shot cross-dataset transfer,

    R. Ranftl, K. Lasinger, D. Hafner, K. Schindler, and V . Koltun, “Towards robust monocular depth estimation: Mixing datasets for zero- shot cross-dataset transfer,” TPAMI, 2020

    2020

  58. [66]

    Segment anything,

    A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y . Lo, et al. , “Segment anything,” in ICCV, 2023

    2023

  59. [67]

    Scaling open-vocabulary object detection,

    M. Minderer, A. Gritsenko, and N. Houlsby, “Scaling open-vocabulary object detection,” NeuRIPS, 2024

    2024

  60. [68]

    Language-driven grasp detection with mask-guided at- tention,

    T. Van V o, M. N. Vu, B. Huang, A. Vuong, N. Le, T. V o, and A. Nguyen, “Language-driven grasp detection with mask-guided at- tention,” in IROS, 2024

    2024

  61. [69]

    Grasp- mamba: A mamba-based language-driven grasp detection framework with hierarchical feature learning,

    H. H. Nguyen, A. Vuong, A. Nguyen, I. Reid, and M. N. Vu, “Grasp- mamba: A mamba-based language-driven grasp detection framework with hierarchical feature learning,” arXiv:2409.14403, 2024

    2024 arXiv

  62. [70]

    Graspsam: When segment anything model meets grasp detection,

    S. Noh, J. Kim, D. Nam, S. Back, R. Kang, and K. Lee, “Graspsam: When segment anything model meets grasp detection,” arXiv:2409.12521, 2024

    2024 arXiv

  63. [71]

    Maniskill3: Gpu parallelized robotics simulation and rendering for generalizable embodied ai,

    S. Tao, F. Xiang, A. Shukla, Y . Qin, X. Hinrichsen, X. Yuan, C. Bao, X. Lin, Y . Liu, T. kai Chan, Y . Gao, X. Li, T. Mu, N. Xiao, A. Gurha, Z. Huang, R. Calandra, R. Chen, S. Luo, and H. Su, “Maniskill3: Gpu parallelized robotics simulation and rendering for generalizable em...

    arXiv

  64. [72]

    Benchmarking in manipulation research: Using the yale-cmu- berkeley object and model set,

    B. Calli, A. Walsman, A. Singh, S. Srinivasa, P. Abbeel, and A. M. Dollar, “Benchmarking in manipulation research: Using the yale-cmu- berkeley object and model set,” IRAM, 2015

    2015

  65. [73]

    Machine learning-based framework for optimally solving the analytical inverse kinematics for redundant manipulators,

    M. N. Vu, F. Beck, M. Schwegel, C. Hartl-Nesic, A. Nguyen, and A. Kugi, “Machine learning-based framework for optimally solving the analytical inverse kinematics for redundant manipulators,” Mecha- tronics, 2023

    2023

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.