arxiv: 2604.17830 · v1 · submitted 2026-04-20 · 💻 cs.RO

Recognition: unknown

SYMBOLIZER: Symbolic Model-free Task Planning with VLMs

Hermann Blum, Sami Azirar, Zlatan Ajanovic

Authors on Pith no claims yet

Pith reviewed 2026-05-10 05:06 UTC · model grok-4.3

classification 💻 cs.RO

keywords symbolic planningvisual language modelstask planningheuristic searchmodel-free planningroboticsbenchmarks

0 comments

The pith

Domain-independent symbolic search over states grounded by visual-language models solves task planning without action models.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper establishes that visual-language models can convert raw images into usable symbolic states simply by grounding a small collection of lifted predicates that describe object relations. Classical heuristic search then operates directly on those states using only goal-count and width-based methods, with no action models or task-specific training required. This setup generalizes to new problem instances and across different domains. It produces better plans than asking the visual-language model itself to choose actions step by step and reaches the same level as methods that use the model only to score search nodes. A sympathetic reader would care because the approach removes the expensive step of handcrafting or learning symbolic action models that has long limited scalable task planning.

Core claim

We propose a framework that requires only lifted predicates describing relations among objects and uses visual-language models to ground them from images to obtain the symbolic state. Planning is performed with domain-independent heuristic search using goal-count and width-based heuristics, without need for action models. Symbolic search over the visual-language model-grounded state space outperforms direct visual-language model-based planning and performs on par with approaches that use a visual-language model-derived heuristic, achieving state-of-the-art results on the ProDG and ViPlan benchmarks.

What carries the argument

VLM-grounded symbolic states obtained from lifted predicates, searched with domain-independent heuristics (goal-count and width-based) and no action models.

If this is right

Long-horizon planning problems with large combinatorial state spaces become solvable without building domain-specific symbolic action models.
Performance exceeds that obtained by querying visual-language models directly to select actions or generate plans.
Results equal those of methods that rely on visual-language models solely to compute search heuristics.
Only a fixed set of lifted predicates is needed, so the same planner applies to unseen problem instances in multiple domains.
State-of-the-art results are reached on the ProDG and ViPlan benchmarks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

More reliable predicate grounding in future visual-language models would immediately raise success rates on real-robot tasks where lighting or viewpoint changes occur.
The clean separation between perception and search suggests the same structure could be reused in other hybrid planning settings that currently require hand-engineered models.
Adding mechanisms to detect and correct occasional grounding errors would make the overall system more robust without changing the core search procedure.

Load-bearing premise

Visual-language models can accurately and consistently ground a small set of lifted predicates from raw images into a reliable symbolic state without task-specific fine-tuning or handcrafted object descriptions.

What would settle it

A sequence of benchmark images in which the visual-language model produces inconsistent or wrong predicate groundings, causing the symbolic search to return invalid or failed plans.

Figures

Figures reproduced from arXiv: 2604.17830 by Hermann Blum, Sami Azirar, Zlatan Ajanovic.

**Figure 2.** Figure 2: Example of planning with a simulator. From an initial state, the planner [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Figure 3: Flowchart of grounding from images or text to symbolic states. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

**Figure 4.** Figure 4: Example observations and goal specifications from our custom evaluation domains. (a)–(c): PDDLG [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗

**Figure 5.** Figure 5: Example observations and goals from the P [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗

**Figure 6.** Figure 6: Example observations and goals from the V [PITH_FULL_IMAGE:figures/full_fig_p007_6.png] view at source ↗

read the original abstract

Traditional Task and Motion Planning (TAMP) systems depend on physics models for motion planning and discrete symbolic models for task planning. Although physics model are often available, symbolic models (consisting of symbolic state interpretation and action models) must be meticulously handcrafted or learned from labeled data. This process is both resource-intensive and constrains the solution to the specific domain, limiting scalability and adaptability. On the other hand, Visual Language Models (VLMs) show desirable zero-shot visual understanding (due to their extensive training on heterogeneous data), but still achieve limited planning capabilities. Therefore, integrating VLMs with classical planning for long-horizon reasoning in TAMP problems offers high potential. Recent works in this direction still lack generality and depend on handcrafted, task-specific solutions, e.g. describing all possible objects in advance, or using symbolic action models. We propose a framework that generalizes well to unseen problem instances. The method requires only lifted predicates describing relations among objects and uses VLMs to ground them from images to obtain the symbolic state. Planning is performed with domain-independent heuristic search using goal-count and width-based heuristics, without need for action models. Symbolic search over VLM-grounded state-space outperforms direct VLM-based planning and performs on par with approaches that use a VLM-derived heuristic. This shows that domain-independent search can effectively solve problems across domains with large combinatorial state spaces. We extensively evaluate on extensively evaluate our method and achieve state-of-the-art results on the ProDG and ViPlan benchmarks.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

The paper's real move is grounding a fixed set of lifted predicates with a VLM and then running plain domain-independent heuristic search without any action model at all.

read the letter

The approach skips the usual work of building or learning symbolic action models for each new domain. It only needs a small set of lifted predicates that describe object relations, lets the VLM turn raw images into a symbolic state by checking which predicates hold, and then searches that state space with goal-count and width-based heuristics. The abstract says this beats direct VLM planning and matches VLM-heuristic baselines while hitting SOTA on ProDG and ViPlan benchmarks. That combination is the part that is not already in the cited prior work. The search side is standard and clean once the state is available, which is the main practical win if the grounding holds up. The grounding step itself is the obvious weak point. The abstract gives no precision or recall numbers for the predicates, no consistency checks across similar images, and no breakdown of where the VLM fails. If the visual cases in the benchmarks are mostly unambiguous, the reported planning success could be driven more by easy perception than by the search. Without those measurements or ablations, it is hard to know how far the method travels beyond the tested images. This is worth sending to referees for people working on VLM-plus-planning hybrids in robotics. The idea is simple enough that a careful review can quickly check whether the grounding numbers support the claims or whether more work is needed there.

Referee Report

2 major / 2 minor

Summary. The manuscript presents SYMBOLIZER, a framework for task planning in TAMP that avoids handcrafted symbolic models and action models. It uses VLMs to ground lifted predicates describing object relations from images to obtain symbolic states, then performs planning via domain-independent heuristic search employing goal-count and width-based heuristics. The authors claim that this symbolic search over VLM-grounded states outperforms direct VLM planning, matches VLM-heuristic approaches, and achieves SOTA on ProDG and ViPlan benchmarks, demonstrating the effectiveness of domain-independent search in large combinatorial spaces.

Significance. Should the VLM grounding step prove robust, the result would indicate that classical planning techniques can be directly applied to perceptual inputs from VLMs for scalable, generalizable task planning without domain-specific engineering. This has potential to advance robotics by reducing the cost of model specification in TAMP.

major comments (2)

[Evaluation] Evaluation section: The abstract reports benchmark results and performance comparisons but supplies no implementation details, error bars, failure cases, or ablation studies on the VLM grounding step. This makes the data-to-claim link unverifiable, as the grounding of lifted predicates is the sole bridge from perception to the combinatorial state space on which goal-count and width-based search operates.
[Abstract] Abstract and method description: No per-predicate precision/recall, inter-image consistency statistics, or failure-mode analysis is provided for the VLM grounding of relations (e.g., 'on'/'in'). Without these, it remains possible that reported successes depend on low-ambiguity benchmark images rather than the robustness of the domain-independent search.

minor comments (2)

[Abstract] Abstract: repeated phrase 'We extensively evaluate on extensively evaluate our method'.
[Abstract] Abstract: grammatical error 'physics model are often available' should read 'physics models are often available'.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback highlighting the need for greater transparency in our evaluation of the VLM grounding step. We address each major comment below and will revise the manuscript to incorporate the requested details.

read point-by-point responses

Referee: [Evaluation] Evaluation section: The abstract reports benchmark results and performance comparisons but supplies no implementation details, error bars, failure cases, or ablation studies on the VLM grounding step. This makes the data-to-claim link unverifiable, as the grounding of lifted predicates is the sole bridge from perception to the combinatorial state space on which goal-count and width-based search operates.

Authors: We agree that additional details are required to strengthen the verifiability of our claims. The current manuscript emphasizes end-to-end benchmark performance but omits granular information on the grounding process. In the revised version, we will expand the evaluation section with: implementation details of the VLM prompting and parsing procedure for lifted predicates; error bars from repeated trials accounting for VLM stochasticity; qualitative and quantitative analysis of failure cases; and ablation studies that isolate the grounding component (e.g., by comparing against oracle or degraded grounding). These additions will clarify how the domain-independent search operates on the resulting symbolic states. revision: yes
Referee: [Abstract] Abstract and method description: No per-predicate precision/recall, inter-image consistency statistics, or failure-mode analysis is provided for the VLM grounding of relations (e.g., 'on'/'in'). Without these, it remains possible that reported successes depend on low-ambiguity benchmark images rather than the robustness of the domain-independent search.

Authors: We acknowledge the value of these metrics for demonstrating robustness beyond benchmark-specific image clarity. Although the paper reports strong overall results on ProDG and ViPlan, it does not currently include per-predicate statistics or consistency measures. We will add a dedicated subsection to the evaluation (and update the abstract if space permits) providing precision/recall for each lifted predicate against available ground truth, inter-image consistency statistics, and a failure-mode analysis. This will help separate the contributions of the VLM grounding from those of the goal-count and width-based search. revision: yes

Circularity Check

0 steps flagged

No significant circularity; empirical method relies on external VLM grounding and standard search

full rationale

The paper proposes SYMBOLIZER as a framework that uses VLMs to ground a fixed set of lifted predicates from images into symbolic states, followed by domain-independent heuristic search (goal-count and width-based) without action models. No equations, derivations, or fitted parameters are present that reduce to inputs by construction. Performance claims (outperformance over direct VLM planning, parity with VLM-heuristic methods, SOTA on ProDG/ViPlan) are presented as empirical results from benchmark evaluation, not as logical necessities derived from self-referential definitions or self-citations. The grounding step is an external capability assumption rather than a self-defined loop, and the search component uses off-the-shelf heuristics. This is a standard non-circular empirical contribution.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

The central claim rests on the unproven reliability of zero-shot VLM predicate grounding and on the effectiveness of goal-count and width-based heuristics in the absence of action models.

axioms (2)

domain assumption VLMs can reliably ground a small set of lifted predicates from images into accurate symbolic states without domain-specific training
This is the step that converts visual input into the symbolic state used by the planner.
domain assumption Domain-independent heuristics suffice for solving the resulting combinatorial planning problems
Invoked when the paper states that goal-count and width-based search works across domains.

pith-pipeline@v0.9.0 · 5570 in / 1415 out tokens · 49037 ms · 2026-05-10T05:06:33.242870+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

34 extracted references · 10 canonical work pages · 3 internal anchors

[1]

Integrated task and motion planning,

C. R. Garrett, R. Chitnis, R. Holladay, B. Kim, T. Silver, L. P. Kaelbling, and T. Lozano-P´erez, “Integrated task and motion planning,”Annu. Rev. Control Robot. Auton. Syst., vol. 4, no. 1, pp. 265–293, 2021

2021
[2]

OpenVLA: An open- source vision-language-action model,

M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. Foster, G. Lam, P. Sanketiet al., “OpenVLA: An open- source vision-language-action model,” inProc. CoRL, 2024

2024
[3]

LLM-Guided Task- and Affordance-Level Exploration in Reinforcement Learning

J. Luijkx, R. Ma, Z. Ajanovi ´c, and J. Kober, “Llm-guided task-and affordance-level exploration in reinforcement learning,” arXiv:2509.16615, 2025

work page internal anchor Pith review Pith/arXiv arXiv 2025
[4]

Behavior-1k: A benchmark for embodied ai with 1,000 everyday activities and realistic simulation,

C. Liet al., “Behavior-1k: A benchmark for embodied ai with 1,000 everyday activities and realistic simulation,” inConference on Robot Learning. PMLR, 2023, pp. 80–93

2023
[5]

Position: Llms cant plan, but can help planning in llm-modulo frameworks,

S. Kambhampatiet al., “Position: Llms cant plan, but can help planning in llm-modulo frameworks,” inForty-first International Conference on Machine Learning, 2024

2024
[6]

Haslum, N

P. Haslum, N. Lipovetzky, D. Magazzeni, C. Muise, R. Brachman, F. Rossi, and P. Stone,An Introduction to the Planning Domain Definition Language, 2019

2019
[7]

Purely Declarative Action Descriptions are Overrated: Classical Planning with Simulators,

G. Frances, M. Ram ´ırez, N. Lipovetzky, and H. Geffner, “Purely Declarative Action Descriptions are Overrated: Classical Planning with Simulators,” inProc. IJCAI, 2017

2017
[8]

Classical planning with simulators: Results on the atari video games,

N. Lipovetzky, M. Ramirez, and H. Geffner, “Classical planning with simulators: Results on the atari video games,” inProc. IJCAI, 2015

2015
[9]

Planning with pixels in (almost) real time,

W. Bandres, B. Bonet, and H. Geffner, “Planning with pixels in (almost) real time,” inProc. AAAI, 2018

2018
[10]

Classical planning in deep latent space: From unlabeled images to pddl (and back)

M. Asai and A. Fukunaga, “Classical planning in deep latent space: From unlabeled images to pddl (and back).” inNeSy, 2017

2017
[11]

Planning from pixels in atari with learned symbolic representations,

A. Dittadi, F. K. Drachmann, and T. Bolander, “Planning from pixels in atari with learned symbolic representations,”Proc. AAAI, 2021

2021
[12]

Do as i can, not as i say: Grounding language in robotic affordances,

A. Brohan, Y . Chebotar, C. Finn, K. Hausman, A. Herzog, D. Ho, J. Ibarz, A. Irpan, E. Jang, R. Julianet al., “Do as i can, not as i say: Grounding language in robotic affordances,” inCoRL, 2023

2023
[13]

SayCanPay: Heuristic planning with large language models using learnable domain knowl- edge,

R. Hazra, P. Z. Dos Martires, and L. De Raedt, “SayCanPay: Heuristic planning with large language models using learnable domain knowl- edge,” inProc. AAAI, 2024

2024
[14]

Text2Motion: From natural language instructions to feasible plans,

K. Lin, C. Agia, T. Migimatsu, M. Pavone, and J. Bohg, “Text2Motion: From natural language instructions to feasible plans,”Auton. Robots, vol. 47, no. 8, pp. 1345–1365, 2023

2023
[15]

Smirnov, F

P. Smirnov, F. Joublin, A. Ceravola, and M. Gienger, “Generating consis- tent PDDL domains with Large Language Models,”arXiv:2404.07751, 2024

work page arXiv 2024
[16]

Llm-bt: Performing robotic adaptive tasks based on large language models and behavior trees,

H. Zhou, Y . Lin, L. Yan, J. Zhu, and H. Min, “Llm-bt: Performing robotic adaptive tasks based on large language models and behavior trees,” in ICRA, 2024

2024
[17]

Plansformer: Generating symbolic plans using transformers,

V . Pallagani, B. Muppasani, K. Murugesan, F. Rossi, L. Horesh, B. Srivastava, F. Fabiano, and A. Loreggia, “Plansformer: Generating Symbolic Plans using Transformers,”arXiv:2212.08681, 2022

work page arXiv 2022
[18]

LLMs Still Can’t Plan; Can LRMs? A Preliminary Evaluation of OpenAI’s O1 on Plan- Bench,

K. Valmeekam, K. Stechly, and S. Kambhampati, “LLMs Still Can’t Plan; Can LRMs? A Preliminary Evaluation of OpenAI’s O1 on Plan- Bench,”arXiv:2409.13373, 2024

work page arXiv 2024
[19]

Can language models serve as text-based world simulators?

R. Wang, G. Todd, Z. Xiao, X. Yuan, M.-A. Cˆot´e, P. Clark, and P. Jansen, “Can language models serve as text-based world simulators?” inProc. ACL, 2024

2024
[20]

Evaluating the World Model Implicit in a Generative Model,

K. Vafa, J. Y . Chen, A. Rambachan, J. Kleinberg, and S. Mul- lainathan, “Evaluating the World Model Implicit in a Generative Model,” arXiv:2406.03689, 2024

work page arXiv 2024
[21]

Lever- aging pre-trained large language models to construct and utilize world models for model-based task planning,

L. Guan, K. Valmeekam, S. Sreedharan, and S. Kambhampati, “Lever- aging pre-trained large language models to construct and utilize world models for model-based task planning,”NeurIPS, 2023

2023
[22]

LLM+P: Empowering Large Language Models with Optimal Planning Proficiency

B. Liu, Y . Jiang, X. Zhang, Q. Liu, S. Zhang, J. Biswas, and P. Stone, “LLM+P: Empowering Large Language Models with Optimal Planning Proficiency,”arXiv:2304.11477, 2023

work page internal anchor Pith review arXiv 2023
[23]

Planning with Vision- Language Models and a Use Case in Robot-Assisted Teaching,

X. Dang, L. Kudl ´aˇckov´a, and S. Edelkamp, “Planning with Vision- Language Models and a Use Case in Robot-Assisted Teaching,”arXiv preprint arXiv:2501.17665, 2025

work page arXiv 2025
[24]

Vision-language interpreter for robot task planning,

K. Shirai, C. C. Beltran-Hernandez, M. Hamaya, A. Hashimoto, S. Tanaka, K. Kawaharazuka, K. Tanaka, Y . Ushiku, and S. Mori, “Vision-language interpreter for robot task planning,” inICRA, 2024

2024
[25]

DKPROMPT: Domain Knowledge Prompting Vision- Language Models for Open-World Planning,

X. Zhanget al., “DKPROMPT: Domain Knowledge Prompting Vision- Language Models for Open-World Planning,”arXiv:2406.17659, 2024

work page arXiv 2024
[26]

Viplan: A benchmark for visual planning with symbolic predicates and vision-language models,

M. Merleret al., “Viplan: A benchmark for visual planning with symbolic predicates and vision-language models,”arXiv:2505.13180, 2026

work page arXiv 2026
[27]

Efficient Guided Generation for Large Language Models

B. T. Willard and R. Louf, “Efficient guided generation for large language models,”arXiv:2307.09702, 2023

work page internal anchor Pith review arXiv 2023
[28]

Mastering controlled generation with Gemini 1.5: Schema adherence for developers,

L. Liu and T. Koo, “Mastering controlled generation with Gemini 1.5: Schema adherence for developers,” Google for Developers Blog, 2024

2024
[29]

Json mode,

Mistral AI, “Json mode,” https://docs.mistral.ai/capabilities/structured output/json mode, 2024

2024
[30]

Efficient memory management for large language model serving with pagedattention,

W. Kwon, Liet al., “Efficient memory management for large language model serving with pagedattention,” inProc. SOSP, 2023, pp. 611–626

2023
[31]

CRANE: Reasoning with constrained LLM gener- ation,

D. Banerjeeet al., “CRANE: Reasoning with constrained LLM gener- ation,” inPMLR, 2025. 9

2025
[32]

Learning first-order symbolic representations for planning from the structure of the state space,

B. Bonet and H. Geffner, “Learning first-order symbolic representations for planning from the structure of the state space,” inProc. ECAI, 2020

2020
[33]

A survey on in-context learning,

Q. Donget al., “A survey on in-context learning,” inProc. EMNLP, 2024

2024
[34]

Width and serialization of classical planning problems,

N. Lipovetzky and H. Geffner, “Width and serialization of classical planning problems,” inProc. ECAI, 2012

2012