Pith. sign in

REVIEW 3 major objections 6 minor 79 references

A training-free method that simulates each action as an incremental scene-graph update lifts vision-language models' accuracy on error-prone visual planning tasks by several percentage points.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 17:12 UTC pith:QI642LN6

load-bearing objection A new benchmark worth having, and a plausible but under-validated method; the empirical claims overreach what the current evidence shows. the 3 major comments →

arxiv 2512.10342 v3 pith:QI642LN6 submitted 2025-12-11 cs.CV

CoSPlan: Corrective Sequential Planning via Scene Graph Incremental Updates

Shresth Grover , Priyank Pathak , Akash Kumar , Yogesh S Rawat This is my paper
classification cs.CV
keywords Vision-language modelssequential planningscene graphserror detectioncorrective planningvisual reasoning benchmarkplan completiontraining-free prompting
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces CoSPlan, a benchmark for corrective sequential planning in vision: a model must finish a multi-step visual task after detecting and correcting a deliberately inserted bad action. Across four tasks, leading vision-language models perform near random guessing, even with chain-of-thought and static scene graphs. The paper's central claim is SGI, a training-free technique that converts the initial and goal images into scene graphs, then simulates each action one at a time to produce intermediate graphs, and finally scores every candidate completion by how similar its simulated final graph is to the goal graph. SGI outperforms vanilla scene graphs on every task, with average gains of about 4–5 percentage points, and also improves accuracy on static visual question answering and a text-only blockworld planning task. If SGI works as reported, it shows that decomposing a plan into incremental state updates, rather than a single jump from start to goal, is a cheap and effective way to make VLMs more reliable planners.

Core claim

The paper's central discovery is that the main obstacle to VLM planning is the absence of explicit intermediate states. Vanilla scene graphs compress the whole transformation into one graph, forcing the model to interpolate the entire action sequence internally; SGI removes that burden by simulating each action as a change to nodes, attributes, and edges of the scene graph, producing a sequence of intermediate graphs. For each multiple-choice option, SGI simulates the option's actions starting from the last context graph, then asks the VLM to score the resulting graph against the goal graph on a 0–100 scale and picks the option with the highest score. Reported step-completion gains over vani

What carries the argument

Scene Graph Incremental updates (SGI): a training-free procedure that (1) builds scene graphs for the initial and goal images, (2) simulates each already-performed action as an incremental update to the current graph, producing a chain of intermediate graphs, then (3) for every multiple-choice completion option simulates that option's actions from the last context graph and asks the VLM to rate the resulting graph's similarity to the goal graph on a 0–100 scale. The option whose simulated final graph scores highest is the model's answer. This machinery converts planning into a search over explicitly represented intermediate states, rather than a single from-start-to-goal inference.

Load-bearing premise

The whole method rests on trusting the VLM's own 0–100 similarity score between a simulated scene graph and the goal scene graph as a truthful measure of whether that option reaches the goal—if that score rewards phrasing, option position, or generic object overlap, the reported gains do not reflect real planning improvements.

What would settle it

On the deterministic tasks (Maze-E and Blocks-World-E), replace the VLM's similarity score with an exact programmatic graph-difference metric—for instance, the normalized number of node/attribute mismatches between the simulated final graph and the goal graph—and re-rank the options; if the programmatic ranking does not match the VLM scores, or if it predicts the correct option better, the claimed mechanism fails. A simpler probe: shuffle the order of the MCQ options and check whether the similarity scores move with position.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If SGI's gains hold, a pure prompting change—no fine-tuning, no extra data—can improve corrective planning ability in open and closed VLMs alike.
  • SGI turns planning into an inspectable search over candidate action sequences, since every intermediate state is an explicit scene graph.
  • Because SGI also improves error-free planning and static VQA, its intermediate-state reasoning may generalize beyond corrective tasks to any visual decision problem that can be phrased as choosing among options.
  • The CoSPlan benchmark gives future work a standardized way to measure error recovery, with separate scores for detecting an error and for completing the steps.
  • The reported gains on a text-only PlanBench task suggest the technique is not specific to vision and may transfer to language-only planning.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because SGI's core is step-wise simulation rather than scene graphs specifically, the same select-and-simulate loop could be applied to other structured state representations, such as coordinate lists or property vectors, when those are more faithful in a given domain.
  • The reliance on the VLM's self-scored similarity makes the method brittle in principle; a natural improvement would be to run the same option-ranking with a deterministic graph comparator on the synthetic tasks and see whether SGI's advantage comes from the simulation or from the scoring.
  • The near-random baselines on CoSPlan suggest that the error injection is what breaks the models; this opens the door to using CoSPlan as a diagnostic to separate planning ability from perceptual ability in future generations of VLMs.
  • Since error-detection gains were larger than step-completion gains for GPT-4o, the incremental graphs may be acting primarily as a state tracker that exposes inconsistencies—an effect that could be tested by feeding the intermediate graphs to a non-VLM planner.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper presents CoSPlan, a multiple-choice benchmark for corrective sequential planning in vision-language settings, spanning four synthetic and real tasks (Maze-E, Blocks-World-E, Shuffle-E, Robo-VQA-E). Each instance provides an initial state, a goal state, and an 'initial context' containing one deliberately erroneous action; models must detect the error and complete the sequence. The authors evaluate five VLMs under vanilla, CoT, and scene-graph prompting, and propose SGI, a training-free method that builds initial/goal scene graphs, simulates each candidate option action-by-action to an intermediate scene graph, and selects the option whose resulting graph is judged most similar to the goal. They report SGI gains of roughly 4.4–5.2% on CoSPlan and further gains on VQA and PlanBench.

Significance. The benchmark is a useful addition: it is, to my knowledge, the first to combine visual sequential planning with intentionally erroneous context, and the included safeguards against 'cheating' options and the multi-model analysis are valuable. If SGI's mechanism is validated, it would be a cheap and model-agnostic way to improve error-prone visual planning without training. However, the current evidence is insufficient to establish the central claim. The method's selection signal is an unvalidated self-reported similarity score from the same VLM; the numerical gains are presented without variance and often sit at or below sampling noise; and some reported external-dataset results contradict the generality statement. The paper also honestly acknowledges in the supplement that the similarity scores are based on VLM judgment and may not be universal. These issues are fixable with additional experiments, so the work merits revision rather than rejection.

major comments (3)
  1. [§4.3, Tables 3/4/7] The headline 'SGI significantly outperforms the vanilla SG approach across all benchmark tasks/datasets' is not supported by the reported evidence. No error bars, confidence intervals, or repeated runs are given for any cell. At the stated sample sizes, the claimed gains are at or below sampling noise: Robo-VQA-E has n=350 (1 SE ≈ 2.2–2.4 percentage points for accuracies in the 20–50% range), while gains such as Qwen2 Shuffle-E SG→SGI (25.1→25.0), Qwen2 Blocks-World-E (18.8→18.5), and CoG-VLM Maze-E error detection (13.3→11.0, Table 7) are negative. Several positive deltas (Intern-VLM Shuffle-E +1.8, Maze-E +2.0, GPT-4o Blocks-World-E +1.0) are smaller than the Monte Carlo error for the corresponding sample size. I request per-cell repeated runs with seeds/temperature variation, variance estimates, and a statement of which differences are statistically distinguishable.
  2. [Algorithms 1 and 2, Suppl. §9] The selection mechanism is a self-referential VLM judgment. The same model builds S0 and Sg, simulates each option to Sm, and assigns the 0–100 SIMILARITY score whose argmax is the predicted answer (Alg. 1 line 14; threshold in Alg. 2 line 9). The supplement states the score 'is based on VLM judgment, which is not guided by us' and that 'one universal similarity metric may not be applicable on all kind of scene graphs.' This leaves open that the score tracks option position, verbosity, or textual overlap rather than whether the plan reaches the goal. The paper already documents such biases (e.g., Janus selects option A ~94% of the time; models choose 'cheating' options), so the same confounds may be driving SGI's gains. For Maze-E and Blocks-World-E the true state after each action is computable; I request an external validation: compare SGI's similarity ranking with a deterministic grid
  3. [§4.3 and Table 5] The claimed generalization to VQA is internally inconsistent. In Table 5, SGI underperforms SG on CoG-VLM Spatial-Map (36.7→35.8) and Maze-Nav (32.4→31.2), while the text claims 'superior performance of SGI' from that table. Table 6 reports a single PlanBench score (14.7 vs 13.9) with no variance and no repeated trials. Additionally, the aggregate improvement is quoted as ~4.4% in the abstract and ~5.2% in the introduction, and Table 7 shows negative SGI deltas for Qwen2 on Shuffle-E and Blocks-World-E. Please report all per-cell deltas, clarify which aggregate is being quoted, and give confidence intervals for the external-dataset results.
minor comments (6)
  1. [Abstract/Introduction] Please reconcile the aggregate gain: the abstract says ~4.4% while the introduction says ~5.2%.
  2. [§3.5.1] Scene Graphs are cited as [43], but [43] is the CoT reference; the scene-graph baseline appears to lack a proper citation.
  3. [Table 3] The Random row is unclear for Error Detection: expected accuracy depends on initial-context length; specify how the aggregate random baseline was computed.
  4. [Algorithm 2] Step numbering is inconsistent (the third block is labeled '3)' instead of a step number), and the indentation from line 6 to line 8 should be cleaned up.
  5. [Figure 9] The error-free comparison lacks error bars, sample sizes, and a statement of whether the same MCQ design and safeguards were used.
  6. [Throughout] Typos such as 'Plancbench' (intro) and 'comperehensive' (supplement) should be fixed.

Circularity Check

0 steps flagged

No significant circularity: SGI is an empirical scoring heuristic; the self-judged similarity is a validity limitation, not a definitional reduction.

full rationale

The paper's central claim is an empirical comparison: SGI chooses the MCQ option with maximum VLM-assigned similarity between its incrementally simulated scene graph and the goal graph (Algorithm 1 line 14; Algorithm 2 lines 8–13). The benchmark labels are fixed independently by task construction, with the correct option defined by whether it corrects the inserted error and reaches the goal, not by the VLM's similarity score, so the prediction is not defined in terms of the label. The supplementary's admission that 'similarity scores are based on VLM judgment' (Sec. 9) and the absence of a deterministic state comparator are genuine validity and robustness concerns, but they do not amount to a derivation-level circularity under the stated criteria. The only author self-citation, [13] for the MCQ option-count effect (Sec. 3.6), is peripheral and not load-bearing; external VQA and PlanBench evaluations use independent datasets and an off-the-shelf planner. The 0.75 error-detection threshold is presented as an explicit hyperparameter, not as a fitted value renamed into a prediction. Accordingly, no circular step can be exhibited from the paper's own equations or definitions.

Axiom & Free-Parameter Ledger

1 free parameters · 5 axioms · 2 invented entities

The central empirical claim rests on three unverified pillars: (1) VLM scene graphs are faithful enough for step-wise state tracking; (2) the same VLM's similarity scores are a reliable goal-equivalence oracle; (3) the single-error MCQ protocol yields unambiguous ground truth. The 0.75 threshold is the one explicit free parameter, and Shuffle-E drops error detection entirely because 'one initial erroneous swap will cascade' (Sec. 3.1), narrowing the 'corrective' scope of the benchmark.

free parameters (1)
  • similarity threshold for error detection = 0.75
    Algorithm 2 (suppl. §9) and §11 set a hand-chosen 0.75 threshold deciding 'None of the above' versus an error; no sensitivity analysis is reported, and it directly gates all Error Detection accuracy numbers.
axioms (5)
  • domain assumption VLM-extracted scene graphs faithfully represent the true scene state at each step
    SGI's incremental simulation reasons entirely over scene graphs; the paper admits 'we do not supervise SG generation, i.e., nodes and edges can sometimes be noisy' (suppl. §9). If graphs misrepresent the scene, simulated options and similarity scores are wrong.
  • domain assumption The VLM's self-scored 0–100 similarity between simulated SG and goal SG is a valid proxy for whether the option reaches the goal
    Algorithm 1 selects argmax similarity and Algorithm 2 thresholds it; the paper states the scores are 'based on VLM judgment' with no external metric (suppl. §9). This is the load-bearing premise of the entire method.
  • domain assumption Each episode contains exactly one intended erroneous action with a unique ground-truth label
    'Error' is defined loosely as 'a plausible but suboptimal action' (§3); Robo-VQA-E labels are human-curated (§3.1), so ground truth is authorial and may be non-unique for real-world scenes.
  • domain assumption MCQ option-selection accuracy is a valid measure of planning ability
    The paper itself documents option bias ('Janus predicts 100% of the time option A', suppl. §10.2) and 'cheating' (Fig. 13), confounds that are reported but not corrected in the Top-1 accuracy metric.
  • standard math Random baseline for error detection equals E[1/(initial context length + 1)]
    Formula given in §3.4; however main-text Table 3 reports values (25.4/26.1/26.1) inconsistent with this formula and with supplementary Table 7 (15.4/33.3/33.3), so even the random reference is internally contradictory.
invented entities (2)
  • CoSPlan erroneous-action construct (A_E embedded in initial context) no independent evidence
    purpose: Makes visual sequence planning error-prone, enabling measurement of error detection and step completion
    The error-injection protocol, distractor options, and ground-truth labels are internal to the paper; no external standard validates that the labeled error is the uniquely correct one.
  • Intermediate scene graph S_c (and per-option S_m) no independent evidence
    purpose: Represents the state after each simulated action so the model can compare against the goal graph
    No independent handle: S_c is produced and scored by the same VLM, and its fidelity is never checked against a deterministic simulator or ground-truth state tracker.

pith-pipeline@v1.3.0-alltime-deepseek · 28312 in / 23826 out tokens · 233993 ms · 2026-08-03T17:12:37.303672+00:00 · methodology

0 comments
read the original abstract

Vision Language Models (VLMs) have shown promising planning capabilities, yet their success remains confined to the text domain, leaving visual decision-making relatively underexplored. Addressing this gap, we introduce Corrective Sequence Planning (CoSPlan) benchmark, where VLMs must plan a sequence of visual actions from an initial scene to a target scene. CoSPlan evaluates models on their ability to imagine and execute a coherent set of visual steps required to reach the goal (Step Completion). To prevent any shortcuts that simply describe the final scene, we introduce an erroneous action in decision-making, which must be detected (Error Detection) and corrected to reach the goal, enabling a deeper understanding of the task. CoSPlan spans across 4 tasks: maze navigation, block re-arrangement, image reconstruction, and object re-organization. Despite using advanced reasoning strategies such as Chain-of-Thought and Scene Graphs, VLMs struggle on CoSPlan, while still showing promising performance in the text domain. Addressing this, we propose Scene Graph Incremental updates (SGI), a novel training-free method to transform images into `textual' scene graphs, enabling step-by-step reasoning through iterative scene graph refinement. SGI yields an average of ~4.4% improvement on CoSPlan w/ generalization on PlanBench and VQA. Link for solving puzzles on the project page.

Figures

Figures reproduced from arXiv: 2512.10342 by Akash Kumar, Priyank Pathak, Shresth Grover, Yogesh S Rawat.

Figure 1
Figure 1. Figure 1: Corrective Sequential Planning: Given the initial and final states, with already performed actions w/ some errors (initial context), model identifies errors in the provided context, and picks the optimal action steps to reach the final goal, correcting the error. tions increases, errors become more likely ( [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of CoSPlan Benchmark: Given initial (Is) and final state (Ie) and initial set of instructions (orange), the model needs to perform two tasks: Step completion, choosing right set of future paths (green) to complete the task, and Error detection the sub-optimal / erroneous action in past actions (initial context). Shown coordinates (row, column) are 0-indexed. Initial steps visualized as black arrow… view at source ↗
Figure 3
Figure 3. Figure 3: Error Correction Initial context (orange arrows) with error move AE (diagonal (1, 0) → (2, 1)) to cell. Step completion to Ig w/ error correction (yellow) is correct while w/o (pink) is not. ∈ [3×3, 8×8], and up to 5 obstacles,susing OpenCV [17] 1 . Blocks-World-E (Fig. 2b): The goal is to stack blocks in a specific (target) configuration. Inputs is the initial block arrangement (I0) and the final arrangem… view at source ↗
Figure 5
Figure 5. Figure 5: Scene Graph for Robo-VQA-E SG generated via GPT￾4o, with objects as nodes, location as edge, state as attributes. 3.5.1. Baseline Reasoning Chain-of-Thought (CoT [43]) We adapt CoT for our CoSPlan datasets by i) Identify: Providing models with a detailed description of the problem and constraints; ii) Con￾text: Step-by-step description of each action in the initial context; iii) Verify: Ask model to plan a… view at source ↗
Figure 6
Figure 6. Figure 6: (a) VLMs excel in error-free settings, highlighting the complexity of error-prone ones. (b) Errors from within context (scene) are harder than random ones (out-context). (c) VLMs struggle on visual reasoning; however perform exceptionally well on text-only domain. Accuracy Random accuracy Intern-VLM # of MCQ options Accuracy Random accuracy Intern-VLM # of MCQ options Maze-E Accuracy Take k steps towards g… view at source ↗
Figure 7
Figure 7. Figure 7: All tasks (a,b,c) shown for Step completion, with (a) using Scene Graph (SG), and (b,c) using Chain-of-Though (CoT). [PITH_FULL_IMAGE:figures/full_fig_p006_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: SGI 1) Initial and Goal Scene Graphs (SG) are generated. 2) Incremental Scene Update sequentially modifies SG for each action Ai 3) Similarity Comparison matches the resultant SG with Goal graph for searching for the best-aligned sequence. a dynamic approach that adaptively represents evolving scenes, incrementally updating the Scene Graph as actions unfold. Simulating each action generates explicit repres… view at source ↗
Figure 9
Figure 9. Figure 9: Error Free Step Completion 3D component, where scene graphs not only to represent the current scene but also to derive next-time-frame scene graphs. Effectively, SGI interpolates CoT and SG reasoning across sequential scenes, allowing VLMs to reason through evolving scenes rather than interpolating scene-level deci￾sions. In terms of reasoning hierarchy, CoT ⊆ SG ⊆ SGI. 4.3. Results CoSPlan Comparison [PI… view at source ↗
Figure 10
Figure 10. Figure 10: CoSPlan overview: The input context comprises ex￾ecuted actions and both the initial and final states. The model pre￾dicts the optimal action steps to reach the goal (green) and iden￾tifies errors in the provided context (red). The Main Submission also showed an example in [PITH_FULL_IMAGE:figures/full_fig_p011_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: As the number of obstacles increases, the accuracy [PITH_FULL_IMAGE:figures/full_fig_p013_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Models have a strong bias towards picking option A [PITH_FULL_IMAGE:figures/full_fig_p014_12.png] view at source ↗
Figure 15
Figure 15. Figure 15: Same convention as fig 7a) in the main submission. [PITH_FULL_IMAGE:figures/full_fig_p014_15.png] view at source ↗
Figure 13
Figure 13. Figure 13: No of times Intern-VLM (CoT) cheat (pick the option [PITH_FULL_IMAGE:figures/full_fig_p014_13.png] view at source ↗
Figure 16
Figure 16. Figure 16: Initial and Final states for Example 1 SGI Input Prompt - Step 1 Examine a composite image showing two stages: the left side depicts the initial stage of a task, and the right side illustrates the completed task. Analyze the provided initial and goal state images to extract a structured scene graph representation. The scene graph should include objects, their attributes (such as color, size, and material)… view at source ↗
Figure 17
Figure 17. Figure 17: Initial and Goal State Images Alternative Paths - Input Prompts and Outputs • Option A Prompt: Simulate the following actions: (1) Remove tissue from both bowls, (2) Transfer dry fruits out of the bowls, (3) Place apples inside the left bowl, (4) Place oranges inside the right bowl. Output (Match Score: 100%): Actions = {Remove tissue, Transfer dry fruits, Place apples (left), Place oranges (right)} • Opt… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

79 extracted references · 14 linked inside Pith

  1. [1]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774,

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774,

    Pith/arXiv arXiv

  2. [2]

    Photo-realistic blocksworld dataset, 2018

    Masataro Asai. Photo-realistic blocksworld dataset, 2018. 2

    2018

  3. [3]

    Classical planning in deep latent space, 2022

    Masataro Asai, Hiroshi Kajino, Alex Fukunaga, and Chris- tian Muise. Classical planning in deep latent space, 2022. 2

    2022

  4. [4]

    Learning to predict action feasibility for task and motion planning in 3d environments

    Smail Ait Bouhsain, Rachid Alami, and Thierry Simeon. Learning to predict action feasibility for task and motion planning in 3d environments. In2023 IEEE International Conference on Robotics and Automation (ICRA), pages 3736–3742. IEEE, 2023. 2

    2023

  5. [5]

    Measuring and improving chain-of-thought reasoning in vision-language models.arXiv preprint arXiv:2309.04461, 2023

    Ming Chen, Yuan Li, et al. Measuring and improving chain-of-thought reasoning in vision-language models.arXiv preprint arXiv:2309.04461, 2023. 2

    Pith/arXiv arXiv 2023

  6. [6]

    Janus- pro: Unified multimodal understanding and generation with data and model scaling.arXiv preprint arXiv:2501.17811,

    Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus- pro: Unified multimodal understanding and generation with data and model scaling.arXiv preprint arXiv:2501.17811,

    Pith/arXiv arXiv

  7. [7]

    Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24185–24198, 2024. 2, 4, 11

    2024

  8. [8]

    Spatialrgpt: Grounded spatial reasoning in vision language model.OpenReview, 2024

    Anjie Cheng et al. Spatialrgpt: Grounded spatial reasoning in vision language model.OpenReview, 2024. 2

    2024

  9. [9]

    Physbench: Benchmarking and enhancing vision-language models for physical world under- standing.arXiv preprint arXiv:2501.16411, 2025

    Wei Chow, Jiageng Mao, Boyi Li, Daniel Seita, Vitor Guizilini, and Yue Wang. Physbench: Benchmarking and enhancing vision-language models for physical world under- standing.arXiv preprint arXiv:2501.16411, 2025. 2

    Pith/arXiv arXiv 2025

  10. [10]

    Hu- man demonstrations enable efficient solutions to sequential manifold planning problems

    Breanne Crockett, Carl L Mueller, and Bradley Hayes. Hu- man demonstrations enable efficient solutions to sequential manifold planning problems. InProceedings of the 2025 ACM/IEEE International Conference on Human-Robot In- teraction, pages 800–809, 2025. 2

    2025

  11. [11]

    Understanding the role of scene graphs in visual question answering, 2021

    Vinay Damodaran, Sharanya Chakravarthy, Akshay Kumar, Anjana Umapathy, Teruko Mitamura, Yuta Nakashima, Noa Garcia, and Chenhui Chu. Understanding the role of scene graphs in visual question answering, 2021. 2

    2021

  12. [12]

    Imagenet: A large-scale hierarchical im- age database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical im- age database. InComputer Vision and Pattern Recognition,

  13. [13]

    Nav- igating hallucinations for reasoning of unintentional activi- ties, 2024

    Shresth Grover, Vibhav Vineet, and Yogesh S Rawat. Nav- igating hallucinations for reasoning of unintentional activi- ties, 2024. 6

    2024

  14. [14]

    Reasoning with language model is planning with world model.arXiv preprint arXiv:2305.14992, 2023

    Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. Reasoning with language model is planning with world model.arXiv preprint arXiv:2305.14992, 2023. 2

    Pith/arXiv arXiv 2023

  15. [15]

    Large language models cannot self-correct reasoning yet

    Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet. In The Twelfth International Conference on Learning Represen- tations, 2024. 1

    2024

  16. [16]

    Understanding the planning of llm agents: A survey.arXiv preprint arXiv:2402.02716, 2024

    Xu Huang, Weiwen Liu, Xiaolong Chen, Xingmei Wang, Hao Wang, Defu Lian, Yasheng Wang, Ruiming Tang, and Enhong Chen. Understanding the planning of llm agents: A survey.arXiv preprint arXiv:2402.02716, 2024. 2

    Pith/arXiv arXiv 2024

  17. [17]

    Open source computer vision library.https:// github.com/itseez/opencv, 2015

    Itseez. Open source computer vision library.https:// github.com/itseez/opencv, 2015. 4

    2015

  18. [18]

    Words over pixels? rethinking vision in multimodal large language models

    Anubhooti Jain, Mayank Vatsa, and Richa Singh. Words over pixels? rethinking vision in multimodal large language models. InProceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, IJCAI-25, pages 10481–10489. International Joint Conferences on Artificial Intelligence Organization, 2025. Survey Track. 6

    2025

  19. [19]

    Image genera- tion from scene graphs, 2018

    Justin Johnson, Agrim Gupta, and Li Fei-Fei. Image genera- tion from scene graphs, 2018. 2

    2018

  20. [20]

    Position: Llms can’t plan, but can help planning in llm-modulo frameworks

    Subbarao Kambhampati, Karthik Valmeekam, Lin Guan, Mudit Verma, Kaya Stechly, Siddhant Bhambri, Lucas Paul Saldyt, and Anil B Murthy. Position: Llms can’t plan, but can help planning in llm-modulo frameworks. InForty-first International Conference on Machine Learning, 2024. 2

    2024

  21. [21]

    Em- bodied semantic scene graph generation

    Xinghang Li, Di Guo, Huaping Liu, and Fuchun Sun. Em- bodied semantic scene graph generation. In5th Annual Con- ference on Robot Learning, 2021. 2

    2021

  22. [22]

    Optimal robotic assembly sequence planning: A sequential decision-making approach,

    Kartik Nagpal and Negar Mehr. Optimal robotic assembly sequence planning: A sequential decision-making approach,

  23. [23]

    MAP-THOR: Benchmarking long-horizon multi-agent planning frameworks in partially observable en- vironments

    Siddharth Nayak, Adelmo Morrison Orozco, Marina Ten Have, Vittal Thirumalai, Jackson Zhang, Darren Chen, Aditya Kapoor, Eric Robinson, Karthik Gopalakrishnan, Brian Ichter, James Harrison, Anuj Mahajan, and Hamsa Balakrishnan. MAP-THOR: Benchmarking long-horizon multi-agent planning frameworks in partially observable en- vironments. InMulti-modal Foundati...

    2024

  24. [24]

    Sequential planning in large partially observable environments guided by llms, 2023

    Swarna Kamal Paul. Sequential planning in large partially observable environments guided by llms, 2023. 1

    2023

  25. [25]

    Capture: Evaluating spatial reasoning in vision lan- guage models via occluded object counting.arXiv preprint arXiv:2504.15485, 2025

    Atin Pothiraj, Elias Stengel-Eskin, Jaemin Cho, and Mohit Bansal. Capture: Evaluating spatial reasoning in vision lan- guage models via occluded object counting.arXiv preprint arXiv:2504.15485, 2025. 3

    Pith/arXiv arXiv 2025

  26. [26]

    Does spatial cog- nition emerge in frontier models?arXiv preprint arXiv:2410.06468, 2024

    Santhosh Kumar Ramakrishnan, Erik Wijmans, Philipp Kraehenbuehl, and Vladlen Koltun. Does spatial cog- nition emerge in frontier models?arXiv preprint arXiv:2410.06468, 2024. 1, 2

    Pith/arXiv arXiv 2024

  27. [27]

    Sayplan: Ground- ing large language models using 3d scene graphs for scalable 9 task planning

    Krishan Rana, Jesse Haviland, Sourav Garg, Jad Abou- Chakra, Ian Reid, and Niko Suenderhauf. Sayplan: Ground- ing large language models using 3d scene graphs for scalable 9 task planning. In7th Annual Conference on Robot Learning,

  28. [28]

    Learning general policies for planning through gpt models.Proceedings of the International Conference on Automated Planning and Scheduling, 34(1):500–508, 2024

    Nicholas Rossetti, Massimiliano Tummolo, Alfonso Emilio Gerevini, Luca Putelli, Ivan Serina, Mattia Chiari, and Mat- teo Olivato. Learning general policies for planning through gpt models.Proceedings of the International Conference on Automated Planning and Scheduling, 34(1):500–508, 2024. 2

    2024

  29. [29]

    Visual cognition in multimodal large language models.Nature Machine Intelligence, 7(1):96–106, 2025

    Luca M Schulze Buschoff, Elif Akata, Matthias Bethge, and Eric Schulz. Visual cognition in multimodal large language models.Nature Machine Intelligence, 7(1):96–106, 2025. 6

    2025

  30. [30]

    Sener, D

    F. Sener, D. Chatterjee, D. Shelepov, K. He, D. Singhania, R. Wang, and A. Yao. Assembly101: A large-scale multi-view video dataset for understanding procedural activities.CVPR 2022, 2022. 1, 2

    2022

  31. [31]

    Robovqa: Multimodal long-horizon reasoning for robotics

    Pierre Sermanet, Tianli Ding, Jeffrey Zhao, Fei Xia, De- bidatta Dwibedi, Keerthana Gopalakrishnan, Christine Chan, Gabriel Dulac-Arnold, Sharath Maddineni, Nikhil J Joshi, et al. Robovqa: Multimodal long-horizon reasoning for robotics. In2024 IEEE International Conference on Robotics and Automation (ICRA), pages 645–652. IEEE,

  32. [32]

    The illusion of thinking: Understanding the strengths and limitations of reasoning models via the lens of problem complexity, 2025

    Parshin Shojaee, Iman Mirzadeh, Keivan Alizadeh, Maxwell Horton, Samy Bengio, and Mehrdad Farajtabar. The illusion of thinking: Understanding the strengths and limitations of reasoning models via the lens of problem complexity, 2025. 1, 2

    2025

  33. [33]

    Alfworld: Aligning text and embodied environments for in- teractive learning.arXiv preprint arXiv:2010.03768, 2020

    Mohit Shridhar, Xingdi Yuan, Marc-Alexandre C ˆot´e, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning text and embodied environments for in- teractive learning.arXiv preprint arXiv:2010.03768, 2020. 2

    Pith/arXiv arXiv 2010

  34. [34]

    To cot or not to cot? chain-of-thought helps mainly on math and sym- bolic reasoning.arXiv preprint arXiv:2409.12183, 2024

    Zayne Sprague, Fangcong Yin, Juan Diego Rodriguez, Dongwei Jiang, Manya Wadhwa, Prasann Singhal, Xinyu Zhao, Xi Ye, Kyle Mahowald, and Greg Durrett. To cot or not to cot? chain-of-thought helps mainly on math and sym- bolic reasoning.arXiv preprint arXiv:2409.12183, 2024. 2

    Pith/arXiv arXiv 2024

  35. [35]

    Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023. 8

    Pith/arXiv arXiv 2023

  36. [36]

    On the planning abilities of large language models - a critical investigation

    Karthik Valmeekam, Matthew Marquez, Sarath Sreedharan, and Subbarao Kambhampati. On the planning abilities of large language models - a critical investigation. InAdvances in Neural Information Processing Systems, pages 75993– 76005. Curran Associates, Inc., 2023. 2

    2023

  37. [37]

    On the planning abilities of large language models-a critical investigation.Advances in Neural Information Processing Systems, 36:75993–76005,

    Karthik Valmeekam, Matthew Marquez, Sarath Sreedharan, and Subbarao Kambhampati. On the planning abilities of large language models-a critical investigation.Advances in Neural Information Processing Systems, 36:75993–76005,

  38. [38]

    Q*: Improving multi-step reasoning for llms with deliberative planning, 2024

    Chaojie Wang, Yanchen Deng, Zhiyi Lyu, Liang Zeng, Jujie He, Shuicheng Yan, and Bo An. Q*: Improving multi-step reasoning for llms with deliberative planning, 2024. 1

    2024

  39. [39]

    Is a picture worth a thou- sand words? delving into spatial reasoning for vision lan- guage models.Advances in Neural Information Processing Systems, 37:75392–75421, 2024

    Jiayu Wang, Yifei Ming, Zhenmei Shi, Vibhav Vineet, Xin Wang, Sharon Li, and Neel Joshi. Is a picture worth a thou- sand words? delving into spatial reasoning for vision lan- guage models.Advances in Neural Information Processing Systems, 37:75392–75421, 2024. 12

    2024

  40. [40]

    Is a picture worth a thou- sand words? delving into spatial reasoning for vision lan- guage models, 2024

    Jiayu Wang, Yifei Ming, Zhenmei Shi, Vibhav Vineet, Xin Wang, Yixuan Li, and Neel Joshi. Is a picture worth a thou- sand words? delving into spatial reasoning for vision lan- guage models, 2024. 2, 3, 8

    2024

  41. [41]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024. 2, 4, 11

    Pith/arXiv arXiv 2024

  42. [42]

    Cogvlm: Visual expert for pretrained language models.arXiv preprint arXiv:2311.03079, 2023

    Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et al. Cogvlm: Visual expert for pretrained language models.arXiv preprint arXiv:2311.03079, 2023. 2, 4, 11

    Pith/arXiv arXiv 2023

  43. [43]

    Chain-of-thought prompting elicits reasoning in large lan- guage models.Advances in neural information processing systems, 35:24824–24837, 2022

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large lan- guage models.Advances in neural information processing systems, 35:24824–24837, 2022. 2, 5, 13

    2022

  44. [44]

    Flow- Bench: Revisiting and benchmarking workflow-guided plan- ning for LLM-based agents

    Ruixuan Xiao, Wentao Ma, Ke Wang, Yuchuan Wu, Junbo Zhao, Haobo Wang, Fei Huang, and Yongbin Li. Flow- Bench: Revisiting and benchmarking workflow-guided plan- ning for LLM-based agents. InFindings of the Association for Computational Linguistics: EMNLP 2024, pages 10883– 10900, Miami, Florida, USA, 2024. Association for Compu- tational Linguistics. 2

    2024

  45. [45]

    Ing-vp: Mllms can- not play easy vision-based games yet, 2024

    Haoran Zhang, Hangyu Guo, Shuyue Guo, Meng Cao, Wen- hao Huang, Jiaheng Liu, and Ge Zhang. Ing-vp: Mllms can- not play easy vision-based games yet, 2024. 2

    2024

  46. [46]

    Fltrnn: Faithful long-horizon task planning for robotics with large language models

    Jiatao Zhang, Lanling Tang, Yufan Song, Qiwei Meng, Haofu Qian, Jun Shao, Wei Song, Shiqiang Zhu, and Ja- son Gu. Fltrnn: Faithful long-horizon task planning for robotics with large language models. In2024 IEEE Inter- national Conference on Robotics and Automation (ICRA), pages 6680–6686. IEEE, 2024. 2

    2024

  47. [47]

    Der- panis, Richard P

    He Zhao, Isma Hadji, Nikita Dvornik, Konstantinos G. Der- panis, Richard P. Wildes, and Allan D. Jepson. P3iv: Prob- abilistic procedure planning from instructional videos with weak supervision, 2022. 2

    2022

  48. [48]

    Large language models are not robust multi- ple choice selectors.arXiv preprint arXiv:2309.03882, 2023

    Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. Large language models are not robust multi- ple choice selectors.arXiv preprint arXiv:2309.03882, 2023. 5

    Pith/arXiv arXiv 2023

  49. [49]

    Le, Ed H

    Huaixiu Steven Zheng, Swaroop Mishra, Hugh Zhang, Xinyun Chen, Minmin Chen, Azade Nova, Le Hou, Heng- Tze Cheng, Quoc V . Le, Ed H. Chi, and Denny Zhou. Nat- ural plan: Benchmarking llms on natural language planning,

  50. [50]

    Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig

    Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. We- barena: A realistic web environment for building au- tonomous agents, 2024. 2 10 CoSPlan: Corrective Sequential Planning via Scene Graph Incremental Updates Supplementary Material Initial Context...

    2024

  51. [52]

    The model pre- dicts the optimal action steps to reach the goal (green) and iden- tifies errors in the provided context (red)

    Pick the pen à 2) Put the holder upside down à ? Figure 10.CoSPlanoverview: The input context comprises ex- ecuted actions and both the initial and final states. The model pre- dicts the optimal action steps to reach the goal (green) and iden- tifies errors in the provided context (red). The Main Submission also showed an example in Figure 1

  52. [53]

    Clarification on Step Completion Design Current VLMs struggle with real-world deployment be- cause their self-supervised training rarely includes the sub- optimal steps or execution errors common in autonomous navigation and robotics.CoSPlanmimics this setting to test recovery capabilities. A key design choice is the inte- gration of error correction into...

  53. [54]

    None of the above

    Vision-Language Models Overview We employ a suite of state-of-the-art vision-language mod- els (VLMs) to address visual reasoning tasks, including both proprietary and open-source solutions. These models exhibit diverse architectural characteristics for multimodal understanding. 8.1. GPT-4o GPT-4o [1] is a general-purpose VLM that uses textual in- puts an...

  54. [55]

    Com- pare the resulting scene graph with the goal scene graph to identify incorrect relationships, misplaced objects, or un- met constraints. Select the best-aligned plan

    Scene Graph Incremental update Details The Scene Graph Incremental update (SGI) frame- work enhances the decision-making of VLMs in sequen- tial instruction-following, particularly when handling in- complete plans or embedded errors (A E ). Unlike conven- tional Chain-of-Thought (CoT) approaches that infer the transformation fromI 0 toI g in a single step...

  55. [56]

    Unicorn Umbrellas

    Results 10.1. External Dataset Description VQA External Dataset DetailsWang et al. proposed a se- ries of visual question-answering tasks to test VLM’s ability on visual reasoning (different from our sequence planning tasks, as they involves no intial context).i) Spatial-Map: Tests spatial relationships between objects with unique lo- cation names (e.g.“U...

  56. [57]

    The batch size was set 1

    Hyperparameters (Reproducibility) We set a threshold of 0.75 for similarity in error detection (Algorithm 2). The batch size was set 1. Number of GPUs used was 1, 48Gb on a NVIDIA RTX A6000 GPU. We would additionally release our code base for task genera- tion and evaluation, along with our SGI algorithm. Ethical Statement TheCoSPlanbenchmark includes bot...

  57. [58]

    However, all VLMs struggle withvisual + textbased sequence planning tasks, further complicated by the addition ofjust one basic error

    Future work Testing the robustness of the SGI algorithm to the noises in faulty Scene graph generation (node, state edges) can better assess the practical value of our algorithm. However, all VLMs struggle withvisual + textbased sequence planning tasks, further complicated by the addition ofjust one basic error. In such scenarios, diving deeper into the m...

  58. [59]

    Task-Specific CoT Examples 13.1. Maze-E 15 Maze-E CoT prompt Please analyze this grid-based image following these specific instructions along with the provided context: You are looking at a grid pattern image that contains: • A grid structure (e.g., 3x3 or 4x4 etc) • A designated start point (e.g., marked by a green circle) • A designated end point (e.g.,...

  59. [60]

    The initial configuration of blocks

  60. [61]

    Which blocks are currently movable (unobstructed)

  61. [62]

    The feasibility of each move

  62. [63]

    Provide only the label of the correct option (e.g.,A, B, C, or D) without any explanation

    Whether the sequence of moves results in the depicted final state. Provide only the label of the correct option (e.g.,A, B, C, or D) without any explanation. 16 13.3. Shuffle-E Shuffle-E CoT Prompt Given information about image patches, determine the subsequent steps required to rearrange a shuffled image into a target image. The image is composed of patc...

  63. [64]

    Scene Graph Examples 17 Figure 16. Initial and Final states for Example 1 SGI Input Prompt - Step 1 Examine a composite image showing two stages: the left side depicts the initial stage of a task, and the right side illustrates the completed task. Analyze the provided initial and goal state images to extract a structured scene graph representation. The sc...

  64. [65]

    Two bowls are placed on the table, spaced apart to accommodate the fruit

  65. [66]

    Tissue or decorative paper is placed inside each bowl to act as lining

  66. [67]

    Open the containers containing dry fruits

  67. [68]

    Put some dry fruits in each bowl

  68. [69]

    initial_scene_graph

    Place another tissue over the dry fruits in both bowls Given the starting scene graph, simulate the sequential execution of the provided steps and update the scene graph at each stage. Ensure that object states and positions are modified based on their interactions. Output - Step 2: Intermediate State Intermediate Scene Graph:    Ob...

  69. [70]

    Move up from [5, 0] to [4, 0]

  70. [71]

    Move right from [4, 0] to [4, 1]

  71. [72]

    intermediate_scene_graph

    Move right from [4, 1] to [4, 2] Ensure the agent’s position is updated while maintaining the positions of the goal and obstacles. Output - Step 2: Intermediate Scene Graph Intermediate Scene Graph: { "intermediate_scene_graph": { "objects": [ {"name": "agent", "type": "green_circle", "state": "active", "position": [4, 2], "properties": {}}, {"name": "goa...

  72. [73]

    Move block 0 from column 2 to column 5

  73. [74]

    objects": [ {

    Move block 2 from column 1 to column 2. Ensure to track state changes and validate each move. Output - Step 2: Intermediate Scene Graphs After Action 0: Move block 0 from column 2 to column 5 { "objects": [ {"name": "block_2", "type": "block", "state": "resting", "position": "column_1, level_1", "properties": {"color": "green"}}, {"name": "block_0", "type...

  74. [77]

    Option B:

    Move block 5 from column 0 to column 1. Option B:

  75. [78]

    Move block 6 from column 3 to column 2

  76. [79]

    Move block 2 from column 2 to column 1

  77. [80]

    Move block 5 from column 3 to column 0

  78. [81]

    scene_graph

    Move block 5 from column 0 to column 1. For each option, provide the final scene graph and compute the similarity score based on goal state achievement, constraint satisfaction, solution efficiency, and resource optimization. Output - Step 3: Option Evaluations Option A: { 30 "scene_graph": { "objects": [ {"name": "block_2", "type": "block", "state": "res...

  79. [2009]

    IEEE Conference on, pages 248–255

    CVPR 2009. IEEE Conference on, pages 248–255. IEEE, 2009. 3, 4

    2009