LLaPa: A Vision-Language Model Framework for Counterfactual-Aware Procedural Planning

Dechen Zhan; Donglin Di; Lanshun Nie; Lei Fan; Mingjie Wei; Shibo Sun; Wei-Nan Zhang; Xue Li; Yang Song

REVIEW 2 major objections 6 minor 72 references

LLaPa: A Vision-Language Model Framework for Counterfactual-Aware Procedural Planning

T0 review · 2 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LLaPa, a VLM framework that reranks task-relevant image patches and retrieves counterfactual visual tokens in one forward pass, reports state-of-the-art executability, LCS, and correctness on ActPlan-1K, beating GPT-4o and fine-tuned open…

desk verdict A genuinely modular and useful planning architecture, but the empirical reporting has inconsistencies that need fixing before the SOTA claim is trusted. read the letter →

arxiv 2507.08496 v1 pith:D6C2HRXB submitted 2025-07-11 cs.CL

Shibo Sun , Xue Li , Donglin Di , Mingjie Wei , Lanshun Nie , Wei-Nan Zhang , Dechen Zhan , Yang Song

show 1 more author

Lei Fan

This is my paper

classification cs.CL

keywords proceduralplanningvision-languagemodelscounterfactualreasoningtask-orientedsegmentationembodiedAIactionsequencegenerationLLaPamultimodal

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that procedural planning for embodied agents improves when a vision-language model is guided by two lightweight add-on modules: one that reranks image features according to task-relevant objects discovered by segmentation, and one that detects counterfactual clauses in the text and pulls the corresponding visual tokens into the prompt. A sympathetic reader would care because household tasks often involve 'if unexpected things are present' conditions, and generic VLMs tend to plan as if the scene were normal. On the ActPlan-1K benchmark, the proposed framework, LLaPa, reports higher executability, action-sequence similarity (LCS), and correctness than strong baselines including GPT-4o, with the two modules transferring to other VLMs. The central claim is that explicit task-object alignment plus one-pass counterfactual retrieval is sufficient to make multimodal procedural planning more reliable without extra planning loops.

What carries the argument

The load-bearing identity is that patch-level binary masks, produced by a frozen promptable segmenter and OR-aggregated over sentence-level constraints, can be added as logits inside self-attention to obtain a reranked visual representation; then conditional pooling over the subset of those masks selected by a counterfactual classifier yields compact tokens that carry the counterfactual evidence into the language model. The formula that carries the argument is the attention modification $\mathrm{Attn}(v,W) = \mathrm{softmax}(QK^\top/\sqrt{d} + \log(W))V$, where $\log(W)$ shifts attention scores, and the pooling rule $V^{\mathrm{cf}}_{i,g} = (1/|\Omega_g|)\sum_{p \in \Omega_g} v'(p)$ that drops regions with empty masks, so that a counterfactual condition such as 'burnt cookies in the microwave' is represented only by the visual patches that actually show cookies.

What would settle it

Filter ActPlan-1K counterfactual tasks whose ground-truth counterfactual object is not present in the scene (for example, the absent ashcan in Figure 3): if LLaPa's correctness on that subset is no better than a plain fine-tuned VLM, then counterfactual retrieval is not generalizing beyond visible objects.

Watch

Extended reading notes

Core claim

LLaPa is a cascaded framework in which every task sentence is turned into a binary segmentation mask that is OR-combined across images and injected additively into the ViT self-attention logits, reranking patch features toward task-relevant objects; a trained classifier then splits the text into clauses and labels each as counterfactual or not, and the masks belonging to counterfactual clauses are used to pool sixteen sub-grid visual tokens per image ($K=4$) that are fed, with a special prompt token, to the language model. The discovery is that this two-stage attention-and-retrieval mechanism lets a single forward pass handle both normal and counterfactual activities, and that it improves performance on ActPlan-1K counterfactual tasks to 53.2% executability, 0.57 LCS, and 36.1% correctness, surpassing GPT-4o (21.4% correctness) and open-set baselines such as Qwen2-VL (27.9% correctness).

Load-bearing premise

For every counterfactual condition named in the task text, the object it refers to must be visible in one of the input images and must be successfully segmented; if the mask is empty, pooling produces zero features and the framework simply ignores that condition.

Editorial extensions

If this is right

ActPlan-1K counterfactual correctness reaches 36.1% and executability 53.2%, beating GPT-4o, Qwen2-VL, and other baselines in Table 1.
The TER and CAR modules are transferable: plugging them into LLaVA-OV and Qwen2-VL raises correctness by about 9 and 8 points respectively (Table 4).
Ablations show that removing CAR costs 5.4 points of counterfactual correctness and removing TER costs 9.8 points of executability, while bitwise-OR mask aggregation beats summation and similarity-based alternatives (Tables 2 and 3).
Counterfactual handling happens in one forward pass, so the framework needs no replanning loops or iterative self-reflection.
The framework also improves results on ALFRED and MFE-ETP, indicating that task-object alignment benefits more than just counterfactual scenarios.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

An implicit consequence is that the model's counterfactual reasoning is bounded by the recall of the frozen segmenter, so swapping in a stronger open-vocabulary segmenter should directly raise the ceiling on ActPlan-1K; the paper's own Figure 3 shows what happens when a referent object is absent.
A testable extension would be to relax the pooling rule to fall back to text-only counterfactual evidence (for example, from the clause embedding) when a mask is empty, which could recover the 'ashcan' case without sacrificing visual grounding on visible objects.
The transfer results suggest TER and CAR are a general-purpose plug-in for household multimodal planning, so they might benefit other embodied benchmarks beyond the ones reported, and could be applied to unseen scenes without retraining the backbone.
A natural stress test would be to run LLaPa on tasks whose counterfactual conditions mention objects that are occluded or out of frame, where the paper's mechanism currently has no explicit recovery path.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is a useful modular idea wrapped in a paper with several mechanical reporting problems. The core claim, that task-oriented segmentation masks can rerank visual features and a clause-level classifier can pull counterfactual-relevant tokens, is genuinely new relative to the cited prior work. The transfer results to LLaVA-OV and Qwen2-VL and the zero-shot numbers on ALFRED and MFE give the method some independent plausibility. The code is released, which makes the issues checkable.

The soft spots are real and need attention before the SOTA claim is taken at face value. The same configuration, LLaPa on ActPlan-1K counterfactual tasks, appears as 53.2 Executability in Table 1, 52.8 for K=4 in Table 2-left, and 53.9 in Table 2-right. That's a factual inconsistency that makes it hard to know which number is right. Equation (2) adds log(W) to attention logits, but W contains zeros, so log(0) is undefined; the paper doesn't say how they handle it (do they mask, add epsilon, or zero the log?).

The stress-test concern about absent referents is also on target. Figure 3 shows an 'ashcan' that isn't there gives an all-zero mask. Then Eq. (4) outputs zero vectors for every sub-grid, and those zeros are concatenated and fed to the LLM. There is no fallback for missing or unsegmented referents. If a meaningful fraction of ActPlan-1K counterfactual tasks have referents that fail to segment, the reported gains could come from text-only reasoning rather than from CAR's visual grounding. The paper doesn't report mask coverage or segmentation success rate, so the mechanism's actual contribution is unquantified. The 'counterfactual' label also oversells what is a supervised clause classification task on explicitly stated conditions; it's useful, but it's not counterfactual reasoning in the usual sense.

On the positive side, the authors do ablate the components, and the gains from removing CAR (5.4% correctness drop) and TER (9.8% executability drop) are consistent with the intended mechanism. The hyperparameter K is tuned on the evaluation benchmark and no error bars are given, so those ablations are not as strong as they look.

For whom: this is a systems paper for embodied AI / VLM planning researchers. The modular design is worth knowing about, and the released code makes it easy to reproduce or extend. But I'd want a revision that fixes the table numbers, clarifies the log(0) handling, reports mask coverage or adds a fallback, and reports variance. With those changes it's a solid within-subfield contribution. I'd send it to serious peer review rather than desk reject, but I'd push for conditional acceptance rather than a clean pass.

Referee Report

2 major / 6 minor

Summary. The paper proposes LLaPa, a vision-language model framework for procedural planning that combines an InternVL2-8B VLM backbone with two auxiliary modules: a Task-Environment Reranker (TER), which uses Grounded-SAM segmentation masks to reweight visual features toward task-relevant objects, and a Counterfactual Activities Retriever (CAR), which classifies textual clauses as counterfactual conditions and extracts masked visual tokens to emphasize those conditions. The model is trained in two stages and evaluated on ActPlan-1K, ALFRED, and MFE-ETP, reporting state-of-the-art results on ActPlan-1K counterfactual tasks (53.2% Executability, 0.57 LCS, 36.1% Correctness) and improved performance when the modules are transferred to LLaVA-OV and Qwen2-VL.

Significance. If the reported results hold, LLaPa would be a useful contribution: it is, to my knowledge, the first procedural-planning framework to combine task-oriented segmentation and counterfactual-condition detection in a single forward pass, and the transferability experiments in Table 4 show that the proposed modules help two different VLM backbones. The paper also provides code and models, which is a concrete reproducibility asset. The central ideas are clearly described and the experimental design covers several benchmarks. However, the empirical support is weakened by inconsistent numbers for the same configuration across tables, and the counterfactual visual mechanism has a degeneracy that is acknowledged in Figure 3 but not addressed in the method or the limitations. These issues need to be resolved before the claims can be accepted.

major comments (2)

[Tables 1 and 2 (Sections 4.2 and 4.4)] The same configuration, namely LLaPa with the full TER and CAR modules on ActPlan-1K counterfactual tasks, is reported as 53.2% Executability in Table 1, 52.8% for K=4 in the left half of Table 2, and 53.9% for 'Full' in the right half of Table 2. These numbers should be identical. This inconsistency makes it impossible to determine the actual performance and affects the claimed margin over the strongest baseline, InternVL2(Plasma), which is reported as 48.2%. Please reconcile the numbers and specify which table reflects the final model.
[Section 3.3, Eq. (4), and Figure 3] The counterfactual visual feature extraction degenerates when the segmentation mask for a counterfactual condition's referent is empty. Figure 3 itself states that an absent ashcan yields an all-zero mask matrix, and in that case Eq. (4) produces zeros for every sub-grid because every Omega_g is empty. No fallback is described in Sections 3.3 or 3.4, and the Limitations paragraph at the end of Section 5 does not mention this failure mode. Since the headline claim is state-of-the-art performance on ActPlan-1K counterfactual tasks, the paper needs to report the fraction of counterfactual clauses whose masks are non-empty, or provide an ablation restricted to cases with non-empty masks, to support the attribution of the observed gains to CAR's visual grounding rather than to the LLM's text-only reasoning.

minor comments (6)

[Section 4.4] The text refers to 'Table 3' for both the K-ablation and the component ablation, but the K-ablation is presented in the left half of Table 2 and the component ablation in the right half of Table 2. The table references should be corrected.
[Section 3.2, Eq. (2)] Eq. (2) adds log(W) to the attention logits, but W is defined as a binary matrix with entries in {0,1}. If the intended behavior is to mask out zero-weight patches by using -inf before softmax, this should be stated explicitly; otherwise the formula is undefined at W=0.
[Table 2 (right)] The 'w/o CAR' row reports higher normal-task Executability (63.8 vs 62.7) and Correctness (45.8 vs 45.2) than the full model. This is surprising and deserves a brief explanation, since it suggests that CAR may slightly hurt normal tasks.
[Figure 5 and Section 4.3] The error-analysis methodology is described only briefly: it samples 40 activity types and classifies errors into two categories, but the sampling procedure, the number of plans inspected, and the inter-annotator agreement are not reported. Please provide these details, at least in an appendix.
[Section 3.4, Eq. (7)] The symbol s_cf is described as a 'prompt embedding' such as 'With counterfactual conditions' environmental features...'. Please clarify whether s_cf is a learned embedding, a token from the tokenizer, or a separate prompt template, since this affects how Eq. (7) is implemented.
[General] The paper states that 'Detailed dataset information is provided in Appendix,' but no appendix appears in the arXiv version. Please either include the appendix or remove the reference.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: LLaPa's reported gains rest on standard supervised training/evaluation splits, external segmentation and VLM backbones, and component ablations; the zero-mask failure mode is a limitation, not a circular derivation.

full rationale

The paper is an empirical systems paper rather than a formal derivation, and the claimed results do not reduce to their inputs by construction. The TER module uses Grounded-Segment Anything, an externally trained segmentation model, to produce spatial masks from task text, and then reweights visual features; this is an external visual-grounding input, not a self-defined target. The CAR classifier is trained on COPLAN and the ActPlan-1K training set to detect counterfactual clauses, and the model is evaluated on ActPlan-1K, ALFRED, and MFE-ETP; this is a standard supervised train/test setup, not fitting a parameter and then predicting the same quantity. The ablations (Full vs. w/o CAR vs. w/o TER vs. Only SFT) directly test the modules' contributions rather than assuming them. The paper's self-citations (e.g., [9,10,11,12,68]) appear in related-work and background contexts and are not load-bearing for the central architecture or the reported improvements. The notable failure mode in Figure 3 and Eq. (4), where an absent object (ashcan) yields an all-zero mask and hence zero counterfactual visual features, is a real robustness limitation, but it is not circularity: the visual feature is not defined as the prediction, and the model still receives the textual clause; the concern belongs under empirical correctness and coverage, not under self-referential derivation. No equation is shown to equal its own input, and no cited 'uniqueness' result is used to force the design. Therefore the paper exhibits no significant circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 2 invented entities

The central claim rests on the reliability of a frozen segmentation model, on the assumption that extra conditions are always stated in text as separable clauses, and on the assumption that the visual evidence for those conditions lies inside the segmentation masks. The sub-grid size K and the hand-crafted prompt token are free parameters tuned or chosen for the ActPlan-1K benchmark. The invented internal representations (v_cf and v_rerank) have no independent falsifiable handles, so their utility is entirely measured through the benchmark outcomes.

free parameters (3)

Sub-grid division K = 4
Number of sub-grids per patch block in CAR conditional pooling (Eq. 4); chosen by ablation on ActPlan-1K, with K=2, K=8, and K=16 all giving worse results.
Counterfactual prompt token s_cf = "With counterfactual conditions' environmental features..."
Hand-crafted prompt string injected during inference (Eq. 7); not learned or validated across alternative phrasings.
Counterfactual classification threshold = not stated (implicit 0.5)
The sigmoid output p_k in Eq. (3) needs a threshold to define the set K_cf; the paper does not state the threshold or tune it.

assumptions (4)

domain assumption Grounded-SAM masks reliably identify task-relevant objects from textual descriptions, including objects mentioned in extra conditions.
TER depends on the frozen segmentation model's masks, used in Eq. (1) and (2) and propagated to CAR. If the masks are wrong or empty, reranking and counterfactual token extraction degrade.
domain assumption Every counterfactual condition is explicitly stated as a separate sentence or clause in the task text, so a sentence-level classifier can detect it.
CAR segments text by delimiters and classifies clauses (Eq. 3); conditions implied by the image alone, or phrased across multiple sentences, would be missed. This matches ActPlan-1K's format but may not generalize.
ad hoc to paper Counterfactual-relevant visual evidence is always located inside the segmentation masks generated from the textual description.
CAR selects tokens only from masked regions (Eq. 4); Figure 3 shows an absent ashcan yields an all-zero mask, so the corresponding counterfactual feature is zero.
domain assumption A single forward pass with reranked and condition-augmented features is sufficient for planning adaptation, without iterative replanning.
The paper contrasts with text-centric replanning baselines; the method assumes one-shot generation can handle deviations from the normal plan.

invented entities (2)

Counterfactual visual tokens v_cf
purpose: Compact per-image feature set representing visual regions associated with detected counterfactual clauses; concatenated into the LLM input (Eq. 7).
Internal representation with no external falsifiable handle; its value is measured only through downstream plan quality.
Task-sensitive reranked feature space v_rerank
purpose: Recalibrated image embeddings emphasizing task-relevant patches, produced by masked self-attention (Eq. 2).
Internal to the model; no independent measurement outside the planning benchmark.

how reviews work

0 comments

Cite this review

Pith. "Pith review of LLaPa: A Vision-Language Model Framework for Counterfactual-Aware Procedural Planning." pith.science (2026). https://pith.science/paper/D6C2HRXB

@misc{pith2026250708496,
  author       = {Pith},
  title        = {Pith review of: LLaPa: A Vision-Language Model Framework for Counterfactual-Aware Procedural Planning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/D6C2HRXB}},
  note         = {Machine review of arXiv:2507.08496}
}

read the original abstract

While large language models (LLMs) have advanced procedural planning for embodied AI systems through strong reasoning abilities, the integration of multimodal inputs and counterfactual reasoning remains underexplored. To tackle these challenges, we introduce LLaPa, a vision-language model framework designed for multimodal procedural planning. LLaPa generates executable action sequences from textual task descriptions and visual environmental images using vision-language models (VLMs). Furthermore, we enhance LLaPa with two auxiliary modules to improve procedural planning. The first module, the Task-Environment Reranker (TER), leverages task-oriented segmentation to create a task-sensitive feature space, aligning textual descriptions with visual environments and emphasizing critical regions for procedural execution. The second module, the Counterfactual Activities Retriever (CAR), identifies and emphasizes potential counterfactual conditions, enhancing the model's reasoning capability in counterfactual scenarios. Extensive experiments on ActPlan-1K and ALFRED benchmarks demonstrate that LLaPa generates higher-quality plans with superior LCS and correctness, outperforming advanced models. The code and models are available https://github.com/sunshibo1234/LLaPa.

Figures

Figures reproduced from arXiv: 2507.08496 by the authors.

**Figure 2.** Overview of the LLaPa architecture. It consists of three components: the VLM backbone, the Task-Environment [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** An example of task-driven segmentation. Although [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: Qualitative comparison between LLaPa and other models. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

**Figure 5.** Figure 5: Error analysis. Compared to other models, LLaPa [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

72 extracted references · 41 canonical work pages

[1]

Michael Ahn, Debidatta Dwibedi, Chelsea Finn, Montse Gonzalez Arenas, Keerthana Gopalakrishnan, Karol Hausman, Brian Ichter, Alex Irpan, Nikhil Joshi, Ryan Julian, et al. 2024. Autort: Embodied foundation models for large scale orchestration of robotic agents. arXiv preprint arXiv:2401.12963 (2024)

arXiv 2024
[2]

Faeze Brahman, Chandra Bhagavatula, Valentina Pyatkin, Jena D Hwang, Xi- ang Lorraine Li, Hirona Jacqueline Arai, Soumya Sanyal, Keisuke Sakaguchi, Xiang Ren, and Yejin Choi. 2024. PlaSma: Procedural Knowledge Models for Language-based Planning and Re-Planning. In The Twelfth International Confer- ence on Learning Representations

work page 2024
[3]

Chien-Yi Chang, De-An Huang, Danfei Xu, Ehsan Adeli, Li Fei-Fei, and Juan Car- los Niebles. 2020. Procedure planning in instructional videos. In European Con- ference on Computer Vision . Springer, 334–350

work page 2020
[4]

Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. 2024. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. Science China Information Sciences 67, 12 (2024), 220101

2024
[5]

Bowen Cheng, Alex Schwing, and Alexander Kirillov. 2021. Per-pixel classifica- tion is not all you need for semantic segmentation.Advances in neural information processing systems 34 (2021), 17864–17875

2021
[6]

Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. 2024. Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms. arXiv preprint arXiv:2406.07476 (2024)

arXiv 2024
[7]

Wenliang Dai, Nayeon Lee, Boxin Wang, Zhuolin Yang, Zihan Liu, Jon Barker, Tuomas Rintamaki, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. 2024. Nvlm: Open frontier-class multimodal llms. arXiv preprint arXiv:2409.11402 (2024)

arXiv 2024
[8]

Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, et al. 2023. Palm-e: An embodied multimodal language model. (2023)

2023

Show all 72 references

[9]

Lei Fan, Yiwen Ding, Dongdong Fan, Yong Wu, Maurice Pagnucco, and Yang Song
[10]

Lei Fan, Dongdong Fan, Yiwen Ding, Yong Wu, Hongxia Chu, Maurice Pagnucco, and Yang Song. 2023. AV4GAInsp: An efficient dual-camera system for identifying defective kernels of cereal grains.IEEE Robotics and Automation Letters 9, 1 (2023), 851–858

2023
[11]

Lei Fan, Dongdong Fan, Yiwen Ding, Yong Wu, Donglin Di, Maurice Pagnucco, and Yang Song. 2025. GrainBrain: Multiview Identification and Stratification of Defective Grain Kernels. IEEE Transactions on Industrial Informatics (2025)

2025
[12]

Lei Fan, Dongdong Fan, Zhiguang Hu, Yiwen Ding, Donglin Di, Kai Yi, Maurice Pagnucco, and Yang Song. 2025. Manta: A large-scale multi-view and visual-text anomaly detection dataset for tiny objects. In Proceedings of the Computer Vision and Pattern Recognition Conference . 25518–25527

2025
[13]

Roya Firoozi, Johnathan Tucker, Stephen Tian, Anirudha Majumdar, Jiankai Sun, Weiyu Liu, Yuke Zhu, Shuran Song, Ashish Kapoor, Karol Hausman, et al. 2023. Foundation models in robotics: Applications, challenges, and the future. The International Journal of Robotics Research (2...

2023
[14]

Jonathan Francis, Nariaki Kitamura, Felix Labelle, Xiaopeng Lu, Ingrid Navarro, and Jean Oh. 2022. Core challenges in embodied vision-language planning. Journal of Artificial Intelligence Research 74 (2022), 459–515

2022
[15]

Lin Guan, Karthik Valmeekam, Sarath Sreedharan, and Subbarao Kambhampati
[16]

Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, et al. 2022. A survey on vision transformer. IEEE transactions on pattern analysis and machine intelligence 45, 1 (2022), 87–110

2022
[17]

Advances in Neural Information Processing Systems 36 (2023), 79081–79094

Leveraging pre-trained large language models to construct and utilize world models for model-based task planning. Advances in Neural Information Processing Systems 36 (2023), 79081–79094

2023
[18]

Koen Holtman. 2021. Counterfactual planning in AGI systems. arXiv preprint arXiv:2102.00834 (2021)

2021 arXiv
[19]

Junjie He, Pengyu Li, Yifeng Geng, and Xuansong Xie. 2023. Fastinst: A simple query-based model for real-time instance segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 23663–23672

2023
[20]

Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. 2022. Lan- guage models as zero-shot planners: Extracting actionable knowledge for embod- ied agents. In International conference on machine learning . PMLR, 9118–9147

2022
[21]

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2022. Lora: Low-rank adaptation of large language models. ICLR 1, 2 (2022), 3

2022
[22]

Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. 2024. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326 (2024)

2024 arXiv
[23]

Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al . 2022. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2207.05608 (2022)

2022 arXiv
[24]

Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. 2024. Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models. arXiv preprint arXiv:2407.07895 (2024)

2024 arXiv
[25]

Chengshu Li, Ruohan Zhang, Josiah Wong, Cem Gokmen, Sanjana Srivastava, Roberto Martín-Martín, Chen Wang, Gabrael Levine, Michael Lingelbach, Jiankai Sun, et al. 2023. Behavior-1k: A benchmark for embodied ai with 1,000 everyday activities and realistic simulation. In Conferen...

2023
[26]

Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi, and Song Han. 2024. Vila: On pre-training for visual language models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 26689–26699

2024
[27]

Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Rui- hang Chu, Shaoteng Liu, and Jiaya Jia. 2024. Mini-gemini: Mining the potential of multi-modality vision language models. arXiv preprint arXiv:2403.18814 (2024)

2024 arXiv
[28]

Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 26296–26306

2024
[29]

Ziyi Lin, Chris Liu, Renrui Zhang, Peng Gao, Longtian Qiu, Han Xiao, Han Qiu, Chen Lin, Wenqi Shao, Keqin Chen, et al . 2023. Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models. arXiv preprint arXiv:2311.07575 (2023)

2023 arXiv
[30]

Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al . 2023. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499 (2023)

2023 arXiv
[31]

Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. 2024. LLaVA-NeXT: Improved reasoning, OCR, and world knowl- edge

2024
[32]

Yiming Lu, Yebowen Hu, Hassan Foroosh, Wei Jin, and Fei Liu. 2024. STRUX: An LLM for Decision-Making with Structured Explanations. arXiv preprint arXiv:2410.12583 (2024)

2024 arXiv
[33]

Jiasen Lu, Vedanuj Goswami, Marcus Rohrbach, Devi Parikh, and Stefan Lee. 2020. 12-in-1: Multi-task vision and language representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 10437–10446

2020
[34]

Gen Luo, Yiyi Zhou, Yuxin Zhang, Xiawu Zheng, Xiaoshuai Sun, and Rongrong Ji. 2024. Feast your eyes: Mixture-of-resolution adaptation for multimodal large language models. arXiv preprint arXiv:2403.03003 (2024)

2024 arXiv
[35]

Yujie Lu, Pan Lu, Zhiyu Chen, Wanrong Zhu, Xin Eric Wang, and William Yang Wang. 2023. Multimodal procedural planning via dual text-image prompting. arXiv preprint arXiv:2305.01795 (2023)

2023 arXiv
[36]

OpenAI. 2023. GPT-4o System Card

2023
[37]

Yao Mu, Qinglong Zhang, Mengkang Hu, Wenhai Wang, Mingyu Ding, Jun Jin, Bin Wang, Jifeng Dai, Yu Qiao, and Ping Luo. 2023. Embodiedgpt: Vision-language pre-training via embodied chain of thought. Advances in Neural Information Processing Systems 36 (2023), 25081–25094

2023
[38]

Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. 2024. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714 (2024)

2024 arXiv
[39]

Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Torralba. 2018. Virtualhome: Simulating household activities via programs. In Proceedings of the IEEE conference on computer vision and pattern recognition. 8494–8502

2018
[40]

Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, et al. 2024. Grounded sam: Assembling open-world models for diverse visual tasks. arXiv preprint arXiv:2401.14159 (2024)

2024 arXiv
[41]

Tianhe Ren, Qing Jiang, Shilong Liu, Zhaoyang Zeng, Wenlong Liu, Han Gao, Hongjie Huang, Zhengyu Ma, Xiaoke Jiang, Yihao Chen, et al . 2024. Ground- ing dino 1.5: Advance the" edge" of open-set object detection. arXiv preprint arXiv:2405.10300 (2024)

2024 arXiv
[42]

Johann Sawatzky, Yaser Souri, Christian Grund, and Jurgen Gall. 2019. What object should i use?-task driven object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 7605–7614

2019
[43]

Shamik Roy, Sailik Sengupta, Daniele Bonadiman, Saab Mansour, and Arshit Gupta. 2024. FLAP: Flow-adhering planning with constrained decoding in LLMs. arXiv preprint arXiv:2403.05766 (2024)

2024 arXiv
[44]

Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. 2020. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the IEEE/CVF conference on computer vision and pa...

2020
[45]

Dhruv Shah, Błażej Osiński, Sergey Levine, et al. 2023. Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action. In Conference on MM’25, October 27–31, 2025, Dublin, Ireland Sun, et al. robot learning. PMLR, 492–504

2023
[46]

Tom Silver, Varun Hariprasad, Reece S Shuttleworth, Nishanth Kumar, Tomás Lozano-Pérez, and Leslie Pack Kaelbling. 2022. PDDL planning with pretrained large language models. In NeurIPS 2022 foundation models for decision making workshop

2022
[47]

Tom Silver, Soham Dan, Kavitha Srinivas, Joshua B Tenenbaum, Leslie Kaelbling, and Michael Katz. 2024. Generalized planning in pddl domains with pretrained large language models. In Proceedings of the AAAI conference on artificial intelli- gence, Vol. 38. 20256–20264

2024
[48]

Xiu Su, Tao Huang, Yanxi Li, Shan You, Fei Wang, Chen Qian, Changshui Zhang, and Chang Xu. 2021. Prioritized architecture sampling with monto-carlo tree search. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 10968–10977

2021
[49]

Chan Hee Song, Jiaman Wu, Clayton Washington, Brian M Sadler, Wei-Lun Chao, and Yu Su. 2023. Llm-planner: Few-shot grounded planning for embodied agents with large language models. In Proceedings of the IEEE/CVF international conference on computer vision . 2998–3009

2023
[50]

Ying Su, Zhan Ling, Haochen Shi, Jiayang Cheng, Yauwai Yim, and Yangqiu Song. 2024. Actplan-1k: Benchmarking the procedural planning ability of visual language models in household activities. arXiv preprint arXiv:2410.03907 (2024)

2024 arXiv
[51]

Xiu Su, Shan You, Jiyang Xie, Mingkai Zheng, Fei Wang, Chen Qian, Changshui Zhang, Xiaogang Wang, and Chang Xu. 2022. ViTAS: Vision transformer archi- tecture search. In European Conference on Computer Vision . Springer, 139–157

2022
[52]

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017)

2017
[53]

Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 (2023)

2023 arXiv
[54]

Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. 2024. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191 (2024)

2024 arXiv
[55]

Hanlin Wang, Chak Tou Leong, Jian Wang, and Wenjie Li. 2024. E2CL: Exploration-based Error Correction Learning for Embodied Agents.arXiv preprint arXiv:2409.03256 (2024)

2024 arXiv
[56]

Yuqing Wang, Zhaoliang Xu, Xinlong Wang, Chunhua Shen, Baoshan Cheng, Hao Shen, and Huaxia Xia. 2021. End-to-end video instance segmentation with transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 8741–8750

2021
[57]

Xudong Wang, Rohit Girdhar, Stella X Yu, and Ishan Misra. 2023. Cut and learn for unsupervised object detection and instance segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 3124–3134

2023
[58]

Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, et al. 2024. Deepseek- vl2: Mixture-of-experts vision-language models for advanced multimodal under- standing. arXiv preprint arXiv:2412.10302 (2024)

2024 arXiv
[59]

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems 35 (2022), 24824–24837

2022
[60]

Le Xue, Manli Shu, Anas Awadalla, Jun Wang, An Yan, Senthil Purushwalkam, Honglu Zhou, Viraj Prabhu, Yutong Dai, Michael S Ryoo, et al . 2024. xgen- mm (blip-3): A family of open large multimodal models. arXiv preprint arXiv:2408.08872 (2024)

2024
[61]

Zhongze Wu, Hongyan Xu, Yitian Long, Shan You, Xiu Su, Jun Long, Yueyi Luo, and Chang Xu. 2024. Detecting Any instruction-to-answer interaction relationship: Universal Instruction-to-Answer Navigator for Med-VQA. In Forty- first International Conference on Machine Learning

2024
[62]

Qirui Yang, Peng-Tao Jiang, Hao Zhang, Jinwei Chen, Bo Li, Huanjing Yue, and Jingyu Yang. 2024. Learning Adaptive Lighting via Channel-Aware Guidance. arXiv preprint arXiv:2412.01493 (2024)

2024 arXiv
[63]

Qirui Yang, Qihua Cheng, Huanjing Yue, Le Zhang, Yihao Liu, and Jingyu Yang
[64]

Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR)

2023
[65]

Duzhen Zhang, Yahan Yu, Jiahua Dong, Chenxing Li, Dan Su, Chenhui Chu, and Dong Yu. 2024. Mm-llms: Recent advances in multimodal large language models. arXiv preprint arXiv:2401.13601 (2024)

2024 arXiv
[66]

Qirui Yang, Fangpu Zhang, Yeying Jin, Qihua Cheng, Pengtao Jiang, Huanjing Yue, and Jingyu Yang. 2025. DSDNet: Raw Domain Demoir\’eing via Dual Color-Space Synergy. arXiv preprint arXiv:2504.15756 (2025)

2025
[67]

Zhehua Zhou, Jiayang Song, Kunpeng Yao, Zhan Shu, and Lei Ma. 2024. Isr- llm: Iterative self-refined large language model for long-horizon sequential task planning. In 2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2081–2088

2024
[68]

Zhijie Zhu, Lei Fan, Maurice Pagnucco, and Yang Song. 2025. Interpretable Image Classification via Non-parametric Part Prototype Learning. In Proceedings of the Computer Vision and Pattern Recognition Conference . 9762–9771

2025
[69]

Min Zhang, Xian Fu, Jianye Hao, Peilong Han, Hao Zhang, Lei Shi, Hongyao Tang, and Yan Zheng. 2024. MFE-ETP: A Comprehensive Evaluation Benchmark for Multi-modal Foundation Models on Embodied Task Planning. arXiv preprint arXiv:2407.05047 (2024)

2024 arXiv
[72]

Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, et al. 2023. Rt-2: Vision-language- action models transfer web knowledge to robotic control. In Conference on Robot Learning. PMLR, 2165–2183

2023
[2023]

Identifying the defective: Detecting damaged grains for cereal appearance inspection. In ECAI. IOS Press, 660–667
[2025]

IEEE Transactions on Image Processing (2025)

Learning to see low-light images via feature domain adaptation. IEEE Transactions on Image Processing (2025)

2025

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Michael Ahn, Debidatta Dwibedi, Chelsea Finn, Montse Gonzalez Arenas, Keerthana Gopalakrishnan, Karol Hausman, Brian Ichter, Alex Irpan, Nikhil Joshi, Ryan Julian, et al. 2024. Autort: Embodied foundation models for large scale orchestration of robotic agents. arXiv preprint arXiv:2401.12963 (2024)

arXiv 2024

[2] [2]

Faeze Brahman, Chandra Bhagavatula, Valentina Pyatkin, Jena D Hwang, Xi- ang Lorraine Li, Hirona Jacqueline Arai, Soumya Sanyal, Keisuke Sakaguchi, Xiang Ren, and Yejin Choi. 2024. PlaSma: Procedural Knowledge Models for Language-based Planning and Re-Planning. In The Twelfth International Confer- ence on Learning Representations

work page 2024

[3] [3]

Chien-Yi Chang, De-An Huang, Danfei Xu, Ehsan Adeli, Li Fei-Fei, and Juan Car- los Niebles. 2020. Procedure planning in instructional videos. In European Con- ference on Computer Vision . Springer, 334–350

work page 2020

[4] [4]

Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. 2024. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. Science China Information Sciences 67, 12 (2024), 220101

2024

[5] [5]

Bowen Cheng, Alex Schwing, and Alexander Kirillov. 2021. Per-pixel classifica- tion is not all you need for semantic segmentation.Advances in neural information processing systems 34 (2021), 17864–17875

2021

[6] [6]

Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. 2024. Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms. arXiv preprint arXiv:2406.07476 (2024)

arXiv 2024

[7] [7]

Wenliang Dai, Nayeon Lee, Boxin Wang, Zhuolin Yang, Zihan Liu, Jon Barker, Tuomas Rintamaki, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. 2024. Nvlm: Open frontier-class multimodal llms. arXiv preprint arXiv:2409.11402 (2024)

arXiv 2024

[8] [8]

Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, et al. 2023. Palm-e: An embodied multimodal language model. (2023)

2023