Pith. sign in

REVIEW 4 major objections 5 minor 31 references

Hierarchical Instruction-aware Embodied Visual Tracking

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A spatial-goal intermediary lets one offline-trained tracker obey natural-language commands in unseen environments.

desk verdict A sensible task and a clean two-module architecture, but the closed four-goal benchmark cannot support the 'diverse instruction comprehension' headline. read the letter →

arxiv 2505.20710 v1 pith:ZRHJYSTF submitted 2025-05-27 cs.CV

Kui Wu , Hao Chen , Churan Wang , Fakhri Karray , Zhoujun Li , Yizhou Wang , Fangwei Zhong This is my paper · ORCID
classification cs.CV
keywords hierarchicalinstruction-awareembodiedvisualtrackingspatialgoalrepresentationLLMsemantic-spatialalignerofflinereinforcementlearninggoal-conditionedpolicyfoundationmodelsuser-centricsim-to-realtransfer
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that the gap between a user's natural-language instruction and an embodied tracker's low-level actions can be closed by inserting an intermediate spatial goal: a bounding box that says where the target should appear in the camera view. The proposed system, HIEVT, first uses a large-language-model-based Semantic-Spatial Goal Aligner to turn instructions such as "move closer" into a concrete bounding-box goal, then uses an offline-trained, goal-conditioned reinforcement-learning policy to drive the agent so the real target aligns with that box. The load-bearing assertion is that a single offline policy, never retrained, can execute a variety of instructions in nine unseen environments and in real robot deployments, at roughly 50 frames per second policy rate, while large-model baselines fail on speed or generalization. If that claim holds, instruction-aware tracking becomes practical without per-environment tuning or per-instruction retraining.

What carries the argument

The central object is the intermediate spatial goal $G^*=[x,y,w,h]$, a normalized bounding box encoding where the target should appear in the tracker's first-person view. It carries the argument by decomposing the instruction-following problem into two subproblems: mapping language to a box, handled by the LLM-based parser with retrieval correction, and mapping a box to actions, handled by the offline goal-conditioned policy. The policy's goal-state aligner fuses the latest available goal with a vision-foundation-model-processed observation, a recurrent LSTM policy outputs linear and angular velocities, and auxiliary reward regression forces the aligner to recognize states with high alignment reward. This design is what allows a single policy to serve many goals and many environments without retraining.

What would settle it

Run the system with instructions drawn from outside the four-goal taxonomy, such as "keep the target behind you" or "stay at two meters on the right", and score episodes by human judgment of whether the final geometric relationship matches the instruction; if success drops substantially relative to the four-goal set, the reported generalization to complex instructions is an artifact of the restricted instruction set.

Watch

Extended reading notes

Core claim

The central claim is that instruction understanding and action generation in embodied visual tracking can be cleanly separated by a spatial goal, and that this separation yields both speed and generalization. Formally, the paper writes the instruction-to-state distance as $D(I_t,S_t)\approx D(I_t,G_{\mathrm{inter}})+D(G_{\mathrm{inter}},S_t)$, where $G_{\mathrm{inter}}$ is a bounding-box spatial goal. The Semantic-Spatial Goal Aligner performs semantic parsing, chain-of-thought-based spatial-goal generation, and retrieval-augmented correction to produce that box from an instruction and the current observation; the Adaptive Goal-Aligned Policy, trained with offline conservative Q-learning on ten million steps of goal-randomized trajectories with an IoU-based reward and an auxiliary reward-regression head, minimizes the second term. The system processes goal generation and policy execution asynchronously, so a slow large-language-model parser does not throttle the fast control policy. In the reported evaluations it achieves accumulated reward 278 and success rate 1.0 in the training environment, success rates between 0.58 and 0.93 in nine unseen environments, and maintains a 0.84 success rate when the target moves at 2.0 m/s.

Load-bearing premise

The load-bearing premise is that every user instruction can be reduced to one of four spatial goals (close, far, left, right), and that success can be measured by whether the episode lasts 500 steps rather than by whether the instruction was actually satisfied.

Editorial extensions

If this is right

  • A single offline-trained policy can follow spatial-goal instructions in unseen environments without any environment-specific fine-tuning, as shown by success rates of at least 0.58 in all nine unseen test environments.
  • Decoupling instruction parsing from control keeps the system real-time even when the parser is a slow large language model: the policy runs at 50 FPS and the asynchronous design lets the tracker keep moving while a new goal is being computed.
  • The system adapts to instruction changes within about 220 ms, correcting a goal switch from a 67-pixel deviation to 2 pixels in roughly eleven steps.
  • The system maintains a 0.84 success rate at target speed 2.0 m/s, where the compared large-model baselines fail completely due to inference latency.
  • The four goal positions plus their textual paraphrases suffice to benchmark instruction-aware tracking quantitatively.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the four-goal taxonomy is the true boundary of the parser, the reported success rates may overstate open-vocabulary instruction comprehension; a denser, continuous goal space or an instruction-verification layer would be a natural extension.
  • The same spatial-goal decomposition could be applied to other embodied tasks with verbal commands, such as 'search to the left' or 'follow at a distance', by training the goal-conditioned policy on a wider goal distribution and measuring zero-shot transfer.
  • The asynchronous design implies that policy performance should degrade gracefully as the LLM parser slows; a direct test is to add artificial latency to goal generation and measure tracking interruption, which would quantify the decoupling claim.
  • Because the policy is conditioned on vision-foundation-model masks, its generalization likely depends on segmentation quality; a testable extension is to corrupt masks at inference and measure how tracking success degrades.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes HIEVT, a hierarchical system for instruction-aware embodied visual tracking. An LLM-based Semantic-Spatial Goal Aligner converts natural-language instructions into intermediate spatial goals in bounding-box form, and a goal-conditioned offline RL policy (AGAP) trained with CQL plus a reward-regression auxiliary task then controls the tracker. The system is trained in the FlexibleRoom environment and evaluated in nine unseen virtual environments plus real-world deployment. The paper claims large-scale data collection (over ten million trajectories), real-time 50 FPS policy inference, and robustness to diverse environments, target dynamics, and complex instruction combinations.

Significance. If the claims were fully supported, HIEVT would be a practically useful architecture: it decouples language understanding from low-level control, runs the policy at 50 FPS asynchronously with LLM goal generation, and avoids per-environment retraining. The open release of code and datasets, the 10-environment benchmark, and the real-robot deployments are clear strengths. However, the current evidence falls short of the headline claims because the quantitative evaluation is closed around four hand-chosen spatial goals and the main success metric does not measure instruction satisfaction. The paper is worth revising, but the evaluation needs to be substantially strengthened before the central claims can be accepted.

major comments (4)
  1. [Section 4.1 / Appendix B.3–B.4] The quantitative evaluation is closed around four hand-chosen spatial goals. Appendix B.3 defines the evaluation goal list as [g_close, g_far, g_left, g_right], and Appendix B.4's instruction list (Table 4) is generated entirely from those four goals, with relative adjustments such as (Δρ,Δθ)=(-150°,0°) or (0,±20°). The switch protocol samples one of these four goals without replacement, so no episode contains a combined instruction such as "closer and to the left." Consequently, the abstract's claims of handling "diverse human instructions" and "complex instruction combinations" are not supported by Table 1 or Figure 9; the experiments only test paraphrases of four elementary spatial directives.
  2. [Section 4.1, Evaluation Metric] Success Rate is defined as the percentage of episodes reaching 500 steps, not as the percentage of episodes that satisfy the user's instruction. Because the episode termination conditions are tied to keeping the target in view, a tracker that ignores language entirely but successfully tracks can achieve a high SR. The high SR values in Tables 1 and 2 therefore do not by themselves establish instruction following. The AR metric does reflect the pre-assigned goals, but it inherits the limitation of the four-goal taxonomy and cannot measure comprehension of instructions outside that taxonomy.
  3. [Appendix E.1 / Figure 12] The parser evaluation is partly circular. The correctness rules in Appendix E.1—e.g., for (ρ*,θ*)=(200,0°) requiring x ∈ (0.4,0.6) and area ∈ (0.06,0.3), and for (Δρ,Δθ)=(-150,0°) requiring positive Δw and Δh—are exactly the spatial heuristics encoded in the system prompt of Figure 12, which instructs the model to increase w,h when getting closer and to shift x when moving left or right. Scoring the parser against rules that mirror the prompt's own instructions measures prompt compliance, not semantic generalization. An independent annotation protocol or held-out instruction types are needed to support the parser accuracy claims in Figure 9.
  4. [Abstract / Section 3.2 / Appendix C] The data-scale claims are inconsistent. The abstract and contributions state "over ten million trajectories," Section 3.2 states that the final training dataset comprises 10 million steps, and Appendix C states that the offline dataset used to train the method is 1,750,000 steps. These numbers differ by orders of magnitude and cannot all be correct. Since the large-scale dataset is a stated contribution and underpins the generalization argument, the authors must reconcile this discrepancy and report the actual number of steps and episodes.
minor comments (5)
  1. [Section 3.2] There are typos in this section: "Architechture" should be "Architecture," and "Specificallu" should be "Specifically."
  2. [Section 4.3] "HIAEVT" is used once instead of "HIEVT"; please fix the inconsistent acronym.
  3. [Section 4.1] The heading "Experiment" should be "Experiments" (or "Experimental Setup") for consistency with the other section headings.
  4. [Appendix D.3 / Table 10] The caption's explanation of the LSTM sizes is vague: "LSTM64 indicates that all the sizes in the LSTM unit are 64" does not specify input, hidden, and output dimensions. Please give the exact dimensions used.
  5. [Appendix D.2] The description "CQL-SAC algorithm" is imprecise: CQL is a conservative Q-learning update that can be combined with an SAC-style actor-critic, but it is not itself an algorithm named CQL-SAC. Please clarify the relationship.

Circularity Check

2 steps flagged · score 6.0 of 10

Parser accuracy and end-to-end instruction reward are scored against the same four-goal template used to generate the instruction set and prompt heuristics, leaving diverse-instruction comprehension untested.

  1. self definitional [Section 4.1 (Instruction Set Creation, Evaluation Metric) with Appendix B.3/B.4 (Goal Randomization, Instructions)]
    "Due to computational constraints, we evaluate our system using a diverse but finite set of natural language instructions mapped to four representative spatial positions ... we ensure objective evaluation by calculating rewards based on pre-defined spatial goals corresponding to each instruction ... These instructions set are extend from previous mentioned four discrete goal positions."

    The quantitative reward r(I_t,s_t)=1-|rho-rho*|/rho_max-|theta-theta*|/theta_max uses the (rho*,theta*) pre-assigned to each instruction by Table 4, while Table 4 itself is generated from the same four-goal list [g_close, g_far, g_left, g_right]. Thus the evaluation cannot detect any spatial intent outside those four anchors; an instruction such as 'closer and to the left' never appears in the benchmark. The claimed robustness to 'complex instruction combinations' is therefore measured only as proximity to a goal chosen from the same closed taxonomy that generated the instruction set.

  2. self definitional [Appendix E.1 (Quality Evaluation) and Figure 12 (System Prompt)]
    "(Delta rho, Delta theta) = (-150, 0 deg): Valid if both Delta w and Delta h are positive. ... if the instruction wants to get closer to the target, the bounding box size should be larger without horizontal change and a slight increment in vertical position, which should be increased Delta w, Delta h and Delta cy."

    Parser correctness in Appendix E.1 is defined by the exact geometric inequalities written into the system prompt of Figure 12: 'get closer' is valid iff Delta w and Delta h are positive, and the prompt instructs the model to output positive Delta w, Delta h for 'get closer'; left/right validity (Delta x < 0 or > 0) mirrors the prompt's 'move to the left -> negative Delta x' rule. The reported 82-100% parser accuracy therefore measures self-consistency with the prompt's own heuristics, not independent language-to-spatial understanding.

full rationale

The hierarchical decomposition D(I_t,S_t) ~ D(I_t,G_inter) + D(G_inter,S_t) is a design choice rather than a circular derivation, and the RL policy's environment generalization is genuinely tested against external baselines (Bbox PID, Ensembled RL, OpenVLA, GPT-4o) in ten environments. However, the paper's central claim that HIEVT comprehends diverse human instructions is not independently falsifiable in the quantitative evaluation: the instruction list, the parser validity rules, and the end-to-end reward all derive from the same four discrete goals. The parser step is self-definitional because the correctness predicate is the prompt's own heuristic; the end-to-end step is self-definitional because the reward's target (rho*,theta*) is the same template used to generate the instruction set. Additionally, the Success Rate metric counts episodes reaching 500 steps rather than instruction satisfaction, which weakens the reported robustness but is not itself circular. Self-citations to [10] and [26] provide dataset/environment infrastructure and are not load-bearing for the comprehension claim. Overall, the environment-generalization result retains independent content, but the instruction-comprehension result reduces by construction to a closed four-goal benchmark, giving a partial circularity score of 6.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on several human-made choices rather than learned or externally benchmarked facts: the four-goal evaluation taxonomy, the PID-generated offline dataset, the hand-written LLM prompt, and the VFM segmentation reliability. The parser evaluation is partly circular because its validity rules are the same heuristics written into the prompt. The low-level tracking reward uses ground-truth UnrealCV positions, which gives some independent grounding, but the instruction-to-goal mapping remains self-referential.

free parameters (4)
  • Evaluation goal quartet and parser thresholds = gclose=[200,0], gfar=[450,0], gleft=[350,-20], gright=[350,20]; parser thresholds e.g.
    Hand-chosen evaluation constants; all instruction understanding is measured against these four goals, so the central claim depends on them.
  • PID gains for data collection = Speed Kp=5, Ki=0.1, Kd=0.05; Angle Kp=1, Ki=0.01, Kd=0
    Hand-tuned; the offline dataset used to train the policy is generated by this PID with noise, so policy behavior inherits these choices.
  • LLM prompt increment fraction = 20% of bounding box proportion per adjustment
    Hand-written heuristic in system prompt (Figure 12) that defines how instructions map to bounding box changes; not learned or derived.
  • RAG IoU threshold = 0.5
    Hand-chosen threshold in Eq (3) that decides when to replace the generated goal with a retrieved historical mask.
assumptions (4)
  • domain assumption Each user instruction maps to a unique spatial goal (rho*, theta*) within a four-goal taxonomy
    Used to define evaluation reward r(It, st) in Section 4.1; if instructions do not map cleanly to these goals, the metric does not measure instruction following.
  • domain assumption Image-space bounding box alignment (IoU) is a valid proxy for physical relative distance and angle
    Training reward in Eq (4) and goal representation rely on this; it breaks if target sizes or categories vary since the same box area corresponds to different physical distances.
  • domain assumption Vision foundation model segmentation (SAM/VFM) provides reliable target masks in all test environments and in the real world
    The whole pipeline depends on VFM masks for goal generation and reward; no quantitative evaluation of segmentation failures is provided.
  • domain assumption The offline dataset generated by a PID controller with noise covers the goal-conditioned state-action distribution well enough for CQL to generalize
    If the PID-generated trajectories do not cover the state distribution induced by diverse spatial goals, the offline policy may fail outside the training distribution.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Hierarchical Instruction-aware Embodied Visual Tracking." pith.science (2026). https://pith.science/paper/ZRHJYSTF

@misc{pith2026250520710,
  author       = {Pith},
  title        = {Pith review of: Hierarchical Instruction-aware Embodied Visual Tracking},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZRHJYSTF}},
  note         = {Machine review of arXiv:2505.20710}
}
read the original abstract

User-Centric Embodied Visual Tracking (UC-EVT) presents a novel challenge for reinforcement learning-based models due to the substantial gap between high-level user instructions and low-level agent actions. While recent advancements in language models (e.g., LLMs, VLMs, VLAs) have improved instruction comprehension, these models face critical limitations in either inference speed (LLMs, VLMs) or generalizability (VLAs) for UC-EVT tasks. To address these challenges, we propose \textbf{Hierarchical Instruction-aware Embodied Visual Tracking (HIEVT)} agent, which bridges instruction comprehension and action generation using \textit{spatial goals} as intermediaries. HIEVT first introduces \textit{LLM-based Semantic-Spatial Goal Aligner} to translate diverse human instructions into spatial goals that directly annotate the desired spatial position. Then the \textit{RL-based Adaptive Goal-Aligned Policy}, a general offline policy, enables the tracker to position the target as specified by the spatial goal. To benchmark UC-EVT tasks, we collect over ten million trajectories for training and evaluate across one seen environment and nine unseen challenging environments. Extensive experiments and real-world deployments demonstrate the robustness and generalizability of HIEVT across diverse environments, varying target dynamics, and complex instruction combinations. The complete project is available at https://sites.google.com/view/hievt.

Figures

Figures reproduced from arXiv: 2505.20710 by the authors.

Figure 1
Figure 1. Examples of User-Centric embodied visual tracking with diverse instructions. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of the Hierarchical Instruction-aware Embodied Visual Tracker (HIEVT). Given a [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. The examples of virtual and real-world environments used in our experiments. The [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (9 more)
Figure 5
Figure 5. Figure 5: We deploy the agent into a wheel robot in a real-world scenario. The sequence shows [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: 18 humanoid models are used in data collection and evaluation. [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: Training curves for OpenVLA fine-tuning on our dataset over 50k steps. The left panel [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]
Figure 8
Figure 8. Figure 8: Illustration of the target bounding box (yellow box) and goal bounding box (red box) used [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: The accuracy of generated bounding boxed based on textual instructions and spatial goals [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]
Figure 10
Figure 10. Figure 10: In this real-world deployment, the robot’s onboard camera captures visual data, which is [PITH_FULL_IMAGE:figures/full_fig_p021_10.png]
Figure 11
Figure 11. Figure 11: A real-time interactive GUI snapshot. The left panel displays the tracker’s first-person [PITH_FULL_IMAGE:figures/full_fig_p021_11.png]
Figure 12
Figure 12. Figure 12: System prompt used in instruction parser module. [PITH_FULL_IMAGE:figures/full_fig_p023_12.png]
Figure 13
Figure 13. Figure 13: System prompt used in baseline method GPT4-o. [PITH_FULL_IMAGE:figures/full_fig_p024_13.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

31 extracted references · 21 canonical work pages

  1. [1]

    The human-following strategy for mobile robots in mixed environments.Robotics and Autonomous Systems, 160:104317, 2023

    Nguyen Van Toan, Minh Do Hoang, Phan Bui Khoi, and Soo-Yeong Yi. The human-following strategy for mobile robots in mixed environments.Robotics and Autonomous Systems, 160:104317, 2023

    work page 2023

  2. [2]

    Animaltrack: A benchmark for multi-animal tracking in the wild.International Journal of Computer Vision, 131(2):496–513, 2023

    Libo Zhang, Junyuan Gao, Zhen Xiao, and Heng Fan. Animaltrack: A benchmark for multi-animal tracking in the wild.International Journal of Computer Vision, 131(2):496–513, 2023

    work page 2023

  3. [3]

    Exploring the role of human-following robots in supporting the mobility and wellbeing of older people.Scientific reports, 13(1):6512, 2023

    Shuo Li, Kirsty Milligan, Phil Blythe, Yanghanzi Zhang, Simon Edwards, Nic Palmarini, Lynne Corner, Yanjie Ji, Fan Zhang, and Anil Namdeo. Exploring the role of human-following robots in supporting the mobility and wellbeing of older people.Scientific reports, 13(1):6512, 2023

    work page 2023

  4. [4]

    Tenenbaum, and Chuang Gan

    Qinhong Zhou, Sunli Chen, Yisong Wang, Haozhe Xu, Weihua Du, Hongxin Zhang, Yilun Du, Joshua B. Tenenbaum, and Chuang Gan. HAZARD challenge: Embodied decision making in dynamically changing environments. InThe Twelfth International Conference on Learning Representations, 2024

    work page 2024

  5. [5]

    ChimpACT: A longitudinal dataset for understanding chim- panzee behaviors

    Xiaoxuan Ma, Stephan Paul Kaufhold, Jiajun Su, Wentao Zhu, Jack Terwilliger, Andres Meza, Yixin Zhu, Federico Rossano, and Yizhou Wang. ChimpACT: A longitudinal dataset for understanding chim- panzee behaviors. InThirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023

    work page 2023

  6. [6]

    End-to-end active object tracking and its real-world deployment via reinforcement learning.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019

    Wenhan Luo, Peng Sun, Fangwei Zhong, Wei Liu, Tong Zhang, and Yizhou Wang. End-to-end active object tracking and its real-world deployment via reinforcement learning.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019

    work page 2019

  7. [7]

    Ad-vat+: An asymmetric dueling mechanism for learning and understanding visual active tracking.IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(5):1467–1482, 2019

    Fangwei Zhong, Peng Sun, Wenhan Luo, Tingyun Yan, and Yizhou Wang. Ad-vat+: An asymmetric dueling mechanism for learning and understanding visual active tracking.IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(5):1467–1482, 2019

    2019

  8. [8]

    Towards distraction-robust active visual tracking

    Fangwei Zhong, Peng Sun, Wenhan Luo, Tingyun Yan, and Yizhou Wang. Towards distraction-robust active visual tracking. InInternational Conference on Machine Learning, pages 12782–12792. PMLR, 2021

    work page 2021

Show all 31 references
  1. [9]

    Rspt: reconstruct surroundings and predict trajectory for generalizable active object tracking

    Fangwei Zhong, Xiao Bi, Yudi Zhang, Wei Zhang, and Yizhou Wang. Rspt: reconstruct surroundings and predict trajectory for generalizable active object tracking. InProceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 3705–3714, 2023

    2023

  2. [10]

    Empowering embodied visual tracking with visual foundation models and offline rl

    Fangwei Zhong, Kui Wu, Hai Ci, Churan Wang, and Hao Chen. Empowering embodied visual tracking with visual foundation models and offline rl. InEuropean Conference on Computer Vision, pages 139–155. Springer, 2024

    2024

  3. [11]

    Chain-of-thought prompting elicits reasoning in large language models.Advances in neural information processing systems, 35:24824–24837, 2022

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models.Advances in neural information processing systems, 35:24824–24837, 2022. 10

    2022

  4. [12]

    Motchallenge 2015: Towards a benchmark for multi-target tracking.arXiv preprint arXiv:1504.01942, 2015

    Laura Leal-Taixé, Anton Milan, Ian Reid, Stefan Roth, and Konrad Schindler. Motchallenge 2015: Towards a benchmark for multi-target tracking.arXiv preprint arXiv:1504.01942, 2015

    2015 arXiv

  5. [13]

    The cityscapes dataset for semantic urban scene understanding

    Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. InProceedings of the IEEE conference on Computer Vision and Pattern Recog...

    2016

  6. [14]

    Conservative q-learning for offline reinforcement learning.Advances in Neural Information Processing Systems, 33:1179–1191, 2020

    Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning.Advances in Neural Information Processing Systems, 33:1179–1191, 2020

    2020

  7. [15]

    Unrealcv: Virtual worlds for computer vision

    Weichao Qiu, Fangwei Zhong, Yi Zhang, Siyuan Qiao, Zihao Xiao, Tae Soo Kim, Yizhou Wang, and Alan Yuille. Unrealcv: Virtual worlds for computer vision. InProceedings of the 2017 ACM on Multimedia Conference, pages 1221–1224, 2017

    2017

  8. [16]

    OpenVLA: An open-source vision-language-action model

    Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan P Foster, Pannag R Sanketi, Quan Vuong, Thomas Kollar, Benjamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn. OpenVLA: An op...

    2024

  9. [17]

    Segment and track anything.arXiv preprint arXiv:2305.06558, 2023

    Yangming Cheng, Liulei Li, Yuanyou Xu, Xiaodi Li, Zongxin Yang, Wenguan Wang, and Yi Yang. Segment and track anything.arXiv preprint arXiv:2305.06558, 2023

    2023 arXiv

  10. [18]

    AD-V AT: An asymmet- ric dueling mechanism for learning visual active tracking

    Fangwei Zhong, Peng Sun, Wenhan Luo, Tingyun Yan, and Yizhou Wang. AD-V AT: An asymmet- ric dueling mechanism for learning visual active tracking. InInternational Conference on Learning Representations, 2019

    2019

  11. [19]

    Tracking anything with decoupled video segmentation

    Ho Kei Cheng, Seoung Wug Oh, Brian Price, Alexander Schwing, and Joon-Young Lee. Tracking anything with decoupled video segmentation. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 1316–1326, 2023

    2023

  12. [20]

    Touchdown: Natural language navigation and spatial reasoning in visual street environments

    Howard Chen, Alane Suhr, Dipendra Misra, Noah Snavely, and Yoav Artzi. Touchdown: Natural language navigation and spatial reasoning in visual street environments. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12538–12547, 2019

    2019

  13. [21]

    Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments

    Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton Van Den Hengel. Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. InProceedings of the IEEE conference ...

    2018

  14. [22]

    Iterative vision-and-language navigation

    Jacob Krantz, Shurjo Banerjee, Wang Zhu, Jason Corso, Peter Anderson, Stefan Lee, and Jesse Thomason. Iterative vision-and-language navigation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14921–14930, 2023

    2023

  15. [23]

    Palm-e: An embodied multimodal language model

    Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. InInternational Conference on Machine Learning, pages 8469–8488. PMLR, 2023

    2023

  16. [24]

    Avlen: Audio-visual-language embodied navigation in 3d environments.Advances in Neural Information Processing Systems, 35:6236–6249, 2022

    Sudipta Paul, Amit Roy-Chowdhury, and Anoop Cherian. Avlen: Audio-visual-language embodied navigation in 3d environments.Advances in Neural Information Processing Systems, 35:6236–6249, 2022

    2022

  17. [25]

    Vima: General robot manipulation with multimodal prompts.arXiv preprint arXiv:2210.03094, 2(3):6, 2022

    Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan. Vima: General robot manipulation with multimodal prompts.arXiv preprint arXiv:2210.03094, 2(3):6, 2022

    arXiv

  18. [26]

    Keep the person in the close center

    Fangwei Zhong, Kui Wu, Churan Wang, Hao Chen, Hai Ci, Zhoujun Li, and Yizhou Wang. Unrealzoo: Enriching photo-realistic virtual worlds for embodied ai, 2024. 11 A Related Works Embodied Visual Tracking (EVT)is a foundational skill of embodied AI. It has been a focal point for ...

    2024

  19. [27]

    Get closer to the person,

    **Instruction:** A natural language command describing the desired change in the tracking of the target (e.g., "Get closer to the person," "Move further from the car," "Keep the dog in the center," or "Keep the object on the left")

  20. [28]

    Target position: [cx, cy, w, h]

    **Current bounding box: ** The current bounding box coordinates and size of the target in your field of view relative to the image size, normalized to [0, 1] (e.g ., "Target position: [cx, cy, w, h]"). Task Definition: Your task is to: Extract the target category from the inst...

  21. [29]

    **Goal bounding box: ** This is provided by the user to indicate the expected distance and angle between the target and the tracker, which is a bounding box format, the agent should try to align the target bounding box with the goal bounding box as much as possible

  22. [30]

    Task Definition: Your task is to give a suitable action from Control actions, and try to align the Goal bounding box with the target bounding box as much as possible

    **Target bounding box: ** This is provided by user to indicate the current target position in the image, in the bounding box format. Task Definition: Your task is to give a suitable action from Control actions, and try to align the Goal bounding box with the target bounding bo...

  23. [31]

    **Actions:** Based on the given Goal bounding box and Target bounding box, you should provide the best control action from the control actions to align the target bounding box with the goal bounding box as much as possible. Strategy Considerations: The target bounding box size...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.