Pith. sign in

REVIEW 4 major objections 6 minor 4 cited by

TrackVLA: Embodied Visual Tracking in the Wild

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read TrackVLA claims a single vision-language-action model can recognize a described target and follow it in unseen environments, including real robots, without fine-tuning.

desk verdict A real VLA-for-tracking contribution with a strong public-benchmark zero-shot result; the self-built benchmark and undisclosed expert-trajectory policy are the soft spots that need a referee's scrutiny. read the letter →

arxiv 2505.23189 v1 pith:6N37WMPZ submitted 2025-05-29 cs.RO cs.CV

Shaoan Wang , Jiazhao Zhang , Minghan Li , Jiahang Liu , Anqi Li , Kui Wu , Fangwei Zhong , Junzhi Yu
show 2 more authors
Zhizheng Zhang He Wang
This is my paper
classification cs.ROcs.CV
keywords EmbodiedVisualTrackingVision-Language-ActionModelAnchor-basedDiffusionZero-shotTransferEVT-BenchSim-to-realVideoQuestionAnsweringHumanFollowing
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

TrackVLA is a vision-language-action (VLA) model trained to do two things in one forward pass: recognize the target described in a natural-language instruction and output a robot trajectory that keeps following that target. It shares one LLM backbone across both skills, using a language-modeling head for recognition and an anchor-based diffusion head for trajectory planning, so the skills are learned jointly rather than chained as separate detection and planning modules. The authors construct a 1.7 million-sample dataset of tracking episodes and video question-answering samples, then evaluate the same model zero-shot on public benchmarks and on a real quadruped robot. Their central claim is that joint training produces synergy: TrackVLA outperforms prior modular and VLA baselines in simulation and tracks previously unseen targets, including animals, in new environments at 10 FPS with no fine-tuning. A sympathetic reader would care because robust target recognition combined with reactive planning under occlusion is a core bottleneck for robots that must follow people or objects in the wild.

What carries the argument

The load-bearing object is a shared LLM backbone with two decoding heads: a language-modeling head for recognition, which predicts text tokens in the usual autoregressive way, and an anchor-based diffusion head for planning. The planning head clusters every expert trajectory in the training set with K-means into M=40 trajectory anchors, adds Gaussian noise to those anchors, and runs a diffusion transformer over the noised anchors conditioned on the LLM's hidden state; at inference, two denoising steps select the top-scoring trajectory. A special [Track] token routes the shared computation to the action head, and a combined text-loss plus track-loss objective trains both heads simultaneously. This machinery is what lets recognition and planning share visual and linguistic features rather than accumulate errors across separate modules.

What would settle it

Run TrackVLA zero-shot on EVT-Bench with human-written target descriptions substituted for the auto-generated avatar captions while keeping the same episodes; if success rates on the distracted- and ambiguity-tracking tasks drop materially from the reported 57.6% and 50.2%, the recognition abilities are partly an artifact of training and evaluation sharing the same captions.

Watch

Extended reading notes

Core claim

The paper's central claim is that embodied visual tracking does not require a separate detector and planner; a single VLA can learn both in one forward pass. TrackVLA feeds a short egocentric video and an instruction into a frozen vision encoder plus a 7B-parameter LLM; when a special [Track] token is present, the model runs a single autoregressive step and passes the last hidden state to an anchor-based diffusion head that denoises a waypoint trajectory in two steps, while the same visual tokens answered without [Track] recognition questions. The authors report state-of-the-art results on Gym-UnrealCV in a zero-shot setting, including perfect 500-step episodes in single-target and unseen-object tasks, and large gains on their own EVT-Bench over reinforcement-learning and imitation-learning baselines. They additionally demonstrate real-world deployment on a quadruped robot where TrackVLA outperformed a commercial tracking drone in medium and hard scenarios, supporting the claim that joint recognition and planning training transfers to the physical world.

Load-bearing premise

Every learned skill in TrackVLA depends on auto-generated training labels: the appearance descriptions of the 100 synthetic avatars come from an automated vision-language model with no reported human-verification rate, and the expert trajectories used as ground truth are not described; if those labels are systematically wrong, both recognition and policy degrade in deployment in ways the synthetic benchmark cannot reveal.

Editorial extensions

If this is right

  • A robot using TrackVLA can, in one forward pass, identify a target from a free-form description and plan a follow trajectory, so a separate detector, tracker, and controller chained together become unnecessary.
  • Zero-shot transfer to Gym-UnrealCV, which shares no training data with TrackVLA, indicates the learned recognition-planning coupling generalizes across simulators, not just within the training simulator.
  • Real-world experiments on a quadruped outperforming a commercial tracking drone in occlusion and fast-motion settings suggest the approach can be deployed in the field at interactive speed.
  • Joint training with open-world video-question-answering data improves both recognition accuracy and downstream tracking, consistent with a data-scaling benefit for VLA models.
  • Anchor-based diffusion gives a fivefold inference speedup over a vanilla diffusion policy and enables 10 FPS closed-loop control, making the policy practical for dynamic targets.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: because the training and evaluation pipelines share the same auto-generated avatar captions, the reported benchmark results likely overstate how the model handles human-written descriptions; a test with human-verified captions would separate real recognition from caption memorization.
  • Editorial extension: the 1:1 data-ratio finding hints that for dynamic embodied tasks the perception and policy data budgets should be balanced; scaling either alone may saturate sooner than scaling both together.
  • Editorial extension: if the expert trajectories were produced with privileged simulator information, such as future target positions, the same policy may need trajectory-level sim-to-real correction before reaching the reported real-world ceiling.
  • Editorial extension: because both recognition and planning consume the same visual tokens, the architecture suggests a path toward other coupled embodied skills, such as instruction-following navigation, social-aware following, or object search, without adding new modules.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes TrackVLA, a vision-language-action model for embodied visual tracking that learns target recognition and trajectory planning jointly through a shared LLM backbone, with a language modeling head for recognition and an anchor-based diffusion head for waypoint trajectory generation. The authors construct an Embodied Visual Tracking Benchmark (EVT-Bench) in Habitat 3.0 with 100 SMPL-X avatars and 25,986 episodes, collect 855K recognition samples, and train a Vicuna-7B-based model on 1.7M samples. Experiments report zero-shot state-of-the-art results on the public Gym-UnrealCV benchmark, strong results on EVT-Bench, a recognition comparison against RexSeek/LISA++/SoM+GPT-4o, ablations of data ratio, action head, history length, and waypoint count, and qualitative plus quantitative real-world deployment on a Unitree GO2 quadruped. The central claim is that a single VLA trained only on simulation and pasted-image data can recognize a described target and follow it in unseen simulators and real environments at 10 FPS with no fine-tuning.

Significance. If the results hold, TrackVLA is a meaningful advance: it provides a unified architecture for recognition and planning in a highly dynamic task, runs at interactive speed, and demonstrates zero-shot transfer to an external simulator and a real robot. The public Gym-UnrealCV results are the strongest evidence, and the proposed 1.7M-sample dataset and EVT-Bench are potentially valuable community resources. The strongest claims are, however, conditional on the transparency of the data-generation process: the expert trajectories that supervise the policy are not described, the avatar captions are machine-generated without reported human verification, and the real-world evaluation is anecdotal. These gaps need to be addressed before the headline claims can be taken at face value.

major comments (4)
  1. [Sec. 4.1, Eq. (3)] The manuscript never describes or cites the policy that generated the expert trajectories tau_gt for the 885K tracking samples. Appendix C.1 specifies only the target avatar's motion (start point, intermediate waypoints, end point) and the agent's initial placement, not the robot's demonstrated action sequence. Since the tracking loss in Eq. (3) regresses the predicted trajectories to tau_gt, the entire learned policy is a function of labels whose causal realizability is unexamined. If tau_gt was produced with future target positions, global map knowledge, or other simulator privileges unavailable at deployment, then the EVT-Bench scores could reflect label leakage, the zero-shot Gym-UnrealCV transfer would not establish that TrackVLA learned a realizable visuomotor policy, and the real-world results in Table 13 could not be attributed to the learned model. Please specify the trajectory generator, its observation space, and whether it is causal; if privileged information was used, provide an analysis of deployability or retrain with causal demonstrations.
  2. [Sec. 4.1, 'Humanoid Avatar Generation'] The avatar descriptions used as tracking instructions are produced automatically by Qwen-VL2.5 with no reported human verification rate. These captions are used both as training instructions and as the basis of EVT-Bench test instructions, and the recognition branch is trained on the same automated label source. Systematic caption errors about colors or clothing would therefore degrade real-world language grounding while leaving EVT-Bench scores optimistic, because the benchmark shares the same flawed labels. Please report a human-verified accuracy sample on the generated captions; if verification reveals a non-negligible error rate, the captions should be filtered or corrected and the experiments rerun.
  3. [Sec. 5.4, Fig. 6, Tables 10-11] The central benchmark EVT-Bench is constructed by the authors and used for training, evaluation, and hyperparameter selection: the 1:1 data ratio, history length k=32, waypoint count Nw=10, number of anchors M=40, and loss weights are all chosen on the same benchmark's DT task. This makes the EVT-Bench comparisons partly self-referential and weakens the claim that the design choices are principled. The public Gym-UnrealCV zero-shot results mitigate the concern for generalization, but the paper should state explicitly that EVT-Bench numbers reflect in-distribution tuning and should report sensitivity of the selected hyperparameters on a held-out public benchmark if possible.
  4. [Sec. H, Table 13] The real-world evaluation compares a quadruped robot running TrackVLA with a DJI Flip drone over 10 trials per condition, but it does not specify the success criterion, the target motion protocol, the environment layout, or the definition of 'Easy', 'Medium', and 'Hard'. No error bars or per-trial results are given, and the comparison is between fundamentally different platforms (a ground robot with LiDAR-based obstacle avoidance versus a commercial aerial drone). Since the abstract claims robustness 'in real-world scenarios', this evidence is currently anecdotal. Please provide a detailed protocol, a clear success definition, per-trial data, and ideally a comparison with a ground-robot baseline; otherwise the real-world claim should be softened.
minor comments (6)
  1. [Sec. 1 vs. Sec. 4.1] The abstract and introduction state 855K embodied visual tracking samples, while Sec. 4.1 says 885K tracking samples were collected; please reconcile these numbers.
  2. [Sec. C.3] The metric definition contains a typo: 'the aggent' should be 'the agent'.
  3. [Fig. 10 and Sec. H] The figure captions use 'UVA views' where 'UAV views' is intended.
  4. [Tables 1, 6-8] The Gym-UnrealCV results report perfect or near-perfect scores over 100 episodes without per-episode variance or confidence intervals; please add standard errors or bootstrap intervals so the reader can assess the stability of the zero-shot claims.
  5. [Fig. 4 and Sec. 4.2] The figure contains an incomplete reference label 'Pandas[]' and the text refers to 'MovieNet' while the reference list and figure use 'MovieChat'; please correct the names and complete the citation.
  6. [Sec. 5.4, Fig. 6] The ablation curves in Fig. 6 appear smooth but no error bars or number of seeds are reported; please state whether the results are averaged over multiple random seeds.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central zero-shot claim is validated on the external Gym-UnrealCV benchmark, and EVT-Bench uses unseen scenes/avatars with metrics distinct from the training loss.

full rationale

The paper's central claims are not circular. TrackVLA is trained on the authors' EVT-Bench training split (21,771 episodes, 703 scenes) plus 855K VQA samples, and its main zero-shot generalization claim is validated on the public Gym-UnrealCV benchmark (Table 1), on which no TrackVLA training data or hyperparameters are fitted. EVT-Bench is self-constructed, but its test split uses 101 unseen scenes and non-overlapping avatars, and the reported metrics (SR, TR, CR) are not the training objective; the training loss (Eq. 3) is an MSE+BCE anchor-diffusion loss, while SR/TR/CR are geometric success criteria evaluated in a closed-loop simulator. Hyperparameters such as the 1:1 data ratio, k=32 history window, Nw=10 waypoints, and M=40 anchors are selected on the EVT-Bench DT task (Fig. 6, Tables 10-11), which weakens the strength of the EVT-Bench numbers as an independent claim, but this is model selection rather than a definitional equivalence between the prediction and its inputs. The SYNTH-PEDES recognition test uses unseen images from the same dataset that supplied training VQA compositions; this is an in-distribution evaluation, not a zero-shot claim, and the paper does not misrepresent it as circular. The only notable gap is that Sec. 4.1 states each tracking sample includes 'the corresponding expert trajectory tau_gt' without describing the policy that generated it; if tau_gt used privileged simulator state, real-world and zero-shot conclusions could be weakened, but the paper does not define tau_gt in terms of TrackVLA's output, so this is a reproducibility/correctness concern rather than circularity. Self-citations (Uni-NaVid [14], EVT [6]) are used only as baselines and related work, not as load-bearing justifications for TrackVLA's design. No equation, label, or benchmark criterion reduces by construction to TrackVLA's own outputs, so there are no circular steps to report.

Assumptions & free parameters 5 free parameters · 6 assumptions · 3 invented entities

The central claims rest on pretrained VLM backbones, auto-generated synthetic supervision, and several empirically tuned hyperparameters. The load-bearing free parameters (data ratio, history length, waypoint count, anchor count, loss weights) are selected on the same EVT-Bench evaluation used for the headline results. The main independent check, the Gym-UnrealCV zero-shot evaluation, is external and does not share these fitted choices.

free parameters (5)
  • training data ratio |Ntrack|:|Nrecog| = 1:1 = 1:1
    The 1:1 ratio is selected on the DT task of EVT-Bench (Fig 6) and stated to yield the best performance, so the headline model is tuned on the evaluation task distribution.
  • history window length k = 32
    Empirically selected on EVT-Bench DT (Table 10): k=0 drops SR from 57.6 to 29.9, and k=64 is slightly worse than k=32.
  • trajectory waypoint count Nw = 10
    Empirically selected on EVT-Bench DT (Table 11): Nw=1 gives SR 44.3 and Nw=20 gives 51.3, both below Nw=10.
  • number of trajectory anchors M = 40
    K-means cluster count M=40 on training trajectories (Sec 3.1); no ablation justifies this number and it directly determines action-head capacity and inference cost.
  • loss balancing weights lambda and alpha = lambda = 100, alpha = 1
    Hand-set weights in the combined loss L = Ltrack + alpha*Ltext (Eq 3, Sec 3.1); no sensitivity analysis is reported for lambda=100 or alpha=1.
assumptions (6)
  • domain assumption Pretrained Vicuna-7B and EVA-CLIP embeddings transfer to egocentric robot video and support both language and action decoding.
    Invoked throughout Sec 3.1-3.2; frozen vision encoder and LLM initialization are treated as reliable foundations without re-validation on robot data.
  • domain assumption Synthetic avatars (SMPL-X bodies with random ATLAS textures, moving with ORCA collision avoidance in Habitat 3.0 scenes) approximate the appearance and dynamics of real humans closely enough for zero-shot sim-to-real transfer.
    Sec 4.1; the entire zero-shot real-world claim depends on this distribution match.
  • domain assumption Qwen-VL2.5-generated avatar descriptions are accurate enough to serve as ground truth for target recognition and for the Distracted Tracking task.
    Sec 4.1 'Humanoid Avatar Generation'; no human verification rate is reported for the generated captions.
  • domain assumption A valid expert trajectory tau_gt exists for every tracking sample and is realizable by the real robot.
    Sec 4.1 'Tracking Data Collection'; the expert policy that produced tau_gt is never described or cited.
  • domain assumption The defined success metrics (oriented toward target within 1-3 m, or the Gym-UnrealCV fan-sector criterion) capture what tracking means in real use.
    Sec C.3 and D.2; the real-world experiments never define formal success criteria.
  • domain assumption SYNTH-PEDES compositions (human images pasted onto random backgrounds) train a recognition capability that transfers to real scenes.
    Sec 4.2; the recognition test in Table 3 samples the same SYNTH-PEDES distribution used in training, so cross-domain recognition is only qualitatively validated.
invented entities (3)
  • EVT-Bench benchmark independent evidence
    purpose: Self-built simulator benchmark with 25,986 episodes across 804 scenes, graded into single-target, distracted, and ambiguous tracking; used to train TrackVLA and to claim state-of-the-art results in Table 2.
    Once released, other groups can run the same episodes and avatars, giving an external falsifiable handle; at submission it is not yet public and its difficulty design is defined by the authors, so the handle is prospective rather than realized.
  • K-means trajectory anchors (M = 40)
    purpose: Typical trajectory shapes computed from TrackVLA's own training data that the diffusion head denoises from, reducing denoising to two steps.
    The anchors are fitted to the paper's own trajectories and have no falsifiable handle outside the paper; they are validated only by the in-benchmark ablation in Table 4.
  • Special [Track] language token
    purpose: Task router that makes the shared LLM emit a single hidden state for the diffusion action head instead of autoregressive text.
    An architectural routing invention with no external falsifiable handle; standard practice in VLA task routing and measured only through the joint-training system as a whole.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TrackVLA: Embodied Visual Tracking in the Wild." pith.science (2026). https://pith.science/paper/6N37WMPZ

@misc{pith2026250523189,
  author       = {Pith},
  title        = {Pith review of: TrackVLA: Embodied Visual Tracking in the Wild},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6N37WMPZ}},
  note         = {Machine review of arXiv:2505.23189}
}
read the original abstract

Embodied visual tracking is a fundamental skill in Embodied AI, enabling an agent to follow a specific target in dynamic environments using only egocentric vision. This task is inherently challenging as it requires both accurate target recognition and effective trajectory planning under conditions of severe occlusion and high scene dynamics. Existing approaches typically address this challenge through a modular separation of recognition and planning. In this work, we propose TrackVLA, a Vision-Language-Action (VLA) model that learns the synergy between object recognition and trajectory planning. Leveraging a shared LLM backbone, we employ a language modeling head for recognition and an anchor-based diffusion model for trajectory planning. To train TrackVLA, we construct an Embodied Visual Tracking Benchmark (EVT-Bench) and collect diverse difficulty levels of recognition samples, resulting in a dataset of 1.7 million samples. Through extensive experiments in both synthetic and real-world environments, TrackVLA demonstrates SOTA performance and strong generalizability. It significantly outperforms existing methods on public benchmarks in a zero-shot manner while remaining robust to high dynamics and occlusion in real-world scenarios at 10 FPS inference speed. Our project page is: https://pku-epic.github.io/TrackVLA-web.

Figures

Figures reproduced from arXiv: 2505.23189 by the authors.

Figure 1
Figure 1. TrackVLA is a vision-language-action model capable of simultaneous object recognition and visual tracking, trained on a dataset of 1.7 million samples. It demonstrates robust tracking, long-horizon tracking, and cross-domain generalization across diverse challenging environments. 1 Introduction Embodied visual tracking (EVT) [1, 2, 3, 4, 5, 6] requires the agent to persistently track a given target, which is a funda… view at source ↗
Figure 2
Figure 2. Overall pipeline of TrackVLA. Given a video and a language instruction, TrackVLA outputs either a tracking trajectory for the robot or an answer to the recognition question. robust perception and flexible motion control, and demonstrating superior embodied visual tracking performance in real-world deployments. Embodied Navigation. Embodied navigation [31, 32, 33, 34, 35, 36] is a fundamental topic in embodied AI, re… view at source ↗
Figure 3
Figure 3. Anchor-based Diffusion Action Model. Anchor-based Diffusion Action Model. We em￾ploy an anchor-based diffusion model [60] that per￾forms denoising from predefined anchors to generate waypoints. These predefined anchors provide ini￾tial coarse trajectories that significantly reduce the required denoising iterations, yielding a 5× speedup compared to vanilla diffusion policies [33, 61]. As shown in [PITH_FULL_IMAGE:f… view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Overview of the training datasets used in TrackVLA. We collect 855K embodied visual tracking samples and 855K open-world recognition samples to jointly enhance the robust recognition and tracking capabilities of TrackVLA. • Single-Target Tracking (STT) evaluates the mo…
Figure 5
Figure 5. Figure 5: Real-world qualitative results of TrackVLA. TrackVLA is deployed in a zero-shot man￾ner across diverse environments, executing diverse tracking instructions in challenging scenarios. tasks, and (D) multi-person recognition. The experimental results demonstrate that Tra…
Figure 6
Figure 6. Figure 6: Comparison of different data scales and ratios. Data Scale and Data Ratio. We conduct an ablation study on the DT task in EVT-Bench to investigate the in￾fluence of training sample scale and ratio. Here, we de￾note the number of embodied visual tracking samples and ope…
Figure 7
Figure 7. Figure 7: Visualization of the custom humanoid avatars with captions. [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: Examples of humanoid avatars used in Gym-UnrealCV. [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]
Figure 9
Figure 9. Figure 9: Real-world system architecture. TrackVLA is deployed on a remote server, and the robot communicates with it via the Internet. G.1 Robot Platform We provide a visualization of our robotic platform in [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]
Figure 10
Figure 10. Figure 10: Visualization of the real-world experiments. TrackVLA demonstrates robust tracking performance under challenging conditions such as occlusions and fast target motion, outperforming existing commercial tracking drones. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_10.png]
Figure 11
Figure 11. Figure 11: Visualization of the training set of EVT-Bench. [PITH_FULL_IMAGE:figures/full_fig_p023_11.png]
Figure 12
Figure 12. Figure 12: Visualization of TrackVLA on EVT-Bench. 24 [PITH_FULL_IMAGE:figures/full_fig_p024_12.png]
Figure 13
Figure 13. Figure 13: Visualization of TrackVLA on Gym-UnrealCV. [PITH_FULL_IMAGE:figures/full_fig_p025_13.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. UESF-Bench: Benchmarking and Probing for Unified Embodied Seeking and Following

    cs.AI 2026-07 conditional novelty 6.0 of 10

    UESF-Bench is a 1.43M-sample simulated benchmark for embodied agents that must first find a language-described person and then follow them; SeekFollow-VLA with task-driven routing outperforms the paper's internal baselines.

  2. ABot-N1: Toward a General Visual Language Navigation Foundation Model

    cs.CV 2026-07 unverdicted novelty 6.0 of 10

    A slow–fast VLN model that routes five navigation tasks through CoT plus image-space pixel goals reaches SOTA on established and new urban benchmarks, including 77.3% POI arrival.

  3. Structured Observation Language for Efficient and Generalizable Vision-Language Navigation

    cs.CV 2026-03 reject novelty 6.0 of 10

    SOL-Nav encodes RGB-D observations as grid-organized text and uses a 0.6B text-embedding model with four classification heads to predict navigation action blocks, reporting SOTA/comparable R2R-CE/RxR-CE results with 1...

  4. LOVON: Legged Open-Vocabulary Object Navigator

    cs.RO 2025-07 reject novelty 4.0 of 10

    LOVON integrates an LLM planner, a blur-filtered object detector, and a small learned motion model to navigate legged robots to user-specified objects over long horizons, claiming near-perfect simulation success and r...

Reference graph

Works this paper leans on

80 extracted references · 40 canonical work pages · cited by 4 Pith papers

  1. [1]

    Maalouf, N

    A. Maalouf, N. Jadhav, K. M. Jatavallabhula, M. Chahine, D. M. V ogt, R. J. Wood, A. Torralba, and D. Rus. Follow anything: Open-set detection, tracking, and following in real-time. IEEE Robotics and Automation Letters, 9(4):3283–3290, 2024

    work page 2024

  2. [2]

    Zhang, K

    W. Zhang, K. Song, X. Rong, and Y . Li. Coarse-to-fine uav target tracking with deep reinforce- ment learning. IEEE Transactions on Automation Science and Engineering, 16(4):1522–1530, 2018

    work page 2018

  3. [3]

    Zhong, P

    F. Zhong, P. Sun, W. Luo, T. Yan, and Y . Wang. Ad-vat: An asymmetric dueling mechanism for learning visual active tracking. In International Conference on Learning Representations, 2019

    work page 2019

  4. [4]

    Zhong, P

    F. Zhong, P. Sun, W. Luo, T. Yan, and Y . Wang. Ad-vat+: An asymmetric dueling mechanism for learning and understanding visual active tracking. IEEE transactions on pattern analysis and machine intelligence, 43(5):1467–1482, 2019

    work page 2019

  5. [5]

    Zhong, X

    F. Zhong, X. Bi, Y . Zhang, W. Zhang, and Y . Wang. Rspt: reconstruct surroundings and predict trajectory for generalizable active object tracking. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 3705–3714, 2023

    work page 2023

  6. [6]

    Zhong, K

    F. Zhong, K. Wu, H. Ci, C. Wang, and H. Chen. Empowering embodied visual tracking with visual foundation models and offline rl. In European Conference on Computer Vision, pages 139–155. Springer, 2024

    work page 2024

  7. [7]

    J. Duan, S. Yu, H. L. Tan, H. Zhu, and C. Tan. A survey of embodied ai: From simulators to research tasks. IEEE Transactions on Emerging Topics in Computational Intelligence , 6(2): 230–244, 2022

    work page 2022

  8. [8]

    Mavrogiannis, F

    C. Mavrogiannis, F. Baldini, A. Wang, D. Zhao, P. Trautman, A. Steinfeld, and J. Oh. Core challenges of social robot navigation: A survey. ACM Transactions on Human-Robot Interac- tion, 12(3):1–39, 2023

    2023

Show all 80 references
  1. [9]

    X. Puig, E. Undersander, A. Szot, M. D. Cote, T.-Y . Yang, R. Partsey, R. Desai, A. W. Clegg, M. Hlavac, S. Y . Min, et al. Habitat 3.0: A co-habitat for humans, avatars and robots. arXiv preprint arXiv:2310.13724, 2023

    2023 arXiv

  2. [10]

    J. Li, J. Xu, F. Zhong, X. Kong, Y . Qiao, and Y . Wang. Pose-assisted multi-camera col- laboration for active object tracking. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 759–766, 2020

    2020

  3. [11]

    Kirillov, E

    A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y . Lo, et al. Segment anything. InProceedings of the IEEE/CVF international conference on computer vision, pages 4015–4026, 2023

    2023

  4. [12]

    N. Ravi, V . Gabeur, Y .-T. Hu, R. Hu, C. Ryali, T. Ma, H. Khedr, R. R ¨adle, C. Rolland, L. Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714, 2024. 9

    2024 arXiv

  5. [13]

    S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, Q. Jiang, C. Li, J. Yang, H. Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection.arXiv preprint arXiv:2303.05499, 2023

    2023 arXiv

  6. [14]

    Zhang, K

    J. Zhang, K. Wang, S. Wang, M. Li, H. Liu, S. Wei, Z. Wang, Z. Zhang, and H. Wang. Uni- navid: A video-based vision-language-action model for unifying embodied navigation tasks. arXiv preprint arXiv:2412.06224, 2024

    2024 arXiv

  7. [15]

    W. Luo, P. Sun, F. Zhong, W. Liu, T. Zhang, and Y . Wang. End-to-end active object tracking via reinforcement learning. In International conference on machine learning , pages 3286–3295. PMLR, 2018

    2018

  8. [16]

    J. Zuo, J. Hong, F. Zhang, C. Yu, H. Zhou, C. Gao, N. Sang, and J. Wang. Plip: Language- image pre-training for person representation learning. Advances in Neural Information Pro- cessing Systems, 37:45666–45702, 2024

    2024

  9. [17]

    E. Song, W. Chai, G. Wang, Y . Zhang, H. Zhou, F. Wu, H. Chi, X. Guo, T. Ye, Y . Zhang, et al. Moviechat: From dense token to sparse memory for long video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18221–18232, 2024

    2024

  10. [18]

    T.-S. Chen, A. Siarohin, W. Menapace, E. Deyneka, H.-w. Chao, B. E. Jeon, Y . Fang, H.-Y . Lee, J. Ren, M.-H. Yang, et al. Panda-70m: Captioning 70m videos with multiple cross- modality teachers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognit...

    2024

  11. [19]

    Z. Yu, D. Xu, J. Yu, T. Yu, Z. Zhao, Y . Zhuang, and D. Tao. Activitynet-qa: A dataset for under- standing complex web videos via question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 9127–9134, 2019

    2019

  12. [20]

    W. Qiu, F. Zhong, Y . Zhang, S. Qiao, Z. Xiao, T. S. Kim, and Y . Wang. Unrealcv: Virtual worlds for computer vision. In Proceedings of the 25th ACM international conference on Multimedia, pages 1221–1224, 2017

    2017

  13. [21]

    H. Ye, J. Zhao, Y . Zhan, W. Chen, L. He, and H. Zhang. Person re-identification for robot person following with online continual learning.IEEE Robotics and Automation Letters, 2024

    2024

  14. [22]

    H. Ye, K. Cai, Y . Zhan, B. Xia, A. Ajoudani, and H. Zhang. Rpf-search: Field-based search for robot person following in unknown dynamic environments. arXiv preprint arXiv:2503.02188, 2025

    2025 arXiv

  15. [23]

    Francis, C

    A. Francis, C. P ´erez-d’Arpino, C. Li, F. Xia, A. Alahi, R. Alami, A. Bera, A. Biswas, J. Biswas, R. Chandra, et al. Principles and guidelines for evaluating social robot navigation algorithms. ACM Transactions on Human-Robot Interaction, 14(2):1–65, 2025

    2025

  16. [24]

    W. Luo, P. Sun, F. Zhong, W. Liu, T. Zhang, and Y . Wang. End-to-end active object tracking and its real-world deployment via reinforcement learning. IEEE transactions on pattern analysis and machine intelligence, 42(6):1317–1332, 2019

    2019

  17. [25]

    A. Devo, A. Dionigi, and G. Costante. Enhancing continuous control of mobile robots for end-to-end visual active tracking. Robotics and Autonomous Systems, 142:103799, 2021

    2021

  18. [26]

    K.-H. Zeng, Z. Zhang, K. Ehsani, R. Hendrix, J. Salvador, A. Herrasti, R. Girshick, A. Kem- bhavi, and L. Weihs. Poliformer: Scaling on-policy rl with transformers results in masterful navigators. arXiv preprint arXiv:2406.20083, 2024

    2024 arXiv

  19. [27]

    Zhong, P

    F. Zhong, P. Sun, W. Luo, T. Yan, and Y . Wang. Towards distraction-robust active visual tracking. In International Conference on Machine Learning , pages 12782–12792. PMLR, 2021. 10

    2021

  20. [28]

    Bajcsy, A

    A. Bajcsy, A. Loquercio, A. Kumar, and J. Malik. Learning vision-based pursuit-evasion robot policies. In 2024 IEEE International Conference on Robotics and Automation (ICRA) , pages 9197–9204. IEEE, 2024

    2024

  21. [29]

    Scofano, A

    L. Scofano, A. Sampieri, T. Campari, V . Sacco, I. Spinelli, L. Ballan, and F. Galasso. Following the human thread in social navigation. arXiv preprint arXiv:2404.11327, 2024

    2024 arXiv

  22. [30]

    D. Shah, A. Bhorkar, H. Leen, I. Kostrikov, N. Rhinehart, and S. Levine. Offline reinforcement learning for visual navigation. arXiv preprint arXiv:2212.08244, 2022

    2022 arXiv

  23. [31]

    Zhang, Z

    Y . Zhang, Z. Ma, J. Li, Y . Qiao, Z. Wang, J. Chai, Q. Wu, M. Bansal, and P. Kordjamshidi. Vision-and-language navigation today and tomorrow: A survey in the era of foundation mod- els. ArXiv, abs/2407.07035, 2024. URL https://api.semanticscholar.org/CorpusID: 271064503

    2024 arXiv

  24. [32]

    Y . Wu, P. Zhang, M. Gu, J. Zheng, and X. Bai. Embodied navigation with multi-modal infor- mation: A survey from tasks to methodology. Information Fusion, page 102532, 2024

    2024

  25. [33]

    Sridhar, D

    A. Sridhar, D. Shah, C. Glossop, and S. Levine. Nomad: Goal masked diffusion policies for navigation and exploration. In 2024 IEEE International Conference on Robotics and Automa- tion (ICRA), pages 63–70. IEEE, 2024

    2024

  26. [34]

    Y . Long, W. Cai, H. Wang, G. Zhan, and H. Dong. Instructnav: Zero-shot system for generic instruction navigation in unexplored environment. arXiv preprint arXiv:2406.04882, 2024

    2024 arXiv

  27. [35]

    D. Shah, A. Sridhar, A. Bhorkar, N. Hirose, and S. Levine. Gnm: A general navigation model to drive any robot. In2023 IEEE International Conference on Robotics and Automation (ICRA), pages 7226–7233. IEEE, 2023

    2023

  28. [36]

    Zhang, K

    J. Zhang, K. Wang, R. Xu, G. Zhou, Y . Hong, X. Fang, Q. Wu, Z. Zhang, and H. Wang. Navid: Video-based vlm plans the next step for vision-and-language navigation. Robotics: Science and Systems, 2024

    2024

  29. [37]

    G. Zhou, Y . Hong, Z. Wang, X. E. Wang, and Q. Wu. Navgpt-2: Unleashing navigational reasoning capability for large vision-language models. In European Conference on Computer Vision, pages 260–278. Springer, 2025

    2025

  30. [38]

    Kuang, H

    Y . Kuang, H. Lin, and M. Jiang. Openfmnav: Towards open-set zero-shot object navigation via vision-language foundation models. arXiv preprint arXiv:2402.10670, 2024

    2024 arXiv

  31. [39]

    D. S. Chaplot, D. P. Gandhi, A. Gupta, and R. R. Salakhutdinov. Object goal navigation using goal-oriented semantic exploration. Advances in Neural Information Processing Systems, 33: 4247–4258, 2020

    2020

  32. [40]

    Zhang, L

    J. Zhang, L. Dai, F. Meng, Q. Fan, X. Chen, K. Xu, and H. Wang. 3d-aware object goal navigation via simultaneous exploration and identification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6672–6682, 2023

    2023

  33. [41]

    Y . Cao, J. Zhang, Z. Yu, S. Liu, Z. Qin, Q. Zou, B. Du, and K. Xu. Cognav: Cognitive process modeling for object goal navigation with llms. arXiv preprint arXiv:2412.10439, 2024

    2024 arXiv

  34. [42]

    M. M. Islam, A. Gladstone, R. Islam, and T. Iqbal. Eqa-mx: Embodied question answering using multimodal expression. In The Twelfth International Conference on Learning Represen- tations, 2023

    2023

  35. [43]

    Majumdar, A

    A. Majumdar, A. Ajay, X. Zhang, P. Putta, S. Yenamandra, M. Henaff, S. Silwal, P. Mcvay, O. Maksymets, S. Arnaud, et al. Openeqa: Embodied question answering in the era of foun- dation models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition...

    2024

  36. [44]

    G. Team, R. Anil, S. Borgeaud, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  37. [45]

    X. Shen, Y . Xiong, C. Zhao, L. Wu, J. Chen, C. Zhu, Z. Liu, F. Xiao, B. Varadarajan, F. Bordes, et al. Longvu: Spatiotemporal adaptive compression for long video-language understanding. arXiv preprint arXiv:2410.17434, 2024

    2024 arXiv

  38. [46]

    Steiner, A

    A. Steiner, A. S. Pinto, M. Tschannen, D. Keysers, X. Wang, Y . Bitton, A. Gritsenko, M. Min- derer, A. Sherbondy, S. Long, et al. Paligemma 2: A family of versatile vlms for transfer.arXiv preprint arXiv:2412.03555, 2024

    2024 arXiv

  39. [47]

    Chiang, Z

    W.-L. Chiang, Z. Li, Z. Lin, Y . Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y . Zhuang, J. E. Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. lmsys. org (accessed 14 April 2023), 2023

    2023

  40. [48]

    Black, N

    K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. Ichter, et al. π0: A vision-language-action flow model for general robot control. arXiv preprint arXiv:2410.24164, 2024

    2024 arXiv

  41. [49]

    Intelligence, K

    P. Intelligence, K. Black, N. Brown, J. Darpinian, K. Dhabalia, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, et al. π0.5: a vision-language-action model with open-world generalization. arXiv preprint arXiv:2504.16054, 2025

    2025 arXiv

  42. [50]

    Q. Li, Y . Liang, Z. Wang, L. Luo, X. Chen, M. Liao, F. Wei, Y . Deng, S. Xu, Y . Zhang, et al. Cogact: A foundational vision-language-action model for synergizing cognition and action in robotic manipulation. arXiv preprint arXiv:2411.19650, 2024

    2024 arXiv

  43. [51]

    M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. Foster, G. Lam, P. Sanketi, et al. Openvla: An open-source vision-language-action model. arXiv preprint arXiv:2406.09246, 2024

    2024 arXiv

  44. [52]

    D. Qu, H. Song, Q. Chen, Y . Yao, X. Ye, Y . Ding, Z. Wang, J. Gu, B. Zhao, D. Wang, et al. Spatialvla: Exploring spatial representations for visual-language-action model. arXiv preprint arXiv:2501.15830, 2025

    2025 arXiv

  45. [53]

    Zhong, X

    Y . Zhong, X. Huang, R. Li, C. Zhang, Y . Liang, Y . Yang, and Y . Chen. Dexgraspvla: A vision-language-action framework towards general dexterous grasping. arXiv preprint arXiv:2502.20900, 2025

    2025

  46. [54]

    Y . Ding, H. Geng, C. Xu, X. Fang, J. Zhang, S. Wei, Q. Dai, Z. Zhang, and H. Wang. Open6dor: Benchmarking open-instruction 6-dof object rearrangement and a vlm-based approach. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 7359–7366...

    2024

  47. [55]

    Cheng, Y

    A.-C. Cheng, Y . Ji, Z. Yang, Z. Gongye, X. Zou, J. Kautz, E. Bıyık, H. Yin, S. Liu, and X. Wang. Navila: Legged robot vision-language-action model for navigation. arXiv preprint arXiv:2412.04453, 2024

    2024 arXiv

  48. [56]

    Y . Li, C. Wang, and J. Jia. Llama-vid: An image is worth 2 tokens in large language models. arXiv preprint arXiv:2311.17043, 2023

    2023 arXiv

  49. [57]

    Q. Sun, Y . Fang, L. Wu, X. Wang, and Y . Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303.15389, 2023

    2023 arXiv

  50. [58]

    H. Liu, C. Li, Q. Wu, and Y . J. Lee. Visual instruction tuning. In NeurIPS, 2023

    2023

  51. [59]

    Touvron, T

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi `ere, N. Goyal, E. Hambro, F. Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. 12

    2023 arXiv

  52. [60]

    B. Liao, S. Chen, H. Yin, B. Jiang, C. Wang, S. Yan, X. Zhang, X. Li, Y . Zhang, Q. Zhang, et al. Diffusiondrive: Truncated diffusion model for end-to-end autonomous driving. arXiv preprint arXiv:2411.15139, 2024

    2024 arXiv

  53. [61]

    C. Chi, Z. Xu, S. Feng, E. Cousineau, Y . Du, B. Burchfiel, R. Tedrake, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, page 02783649241273668, 2023

    2023

  54. [62]

    Arthur and S

    D. Arthur and S. Vassilvitskii. k-means++: The advantages of careful seeding. Technical report, Stanford, 2006

    2006

  55. [63]

    Peebles and S

    W. Peebles and S. Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4195–4205, 2023

    2023

  56. [64]

    J. Song, C. Meng, and S. Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020

    2010 arXiv

  57. [65]

    Y . Liu, J. Zhu, J. Tang, S. Zhang, J. Zhang, W. Cao, C. Wang, Y . Wu, and D. Huang. Tex- dreamer: Towards zero-shot high-fidelity 3d human texture generation. In European Confer- ence on Computer Vision, pages 184–202. Springer, 2024

    2024

  58. [66]

    S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025

    2025 arXiv

  59. [67]

    R. L. Knoblauch, M. T. Pietrucha, and M. Nitzburg. Field studies of pedestrian walking speed and start-up time. Transportation research record, 1538(1):27–38, 1996

    1996

  60. [68]

    Van Den Berg, S

    J. Van Den Berg, S. J. Guy, M. Lin, and D. Manocha. Reciprocal n-body collision avoidance. In Robotics Research: The 14th International Symposium ISRR, pages 3–19. Springer, 2011

    2011

  61. [69]

    S. K. Ramakrishnan, A. Gokaslan, E. Wijmans, O. Maksymets, A. Clegg, J. M. Turner, E. Un- dersander, W. Galuba, A. Westbury, A. X. Chang, et al. Habitat-matterport 3d dataset (hm3d): 1000 large-scale 3d environments for embodied ai. In Thirty-fifth Conference on Neural Infor- ...

    2021

  62. [70]

    Chang, A

    A. Chang, A. Dai, T. Funkhouser, M. Halber, M. Niebner, M. Savva, S. Song, A. Zeng, and Y . Zhang. Matterport3d: Learning from rgb-d data in indoor environments. In 2017 Interna- tional Conference on 3D Vision (3DV), pages 667–676. IEEE, 2017

    2017

  63. [71]

    Gupta, S

    M. Gupta, S. Kumar, L. Behera, and V . K. Subramanian. A novel vision-based tracking algo- rithm for a human-following mobile robot. IEEE Transactions on Systems, Man, and Cyber- netics: Systems, 47(7):1415–1427, 2016

    2016

  64. [72]

    G. Bhat, M. Danelljan, L. V . Gool, and R. Timofte. Learning discriminative model prediction for tracking. In Proceedings of the IEEE/CVF international conference on computer vision , pages 6182–6191, 2019

    2019

  65. [73]

    J. Yang, H. Zhang, F. Li, X. Zou, C. Li, and J. Gao. Set-of-mark prompting unleashes extraor- dinary visual grounding in gpt-4v. arXiv preprint arXiv:2310.11441, 2023

    2023 arXiv

  66. [74]

    Introducing 4o image generation

    OpenAI. Introducing 4o image generation. https://openai.com/index/ introducing-4o-image, 2024. Accessed: 2025-04-29

    2024

  67. [75]

    Jiang, L

    Q. Jiang, L. Wu, Z. Zeng, T. Ren, Y . Xiong, Y . Chen, Q. Liu, and L. Zhang. Referring to any person. arXiv preprint arXiv:2503.08507, 2025

    2025 arXiv

  68. [76]

    S. Yang, T. Qu, X. Lai, Z. Tian, B. Peng, S. Liu, and J. Jia. Lisa++: An improved baseline for reasoning segmentation with large language model. arXiv preprint arXiv:2312.17240, 2023. 13

    2023 arXiv

  69. [77]

    Misra, A

    I. Misra, A. Shrivastava, A. Gupta, and M. Hebert. Cross-stitch networks for multi-task learn- ing. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3994–4003, 2016

    2016

  70. [78]

    Zheng, S

    D. Zheng, S. Huang, L. Zhao, Y . Zhong, and L. Wang. Towards learning a generalist model for embodied navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13624–13634, 2024

    2024

  71. [79]

    C. Lu, M. Liu, Z. Luan, Y . He, and B. Chen. Multi-view spatial context and state constraints for object-goal navigation. IEEE Robotics and Automation Letters, 2025

    2025

  72. [80]

    Follow the first person you see

    P. Roth, J. Nubert, F. Yang, M. Mittal, and M. Hutter. Viplanner: Visual semantic impera- tive learning for local navigation. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 5243–5249. IEEE, 2024. A Training Details Similar to conventional vision...

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.