Pith. sign in

REVIEW 4 major objections 5 minor 53 references

SocialNav-SUB: Benchmarking VLMs for Scene Understanding in Social Robot Navigation

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read State-of-the-art vision-language models underperform a simple rule-based baseline and human consensus on scene understanding for social robot navigation, according to the new SocialNav-SUB benchmark.

desk verdict A useful new VQA benchmark for social robot navigation with reproducible code and data, but the headline VLM-vs-rule-based gap is not statistically supported once you cluster by scene; the human-oracle gap is the real finding. read the letter →

arxiv 2509.08757 v1 pith:S75IG4BE submitted 2025-09-10 cs.RO cs.CV

Michael J. Munje , Chen Tang , Shuijing Liu , Zichao Hu , Yifeng Zhu , Jiaxun Cui , Garrett Warnell , Joydeep Biswas
show 1 more author
Peter Stone
This is my paper · ORCID
classification cs.ROcs.CV
keywords socialrobotnavigationvision-languagemodelsvisualquestionansweringbenchmarkspatialreasoningspatiotemporalbird's-eyeviewhumangroundtruth
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces SocialNav-SUB, a visual question-answering benchmark that tests whether large vision-language models (VLMs) understand social robot navigation scenes. It asks spatial, spatiotemporal, and social-reasoning questions about 60 crowded real-world scenarios, with human answers serving as ground truth. The central empirical finding is that the best tested VLM (OpenAI o4-mini) agrees with humans 62% of the time, while a hand-crafted rule-based baseline reaches 64% and a human oracle reaches 74%. This matters because VLMs are being proposed as scene-understanding modules for socially compliant robot navigation, and the result suggests they are not yet reliable enough for that role. The benchmark also shows where the gap lives: spatial and spatiotemporal reasoning are the weakest areas, and social reasoning improves when spatial information is accurate.

What carries the argument

The carrying object is the benchmark itself: a VQA dataset plus evaluation protocol. Each item pairs a front-view RGB clip with a bird's-eye-view (BEV) image in which pedestrians are tracked by the PHALP algorithm (a monocular 3D human tracker), Kalman-smoothed, and projected from robot odometry into numbered color-coded circles; this object-centric representation is shown to both humans and VLMs. Two agreement metrics, probability of agreement (PA) and consensus-weighted probability of agreement (CWPA), score answers against the distribution of at least five human responses per question. The comparator that produces the headline result is a simple rule-based baseline that answers from the same pedestrian position data using hand-crafted cutoffs and line-intersection tests. Chain-of-thought prompting is the query mechanism that links questions sequentially, and ablations of BEV and CoT show their contributions.

What would settle it

Re-run the 60 scenarios with human answers and rule-based answers computed from a precise tracking system (for example, motion capture or multi-camera calibration) instead of the monocular tracking pipeline, and give VLMs the same corrected bird's-eye views; if their agreement rises above the rule-based baseline, the underperformance result is an artifact of the tracking errors rather than of VLM reasoning.

Watch

Extended reading notes

Core claim

SocialNav-SUB is the first VQA benchmark built specifically for social robot navigation scene understanding. It provides 4,968 human-labeled multiple-choice questions derived from 60 SCAND scenarios, each represented as a 2.5-second multi-view clip with numbered, color-coded pedestrians in front-view and bird's-eye-view images. Across the tested VLMs—Gemini 2.0 and 2.5, GPT-4o, OpenAI o4-mini, and LLaVa-Next-Video—the best model, o4-mini, achieves a probability of agreement (PA) of 0.62, below the rule-based baseline's 0.64 and the human oracle's 0.74, while average human PA is 0.60. The largest shortfalls are in spatial and spatiotemporal reasoning, while social reasoning comes closest to human-level agreement. The paper also reports that chain-of-thought prompting improves social reasoning and that supplying ground-truth spatial answers improves social-reasoning performance, indicating that poor spatial grounding is a bottleneck.

Load-bearing premise

The ground-truth human answers and the rule-based baseline are both built from estimated pedestrian positions in the bird's-eye view, produced by monocular tracking with an average displacement error of 0.67 meters, so if those positions are systematically wrong, the benchmark's comparison is distorted.

Editorial extensions

If this is right

  • A robot using current VLMs as its only scene-understanding module would agree with human judgments less often than a robot using the simple rule-based baseline, so near-term deployment should pair VLMs with dedicated perception and reasoning modules.
  • Because supplying correct spatial and spatiotemporal answers to the VLM raised its social-reasoning performance, spatial grounding is a bottleneck: improving spatial and dynamic perception should improve higher-level social inference.
  • Query format materially changes measured capability: chain-of-thought reliably helps social reasoning, while BEV overlays help some models and not others, so reported VLM performance is tied to the prompt interface.
  • The waypoint-selection experiment links scene understanding to action: when social scene context came from the human oracle rather than from random or model-generated context, all evaluated VLMs selected the human operator's waypoint more often.
  • Failure rates are uneven: models err more often at blind corners and indoors and on 'yielding to' and 'overtaking' labels, identifying targeted environments and action categories for future data collection or fine-tuning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the paper's monocular tracking has systematic rather than random error, the human labels and rule-based baseline may be biased in the same direction; re-scoring with ground-truth tracks could narrow or eliminate the VLM gap. This is a testable reconstruction, not a claim the paper makes.
  • Because human annotators often disagree on the same scene, exact-match agreement may understate a VLM's competence on genuinely subjective questions; a distributional scoring scheme could tell whether remaining errors are wrong or merely one plausible reading.
  • The same BEV-plus-numbered-pedestrian VQA template could be ported to other dynamic robot domains, such as driving or drone navigation, to test whether the observed underperformance is specific to social scenes or a general deficit in spatiotemporal reasoning.
  • Making the human labels publicly available turns the benchmark into training material, not just an evaluation: fine-tuning or reinforcement learning from these answers could plausibly close part of the gap that zero-shot prompting has not closed.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces SocialNav-SUB, a VQA benchmark for evaluating vision-language models (VLMs) on scene understanding for social robot navigation. Built from 60 curated SCAND scenarios, it uses PHALP-based 3D tracking and Kalman smoothing to produce annotated front-view images and bird's-eye views, together with 4,968 multiple-choice questions probing spatial, spatiotemporal, and social reasoning. Human responses from a Prolific study serve as ground truth, with PA and CWPA metrics and average-human and human-oracle baselines. The authors evaluate several closed- and open-source VLMs and report that the best VLM, OpenAI o4-mini, reaches PA 0.62, below a rule-based baseline (0.64) and the human oracle (0.74). They also perform ablations on chain-of-thought and BEV prompts, a waypoint-selection experiment, and a failure-case analysis. The manuscript claims this is the first VQA benchmark for social robot navigation evaluating these reasoning dimensions against human baselines.

Significance. If the empirical claims hold, SocialNav-SUB would be a useful community resource: it provides a human-labeled VQA benchmark in a domain that currently lacks systematic evaluation, with multiple annotators per scenario, two agreement metrics, object-centric/BEV visual prompts, and a public-release plan. The ablation studies and the waypoint-selection validation are constructive steps toward understanding how VLMs can be integrated into social navigation stacks. The benchmark also generates falsifiable, field-relevant questions about VLM spatial and social reasoning. However, the headline comparison against the rule-based baseline is currently not statistically supported, and the rule-based baseline uses privileged tracker state rather than the visual inputs given to VLMs and humans, so the central claim needs re-analysis and reframing before the benchmark's conclusions can be accepted.

major comments (4)
  1. [Section 4.2, Table 1] The headline comparison between o4-mini (PA 0.62 ± 0.01) and the rule-based baseline (PA 0.64 ± 0.00) treats the 4,968 questions as independent, but these questions are nested in only 60 SCAND scenarios, with about 83 questions per scenario (Appendix 7.7). Questions from the same scenario share crowd layout, occlusion patterns, environment, and question-order effects, so the effective sample size is far smaller than 4,968. No cluster-robust standard errors, scenario-level bootstrap, or paired permutation test is reported. With an intra-scenario correlation of even 0.05, the standard error on the difference would more than double, and the 0.02 gap would no longer be significant. The table caption itself concedes that the bolded VLM result 'may be statistically tied.' Because the paper's central claim is that all VLMs underperform the rule-based baseline, this comparison must be re-analyzed with scene-level clustering; the gap against the human oracle (0.74 vs 0.62) is larger and may survive, but the rule-based comparison is not established.
  2. [Section 4.1 and Appendix 7.10] The rule-based baseline is constructed from 'the position data of pedestrians in the scene' and uses hand-crafted rules such as line-intersection checks and distance cutoffs, whereas VLMs and human annotators are given only images (front-view and BEV). This is not a like-for-like test of visual scene understanding: the rule-based baseline effectively receives privileged geometric state from the PHALP tracker. The conclusion that a 'simpler rule-based approach' outperforms VLMs is therefore misleading as stated. The comparison would be more informative if the rule-based baseline were required to operate on the same visual inputs, or if the manuscript clearly framed it as a perception-plus-rules pipeline that upper-bounds what can be done with accurate positions.
  3. [Section 3.2 and Appendix 7.3] The benchmark's spatial questions ask for categorical relations such as ahead, left, right, and behind, but the PHALP-based tracking pipeline reports an average displacement error of 0.67 ± 0.14 m. In crowded scenes, distances between the robot and pedestrians can be on the order of 1–2 m, so this error is large relative to the distinctions being tested. Because the same BEV positions are used to construct the VLM prompts, the human labels, and the rule-based baseline, tracking error is a shared source of label noise. The paper should quantify sensitivity to this error, for example by reporting results on a subset of scenes with high-confidence tracks below a displacement threshold, or by perturbing the BEV positions and measuring the resulting change in PA.
  4. [Section 4.1 and Section 4.2] The query protocol is described inconsistently. Section 4.1 says chain-of-thought 'provides the previous answers of the VLM for future questions,' while Section 4.2 says 'we run our experiments by querying each VLM model once per unique question.' These statements conflict: either each question is answered independently, or prior answers are fed back into the prompt. The ablation rows labeled 'No CoT' also do not clarify whether the question order and inter-question context remain the same. Because the manuscript emphasizes a fair comparison with the sequential human-subject protocol, the exact prompt construction, including whether previous model answers are included and in what order, must be specified precisely for reproducibility.
minor comments (5)
  1. [Section 3.3] The text refers to 'Table 7.7' for question details, but the relevant table is Table 6 and the appendix is numbered 7.7; the cross-reference should be corrected.
  2. [Table 1 and Section 4.2] The Table 1 caption says the bolded VLM result 'may be statistically tied,' but Section 4.2 states that o4-mini 'still has a considerable gap' compared to the rule-based baseline; these statements should be made consistent with the statistical evidence after the clustering analysis is added.
  3. [Appendix 7.10] The rule-based baseline is described qualitatively ('cutoff values,' 'draw a line,' 'if the lines intersect'); for reproducibility, the exact thresholds, coordinate conventions, and decision rules should be provided, ideally as pseudocode or with the released implementation.
  4. [Section 3.4, Equation (1)] The statement that PA 'is essentially the expected cosine similarity between the model's predictions and the distribution of human responses' is not literally correct unless the answer vectors are normalized; please either derive this equivalence or rephrase it as an average one-hot agreement.
  5. [Appendix 7.5] The description of the prompt says the VLM receives 'the next 9 images' in addition to the image shown on the left, but the pipeline is described as a 2.5 s segment sampled at 4 Hz, which would produce 10 frames; please clarify the frame count and ordering.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark is an external evaluation against human labels and an independent rule-based baseline.

full rationale

The paper's derivation chain is an empirical benchmark construction, not a fitted model. Ground truth is collected from 153 Prolific participants (Appendix 7.4), and the rule-based baseline is a hand-crafted comparator (Appendix 7.10) using PHALP positions; VLM answers are externally generated. The only fits are Kalman smoothing hyperparameters tuned on the external CODa dataset (Appendix 7.3), and they are not used to define the reported PA/CWPA scores or the human labels. The waypoint experiment uses oracle answers as input context, but it is explicitly labeled preliminary and tests a conditional effect, not a predicted target derived from those answers. No central claim reduces by construction to its inputs, and the SCAND dataset citation, despite author overlap, is an external released dataset rather than an unverified self-citation. Therefore no circular step is present.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The benchmark rests on tracking accuracy, scene selection choices, and the assumption that human labels define correctness. No new physical entities or fitted parameters are introduced to force the central result.

free parameters (3)
  • Scene selection feature weights = Not reported (weights over crowd size, close proximity, lateral movement)
    The selection of benchmark scenarios is driven by an unreported weighted sum; this affects the difficulty distribution of the benchmark.
  • Kalman smoothing hyperparameters = Tuned on CODa, not numerically specified
    Hyperparameters tuned on a different dataset (CODa) and applied to SCAND; a legitimate transfer but a free choice.
  • Rule-based baseline cutoffs = Not specified; hand-chosen thresholds
    The hand-crafted rules and cutoffs are a free design choice; they affect the strength of the rule-based baseline.
assumptions (4)
  • domain assumption Human responses are ground truth for social scene understanding.
    The benchmark treats human-annotated answers as correct labels; this is standard for VQA but reflects a specific social norm interpretation.
  • domain assumption PHALP 3D pose estimates are accurate enough to construct valid BEVs.
    Appendix 7.3 reports 0.67 m average displacement error on CODa; this error propagates into the BEV inputs.
  • domain assumption The 60 selected scenes represent challenging social navigation.
    Scene selection uses a weighted score with unreported weights; no external validation that the selected scenes are indeed hardest.
  • standard math Standard statistics assume question independence.
    Questions are grouped by scene, so independence may not hold; the paper does not cluster standard errors.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SocialNav-SUB: Benchmarking VLMs for Scene Understanding in Social Robot Navigation." pith.science (2026). https://pith.science/paper/S75IG4BE

@misc{pith2026250908757,
  author       = {Pith},
  title        = {Pith review of: SocialNav-SUB: Benchmarking VLMs for Scene Understanding in Social Robot Navigation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/S75IG4BE}},
  note         = {Machine review of arXiv:2509.08757}
}
read the original abstract

Robot navigation in dynamic, human-centered environments requires socially-compliant decisions grounded in robust scene understanding. Recent Vision-Language Models (VLMs) exhibit promising capabilities such as object recognition, common-sense reasoning, and contextual understanding-capabilities that align with the nuanced requirements of social robot navigation. However, it remains unclear whether VLMs can accurately understand complex social navigation scenes (e.g., inferring the spatial-temporal relations among agents and human intentions), which is essential for safe and socially compliant robot navigation. While some recent works have explored the use of VLMs in social robot navigation, no existing work systematically evaluates their ability to meet these necessary conditions. In this paper, we introduce the Social Navigation Scene Understanding Benchmark (SocialNav-SUB), a Visual Question Answering (VQA) dataset and benchmark designed to evaluate VLMs for scene understanding in real-world social robot navigation scenarios. SocialNav-SUB provides a unified framework for evaluating VLMs against human and rule-based baselines across VQA tasks requiring spatial, spatiotemporal, and social reasoning in social robot navigation. Through experiments with state-of-the-art VLMs, we find that while the best-performing VLM achieves an encouraging probability of agreeing with human answers, it still underperforms simpler rule-based approach and human consensus baselines, indicating critical gaps in social scene understanding of current VLMs. Our benchmark sets the stage for further research on foundation models for social robot navigation, offering a framework to explore how VLMs can be tailored to meet real-world social robot navigation needs. An overview of this paper along with the code and data can be found at https://larg.github.io/socialnav-sub .

Figures

Figures reproduced from arXiv: 2509.08757 by the authors.

Figure 1
Figure 1. Examples of social robot navigation scenarios from SCAND [4]. The ability to de￾termine socially compliant navigation actions requires understanding each dynamic scene by spa￾tiotemporal reasoning (e.g. the movements of people in the scene) and social reasoning (inferring the navigation intentions of people in the scene). of-thought reasoning [5, 6, 7]. Trained in diverse large-scale multimodal datasets that span va… view at source ↗
Figure 2
Figure 2. An overview of SOCIALNAV-SUB, which facilitates the systematic evaluation of VLMs in social robot navigation scenarios. Using SCAND data, human-labeled VQA datasets, and var￾ious VLMs, this framework offers the evaluation of VLMs across multiple dimensions of scene understanding for social robot navigation that can enable advancements in prompt designs, social reasoning, and social robot navigation research in gener… view at source ↗
Figure 3
Figure 3. The data processing pipeline for VQA prompts in SOCIALNAV-SUB. We first mine social robot navigation scenarios from SCAND [4], then use the PHALP algorithm [46] to provide human tracking and estimations of 3D locations, which are used to construct BEV representations of the scene and annotated images. Along with the annotated images and BEV representations, a set of carefully designed questions (more details in Appe… view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: An example of the waypoint selection VQA task. This particular example highlights using scene context from the human oracle. Having no context removes the middle portion of the text prompt that includes the context, and having random context randomizes each relational …
Figure 5
Figure 5. Figure 5: shows four representative examples [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]
Figure 6
Figure 6. Figure 6: CODa example for 3D pose pipeline validation. Left: input image with PHALP bound￾ing box detections. Right: BEV positions and headings after Kalman smoothing. Estimates are generally close ( < 1 m displacement error) to pseudo–ground truth for well-observed pedestrians…
Figure 7
Figure 7. Figure 7: An example of a survey page shown to human participants. Prior to answering the survey questions, human subjects were given human-subject study participation instructions, re￾quirements, and instructions about the survey content. 7.5 VQA Prompt Details To provide fair …
Figure 8
Figure 8. Figure 8: An example of a full VQA prompt shown to VLMs. This context closely resembles the instructions that were provided to human participants for the human-subject study. In addition to the image shown on the left, the VLM also receives the next 9 images in the sequence. 7.6…
Figure 9
Figure 9. Figure 9: Examples of failure cases for VLMs. Top-left: Failing to recognize that person 5 is on the left. Top-right: Failing to recognize that person 4 ends up further away. Bottom-left: Answering that the distant person 3 should be avoided. Bottom-right: Incorrectly answering …
Figure 10
Figure 10. Figure 10: Examples of success cases for VLMs. Top-left: All VLMs correctly infer that person 1 is not obstructing the path to the goal. Top-right: Gemini correctly predicts that person 1 should be avoided (the other VLMs incorrectly predict this). Bottom-left: GPT-4o correctly …

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

53 extracted references · 24 canonical work pages

  1. [1]

    Mavrogiannis, F

    C. Mavrogiannis, F. Baldini, A. Wang, D. Zhao, P. Trautman, A. Steinfeld, and J. Oh. Core challenges of social robot navigation: A survey.J. Hum.-Robot Interact., 12(3), Apr. 2023. doi:10.1145/3583741. URLhttps://doi.org/10.1145/3583741

    doi:10.1145/3583741 2023

  2. [2]

    Francis, C

    A. Francis, C. P ´erez-D’Arpino, C. Li, F. Xia, A. Alahi, R. Alami, A. Bera, A. Biswas, J. Biswas, R. Chandra, H.-T. L. Chiang, M. Everett, S. Ha, J. Hart, J. P. How, H. Karnan, T.-W. E. Lee, L. J. Manso, R. Mirksy, S. Pirk, P. T. Singamaneni, P. Stone, A. V . Taylor, P. Trautman, N. Tsoi, M. V´azquez, X. Xiao, P. Xu, N. Yokoyama, A. Toshev, and R. Mart´ı...

    arXiv 2023

  3. [3]

    A. H. Raj, Z. Hu, H. Karnan, R. Chandra, A. Payandeh, L. Mao, P. Stone, J. Biswas, and X. Xiao. Rethinking social robot navigation: Leveraging the best of two worlds. In2024 IEEE International Conference on Robotics and Automation (ICRA), pages 16330–16337, 2024. doi: 10.1109/ICRA57147.2024.10611710. 9

    arXiv 2024

  4. [4]

    Karnan, A

    H. Karnan, A. Nair, X. Xiao, G. Warnell, S. Pirk, A. Toshev, J. Hart, J. Biswas, and P. Stone. Socially compliant navigation dataset (scand): A large-scale dataset of demonstrations for social navigation, 2022. URLhttps://arxiv.org/abs/2203.15041

    arXiv 2022

  5. [5]

    H. Liu, C. Li, Q. Wu, and Y . J. Lee. Visual instruction tuning, 2023. URLhttps://arxiv. org/abs/2304.08485

    arXiv 2023

  6. [6]

    Hurst, A

    OpenAI, :, A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, and A. C. et. al. Gpt-4o system card, 2024. URLhttps://arxiv.org/abs/2410.21276

    arXiv 2024

  7. [7]

    Team and P

    G. Team and P. G. et. al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024. URLhttps://arxiv.org/abs/2403.05530

    arXiv 2024

  8. [8]

    Z. Hu, Y . Ren, J. Li, and Y . Yin. Viva: A benchmark for vision-grounded decision-making with human values, 2024. URLhttps://arxiv.org/abs/2407.03000

    work page Pith review arXiv 2024

Show all 53 references
  1. [9]

    D. Song, J. Liang, A. Payandeh, A. H. Raj, X. Xiao, and D. Manocha. Vlm-social-nav: Socially aware robot navigation through scoring using vision-language models, 2024. URLhttps: //arxiv.org/abs/2404.00210

    2024 arXiv

  2. [10]

    S. K. Ramakrishnan, E. Wijmans, P. Kraehenbuehl, and V . Koltun. Does spatial cognition emerge in frontier models?, 2024. URLhttps://arxiv.org/abs/2410.06468

    2024 arXiv

  3. [11]

    Kessler, J

    F. Kessler, J. Frankenstein, and C. A. Rothkopf. Human navigation strategies and their errors result from dynamic interactions of spatial uncertainties.Nature Communications, 15(1):5677, 2024

    2024

  4. [12]

    A. D. Ekstrom and P. F. Hill. Spatial navigation and memory: A review of the similar- ities and differences relevant to brain models and age.Neuron, 111(7):1037–1049, 2023. ISSN 0896-6273. doi:https://doi.org/10.1016/j.neuron.2023.03.001. URLhttps://www. sciencedirect.com/scie...

    2023 doi

  5. [13]

    Moussa ¨ıd, N

    M. Moussa ¨ıd, N. Perozo, S. Garnier, D. Helbing, and G. Theraulaz. The walking behaviour of pedestrian social groups and its impact on crowd dynamics.PloS one, 5(4):e10047, 2010

    2010

  6. [14]

    Moussa ¨ıd, D

    M. Moussa ¨ıd, D. Helbing, S. Garnier, A. Johansson, M. Combe, and G. Theraulaz. Experi- mental study of the behavioural mechanisms underlying self-organization in human crowds. Proceedings of the Royal Society B: Biological Sciences, 276(1668):2755–2762, 2009

    2009

  7. [15]

    Karnan, A

    H. Karnan, A. Nair, X. Xiao, G. Warnell, S. Pirk, A. Toshev, J. Hart, J. Biswas, and P. Stone. Socially Compliant Navigation Dataset (SCAND), 2022. URLhttps://doi.org/10. 18738/T8/0PRYRH

    2022

  8. [16]

    Openai o3 and o4-mini system card, 2025

    OpenAI. Openai o3 and o4-mini system card, 2025. URLhttps://openai.com/ index/o3-o4-mini-system-card/

    2025

  9. [17]

    Zhang, B

    Y . Zhang, B. Li, h. Liu, Y . j. Lee, L. Gui, D. Fu, J. Feng, Z. Liu, and C. Li. Llava-next: A strong zero-shot video understanding model, April 2024. URLhttps://llava-vl.github. io/blog/2024-04-30-llava-next-video/

    2024

  10. [18]

    Nasiriany, F

    S. Nasiriany, F. Xia, W. Yu, T. Xiao, J. Liang, I. Dasgupta, A. Xie, D. Driess, A. Wahid, Z. Xu, Q. Vuong, T. Zhang, T.-W. E. Lee, K.-H. Lee, P. Xu, S. Kirmani, Y . Zhu, A. Zeng, K. Hausman, N. Heess, C. Finn, S. Levine, and B. Ichter. Pivot: Iterative visual prompting elicits...

    2024 arXiv

  11. [19]

    Z. Yang, C. Garrett, D. Fox, T. Lozano-P ´erez, and L. P. Kaelbling. Guiding long-horizon task and motion planning with vision language models.arXiv preprint arXiv:2410.02193, 2024. 10

    2024 arXiv

  12. [20]

    Chang, S

    P. Chang, S. Liu, and K. Driggs-Campbell. Learning visual-audio representations for voice- controlled robots. InIEEE International Conference on Robotics and Automation (ICRA), 2023

    2023

  13. [21]

    Chang, S

    P. Chang, S. Liu, T. Ji, N. Chakraborty, K. Hong, and K. R. Driggs-Campbell. A data-efficient visual-audio representation with intuitive fine-tuning for voice-controlled robots. InConfer- ence on Robot Learning (CoRL), 2023

    2023

  14. [22]

    Z. Dong, W. Zhang, X. Huang, H. Ji, X. Zhan, and J. Chen. Hubo-vlm: Unified vision-language model designed for human robot interaction tasks.arXiv preprint arXiv:2308.12537, 2023

    2023 arXiv

  15. [23]

    A. J. Sathyamoorthy, K. Weerakoon, M. Elnoor, A. Zore, B. Ichter, F. Xia, J. Tan, W. Yu, and D. Manocha. Convoi: Context-aware navigation using vision language models in outdoor and indoor environments, 2024. URLhttps://arxiv.org/abs/2403.15637

    2024 arXiv

  16. [24]

    H.-T. L. Chiang, Z. Xu, Z. Fu, M. G. Jacob, T. Zhang, T.-W. E. Lee, W. Yu, C. Schenck, D. Rendleman, D. Shah, F. Xia, J. Hsu, J. Hoech, P. Florence, S. Kirmani, S. Singh, V . Sind- hwani, C. Parada, C. Finn, P. Xu, S. Levine, and J. Tan. Mobility vla: Multimodal in- struction ...

    2024 arXiv

  17. [25]

    Weerakoon, M

    K. Weerakoon, M. Elnoor, G. Seneviratne, V . Rajagopal, S. H. Arul, J. Liang, M. K. M. Jaffar, and D. Manocha. Behav: Behavioral rule guided autonomy using vlms for robot navigation in outdoor scenes, 2024. URLhttps://arxiv.org/abs/2409.16484

    2024 arXiv

  18. [26]

    Hirose, C

    N. Hirose, C. Glossop, A. Sridhar, D. Shah, O. Mees, and S. Levine. Lelan: Learning a language-conditioned navigation policy from in-the-wild videos, 2024. URLhttps: //arxiv.org/abs/2410.03603

    2024 arXiv

  19. [27]

    B. Chen, Z. Xu, S. Kirmani, B. Ichter, D. Driess, P. Florence, D. Sadigh, L. Guibas, and F. Xia. Spatialvlm: Endowing vision-language models with spatial reasoning capabilities, 2024. URL https://arxiv.org/abs/2401.12168

    2024 arXiv

  20. [28]

    Y . Tang, A. Qu, Z. Wang, D. Zhuang, Z. Wu, W. Ma, S. Wang, Y . Zheng, Z. Zhao, and J. Zhao. Sparkle: Mastering basic spatial capabilities in vision language models elicits generalization to composite spatial reasoning, 2024. URLhttps://arxiv.org/abs/2410.16162

    2024

  21. [29]

    Helbing and P

    D. Helbing and P. Molnar. Social force model for pedestrian dynamics.Physical review E, 51 (5):4282, 1995

    1995

  22. [30]

    Mumm and B

    J. Mumm and B. Mutlu. Human-robot proxemics: Physical and psychological distancing in human-robot interaction. In2011 6th ACM/IEEE International Conference on Human-Robot Interaction (HRI), pages 331–338, 2011. doi:10.1145/1957656.1957786

    2011

  23. [31]

    Hirose, D

    N. Hirose, D. Shah, A. Sridhar, and S. Levine. Sacson: Scalable autonomous control for social navigation.IEEE Robotics and Automation Letters, 2023

    2023

  24. [32]

    Zhu and T

    K. Zhu and T. Zhang. Deep reinforcement learning based mobile robot navigation: A review. Tsinghua Science and Technology, 26(5):674–691, 2021

    2021

  25. [33]

    Y . F. Chen, M. Everett, M. Liu, and J. P. How. Socially aware motion planning with deep rein- forcement learning. InIEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1343–1350, 2017

    2017

  26. [34]

    C. Chen, Y . Liu, S. Kreiss, and A. Alahi. Crowd-robot interaction: Crowd-aware robot naviga- tion with attention-based deep reinforcement learning. InIEEE International Conference on Robotics and Automation (ICRA), pages 6015–6022, 2019. 11

    2019

  27. [35]

    S. Liu, P. Chang, Z. Huang, N. Chakraborty, K. Hong, W. Liang, D. L. McPherson, J. Geng, and K. Driggs-Campbell. Intention aware robot crowd navigation with attention-based interaction graph. InIEEE International Conference on Robotics and Automation (ICRA), pages 12015– 12021, 2023

    2023

  28. [36]

    S. Liu, H. Xia, F. C. Pouria, K. Hong, N. Chakraborty, Z. Hu, J. Biswas, and K. Driggs- Campbell. Height: Heterogeneous interaction graph transformer for robot navigation in crowded and constrained environments.arXiv preprint arXiv:2411.12150, 2025. URL https://arxiv.org/abs/2...

    2025

  29. [37]

    D. M. Nguyen, M. Nazeri, A. Payandeh, A. Datar, and X. Xiao. Toward human-like so- cial robot navigation: A large-scale, multi-modal, social human navigation dataset. In2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 7442–

  30. [38]

    Hirose, D

    N. Hirose, D. Shah, A. Sridhar, and S. Levine. Sacson: Scalable autonomous control for social navigation.IEEE Robotics and Automation Letters, 9(1):49–56, 2024. doi:10.1109/LRA.2023. 3329626

    2024 doi

  31. [39]

    Payandeh, D

    A. Payandeh, D. Song, M. Nazeri, J. Liang, P. Mukherjee, A. H. Raj, Y . Kong, D. Manocha, and X. Xiao. Social-llava: Enhancing robot navigation through human-language reasoning in social spaces.arXiv preprint arXiv:2501.09024, 2024

    2024 arXiv

  32. [40]

    Anwar, J

    A. Anwar, J. Welsh, J. Biswas, S. Pouya, and Y . Chang. Remembr: Building and reasoning over long-horizon spatio-temporal memory for robot navigation.arXiv preprint arXiv:2409.13682, 2024

    2024 arXiv

  33. [41]

    Zhang, C

    A. Zhang, C. Eranki, C. Zhang, J.-H. Park, R. Hong, P. Kalyani, L. Kalyanaraman, A. Gamare, A. Bagad, M. Esteva, and J. Biswas. Towards robust robot 3d perception in urban environ- ments: The ut campus object dataset, 2023. URLhttps://arxiv.org/abs/2309. 13549

    2023

  34. [42]

    W. Wang, C. Duan, Z. Peng, Y . Liu, and B. Zhou. Embodied scene understanding for vision language models via metavqa.arXiv preprint arXiv:2501.09167, 2025

    2025 arXiv

  35. [43]

    Sreeram, T.-H

    S. Sreeram, T.-H. Wang, A. Maalouf, G. Rosman, S. Karaman, and D. Rus. Probing multi- modal llms as world models for driving.arXiv preprint arXiv:2405.05956, 2024

    2024 arXiv

  36. [44]

    Chow*, J

    W. Chow*, J. Mao*, B. Li, D. Seita, V . C. Guizilini, and Y . Wang. Physbench: Benchmarking and enhancing vision-language models for physical world understanding. InInternational Conference on Learning Representations, 2025

    2025

  37. [45]

    J. Yang, H. Zhang, F. Li, X. Zou, C. Li, and J. Gao. Set-of-mark prompting unleashes extraor- dinary visual grounding in gpt-4v.arXiv preprint arXiv:2310.11441, 2023

    2023 arXiv

  38. [46]

    Rajasegaran, G

    J. Rajasegaran, G. Pavlakos, A. Kanazawa, and J. Malik. Tracking people by predicting 3d appearance, location and pose. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2740–2749, 2022

    2022

  39. [47]

    Zhang, H

    A. Zhang, H. Sikchi, A. Zhang, and J. Biswas. Creste: Scalable mapless navigation with internet scale priors and counterfactual guidance.arXiv preprint arXiv:2503.03921, 2025

    2025 arXiv

  40. [48]

    Tadic, A

    V . Tadic, A. Toth, Z. Vizvari, M. Klincsik, Z. Sari, P. Sarcevic, J. Sarosi, and I. Biro. Perspec- tives of realsense and zed depth sensors for robotic vision applications.Machines, 10(3):183, 2022

    2022

  41. [49]

    Aharony, A

    N. Aharony, A. Meshurer, M. Krakovski, Y . Parmet, I. Melzer, and Y . Edan. Comparative analysis of cameras and software tools for skeleton tracking.IEEE Sensors Journal, 2024. 12

    2024

  42. [50]

    Prolific.https://www.prolific.com, 2014.Accessed on [date accessed]

    Prolific. Prolific.https://www.prolific.com, 2014.Accessed on [date accessed]

    2014

  43. [51]

    Antol, A

    S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. L. Zitnick, and D. Parikh. Vqa: Visual question answering. InProceedings of the IEEE international conference on computer vision, pages 2425–2433, 2015

    2015

  44. [52]

    D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  45. [53]

    avoiding

    A.-C. Cheng, H. Yin, Y . Fu, Q. Guo, R. Yang, J. Kautz, X. Wang, and S. Liu. Spatialrgpt: Grounded spatial reasoning in vision language models, 2024. URLhttps://arxiv.org/ abs/2406.01584. 13 7 Appendix 7.1 Waypoint Selection Experiments To further demonstrate the practical val...

    2024 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.