Pith. sign in

REVIEW 5 major objections 5 minor 46 references

VLMs cannot yet think ahead, but a fine-tuned 8B model on the new FSU-QA dataset can outperform GPT-5 on driving foresight tasks.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 20:38 UTC pith:3AROFCFS

load-bearing objection A useful new VQA benchmark for foresight in driving, with a strong fine-tuning result, but the headline world-model evaluation protocol is missing the control the abstract promises. the 5 major comments →

arxiv 2511.18735 v3 pith:3AROFCFS submitted 2025-11-24 cs.CV cs.AI

Thinking Ahead: Foresight Intelligence in MLLMs and World Model

Zhantao Gong , Liaoyuan Fan , Qing Guo , Xun Xu , Xulei Yang , Shijie Li This is my paper
classification cs.CV cs.AI
keywords Foresight IntelligenceVQA datasetautonomous drivingvision-language modelsworld modelssemantic coherencecounterfactual reasoningnuScenes
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper proposes FSU-QA, a dataset of 21,000+ visual question-answer pairs built from 850 real driving videos, designed to test whether vision-language models can anticipate future events — speed changes, turns, pedestrian intent, risks, and counterfactual outcomes. On the held-out FSU-Bench, state-of-the-art models average around 41–49% accuracy, with GPT-5 best at 48.66%, showing that foresight reasoning is largely unsolved. The paper then shows that fine-tuning an 8-billion-parameter open model (Qwen3-VL-8B-FI) on FSU-QA lifts it to 59.59%, beating all closed-source models. A second use is evaluating world models: feeding VLM-generated future video and trajectories into VLMs and measuring accuracy gains, with DrivingWorld producing larger gains than Epona. A sympathetic reader should care because the dataset offers a reusable testbed for a capability — anticipating the future rather than describing the present — that is missing from current benchmarks.

Core claim

The central finding is that current VLMs still struggle to reason about future situations — the strongest (GPT-5) scores 48.66% on FSU-Bench — and that a small model fine-tuned on the proposed FSU-QA dataset (Qwen3-VL-8B-FI) reaches 59.59%, surpassing much larger closed-source models. The companion claim is that the semantic coherence of world-model-generated futures can be measured by the accuracy gain a VLM shows when given those futures as additional inputs; by this measure, DrivingWorld's predictions are more informative than Epona's. The paper also introduces the term 'Foresight Intelligence' and organizes it into three pillars: situational modeling, causal/dynamic simulation, and goal-

What carries the argument

FSU-QA/FSU-Bench: a template-generated VQA dataset from nuScenes with 9 task types across 3 cognitive levels (low: speed/turn/lane change, relative distance/position; mid: pedestrian intent, ego-VRU relative position, risk area; high: counterfactual prediction). The evaluation protocol treats a VLM as a 'future judge': answers are scored by exact match to rule-based ground truth, and WM quality is operationalized as the accuracy delta when predicted future frames/trajectories are appended to the historical inputs.

Load-bearing premise

The load-bearing premise is that a VLM's accuracy gain when given a world model's predicted future frames and trajectories measures the semantic coherence of those predictions; the abstract says shuffled-control experiments validate this, but no such controls appear in the main text or the appendix, so the gains could in principle come from low-level artifacts or prompt-length effects rather than semantic alignment.

What would settle it

Take a world-model-generated future, shuffle the order of its frames (or randomly permute trajectory waypoints), feed the scrambled version to the same VLM, and compare accuracy gains to the intact version; the semantic-coherence interpretation predicts a large drop, while artifact- or order-insensitive gains would keep accuracy high. A second check: feed unrelated or random future frames as the 'predicted' input; if gains persist, they are not measuring semantic content.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Foresight reasoning is a measurable, currently open gap: even the best closed VLMs score below 50% on FSU-Bench, leaving clear headroom for dedicated methods.
  • Fine-tuning on FSU-QA transfers: a small open model outperforms all baselines, suggesting the dataset provides effective supervision for foresight tasks.
  • Predicted video and predicted trajectories help in complementary ways — video aids motion-based tasks, trajectories aid relational reasoning.
  • DrivingWorld's generated futures induce larger VLM gains than Epona's, which the paper interprets as higher semantic coherence.
  • World-model outputs can be assessed without human annotation, using VLM accuracy deltas as a proxy for semantic coherence.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A natural extension would be to test whether the accuracy-gain metric survives shuffled controls (e.g., reversed, frame-scrambled, or trajectory-shuffled futures); the abstract says such controls validate the protocol, but the main text and appendix do not report them, so this validation is currently unverifiable from the paper.
  • FSU-QA's rule-based answer generation could be adapted to other driving datasets (e.g., different cities) to test geographic generalization; the paper itself acknowledges the nuScenes geography (Boston, Singapore) as a limitation.
  • Because gains are computed per task, the metric could be turned into a diagnostic: if a world model helps only low-level tasks and not high-level counterfactual ones, that points to where the model lacks causal consistency.
  • A testable extension: replacing world-model outputs with ground-truth future frames and trajectories should yield the upper-bound gain; comparing world-model gains to that bound would quantify how much semantic content each world model preserves.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces FSU-QA, a VQA dataset for 'Foresight Intelligence' in autonomous driving, built from 850 nuScenes videos with more than 21k QA pairs across nine tasks (spatio-temporal dynamics, VRU risk assessment, and counterfactual prediction). The accompanying FSU-Bench is a held-out evaluation set of 3,831 QA pairs. The paper evaluates a range of closed- and open-source VLMs with historical video and trajectories, then augments them with world-model-generated future video/trajectories from Epona and DrivingWorld, using the resulting accuracy gains as a proxy for the semantic coherence of the world-model outputs. The main reported results are that GPT-5 achieves 48.66% baseline accuracy, a small Qwen3-VL-8B fine-tuned on FSU-QA reaches 59.59%, and DrivingWorld yields larger VLM gains than Epona.

Significance. The paper addresses a genuine gap: existing VLM benchmarks mostly test understanding of observed events, not reasoning about unobserved future situations. FSU-QA's task taxonomy, automatic annotation pipeline, and broad evaluation across recent VLMs are valuable assets, and the public project page / evaluation code support reproducibility. The idea of using VLM accuracy gains to judge whether world-model outputs are semantically usable is also promising. However, the validity of the main WM-oriented claim depends on control experiments that are promised in the abstract but not reported, and several conclusions rest on single-run accuracy differences without statistical support. With those issues resolved, the benchmark could be a useful community resource; in its current form the claims outrun the evidence.

major comments (5)
  1. [Abstract; §3.2 Eq. (2)–(3); §5.2] The WM-oriented protocol defines semantic coherence solely as the VLM accuracy gain from adding predicted future video/trajectories. The abstract states that this protocol is validated through shuffled control experiments, but I could not find any shuffled-control condition in the main text, appendices, tables, or figures. Without a control that destroys temporal order while preserving frame appearance and token counts (e.g., temporally shuffled predicted frames and permuted predicted trajectories), the observed gains—and the DrivingWorld > Epona ranking—remain confounded by extra-token effects, low-level visual artifacts, or distributional priors. Please report these control experiments or remove the validation claim.
  2. [Tables 1 and 3–8; §5.1, §5.2] All accuracies are single-point numbers on a 3,831-question test set, with no error bars, confidence intervals, or statistical significance tests. Several differences that support the paper's claims are small (e.g., Table 1: GPT-5 48.66 vs. Claude-Sonnet-4.5 46.38; Qwen2.5-VL-72B 45.86 vs. Qwen3-VL-32B 44.74). The statements that current models 'still struggle' and that the fine-tuned model surpasses advanced models 'by a substantial margin' need at least bootstrapped confidence intervals or repeated-inference variability reported.
  3. [§4.2, §5.1 (finetuning paragraph)] The fine-tuned Qwen3-VL-8B is evaluated on FSU-Bench, a held-out split of FSU-QA generated with the same templates and rule-based labeling as the training split. This can reflect template and answer-distribution overfitting rather than generalizable Foresight Intelligence. The claim that 'even small VLMs fine-tuned on FSU-QA surpass much larger, advanced models' should be tempered or supplemented with a cross-dataset / cross-template evaluation or an explicit analysis of how much of the gain survives distribution shift.
  4. [§5.2 vs. Tables 3–8] The text states that adding predicted inputs produces 'noticeable gains' for 'all models' and that models generally benefit more from DrivingWorld's outputs. The tables contradict this in places: with Epona predicted video (Table 3), Qwen2.5-VL-7B drops from 44.74 to 43.38, GPT-4o Mini from 44.22 to 43.75, and GPT-5 from 48.66 to 47.87 relative to the Table 1 baselines. The WM-oriented ranking is therefore sensitive to the choice of VLM evaluator and input modality. Please report per-model deltas and an explicit treatment of negative-gain cases rather than broad 'all models' statements.
  5. [§4.1 QA Data Generation; §A.1.3] The Counterfactual Prediction labels are generated using 'rule-based criteria and counterfactual verification results,' but the rules or simulation used to decide whether a hypothetical action leads to collision, traffic violation, or a safe outcome are not specified. These ground-truth labels are not directly observable in nuScenes, so the validity of this high-level task rests on an unstated assumption. Please provide the exact verification logic and human-agreement or quality-control statistics (sample sizes and agreement rates) for these labels.
minor comments (5)
  1. [Abstract/Figure 1 vs. §4.2] The teaser says '900+ scenes' while the text and statistics say 850 scenes. Please make the numbers consistent.
  2. [§4.3 vs. Table 1] The Llama 4 models are listed under 'Closed-source Models' in the evaluation setup but under 'Open-source Models' in Table 1. Clarify the categorization.
  3. [Figures 5–6] These figures display only five models while the corresponding tables report all models. State the selection criterion or show all models for comparability.
  4. [§3.2, §5.2] 'Semantic coherence' is used as a central quantity but is never formally defined. Provide an operational definition, especially how it relates to VLM accuracy gains.
  5. [B.2] The temperature for local models is set to 0.1 'to minimize randomness,' but no repeated sampling is reported. State whether the reported numbers are from a single generation or an average over multiple seeds.

Circularity Check

0 steps flagged

No circularity found; the only substantive caveat is that the shuffled-control validation mentioned in the abstract is not reported, which is an evidential gap rather than a circular reduction.

full rationale

The paper's VLM evaluation is a standard benchmark protocol: FSU-Bench is a held-out scene split of FSU-QA, and the fine-tuned model Qwen3-VL-8B-FI is evaluated on that held-out split after training on the remaining FSU-QA scenes. Although the held-out split is generated by the same templates and rule-based pipeline as the training split, this is a generality/robustness limitation, not a circular derivation: the reported accuracy is not constructed to equal the training objective. The WM-oriented evaluation (Eq. 2-3) measures an empirical accuracy delta when predicted future video/trajectories are added to the prompt; the conclusion that DrivingWorld provides higher-quality cues than Epona is an empirical inference from Tables 3-8, not a quantity that is identical to the input by construction. No self-citations, imported uniqueness theorems, or ansatz-by-citation chains are load-bearing. The abstract states 'we validate this evaluation protocol through shuffled control experiments,' but no such control experiment appears in the main text, appendix, or tables. That is a missing validation for the semantic-coherence construct and a real correctness/evidence concern, but it does not make the evaluation circular: the paper does not define 'semantic coherence' as the accuracy gain, and a shuffled control would be an empirical check rather than a logical identity. Therefore, no circular step can be exhibited under the required standard.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 1 invented entities

The central claims rest on design choices (temporal split, sampling rate, filtering criteria), on the accuracy of nuScenes ground truth, on rule-based counterfactual labeling, and on the unvalidated assumption that VLM gains measure WM semantic coherence. The invented construct 'Foresight Intelligence' has no independent evidence outside the benchmark itself.

free parameters (3)
  • Historical/future split = 3 s observation / 12 s future
    Chosen by hand; affects all QA pairs and the difficulty distribution. Not derived from data.
  • Frame sampling rate = 2 Hz
    Historical 3s -> 7 frames, future -> 6 frames. A manual design choice that can influence task difficulty.
  • Task-specific filtering prerequisites = not enumerated in paper
    Each of the nine tasks has entry conditions (Section B.1) that determine which scenes/questions are included, shaping the task distribution; the exact criteria are not fully specified.
axioms (4)
  • domain assumption nuScenes 3D object annotations, HD maps, and trajectories are sufficiently accurate to serve as ground truth for the 21K QA pairs.
    The entire automatic annotation pipeline (Section 4.1) derives answers from these annotations; any systematic error propagates to labels.
  • ad hoc to paper Rule-based criteria and template-generated answers correctly capture ground truth for all nine tasks, including counterfactual outcomes that are not directly observable.
    Counterfactual Prediction labels ('collision', 'violation', 'safe') come from a simulated verification rule using map/trajectory data, not from observed outcomes; the rule's validity is not benchmarked externally.
  • domain assumption Performance gains of a VLM when augmented with WM-generated video/trajectory are a valid proxy for semantic coherence of the WM predictions.
    Section 5.2 quantifies semantic coherence by 'performance gains'. This assumes the VLM cannot exploit spurious cues and that gains reflect true semantic consistency; the claimed shuffled-control validation is absent.
  • domain assumption Human-in-the-loop random sampling review is sufficient to ensure annotation quality across 850 scenes.
    Section 4.1 Quality Control says experts randomly sample scenes; the coverage and error-rate statistics are not reported, so the strength of this assumption is unknown.
invented entities (1)
  • Foresight Intelligence no independent evidence
    purpose: Names the target capability the dataset measures: anticipating and interpreting future events.
    The construct is defined solely through FSU-QA/FSU-Bench; no independent behavioral probe outside this benchmark is provided.

pith-pipeline@v1.3.0-alltime-deepseek · 22791 in / 10880 out tokens · 105280 ms · 2026-08-03T20:38:28.466680+00:00 · methodology

0 comments
read the original abstract

In this work, we define Foresight Intelligence as the capability to anticipate and interpret future events-an ability essential for applications such as autonomous driving, yet largely overlooked by existing research. To bridge this gap, we introduce FSU-QA, a new Visual Question-Answering (VQA) dataset specifically designed to elicit and evaluate Foresight Intelligence. Using FSU-QA, we conduct the first comprehensive study of state-of-the-art Vision-Language Models (VLMs) under foresight-oriented tasks, revealing that current models still struggle to reason about future situations. Beyond serving as a benchmark, FSU-QA also enables the assessment of world models by measuring the semantic coherence of their generated predictions, quantified through performance gains when VLMs are augmented with such outputs. Our experiments further demonstrate that FSU-QA can effectively enhance foresight reasoning: even small VLMs fine-tuned on FSU-QA surpass much larger, advanced models by a substantial margin. Together, these findings position FSU-QA as a principled foundation for developing next-generation models capable of truly anticipating and understanding future events. Furthermore, beyond model performance, we examine whether WM-generated predictions remain semantically consistent by using VLM-based proxy judges, and validate this evaluation protocol through shuffled control experiments. Fine-tuning models on FSU-QA leads to substantial improvements in foresight understanding, demonstrating the dataset's effectiveness and offering a principled foundation for future research.

Figures

Figures reproduced from arXiv: 2511.18735 by Liaoyuan Fan, Qing Guo, Shijie Li, Xulei Yang, Xun Xu, Zhantao Gong.

Figure 1
Figure 1. Figure 1: Overview of FSU-QA and FSU-Bench. Abstract In this work, we define Foresight Intelligence as the capa￾bility to anticipate and interpret future events—an ability essential for applications such as autonomous driving, yet largely overlooked by existing research. To bridge this gap, we introduce FSU-QA, a new Visual Question-Answering (VQA) dataset specifically designed to elicit and evaluate Foresight Intel… view at source ↗
Figure 2
Figure 2. Figure 2: Visualization of FSU-Bench. Yellow boxes represent historical frames, and green boxes represent future frames. The bounding boxes are illustrative only and do not appear in the actual dataset. Content in {} is filled into a sentence template. Note: the questions above are simplified slightly for clarity and brevity. Dataset Collection t Single View Videos Key Frames HD map Trajectory Check tasks Question C… view at source ↗
Figure 3
Figure 3. Figure 3: FSU-QA construction pipeline. Note: the questions above are simplified slightly for clarity and brevity. held out for evaluation (FSU-Bench). An overview of the FSU-QA statistics is provided in [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: FSU-QA Statistics. The distribution of tasks across three levels. of varying parameter sizes and architectural types, to reveal their capabilities in foresight intelligence. For closed-source models, we assess Gemini-2.0-Flash, Gemini-2.5-Flash, Gemini-2.5-Pro [8], GPT-4o-Mini [13], GPT-5, Claude-3.7-Sonnet-Standard, Claude-Sonnet-4.5, Llama-4-Scout, and Llama-4-Maverick [1]. For open-source models, we eva… view at source ↗
Figure 5
Figure 5. Figure 5: Results with data generated by Epona. (a)-(d) correspond to results under four different inputs. Speed Change Turn Change Lane Change Relative Distance Relative Position Pedestrian Intent Ego-VRU Relative Position Risk Area Counterfactual Prediction 18.75 37.50 56.25 25.00 50.00 75.00 25.00 50.00 75.00 18.75 37.50 56.25 18.75 37.50 56.25 18.75 37.50 56.25 18.75 37.50 56.25 25.00 50.00 75.00 18.75 37.50 56.… view at source ↗
Figure 6
Figure 6. Figure 6: Results with data generated by DrivingWorld. (a)-(d) correspond to results under four different inputs. Area (49.33) and Pedestrian Interaction (27.54), suggesting a heightened sensitivity to safety-critical cues. Meanwhile, Claude-3.7-Sonnet shows consistent mid-level performance but lags slightly behind Claude-Sonnet-4.5 in high-level reasoning. In contrast, Gemini models (2.0 Flash, 2.5 Flash, 2.5 Pro) … view at source ↗
Figure 7
Figure 7. Figure 7: QA Template: Speed Change. QA Template: Turn Change Question: You are an expert driving assistant. What is the turn change trend of the ego vehicle in the next {interval} seconds? The answer must be one of: A. Straight B. Left turn C. Right turn D. U-turn Answer: A/B/C/D [PITH_FULL_IMAGE:figures/full_fig_p012_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: QA Template: Turn Change task only if the scene satisfies the necessary prerequisites for that task. B.2. Experiment Details Hyperparameter Settings for Inference. To ensure a fair and robust evaluation, we adopted specific inference strate￾gies tailored to the nature of the models. API-based Models: This category includes both propri￾etary closed-source models (e.g., GPT-5, Claude-Sonnet￾4.5, Gemini-2.5) … view at source ↗
Figure 9
Figure 9. Figure 9: QA Template: Lane Change. QA Template: Relative Distance Question: You are an expert driving assistant. Focus on the nearest {description} vehicle in front. How will the relative distance between your vehicle and this vehicle change over the next {interval} seconds? The answer must be one of: A. Approaching B. Distancing C. Maintaining D. Approaching then distancing Answer: A/B/C/D [PITH_FULL_IMAGE:figure… view at source ↗
Figure 10
Figure 10. Figure 10: QA Template: Relative Distance. QA Template: Relative Position Question: You are an expert driving assistant with localization capabilities. Focus on the nearest {description} vehicle in front. Predict its relative position to the ego-vehicle over the next {interval} seconds. The answer must be one of the following eight: A. Ahead B. Behind C. Right D. Left E. Right-front F. Left-front G. Left-rear H. Rig… view at source ↗
Figure 11
Figure 11. Figure 11: QA Template: Relative Position. uation results of DrivingWorld and VLMs are presented in Tab. 6, Tab. 7 and Tab. 8. D. Broader The development of Foresight Intelligence in VLMs and WMs holds significant implications for the advancement of embodied AI and autonomous systems. In this section, we discuss the potential positive impacts and ethical considera￾tions associated with our work, FSU-QA and FSU-Bench… view at source ↗
Figure 12
Figure 12. Figure 12: QA Template: Pedestrian Intent. QA Template: Ego-VRU Relative Position Question: You are an expert driving assistant with localization capabilities. Focus on the closest VRU in the front view. Predict its relative position to the ego-vehicle after 3 seconds. The answer must be one of the following eight: A. Ahead B. Behind C. Right D. Left E. Right-front F. Left-front G. Left-rear H. Right-rear Answer: A/… view at source ↗
Figure 13
Figure 13. Figure 13: QA Template: Ego-VRU Relative Position. Hyperparameter Value Description Base Model Qwen3-VL-8B-Instruct Initial checkpoint initialized from HuggingFace Tuned Modules LLM + Projector Vision encoder is frozen (tune_mm_vision=False) Optimization Strategy DeepSpeed ZeRO-3 Memory optimization for full fine-tuning Precision BF16 Bfloat16 mixed precision training Learning Rate 2e − 5 Peak learning rate LR Sched… view at source ↗
Figure 14
Figure 14. Figure 14: QA Template: Risk Area. QA Template: Counterfactual Prediction Question: You are an expert driving assistant with predictive capabilities. If the ego vehicle were to {act} over the next 3 seconds, what would be the most likely outcome? This is a multiple-choice question, and the answer can be one or more of: A. Collision B. Driving Out of Legal Area C. Violating Traffic Rules D. Safe Answer: A, B, C (mult… view at source ↗
Figure 15
Figure 15. Figure 15: QA Template: Counterfactual Prediction. data in FSU-QA, inherited from nuScenes, is geographi￾cally limited to specific urban environments (Boston and Singapore). Driving behaviors, traffic rules, and road infras￾tructures vary significantly across different countries and cultures. Consequently, models trained or evaluated on this benchmark may exhibit geographic bias and may not gen￾eralize perfectly to … view at source ↗
Figure 16
Figure 16. Figure 16: Prompt: Baseline [PITH_FULL_IMAGE:figures/full_fig_p016_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Prompt: Base & Pred. Video [PITH_FULL_IMAGE:figures/full_fig_p017_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: Prompt: Base & Pred. Traj [PITH_FULL_IMAGE:figures/full_fig_p018_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: Prompt: Base & Pred. V+T [PITH_FULL_IMAGE:figures/full_fig_p019_19.png] view at source ↗
Figure 20
Figure 20. Figure 20: Speed Change Visualization. Top: Constant Speed. Bottom: Acceleration [PITH_FULL_IMAGE:figures/full_fig_p023_20.png] view at source ↗
Figure 21
Figure 21. Figure 21: Turn Change Visualization. Top: Turn Right. Bottom: Turn Left [PITH_FULL_IMAGE:figures/full_fig_p023_21.png] view at source ↗
Figure 22
Figure 22. Figure 22: Lane Change Visualization. Top: Left Lane Change. Bottom: Right Lane Change [PITH_FULL_IMAGE:figures/full_fig_p023_22.png] view at source ↗
Figure 23
Figure 23. Figure 23: Relative Distance & Relative Position Visualization. The bounding boxes are illustrative only and do not appear in the actual dataset [PITH_FULL_IMAGE:figures/full_fig_p024_23.png] view at source ↗
Figure 24
Figure 24. Figure 24: Pedestrian Intent Visualization. Top: Waiting at curbside. Bottom: Jaywalking. The bounding boxes are illustrative only and do not appear in the actual dataset [PITH_FULL_IMAGE:figures/full_fig_p024_24.png] view at source ↗
Figure 25
Figure 25. Figure 25: Ego-VRU Relative Position Visualization. Top: Motorcyclist. Bottom: Pedestrian. The bounding boxes are illustrative only and do not appear in the actual dataset [PITH_FULL_IMAGE:figures/full_fig_p024_25.png] view at source ↗
Figure 26
Figure 26. Figure 26: Risk Area Visualization. Top: High Occlusion Area. Bottom: Complex Intersection [PITH_FULL_IMAGE:figures/full_fig_p025_26.png] view at source ↗
Figure 27
Figure 27. Figure 27: Counterfactual Prediction Visualization. Top: What would happen if the ego vehicle were to change lanes to the left? Bottom: What would happen if the ego vehicle were to turn right? [PITH_FULL_IMAGE:figures/full_fig_p025_27.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

46 extracted references · 6 linked inside Pith

  1. [1]

    Introducing llama 4: Advancing multimodal intel- ligence, 2025

    Meta AI. Introducing llama 4: Advancing multimodal intel- ligence, 2025. 6

    2025

  2. [2]

    Scanqa: 3d question answering for spatial scene understanding

    Daichi Azuma, Taiki Miyanishi, Shuhei Kurita, and Motoaki Kawanabe. Scanqa: 3d question answering for spatial scene understanding. In2022 IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pages 19107– 19117, 2022. 2

    2022

  3. [3]

    Qwen2.5-vl technical report.arXiv preprint arXiv:2502.13923, 2025

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhao- hai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Jun- yang Lin. Qwen2.5-vl technical repor...

    Pith/arXiv arXiv 2025

  4. [4]

    Ge- nie: generative interactive environments

    Jake Bruce, Michael Dennis, Ashley Edwards, Jack Parker- Holder, Yuge (Jimmy) Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, Yusuf Aytar, Sarah Bechtle, Feryal Behbahani, Stephanie Chan, Nicolas Heess, Lucy Gonzalez, Simon Osindero, Sherjil Ozair, Scott Reed, Jingwei Zhang, Konrad Zolna, Jeff Clune, Nando De Freitas, Sa...

    2024

  5. [5]

    Lang, Sourabh V ora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Gi- ancarlo Baldan, and Oscar Beijbom

    Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh V ora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Gi- ancarlo Baldan, and Oscar Beijbom. nuScenes: A Multi- modal Dataset for Autonomous Driving. In2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, W A, USA, 2020. IEEE. 4

    2020

  6. [6]

    Deepverse: 4d autoregressive video generation as a world model, 2025

    Junyi Chen, Haoyi Zhu, Xianglong He, Yifan Wang, Jianjun Zhou, Wenzheng Chang, Yang Zhou, Zizun Li, Zhoujie Fu, Jiangmiao Pang, and Tong He. Deepverse: 4d autoregressive video generation as a world model, 2025. 3

    2025

  7. [7]

    Spatial- RGPT: Grounded spatial reasoning in vision-language mod- els

    An-Chieh Cheng, Hongxu Yin, Yang Fu, Qiushan Guo, Rui- han Yang, Jan Kautz, Xiaolong Wang, and Sifei Liu. Spatial- RGPT: Grounded spatial reasoning in vision-language mod- els. InThe Thirty-eighth Annual Conference on Neural In- formation Processing Systems, 2024. 2

    2024

  8. [8]

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261, 2025

    Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blis- tein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261, 2025. 6

    Pith/arXiv arXiv 2025

  9. [9]

    Drivellm: Charting the path toward full au- tonomous driving with large language models.IEEE Trans- actions on Intelligent Vehicles, 9(1):1450–1464, 2024

    Yaodong Cui, Shucheng Huang, Jiaming Zhong, Zhenan Liu, Yutong Wang, Chen Sun, Bai Li, Xiao Wang, and Amir Khajepour. Drivellm: Charting the path toward full au- tonomous driving with large language models.IEEE Trans- actions on Intelligent Vehicles, 9(1):1450–1464, 2024. 3

    2024

  10. [10]

    World models

    David Ha and Jürgen Schmidhuber. World models. 2018. 3

    2018

  11. [11]

    Driving- World: Constructing World Model for Autonomous Driving via Video GPT.arXiv preprint arXiv:2412.19505, 2024

    Xiaotao Hu, Wei Yin, Mingkai Jia, Junyuan Deng, Xiaoyang Guo, Qian Zhang, Xiaoxiao Long, and Ping Tan. Driving- World: Constructing World Model for Autonomous Driving via Video GPT.arXiv preprint arXiv:2412.19505, 2024. 3, 6

    Pith/arXiv arXiv 2024

  12. [12]

    Hudson and Christopher D

    Drew A. Hudson and Christopher D. Manning. Gqa: A new dataset for real-world visual reasoning and composi- tional question answering. In2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6693–6702, 2019. 2

    2019

  13. [13]

    Gpt-4o system card,

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perel- man, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Weli- hinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card,

  14. [14]

    Unifying 2d and 3d vision-language un- derstanding

    Ayush Jain, Alexander Swerdlow, Yuzhou Wang, Sergio Ar- naud, Ada Martin, Alexander Sax, Franziska Meier, and Ka- terina Fragkiadaki. Unifying 2d and 3d vision-language un- derstanding. InForty-second International Conference on Machine Learning, 2025. 2

    2025

  15. [15]

    Lawrence Zitnick, and Ross Girshick

    Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning . In2017 IEEE Confer- ence on Computer Vision and Pattern Recognition (CVPR), pages 1988–1997, Los Alamitos, CA, USA, 2017. IEEE Computer Society. 2

    1988

  16. [16]

    Evaluating object hallucination in large vision- language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision- language models. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 292–305, Singapore, 2023. Association for Computational Linguistics. 3

    2023

  17. [17]

    Drivegen: Generalized and robust 3d detection in driving via controllable text-to-image diffusion generation

    Hongbin Lin, Zilu Guo, Yifan Zhang, Shuaicheng Niu, Yafeng Li, Ruimao Zhang, Shuguang Cui, and Zhen Li. Drivegen: Generalized and robust 3d detection in driving via controllable text-to-image diffusion generation. InProceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 27497–27507, 2025. 3

    2025

  18. [18]

    Visual Instruction Tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual Instruction Tuning. InAdvances in Neural Informa- tion Processing Systems, pages 34892–34916. Curran Asso- ciates, Inc., 2023. 2

    2023

  19. [19]

    3dsrbench: A comprehensive 3d spatial reasoning benchmark, 2025

    Wufei Ma, Haoyu Chen, Guofeng Zhang, Yu-Cheng Chou, Jieneng Chen, Celso M de Melo, and Alan Yuille. 3dsrbench: A comprehensive 3d spatial reasoning benchmark, 2025. 3

    2025

  20. [20]

    Video-ChatGPT: Towards detailed video un- derstanding via large vision and language models

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Khan. Video-ChatGPT: Towards detailed video un- derstanding via large vision and language models. InPro- ceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12585–12602, Bangkok, Thailand, 2024. Association for Computational Linguistics. 2

    2024

  21. [21]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10674–10685, 2022. 3

    2022

  22. [22]

    Schacter, Donna Rose Addis, and Randy L

    Daniel L. Schacter, Donna Rose Addis, and Randy L. Buck- ner. Remembering the past to imagine the future: the prospective brain.Nature Reviews Neuroscience, 8(9):657– 661, 2007. 3

    2007

  23. [23]

    Video-xl: Extra-long vision language model for hour-scale video understanding

    Yan Shu, Zheng Liu, Peitian Zhang, Minghao Qin, Jun- jie Zhou, Zhengyang Liang, Tiejun Huang, and Bo Zhao. Video-xl: Extra-long vision language model for hour-scale video understanding. In2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 26160–26169, 2025. 2

    2025

  24. [24]

    Sutton and A.G

    R.S. Sutton and A.G. Barto. Reinforcement learning: An introduction.IEEE Transactions on Neural Networks, 9(5): 1054–1054, 1998. 3

    1998

  25. [25]

    Internvl3.5: Advanc- ing open-source multimodal models in versatility, reasoning, and efficiency.arXiv preprint arXiv:2508.18265, 2025

    Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Sheng- long Ye, Jie Shao, Zhaokai Wang, Zhe Chen, Hongjie Zhang, Ganlin Yang, Haomin Wang, Qi Wei, Jinhui Yin, Wenhao Li, Erfei Cui, Guanzhou Chen, Zichen Ding, Changyao Tian, Zhenyu Wu, Jingjing Xie, Zehao Li, Bowen Yang, Yuchen Duan, Xuehui Wang, Zhi Ho...

    Pith/arXiv arXiv 2025

  26. [26]

    Drivedreamer: Towards real-world- driven world models for autonomous driving.arXiv preprint arXiv:2309.09777, 2023

    Xiaofeng Wang, Zheng Zhu, Guan Huang, Xinze Chen, Jia- gang Zhu, and Jiwen Lu. Drivedreamer: Towards real-world- driven world models for autonomous driving.arXiv preprint arXiv:2309.09777, 2023. 3

    Pith/arXiv arXiv 2023

  27. [27]

    Internvideo: General video foundation models via generative and discriminative learning, 2022

    Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, Sen Xing, Guo Chen, Junting Pan, Jiashuo Yu, Yali Wang, Limin Wang, and Yu Qiao. Internvideo: General video foundation models via generative and discriminative learning, 2022. 2

    2022

  28. [28]

    Orient anything: Learning robust object orientation estimation from rendering 3d mod- els

    Zehan Wang, Ziang Zhang, Tianyu Pang, Chao Du, Heng- shuang Zhao, and Zhou Zhao. Orient anything: Learning robust object orientation estimation from rendering 3d mod- els. InForty-second International Conference on Machine Learning, 2025. 3

    2025

  29. [29]

    Panacea: Panoramic and controllable video generation for autonomous driving

    Yuqing Wen, Yucheng Zhao, Yingfei Liu, Fan Jia, Yan- hui Wang, Chong Luo, Chi Zhang, Tiancai Wang, Xi- aoyan Sun, and Xiangyu Zhang. Panacea: Panoramic and controllable video generation for autonomous driving. In 2024 IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pages 6902–6912, 2024. 3

    2024

  30. [30]

    Next-qa: Next phase of question-answering to explaining temporal actions

    Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question-answering to explaining temporal actions. In2021 IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pages 9772– 9781, 2021. 3

    2021

  31. [31]

    Wong, Zhenguo Li, and Hengshuang Zhao

    Zhenhua Xu, Yujia Zhang, Enze Xie, Zhen Zhao, Yong Guo, Kwan-Yee K. Wong, Zhenguo Li, and Hengshuang Zhao. DriveGPT4: Interpretable End-to-End Autonomous Driving Via Large Language Model.IEEE Robotics and Automation Letters, 9(10):8186–8193, 2024. 2, 3

    2024

  32. [32]

    Videogpt: Video generation using vq-vae and trans- formers.2104.10157, 2021

    Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. Videogpt: Video generation using vq-vae and trans- formers.2104.10157, 2021. 3

    Pith/arXiv arXiv 2021

  33. [33]

    Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie

    Jihan Yang, Shusheng Yang, Anjali W. Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in Space: How Multi- modal Large Language Models See, Remember, and Recall Spaces. In2025 IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition (CVPR), pages 10632–10643, Nashville, TN, USA, 2025. IEEE. 2

    2025

  34. [34]

    Mmsi-bench: A benchmark for multi-image spatial intelli- gence, 2025

    Sihan Yang, Runsen Xu, Yiman Xie, Sizhe Yang, Mo Li, Jingli Lin, Chenming Zhu, Xiaochen Chen, Haodong Duan, Xiangyu Yue, Dahua Lin, Tai Wang, and Jiangmiao Pang. Mmsi-bench: A benchmark for multi-image spatial intelli- gence, 2025. 2

    2025

  35. [35]

    Video-LLaMA: An instruction-tuned audio-visual language model for video un- derstanding

    Hang Zhang, Xin Li, and Lidong Bing. Video-LLaMA: An instruction-tuned audio-visual language model for video un- derstanding. InProceedings of the 2023 Conference on Em- pirical Methods in Natural Language Processing: System Demonstrations, pages 543–553, Singapore, 2023. Associ- ation for Computational Linguistics. 2

    2023

  36. [36]

    Epona: Au- toregressive diffusion world model for autonomous driving

    Kaiwen Zhang, Zhenyu Tang, Xiaotao Hu, Xingang Pan, Xiaoyang Guo, Yuan Liu, Jingwei Huang, Li Yuan, Qian Zhang, Xiao-Xiao Long, Xun Cao, and Wei Yin. Epona: Au- toregressive diffusion world model for autonomous driving. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 27220–27230, 2025. 3, 6

    2025

  37. [37]

    DSI- Bench: A Benchmark for Dynamic Spatial Intelligence

    Ziang Zhang, Zehan Wang, Guanghao Zhang, Weilong Dai, Yan Xia, Ziang Yan, Minjie Hong, and Zhou Zhao. DSI- Bench: A Benchmark for Dynamic Spatial Intelligence. arXiv preprint arXiv:2510.18873, 2025. 3

    arXiv 2025

  38. [38]

    Drivedreamer-2: Llm-enhanced world models for diverse driving video generation

    Guosheng Zhao, Xiaofeng Wang, Zheng Zhu, Xinze Chen, Guan Huang, Xiaoyi Bao, and Xingang Wang. Drivedreamer-2: Llm-enhanced world models for diverse driving video generation. InProceedings of the Thirty- Ninth AAAI Conference on Artificial Intelligence and Thirty- Seventh Conference on Innovative Applications of Artificial Intelligence and Fifteenth Sympo...

    2025

  39. [39]

    Vlm4d: To- wards spatiotemporal awareness in vision language models,

    Shijie Zhou, Alexander Vilesov, Xuehai He, Ziyu Wan, Shuwang Zhang, Aditya Nagachandra, Di Chang, Dongdong Chen, Xin Eric Wang, and Achuta Kadambi. Vlm4d: To- wards spatiotemporal awareness in vision language models,

  40. [42]

    The current moment is defined as the last historical frame

    Predicted future frames: 6 images generated by a world model, representing possible future views of the vehicle in the next 3 seconds after the last historical frame. The current moment is defined as the last historical frame. All trajectory points are historical and correspond to each historical frame. Trajectory points are provided as relative coordinat...

  41. [45]

    The current moment is defined as the last historical frame

    Predicted trajectory:xtrajectory points (relative coordinates and yaw angles) predicted for the next 3 seconds after the last historical frame. The current moment is defined as the last historical frame. All trajectory points are relative to the last historical frame. The predicted trajectory is provided to assist you in answering the following questions ...

  42. [46]

    Each image corresponds to a historical trajectory point, in chronological order

    Historical frames: 7 images captured by the vehicle’s front-facing camera over 3 seconds. Each image corresponds to a historical trajectory point, in chronological order

  43. [47]

    Historical trajectory: 7 trajectory points (relative coordinates and yaw angles) corresponding to the historical frames, all relative to the last historical frame

  44. [48]

    Predicted future frames: 6 images generated by a world model, representing possible future views of the vehicle in the next 3 seconds after the last historical frame

  45. [49]

    These points do not necessarily correspond to the predicted frames one by one

    Predicted trajectory:xtrajectory points (relative coordinates and yaw angles) predicted for the next 3 seconds after the last historical frame. These points do not necessarily correspond to the predicted frames one by one. The current moment is defined as the last historical frame. All historical trajectory points are relative to the last historical frame...

  46. [2025]

    Turn Change

    3 A. Task Descriptions A.1. Task Definition In this section, we provide detailed descriptions of each task defined in the dataset. A.1.1. Spatio-temporal Dynamic Reasoning (Low-level) Speed Change.This task evaluates the model’s ca- pability to perceive and anticipate the longitudinal motion dynamics of the ego vehicle. Given historical visual obser- vati...