Pith. sign in

REVIEW 5 major objections 6 minor 62 references

World-Consistent Data Generation for Vision-and-Language Navigation

T0 review · 5 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that adding explicit camera geometry to diffusion-based data generation produces world-consistent VLN training environments and new state-of-the-art results on R2R, R4R, REVERIE, and CVDN.

desk verdict Useful VLN data augmentation with real SPL gains; the world-consistency story is under-validated and the DPT depth-scale issue needs to be fixed. read the letter →

arxiv 2412.06413 v2 pith:KSMSYT3J submitted 2024-12-09 cs.CV

Yu Zhong , Rui Zhang , Zihao Zhang , Shuo Wang , Chuan Fang , Xishan Zhang , Jiaming Guo , Shaohui Peng
show 4 more authors
Di Huang Yanyang Yan Xing Hu Qi Guo
This is my paper · ORCID
classification cs.CV
keywords vision-and-languagenavigationdataaugmentationworld-consistencypanoramagenerationdiffusionmodelsControlNetpointcloudprojectioninstruction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes WCGEN, a data-augmentation pipeline for Vision-and-Language Navigation (VLN) that generates synthetic training environments which are both diverse and world-consistent. Its central claim is that injecting camera projective geometry into a diffusion-based generator fixes the world-inconsistency that plagues prior augmentation such as PanoGen, and that training the DUET agent on this augmented data yields new state-of-the-art results on R2R, R4R, REVERIE, and CVDN. A reader should care because VLN agents generalize poorly to unseen environments due to data scarcity, and the paper shows a practical way to scale up training data without breaking spatial coherence.

What carries the argument

The load-bearing mechanism is the coupling of two generative modules with camera projective geometry: the Temporal Replenishment Module (TRM), which uses point-cloud reprojection (Eqs. 1–3) to warp the previous reference image into the next viewpoint as a guidance image for ControlNet, and the Spatial Outpainting Module (SOM), which uses spherical rotation of pixels (Eqs. 4–5) to warp already-generated neighboring perspectives into the target view with blurred binary masks, again guiding ControlNet-based outpainting. These modules turn a purely semantic diffusion generator into one that is geometrically grounded at both trajectory and viewpoint levels.

What would settle it

Train DUET on data from an ablated pipeline that replaces the geometrically warped guidance images with randomly perturbed versions, while keeping the same prompts, depth conditioning, and outpainting order; if SPL on R2R validation unseen stays close to the reported 65.8, then world-consistency is not the cause of the gain.

Watch

Extended reading notes

Core claim

The paper's central claim is that world-consistency—spatial coherence across viewpoints along a trajectory and seamless wraparound coherence within each panorama—is the missing ingredient in VLN data augmentation, and that it can be achieved by combining diffusion models with explicit 3D geometry. The WCGEN framework does this in two stages: a trajectory stage in which a Temporal Replenishment Module projects the previously generated reference image into the next viewpoint using point-cloud reprojection and then regenerates it with ControlNet conditioned on the warped guidance and depth; and a viewpoint stage in which a Spatial Outpainting Module warps neighboring perspective images onto the target view by spherical rotation, builds blurred guidance masks, and progressively outpaints the full panorama. New instructions are generated by finetuning mPLUG-2 on the resulting observations. The paper reports that training DUET on this data improves SPL by 5.8% on R2R validation unseen, 5.5% on R2R test, and 3.4% on R4R validation unseen, and achieves new state-of-the-art results on all four benchmarks.

Load-bearing premise

The method's effectiveness rests on the premise that the pretrained monocular depth estimator and ControlNet preserve 3D geometry accurately enough that the projected guidance images and outpainted panoramas stay world-consistent; the paper never measures this consistency directly.

Editorial extensions

If this is right

  • Training DUET with WCGEN data yields SPL improvements of 5.8, 5.5, and 3.4 points on R2R validation unseen, R2R test, and R4R validation unseen, respectively, over the unaugmented baseline.
  • WCGEN outperforms PanoGen by 1.5, 0.6, and 1.0 SPL points on the same three splits, and by 9.1% in Goal Progress on CVDN validation unseen.
  • Ablations show that each component—depth-aware rendering, multi-view angle synthesis, and instruction regeneration—contributes, with removal of either geometric module degrading SPL by roughly 4% or more.
  • Performance continues to rise with the number of generated trajectories beyond the 4,676 trajectories in the R2R training set, suggesting the augmentation does not saturate at current scale.
  • Using augmented data in both pretraining and finetuning gives larger gains than using it in only one phase, with finetuning benefiting more.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's ablations do not isolate the LLM prompt contribution from the geometric guidance, so a fair editorial inference is that some of the gain may come from richer textual conditioning rather than from world-consistency alone.
  • Because the pipeline only needs RGB-D panoramas and camera poses, the same geometry-conditioned augmentation could transfer to other embodied perception tasks that use panoramic observations, such as point-goal navigation or embodied question answering, though the paper does not test this.
  • A direct test of the world-consistency hypothesis would compare WCGEN against an otherwise identical pipeline in which the guidance images are replaced by randomly warped or non-geometric versions; if SPL gains persist, consistency is not the active ingredient.
  • The method's ceiling is tied to the accuracy of the monocular depth estimator and the fidelity of ControlNet in respecting the warped guidance, so stronger geometric priors (e.g., multi-view stereo or sensor depth) could plausibly increase the gains.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes WCGEN, a two-stage data-augmentation framework for Vision-and-Language Navigation. In the trajectory stage, a Temporal Replenishment Module projects the previously generated reference image into the next viewpoint using estimated depth and camera poses to produce a guidance image, which is then refined by ControlNet. In the viewpoint stage, a Spatial Outpainting Module warps neighboring generated perspective images onto a target perspective using spherical rotations and masks, then performs depth-guided outpainting to complete a panorama. Instructions are regenerated by fine-tuning mPLUG-2. The authors evaluate WCGEN by training DUET on augmented R2R, R4R, REVERIE, and CVDN data, reporting SPL improvements over PanoGen on R2R unseen/test and R4R unseen, plus gains on coarse-grained datasets, and they claim new state-of-the-art results. They also provide ablations of the modules, instruction generation, training phases, and the number of generated trajectories.

Significance. If the world-consistency claim were validated, this would be a valuable contribution to VLN data augmentation, since current methods such as PanoGen ignore geometric constraints. The paper is the first in this line to explicitly inject camera geometry into diffusion-based panorama generation, and the downstream empirical gains on multiple benchmarks are potentially useful. The method is self-contained and the comparison against PanoGen under the same DUET backbone is a fair head-to-head. However, the central geometric mechanism is not directly verified, and the reported improvements over PanoGen are small and unreplicated, so the significance is contingent on the authors addressing these gaps.

major comments (5)
  1. [III-A, Eqs. (1)-(3)] The trajectory-stage projection uses a pre-trained transformer-based depth estimator (reference [55], DPT) to obtain depth for the synthesized image. DPT outputs affine-invariant monocular depth, not metric depth. Equations (1)-(3) combine this estimated depth with metric camera intrinsics K and the metric translation T from Matterport3D. Without an explicit scale (and shift) calibration step, which is not described anywhere in Section III-A or the supplementary, the unprojected 3D point cloud has an arbitrary scale, so the reprojection to the next viewpoint is not a physically correct novel view. Figure 5 uses a different metric-depth model (Metric3D v2) only for visualization, not for the generation pipeline. The authors should either describe and validate the scale alignment used in practice, or demonstrate that the guidance image remains sufficiently accurate despite the scale ambiguity; as written, the claim that WCGEN maintains world consistency through accurate 3D viewpoint prediction is unsubstantiated.
  2. [IV-D, Table III and Figure 5] World consistency is never directly measured. Table III reports IS and FID, which are appearance metrics and do not test geometric or trajectory-level consistency. Figure 5 is a qualitative 3D point-cloud visualization computed post-hoc with Metric3D v2 on generated images, and the same model is not used in the generation pipeline. Table IV shows that removing components degrades SPL, but SPL on held-out environments can improve from increased data diversity or instruction regeneration even if the geometry is wrong. A quantitative geometric consistency evaluation is needed, for example by computing reprojection errors on known correspondences across generated viewpoints, or by measuring multi-view consistency of the generated panoramas against a reference.
  3. [IV-E2, Table V] Instruction regeneration is a major confound for the core claim. Table V shows that using WCGEN-generated instructions instead of the original R2R instructions raises validation-unseen SPL from 62.5 to 65.8, a gain of 3.3 SPL points. The total gain over the DUET baseline is 5.8 SPL points (60.0 to 65.8). Thus more than half of the total improvement comes from the instruction module, which is not specific to world-consistent visual generation. Since PanoGen also regenerates instructions, the 1.5-point SPL advantage over PanoGen in Table I may be partially or wholly attributable to instruction quality or other differences in the generation pipeline. The authors should ablate the visual augmentation with and without instruction regeneration in direct comparison to PanoGen under the same settings.
  4. [IV-D2 and Table II] The abstract and conclusion claim 'new state-of-the-art results on all navigation tasks,' but the results on REVERIE in Table II show WCGEN is worse than the DUET baseline on RGSPL (22.33 vs. 22.89), which the authors acknowledge in Section IV-D.2. The claim is also not supported for all metrics on CVDN, where the test GP is essentially tied with PanoGen (7.18 vs. 7.17). The state-of-the-art claim should be qualified to the specific datasets and metrics where the improvement holds, or the wording should be softened.
  5. [IV-D and Table I] The reported gains over PanoGen are small (0.6-1.0 SPL on R2R/R4R and 9.1% on CVDN validation unseen, but only 0.01 GP on CVDN test), and no error bars or multiple-seed results are reported anywhere in the paper. Given the inherent variance in VLN evaluation and the modest size of the differences, the authors should report the mean and standard deviation over at least three independent training runs for the main comparisons, especially for the WCGEN vs. PanoGen differences.
minor comments (6)
  1. [Abstract] The abstract contains a typo: 'data argumentation' should be 'data augmentation.'
  2. [IV-C] There is a typo, 'Stbale Diffusion XL' should be 'Stable Diffusion XL.'
  3. [IV-F] In Section IV-F, 'utilze' should be 'utilize.'
  4. [III-B, Eq. (5)] In Eq. (5), the rotation composition is written as R_{j->i} * R; it would be clearer to specify that R is the extrinsic rotation of the source perspective and R_{j->i} is the relative rotation from perspective j to i, or to define both matrices explicitly.
  5. [Table VII] The text says '4,676, which is the total number of trajectories in the R2R dataset' without specifying that this is the training-split count; clarify the scope to avoid ambiguity.
  6. [IV-F, Figure 5] The caption of Figure 5 mentions using Metric3D v2 for depth estimation, but the main text does not explain why a different depth model is used here versus the DPT model in Eq. (1) of the trajectory stage; this discrepancy should be addressed.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the derivation is an empirical augmentation pipeline whose claimed gains are measured on held-out navigation splits, and no equation or fitted parameter reduces to the reported target.

full rationale

WCGEN is a two-stage data generation pipeline, not a derivation of navigation performance from its own definitions. The trajectory stage (Eqs. 1-3) unprojects a previous perspective image with an estimated depth map and camera poses and reprojects into the next viewpoint; the viewpoint stage (Eqs. 4-5) warps between perspectives under rotation; ControlNet and mPLUG-2 then generate panoramas and instructions. None of these quantities is defined in terms of the reported SPL/SR/GP/NE metrics, and no parameter is fitted to a subset of those metrics and then presented as a prediction. The reported improvements are obtained by training DUET on the augmented data and evaluating on R2R validation unseen/test, R4R validation unseen, and CVDN/REVERIE unseen splits, i.e., external held-out benchmarks. There are no load-bearing self-citations by the present authors: cited pretrained models (DPT, ControlNet, GPT-4, mPLUG-2, DUET, PanoGen) are external and independently published. The paper's weak point is that 'world-consistency' is not directly measured (Table III uses IS/FID appearance metrics, and Fig. 5 uses Metric3D only for visualization), and the trajectory-stage projection uses a monocular depth estimator whose scale is not explicitly calibrated to the metric translation T. These are evidence and validity limitations about whether geometry is preserved, not circular reductions: the SPL gains do not reduce by construction to the inputs. Hence the circularity score is 0.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

No new physical entities are introduced. The framework composes existing neural models and standard projective geometry. The main uncharged assumptions are the reliability of pretrained depth estimation, ControlNet, and video captioning models.

free parameters (4)
  • ControlNet denoising strength = not reported
    A reduced noise strength is mentioned in Section III-A to limit deviation from the guidance image, but the exact value is not given.
  • Number of generated trajectories = 5000
    Selected based on validation-unseen performance in Table VII; the authors note performance keeps rising with scale.
  • Blur kernel for mask = not reported
    Edges of the binary mask Mj->i are blurred in the viewpoint stage, but the kernel size is unspecified.
  • Weighted sum weights for multiple guidance images = not reported
    In the viewpoint stage, when multiple neighboring images guide outpainting, a weighted sum is used without stating the weights.
assumptions (3)
  • domain assumption Pretrained monocular depth estimator produces accurate depth for generated images
    Needed for the point-cloud projection in Eqs. 1-3 of Section III-A; depth errors directly corrupt the guidance images.
  • domain assumption ControlNet with depth conditioning can fill large missing regions while preserving geometry
    The projected guidance image has dense grid-shaped holes, and the authors assume the diffusion model fills them without breaking spatial consistency.
  • domain assumption Pretrained Stable Diffusion XL and mPLUG-2 provide sufficiently realistic and aligned generation
    These models are used as black boxes; their failure modes are not analyzed in the paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of World-Consistent Data Generation for Vision-and-Language Navigation." pith.science (2026). https://pith.science/paper/KSMSYT3J

@misc{pith2026241206413,
  author       = {Pith},
  title        = {Pith review of: World-Consistent Data Generation for Vision-and-Language Navigation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KSMSYT3J}},
  note         = {Machine review of arXiv:2412.06413}
}
read the original abstract

Vision-and-Language Navigation (VLN) is a challenging task that requires an agent to navigate through photorealistic environments following natural-language instructions. One main obstacle existing in VLN is data scarcity, leading to poor generalization performance over unseen environments. Though data argumentation is a promising way for scaling up the dataset, how to generate VLN data both diverse and world-consistent remains problematic. To cope with this issue, we propose the world-consistent data generation (WCGEN), an efficacious data-augmentation framework satisfying both diversity and world-consistency, aimed at enhancing the generalization of agents to novel environments. Roughly, our framework consists of two stages, the trajectory stage which leverages a point-cloud based technique to ensure spatial coherency among viewpoints, and the viewpoint stage which adopts a novel angle synthesis method to guarantee spatial and wraparound consistency within the entire observation. By accurately predicting viewpoint changes with 3D knowledge, our approach maintains the world-consistency during the generation procedure. Experiments on a wide range of datasets verify the effectiveness of our method, demonstrating that our data augmentation strategy enables agents to achieve new state-of-the-art results on all navigation tasks, and is capable of enhancing the VLN agents' generalization ability to unseen environments.

Figures

Figures reproduced from arXiv: 2412.06413 by the authors.

Figure 1
Figure 1. Comparison between our WCGEN and previous data [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Framework of WCGEN. The input is a trajectory of viewpoints together with their panoramic observation, depth values, [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Qualitative results for panoramas and perspective images. We visualize three viewpoints and observe that our results [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Visualization of temporal consistency . Our trajectory-based generation (the third line) exhibits more temporally consistency in styles and semantics than independently generated images (the second line). Matterport3D PanoGen Ours [PITH_FULL_IMAGE:figures/full_fig_p00…
Figure 5
Figure 5. Figure 5: Visualization of spatial consistency. We perform depth estimation on the generated RGB images and further establish the 3D point clouds. Our generation is capable of reserving realistic layouts and exhibiting clear geometry of indoor scenes. we explore the impact of da…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

62 extracted references · 37 canonical work pages

  1. [55]

    Vision transformers for dense prediction,

    R. Ranftl, A. Bochkovskiy, and V . Koltun, “Vision transformers for dense prediction,” CoRR, vol. abs/2103.13413, 2021. [Online]. Available: https://arxiv.org/abs/2103.13413

    arXiv 2021

  2. [1]

    Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments,

    P. Anderson, Q. Wu, D. Teney, J. Bruce, M. Johnson, N. S ¨underhauf, I. Reid, S. Gould, and A. Van Den Hengel, “Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2018, pp. 3674–3683

    work page 2018

  3. [2]

    Matterport3d: Learning from rgb-d data in indoor environments,

    A. Chang, A. Dai, T. Funkhouser, M. Halber, M. Niessner, M. Savva, S. Song, A. Zeng, and Y . Zhang, “Matterport3d: Learning from rgb-d data in indoor environments,” arXiv preprint arXiv:1709.06158 , 2017

    arXiv 2017

  4. [3]

    Learning from unlabeled 3d environments for vision-and-language navigation,

    S. Chen, P.-L. Guhur, M. Tapaswi, C. Schmid, and I. Laptev, “Learning from unlabeled 3d environments for vision-and-language navigation,” in European Conference on Computer Vision . Springer, 2022, pp. 638– 655

    work page 2022

  5. [4]

    A new path: Scaling vision-and- language navigation with synthetic instructions and imitation learning,

    A. Kamath, P. Anderson, S. Wang, J. Y . Koh, A. Ku, A. Waters, Y . Yang, J. Baldridge, and Z. Parekh, “A new path: Scaling vision-and- language navigation with synthetic instructions and imitation learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 10 813–10 823

    work page 2023

  6. [5]

    Scaling data generation in vision-and-language navigation,

    Z. Wang, J. Li, Y . Hong, Y . Wang, Q. Wu, M. Bansal, S. Gould, H. Tan, and Y . Qiao, “Scaling data generation in vision-and-language navigation,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 12 009–12 020

    work page 2023

  7. [6]

    Envedit: Environment editing for vision- and-language navigation,

    J. Li, H. Tan, and M. Bansal, “Envedit: Environment editing for vision- and-language navigation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 15 407–15 417

    work page 2022

  8. [7]

    Simple and effective synthesis of indoor 3d scenes,

    J. Y . Koh, H. Agrawal, D. Batra, R. Tucker, A. Waters, H. Lee, Y . Yang, J. Baldridge, and P. Anderson, “Simple and effective synthesis of indoor 3d scenes,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, no. 1, 2023, pp. 1169–1178

    work page 2023

Show all 62 references
  1. [8]

    Frequency- enhanced data augmentation for vision-and-language navigation,

    K. He, C. Si, Z. Lu, Y . Huang, L. Wang, and X. Wang, “Frequency- enhanced data augmentation for vision-and-language navigation,” in Thirty-seventh Conference on Neural Information Processing Systems, 2023. [Online]. Available: https://openreview.net/forum?id= eKFrXWb0sT

    2023

  2. [9]

    Panogen: Text-conditioned panoramic environment generation for vision-and-language navigation,

    M. B. Jialu Li, “Panogen: Text-conditioned panoramic environment generation for vision-and-language navigation,” in arxiv, 2023

    2023

  3. [11]

    Think global, act local: Dual-scale graph transformer for vision-and-language navigation,

    S. Chen, P.-L. Guhur, M. Tapaswi, C. Schmid, and I. Laptev, “Think global, act local: Dual-scale graph transformer for vision-and-language navigation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 16 537–16 547

    2022

  4. [12]

    Stay on the path: Instruction fidelity in vision-and-language navigation,

    V . Jain, G. Magalhaes, A. Ku, A. Vaswani, E. Ie, and J. Baldridge, “Stay on the path: Instruction fidelity in vision-and-language navigation,” in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , A. Korhonen, D. Traum, and L. M `arquez,...

    2019

  5. [13]

    Reverie: Remote embodied visual referring expression in real indoor environments,

    Y . Qi, Q. Wu, P. Anderson, X. Wang, W. Y . Wang, C. Shen, and A. v. d. Hengel, “Reverie: Remote embodied visual referring expression in real indoor environments,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2020, pp. 9982–9991

    2020

  6. [14]

    Vision-and- dialog navigation,

    J. Thomason, M. Murray, M. Cakmak, and L. Zettlemoyer, “Vision-and- dialog navigation,” in Conference on Robot Learning . PMLR, 2020, pp. 394–406

    2020

  7. [15]

    A survey of embodied ai: From simulators to research tasks,

    J. Duan, S. Yu, H. L. Tan, H. Zhu, and C. Tan, “A survey of embodied ai: From simulators to research tasks,” IEEE Transactions on Emerging Topics in Computational Intelligence , vol. 6, no. 2, pp. 230–244, 2022

    2022

  8. [16]

    The regretful agent: Heuristic-aided navigation through progress estimation,

    C. Ma, Z. Wu, G. AlRegib, C. Xiong, and Z. Kira, “The regretful agent: Heuristic-aided navigation through progress estimation,” CoRR, vol. abs/1903.01602, 2019. [Online]. Available: http://arxiv.org/abs/ 1903.01602

    1903 arXiv

  9. [17]

    Contrastive instruction-trajectory learning for vision-language navigation,

    X. Liang, F. Zhu, Y . Zhu, B. Lin, B. Wang, and X. Liang, “Contrastive instruction-trajectory learning for vision-language navigation,” in Pro- ceedings of the AAAI Conference on Artificial Intelligence, vol. 36, no. 2, 2022, pp. 1592–1600

    2022

  10. [19]

    Reinforced cross-modal matching and self-supervised imitation learning for vision-language navigation,

    X. Wang, Q. Huang, A. Celikyilmaz, J. Gao, D. Shen, Y . Wang, W. Y . Wang, and L. Zhang, “Reinforced cross-modal matching and self-supervised imitation learning for vision-language navigation,” CoRR, vol. abs/1811.10092, 2018. [Online]. Available: http://arxiv.org/ abs/1811.10092

    2018 arXiv

  11. [22]

    History aware multimodal transformer for vision-and-language navigation,

    S. Chen, P. Guhur, C. Schmid, and I. Laptev, “History aware multimodal transformer for vision-and-language navigation,” CoRR, vol. abs/2110.13309, 2021. [Online]. Available: https://arxiv.org/abs/ 2110.13309

    2021 arXiv

  12. [25]

    Episodic transformer for vision-and-language navigation,

    A. Pashevich, C. Schmid, and C. Sun, “Episodic transformer for vision-and-language navigation,” CoRR, vol. abs/2105.06453, 2021. [Online]. Available: https://arxiv.org/abs/2105.06453

    2021 arXiv

  13. [26]

    Improving vision-and-language navigation with image-text pairs from the web,

    A. Majumdar, A. Shrivastava, S. Lee, P. Anderson, D. Parikh, and D. Batra, “Improving vision-and-language navigation with image-text pairs from the web,” CoRR, vol. abs/2004.14973, 2020. [Online]. Available: https://arxiv.org/abs/2004.14973

    2004 arXiv

  14. [27]

    Hop: History- and-order aware pretraining for vision-and-language navigation,

    Y . Qiao, Y . Qi, Y . Hong, Z. Yu, P. Wang, and Q. Wu, “Hop: History- and-order aware pretraining for vision-and-language navigation,” in2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 15 397–15 406

    2022

  15. [28]

    Geovln: Learning geometry-enhanced visual representation with slot attention for vision- and-language navigation,

    J. Huo, Q. Sun, B. Jiang, H. Lin, and Y . Fu, “Geovln: Learning geometry-enhanced visual representation with slot attention for vision- and-language navigation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2023, pp. 23 212...

    2023

  16. [29]

    Improving vision-and-language navigation by gen- erating future-view image semantics,

    J. Li and M. Bansal, “Improving vision-and-language navigation by gen- erating future-view image semantics,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 10 803–10 812

    2023

  17. [30]

    Bevbert: Multimodal map pre-training for language-guided navigation,

    D. An, Y . Qi, Y . Li, Y . Huang, L. Wang, T. Tan, and J. Shao, “Bevbert: Multimodal map pre-training for language-guided navigation,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 2737–2748

    2023

  18. [31]

    Target-driven structured transformer planner for vision-language navigation,

    Y . Zhao, J. Chen, C. Gao, W. Wang, L. Yang, H. Ren, H. Xia, and S. Liu, “Target-driven structured transformer planner for vision-language navigation,” in Proceedings of the 30th ACM International Conference on Multimedia, 2022, pp. 4194–4203

    2022

  19. [32]

    Etpnav: Evolving topological planning for vision-language navigation in continuous environments,

    D. An, H. Wang, W. Wang, Z. Wang, Y . Huang, K. He, and L. Wang, “Etpnav: Evolving topological planning for vision-language navigation in continuous environments,” IEEE Transactions on Pattern Analysis and Machine Intelligence , 2024

    2024

  20. [33]

    Gridmm: Grid memory map for vision-and-language navigation,

    Z. Wang, X. Li, J. Yang, Y . Liu, and S. Jiang, “Gridmm: Grid memory map for vision-and-language navigation,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 15 625–15 636

    2023

  21. [34]

    Grounded entity-landmark adaptive pre-training for vision-and-language naviga- tion,

    Y . Cui, L. Xie, Y . Zhang, M. Zhang, Y . Yan, and E. Yin, “Grounded entity-landmark adaptive pre-training for vision-and-language naviga- tion,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 12 043–12 053

    2023

  22. [35]

    V olumetric environment representation for vision-language navigation,

    R. Liu, W. Wang, and Y . Yang, “V olumetric environment representation for vision-language navigation,” in Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, 2024, pp. 16 317– 16 328

    2024

  23. [36]

    Lookahead exploration with neural radiance representation for con- tinuous vision-language navigation,

    Z. Wang, X. Li, J. Yang, Y . Liu, J. Hu, M. Jiang, and S. Jiang, “Lookahead exploration with neural radiance representation for con- tinuous vision-language navigation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 13 753–13 762

    2024

  24. [37]

    Kerm: Knowledge en- hanced reasoning for vision-and-language navigation,

    X. Li, Z. Wang, J. Yang, Y . Wang, and S. Jiang, “Kerm: Knowledge en- hanced reasoning for vision-and-language navigation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 2583–2592

    2023

  25. [38]

    Over-nav: Elevating itera- tive vision-and-language navigation with open-vocabulary detection and structured representation,

    G. Zhao, G. Li, W. Chen, and Y . Yu, “Over-nav: Elevating itera- tive vision-and-language navigation with open-vocabulary detection and structured representation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2024, pp. 16 2...

    2024

  26. [39]

    Pathdreamer: A world model for indoor navigation,

    J. Y . Koh, H. Lee, Y . Yang, J. Baldridge, and P. Anderson, “Pathdreamer: A world model for indoor navigation,” 2021

    2021

  27. [40]

    Learning vision-and-language navigation from youtube videos,

    K. Lin, P. Chen, D. Huang, T. H. Li, M. Tan, and C. Gan, “Learning vision-and-language navigation from youtube videos,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 8317–8326

    2023

  28. [41]

    Airbert: In- domain pretraining for vision-and-language navigation,

    P.-L. Guhur, M. Tapaswi, S. Chen, I. Laptev, and C. Schmid, “Airbert: In- domain pretraining for vision-and-language navigation,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021, pp. 1634–1643

    2021

  29. [42]

    Generative modeling by estimating gradients of the data distribution,

    Y . Song and S. Ermon, “Generative modeling by estimating gradients of the data distribution,” Advances in neural information processing systems, vol. 32, 2019

    2019

  30. [43]

    Raphael: Text-to-image generation via large mixture of diffusion paths,

    Z. Xue, G. Song, Q. Guo, B. Liu, Z. Zong, Y . Liu, and P. Luo, “Raphael: Text-to-image generation via large mixture of diffusion paths,”Advances in Neural Information Processing Systems , vol. 36, 2024

    2024

  31. [44]

    Repaint: Inpainting using denoising diffusion probabilistic models,

    A. Lugmayr, M. Danelljan, A. Romero, F. Yu, R. Timofte, and L. Van Gool, “Repaint: Inpainting using denoising diffusion probabilistic models,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2022, pp. 11 461–11 471

    2022

  32. [45]

    Latentpaint: Image inpainting in latent space with diffusion models,

    C. Corneanu, R. Gadde, and A. M. Martinez, “Latentpaint: Image inpainting in latent space with diffusion models,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , 2024, pp. 4334–4343

    2024

  33. [46]

    Exploiting diffusion prior for real-world image super-resolution,

    J. Wang, Z. Yue, S. Zhou, K. C. Chan, and C. C. Loy, “Exploiting diffusion prior for real-world image super-resolution,” International Journal of Computer Vision , pp. 1–21, 2024

    2024

  34. [47]

    Stable video diffusion: Scaling latent video diffusion models to large datasets,

    A. Blattmann, T. Dockhorn, S. Kulal, D. Mendelevitch, M. Kilian, D. Lorenz, Y . Levi, Z. English, V . V oleti, A. Lettset al., “Stable video diffusion: Scaling latent video diffusion models to large datasets,” arXiv preprint arXiv:2311.15127, 2023

    2023 arXiv

  35. [48]

    Videocomposer: Compositional video synthesis with motion controllability,

    X. Wang, H. Yuan, S. Zhang, D. Chen, J. Wang, Y . Zhang, Y . Shen, D. Zhao, and J. Zhou, “Videocomposer: Compositional video synthesis with motion controllability,” Advances in Neural Information Processing Systems, vol. 36, 2024

    2024

  36. [49]

    Dreamfusion: Text- to-3d using 2d diffusion,

    B. Poole, A. Jain, J. T. Barron, and B. Mildenhall, “Dreamfusion: Text- to-3d using 2d diffusion,” arXiv preprint arXiv:2209.14988 , 2022

    2022 arXiv

  37. [50]

    Magic3d: High-resolution text-to- 3d content creation,

    C.-H. Lin, J. Gao, L. Tang, T. Takikawa, X. Zeng, X. Huang, K. Kreis, S. Fidler, M.-Y . Liu, and T.-Y . Lin, “Magic3d: High-resolution text-to- 3d content creation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 300–309

    2023

  38. [51]

    Prolific- dreamer: High-fidelity and diverse text-to-3d generation with variational score distillation,

    Z. Wang, C. Lu, Y . Wang, F. Bao, C. Li, H. Su, and J. Zhu, “Prolific- dreamer: High-fidelity and diverse text-to-3d generation with variational score distillation,” Advances in Neural Information Processing Systems , vol. 36, 2024

    2024

  39. [52]

    High- resolution image synthesis with latent diffusion models,

    R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High- resolution image synthesis with latent diffusion models,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 10 684–10 695

    2022

  40. [54]

    Adding conditional control to text-to-image diffusion models,

    L. Zhang, A. Rao, and M. Agrawala, “Adding conditional control to text-to-image diffusion models,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 3836–3847

    2023

  41. [56]

    Gpt-4 technical report,

    O. OpenAI, “Gpt-4 technical report,” Mar 2023

    2023

  42. [57]

    mplug-2: A modularized multi-modal foundation model across text, image and video,

    H. Xu, Q. Ye, M. Yan, Y . Shi, J. Ye, Y . Xu, C. Li, B. Bi, Q. Qian, W. Wanget al., “mplug-2: A modularized multi-modal foundation model across text, image and video,” in International Conference on Machine Learning. PMLR, 2023, pp. 38 728–38 748

    2023

  43. [58]

    Learning to navigate unseen environments: Back translation with environmental dropout,

    H. Tan, L. Yu, and M. Bansal, “Learning to navigate unseen environments: Back translation with environmental dropout,” CoRR, vol. abs/1904.04195, 2019. [Online]. Available: http://arxiv.org/abs/ 1904.04195

    1904 arXiv

  44. [59]

    Towards learning a generic agent for vision-and-language navigation via pre-training,

    W. Hao, C. Li, X. Li, L. Carin, and J. Gao, “Towards learning a generic agent for vision-and-language navigation via pre-training,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 13 137–13 146

    2020

  45. [60]

    Vision- language navigation with random environmental mixup,

    C. Liu, F. Zhu, X. Chang, X. Liang, Z. Ge, and Y .-D. Shen, “Vision- language navigation with random environmental mixup,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021, pp. 1644–1654

    2021

  46. [61]

    Airbert: In-domain pretraining for vision-and-language navigation,

    P. Guhur, M. Tapaswi, S. Chen, I. Laptev, and C. Schmid, “Airbert: In-domain pretraining for vision-and-language navigation,” CoRR, vol. abs/2108.09105, 2021. [Online]. Available: https://arxiv.org/abs/2108. 09105

    2021 arXiv

  47. [62]

    On evalu- ation of embodied navigation agents,

    P. Anderson, A. Chang, D. S. Chaplot, A. Dosovitskiy, S. Gupta, V . Koltun, J. Kosecka, J. Malik, R. Mottaghi, M. Savvaet al., “On evalu- ation of embodied navigation agents,” arXiv preprint arXiv:1807.06757, 2018

    2018 arXiv

  48. [63]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al., “Learning transferable visual models from natural language supervision,” in International conference on machine learning . PMLR, 2021, pp. 8748–8763

    2021

  49. [64]

    Adding conditional control to text-to-image diffusion models,

    L. Zhang, A. Rao, and M. Agrawala, “Adding conditional control to text-to-image diffusion models,” in 2023 IEEE/CVF International Conference on Computer Vision (ICCV) , 2023, pp. 3813–3824

    2023

  50. [65]

    Sdxl: Improving latent diffusion models for high-resolution image synthesis,

    D. Podell, Z. English, K. Lacey, A. Blattmann, T. Dockhorn, J. M ¨uller, J. Penna, and R. Rombach, “Sdxl: Improving latent diffusion models for high-resolution image synthesis,” arXiv preprint arXiv:2307.01952 , 2023

    2023 arXiv

  51. [66]

    Metric3d v2: A versatile monocular geometric foundation model for zero-shot metric depth and surface normal estimation,

    M. Hu, W. Yin, C. Zhang, Z. Cai, X. Long, H. Chen, K. Wang, G. Yu, C. Shen, and S. Shen, “Metric3d v2: A versatile monocular geometric foundation model for zero-shot metric depth and surface normal estimation,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vo...

    2024

  52. [2018]

    Available: http://arxiv.org/abs/1806.02724

    [Online]. Available: http://arxiv.org/abs/1806.02724

    arXiv

  53. [2019]

    Available: http://arxiv.org/abs/1910.02029

    [Online]. Available: http://arxiv.org/abs/1910.02029

    1910 arXiv

  54. [2020]

    Available: https://arxiv.org/abs/2011.13922

    [Online]. Available: https://arxiv.org/abs/2011.13922

    2011 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.