Pith. sign in

REVIEW 4 major objections 4 minor 46 references

Transferable Representation Learning in Vision-and-Language Navigation

T0 review · 4 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Domain-adapted representations from two auxiliary tasks improve competitive navigation agents in Room-to-Room, with the transferred agent exceeding the previous state of the art by 5% absolute SPL.

desk verdict Novel auxiliary-task idea, but the paper's own ablations contradict the central transfer claim; needs matched controls before the SPL gain can be believed. read the letter →

arxiv 1908.03409 v2 pith:GQ2UEEBK submitted 2019-08-09 cs.CV cs.CLcs.LGcs.RO

Haoshuo Huang , Vihan Jain , Harsh Mehta , Alexander Ku , Gabriel Magalhaes , Jason Baldridge , Eugene Ie This is my paper · ORCID
classification cs.CVcs.CLcs.LGcs.RO
keywords vision-and-languagenavigationrepresentationlearningtransfercross-modalalignmentcontrastivepredictivecodingnegativeminingsuccessweightedbypathlength
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that adapting pre-trained vision and language representations to two auxiliary tasks, before any navigation training, makes them more effective for vision-and-language navigation agents, and that transferring them improves measured navigation efficiency. The central evidence is the ALTR agent, which warm-starts the Reinforced Cross-Modal agent with these representations and reports a success rate weighted by path length (SPL) of 45 on the Room-to-Room test set, five absolute points above the previous best published at the time. The same auxiliary model also scores machine-generated instruction-path pairs, so its alignment judgments can be used to keep only high-quality synthetic training data. If this is right, representation adaptation becomes a cheap, annotation-free step that can be layered onto existing embodied language agents.

What carries the argument

The load-bearing object is a two-tower sequence encoder trained on a combined auxiliary loss. One tower is a bidirectional LSTM over instruction tokens initialized with pre-trained word embeddings; the other is an attention-pooled LSTM over egocentric panoramic frames. The cross-modal alignment (CMA) head computes a softmax-then-softmin pooled score over the cross-modal similarity matrix, and the next-visual-scene (NVS) head uses an InfoNCE contrastive loss to predict the latent visual representation one or two steps ahead. Negative instruction-path pairs for both heads are mined without human annotation by path substitution, random walks, and partial reordering. The combined loss $\alpha L_{\text{alignment}} + (1-\alpha)L_{\text{coherence}}$ adapts both towers, and the adapted weights are transferred to warm-start the navigation agents.

What would settle it

Train the exact ALTR pipeline from random initialization with the same optimizer, learning rate schedule, and interleaved behavior-cloning and policy-gradient batches, and compare SPL on the Room-to-Room validation-unseen and test splits; if the scratch agent matches or exceeds ALTR, the transferred representations are not the cause of the reported gain.

Watch

Extended reading notes

Core claim

The paper's discovery is that a model trained jointly on cross-modal alignment (CMA) and next-visual-scene prediction (NVS) learns visual and textual representations that are better suited to the R2R navigation domain than the original pre-trained ones, and that initializing a navigation agent with them improves SPL. In the paper's own account, the ALTR agent, the Reinforced Cross-Modal agent warm-started from these representations, outperforms the known state of the art at the time by 5% absolute SPL on the test set while matching Success Rate and producing shorter paths. The auxiliary training also enables the model to rank machine-augmented instruction-path pairs, so using the top-ranked fraction improves validation performance relative to using the bottom-ranked fraction.

Load-bearing premise

The load-bearing premise is that the reported SPL improvement comes from the transferred representations, not from the different training schedule used for warm-started agents, since no no-pretraining version of the same ALTR pipeline is evaluated on the test set.

Editorial extensions

If this is right

  • Other navigation agents that reuse the same off-the-shelf encoders can be warm-started from these adapted representations without any additional human annotation.
  • The alignment scorer can filter noisy machine-generated instruction-path pairs, so synthetic data becomes more useful for training.
  • Joint CMA and NVS pre-training should produce shorter, more direct agent paths, improving SPL even when raw success rate is unchanged.
  • NVS alone is expected to be a poor pre-training task, while CMA alone helps less than the joint objective; the two losses are complementary.
  • The approach is complementary to other agent-level improvements, so later agents could stack it on top of their own training schemes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One testable extension is to apply the same two-task adaptation to other instruction-following domains with paired trajectories and language, such as outdoor navigation, as long as negative paths can be mined cheaply.
  • The alignment score could be reused online as a shaped reward or a curriculum signal during agent training, not just as a one-time filter over generated data.
  • A controlled comparison in which the scratch and warm-started agents use identical optimizer schedules and interleaving would isolate the contribution of the transferred representations from training dynamics.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. This paper introduces two self-supervised auxiliary tasks, Cross-Modal Alignment (CMA) and Next Visual Scene (NVS), designed to adapt pretrained vision and language representations to the R2R navigation domain without extra human annotations. The learned representations are subsequently used to warm-start the encoders of the Speaker-Follower (SF) and Reinforced Cross-Modal (RCM) navigation agents. The authors report that the resulting ALTR agent achieves SPL 45.0 on the R2R test set, 5 absolute percentage points above the previous state of the art, and attribute this improvement to the transfer of the domain-adapted representations. The paper also provides ablation studies on Validation Seen and Unseen, and an analysis of the auxiliary tasks' effectiveness in scoring generated instruction-path pairs.

Significance. If the central attribution claim were correct, the paper would make a valuable contribution: it would show that cheap, automatically mined discriminative auxiliary tasks can improve downstream VLN agents on unseen environments, and the leaderboard comparison is a useful benchmark result. The paper also offers a plausible methodology for ranking machine-generated instruction-path pairs and demonstrates that the combined CMA+NVS objective improves the CMA scoring model. However, the paper's own ablation tables contradict the transfer claim, and the training schedules are not matched across the compared conditions. As such, the significance of the work cannot currently be recognized beyond the auxiliary-task scoring results, and the main claim requires substantial additional evidence.

major comments (4)
  1. [7.5 / Tables 5 and 6] On Validation Unseen, the no-pretraining RCM baseline (no CMA, no NVS, no warm-start) attains SPL 43.0 and SR 46.1, whereas the jointly warm-started agent (CMA+NVS) attains only SPL 32.7 and SR 41.1. This is the opposite of the claim in Section 7.5 that 'agents benefit the most when initialized with representations learned on both the tasks simultaneously' and that joint pretraining yields 'a consistent 11-12% improvement in SR.' The tables as printed show that transferring these representations degrades generalization to unseen environments; the text and the data must be reconciled, or the tables corrected.
  2. [7.1 / learning rate] The experimental setup states that warm-started agents use a learning rate of 1e-5, while non-warm-started agents use 1e-4. This is a direct confound: the differences between warm-started and no-pretraining conditions in Tables 5 and 6 cannot be attributed to representation transfer alone. Moreover, the test-set comparison in Table 4 has no no-pretraining version of the exact ALTR pipeline, so the reported 5-point SPL gain over RCM could be caused by the different learning-rate schedule or other training-recipe changes rather than by the CMA/NVS representations.
  3. [6.2] The interleaved behavior-cloning and policy-gradient schedule with exponentially decaying K is introduced as an improvement to the RCM training recipe, but this modification is not isolated in the ablations. It is not stated whether the no-pretraining rows in Tables 5 and 6 used this interleaved schedule or the original RCM recipe. Without matched training schedules, the attribution of the test-set gain to the transferred representations is unsupported. The paper needs a matched baseline with identical learning rate, interleaving schedule, and data, but without pretraining, evaluated on both Validation Unseen and the test set.
  4. [7.4] The comparison with the R2R leaderboard reports only the warm-started ALTR agent, not a matched no-pretraining version of the same pipeline. Given that the paper's own Validation Unseen ablations show the no-pretraining agent outperforming the warm-started agent, the leaderboard result cannot be taken as evidence for the transfer hypothesis unless a matched no-pretraining run is also reported on the same test set.
minor comments (4)
  1. [Tables 5 and 6] The checkmark notation is ambiguous; the caption should explicitly state that a checkmark indicates the auxiliary task was included in pretraining and that the corresponding encoder was warm-started.
  2. [Equations (8)-(9)] The description of Eq. (8) as 'summing the columns' is unclear, and 'softmin' in Eq. (9) is not defined. The pooling procedure should be written out precisely.
  3. [Throughout] There are several typos, including 'pre-trainning' in Section 7.5, 'F ollowing' in reference [7], and 'ıncorrect' in Section 8. The manuscript would benefit from a proofreading pass.
  4. [Section 4 / related work] The relationship to the authors' prior work in [22] should be clarified: the reader should be told which components of the negative-mining and alignment model are inherited from that work and which are newly introduced here.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the transfer claim is empirically evaluated on held-out splits and an external leaderboard; the only self-citation is non-load-bearing.

full rationale

The paper's central claim is an empirical transfer-learning result: representations are adapted with auxiliary CMA/NVS tasks trained on R2R train data and then evaluated on previously unseen environments (Validation Unseen) and on the external R2R test leaderboard. No equation in the paper defines an output in terms of the quantity it is supposed to predict; in particular, the navigation SPL and SR numbers are measured on held-out environments, not reconstructed from the auxiliary-task losses. The negative-mining procedure cites the authors' prior work [22], but Section 4 fully specifies the three sampling strategies (PS, RW, PR) in the present paper, so the citation does not carry a load-bearing result whose validity is assumed from the authors themselves. The ablation results in Tables 5 and 6, where the no-pretraining RCM baseline reaches higher Validation Unseen SPL than warm-started variants, weaken the attribution of the test-set gain to the learned representations; however, this is an experimental confound (different learning rates, different training schedules, and a missing no-pretraining version of the exact ALTR pipeline), not a circularity. A confound or an unsupported causal attribution is a correctness risk, not a case in which a prediction reduces by construction to its inputs. Because no self-definitional step, fitted-input-called-prediction step, or self-citation chain forces the claimed outcome, the derivation chain is not circular.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim is an empirical transfer-learning claim, not a derivation. Main tunable quantities are loss weighting, predictive horizon, negative sampling mix, and the learning-rate schedule that differs between warm-started and baseline agents. No new physical or conceptual entities are introduced; the assumptions are dataset and training-signal assumptions.

free parameters (4)
  • alpha = 0.5
    Weights for combined loss alpha * L_alignment + (1-alpha) * L_coherence; chosen by hand in Sections 5.3 and 7.2 and affects both auxiliary tasks but not directly the SPL metric.
  • k = 1, 2
    Number of future steps for NVS/InfoNCE loss; chosen without a reported search in Section 5.2.
  • negative sampling counts = 10 negatives per positive (4 PS, 3 RW, 3 PR)
    Sampling mix for CMA and NVS negatives, chosen in Sections 4 and 7.2; alignment quality depends on the negative distribution.
  • auxiliary-to-agent learning rate ratio = 1e-5 with warm start vs 1e-4 without
    Learning rate during agent training is 10x lower when warm-started (Section 7.1); this confounds the transfer effect in ablations and test comparisons.
assumptions (4)
  • domain assumption Randomly sampled paths from the same environment are valid negatives for a given instruction.
    Section 4 states that given R2R's room and path diversity, original instructions are unlikely to match automatically mined negative paths; the auxiliary tasks rely on this. If false, CMA learns little.
  • domain assumption InfoNCE/CPC contrastive loss creates visual representations that transfer to navigation.
    Section 5.2 assumes predicting future latent visual states with InfoNCE yields useful features for downstream navigation; this is taken from prior work [36].
  • domain assumption Representations fine-tuned on auxiliary tasks can be transferred to navigation agents without requiring additional data.
    Section 6 assumes warm-started encoders improve the agent; the paper's ablations in Tables 5 and 6 test this and do not confirm it.
  • domain assumption SPL is the appropriate primary metric for ranking agents.
    Section 3 asserts SPL is best; changing the metric could change rankings.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Transferable Representation Learning in Vision-and-Language Navigation." pith.science (2026). https://pith.science/paper/GQ2UEEBK

@misc{pith2026190803409,
  author       = {Pith},
  title        = {Pith review of: Transferable Representation Learning in Vision-and-Language Navigation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GQ2UEEBK}},
  note         = {Machine review of arXiv:1908.03409}
}
read the original abstract

Vision-and-Language Navigation (VLN) tasks such as Room-to-Room (R2R) require machine agents to interpret natural language instructions and learn to act in visually realistic environments to achieve navigation goals. The overall task requires competence in several perception problems: successful agents combine spatio-temporal, vision and language understanding to produce appropriate action sequences. Our approach adapts pre-trained vision and language representations to relevant in-domain tasks making them more effective for VLN. Specifically, the representations are adapted to solve both a cross-modal sequence alignment and sequence coherence task. In the sequence alignment task, the model determines whether an instruction corresponds to a sequence of visual frames. In the sequence coherence task, the model determines whether the perceptual sequences are predictive sequentially in the instruction-conditioned latent space. By transferring the domain-adapted representations, we improve competitive agents in R2R as measured by the success rate weighted by path length (SPL) metric.

Figures

Figures reproduced from arXiv: 1908.03409 by the authors.

Figure 1
Figure 1. To overcome the scarcity of high-quality human [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Alignment matrix (Eq. 7) for model trained on the dataset containing (a) PS, PR, RW negatives (b) PS negatives only. Note that darker means higher alignment. the ones trained using low-quality examples. Note that the performance is low in both cases because none of the origi￾nal human-created instructions were used—what is impor￾tant is the relative performance between examples judged higher or lower. This clearly i… view at source ↗
Figure 3
Figure 3. Sample visualizations comparing reference paths [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

46 extracted references · 41 canonical work pages

  1. [1]

    Peter Anderson, Angel Chang, Devendra Singh Chap- lot, Alexey Dosovitskiy, Saurabh Gupta, Vladlen Koltun, Jana Kosecka, Jitendra Malik, Roozbeh Mot- taghi, Manolis Savva, and Amir R. Zamir. On evaluation of embodied navigation agents. 2018. arXiv:1807.06757 [cs.AI] . 2

    arXiv 2018

  2. [2]

    Vision-and- Language Navigation: Interpreting visually-grounded navigation instructions in real environments

    Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko S ¨underhauf, Ian Reid, Stephen Gould, and Anton van den Hengel. Vision-and- Language Navigation: Interpreting visually-grounded navigation instructions in real environments. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018. 1, 2, 5, 7

    work page 2018

  3. [3]

    Antol, A

    S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. L. Zitnick, and D. Parikh. VQA: Visual Question Answering. In 2015 IEEE International Conference on Computer Vision (ICCV), pages 2425–2433, Dec 2015. 2

    work page 2015

  4. [4]

    A framework for behavioural cloning

    Michael Bain and Claude Sammut. A framework for behavioural cloning. In Machine Intelligence 15, In- telligent Agents [St. Catherine’s College, Oxford, July 1995], pages 103–129, Oxford, UK, UK, 1999. Oxford University. 4

    work page 1995

  5. [5]

    Pre- diction, cognition and the brain

    Andreja Bubi, D Cramon, and Ricarda Schubotz. Pre- diction, cognition and the brain. Frontiers in human neuroscience, 4:25, 03 2010. 3

    work page 2010

  6. [6]

    Matterport3D: Learning from RGB-D data in indoor environments

    Angel Chang, Angela Dai, Thomas Funkhouser, Ma- ciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3D: Learning from RGB-D data in indoor environments. International Conference on 3D Vision (3DV), 2017. 2

    work page 2017

  7. [7]

    Fol- lowing formulaic map instructions in a street simu- lation environment

    V olkan Cirik, Yuan Zhang, and Jason Baldridge. Fol- lowing formulaic map instructions in a street simu- lation environment. In 2018 NeurIPS Workshop on Visually Grounded Interaction and Language, 2018. 1

    work page 2018

  8. [8]

    Learning Transferable Policies for Monocular Reactive MAV Control

    Shreyansh Daftry, J. Andrew Bagnell, and Martial Hebert. Learning transferable policies for monocular reactive MA V control.CoRR, abs/1608.00627, 2016. 4

    work page Pith review arXiv 2016

Show all 46 references
  1. [9]

    Moura, Devi Parikh, and Dhruv Batra

    Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, Jos´e M.F. Moura, Devi Parikh, and Dhruv Batra. Visual Dialog. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017. 2

    2017

  2. [10]

    Talk the Walk: Nav- igating New York City through Grounded Dialogue

    Harm de Vries, Kurt Shuster, Dhruv Batra, Devi Parikh, Jason Weston, and Douwe Kiela. Talk the Walk: Nav- igating New York City through Grounded Dialogue. CoRR, abs/1807.03367, 2018. 1

    2018 arXiv

  3. [11]

    Optimal perceived timing: Integrating sensory information with dynamically updated expectations

    Massimiliano Di Luca and Darren Rhodes. Optimal perceived timing: Integrating sensory information with dynamically updated expectations. Scientific reports, 6:28563, July 2016. 3

    2016

  4. [12]

    Donahue, L

    J. Donahue, L. A. Hendricks, M. Rohrbach, S. Venu- gopalan, S. Guadarrama, K. Saenko, and T. Darrell. Long-term recurrent convolutional networks for visual recognition and description. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(4):677– 691, April 2017. 2

    2017

  5. [13]

    H. Fang, S. Gupta, F. Iandola, R. K. Srivastava, L. Deng, P. Dollr, J. Gao, X. He, M. Mitchell, J. C. Platt, C. L. Zitnick, and G. Zweig. From captions to visual concepts and back. In 2015 IEEE Conference on Com- puter Vision and Pattern Recognition (CVPR), pages 1473–1482, Ju...

    2015

  6. [14]

    Speaker-follower models for vision- and-language navigation

    Daniel Fried, Ronghang Hu, V olkan Cirik, Anna Rohrbach, Jacob Andreas, Louis-Philippe Morency, Taylor Berg-Kirkpatrick, Kate Saenko, Dan Klein, and Trevor Darrell. Speaker-follower models for vision- and-language navigation. In Neural Information Pro- cessing Systems (NeurIPS...

    2018

  7. [15]

    End-to-end retrieval in continuous space

    Daniel Gillick, Alessandro Presta, and Gaurav Singh Tomar. End-to-end retrieval in continuous space. 2018. arXiv:1811.08008 [cs.IR] . 3

    2018 arXiv

  8. [16]

    Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik

    Ross B. Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In 2014 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2014, Columbus, OH, USA, June 23-28, 2014, pages 580–5...

    2014

  9. [17]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV , USA, June 27-30, 2016, pages 770–778, 2016. 1

    2016

  10. [18]

    Howard, Nicholas Roy, Anthony Stentz, and Matthew R

    Sachithra Hemachandra, Felix Duvallet, Thomas M. Howard, Nicholas Roy, Anthony Stentz, and Matthew R. Walter. Learning models for following natural language directions in unknown environments. In IEEE International Conference on Robotics and Au- tomation, ICRA 2015, Seattle, W...

    2015

  11. [19]

    Learning to follow directions in street view

    Karl Moritz Hermann, Mateusz Malinowski, Piotr Mirowski, Andras Banki-Horvath, and Raia Hadsell Keith Anderson. Learning to follow directions in street view. CoRR, abs/1903.00401, 2019. 1

    1903 arXiv

  12. [20]

    Long short- term memory

    Sepp Hochreiter and J¨urgen Schmidhuber. Long short- term memory. Neural Comput., 9(8):1735–1780, Nov

  13. [21]

    Segmentation from natural language expressions

    Ronghang Hu, Marcus Rohrbach, and Trevor Darrell. Segmentation from natural language expressions. In Computer Vision - ECCV 2016 - 14th European Con- ference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part I, pages 108–124, 2016. 2

    2016

  14. [22]

    Multi-modal discriminative model for vision-and-language navigation

    Haoshuo Huang, Vihan Jain, Harsh Mehta, Jason Baldridge, and Eugene Ie. Multi-modal discriminative model for vision-and-language navigation. In Proceed- ings of the Combined Workshop on Spatial Language Understanding (SpLU) and Grounded Communication for Robotics (RoboNLP), pa...

    2019

  15. [23]

    Deep visual-semantic alignments for generating image descriptions

    Andrej Karpathy and Fei-Fei Li. Deep visual-semantic alignments for generating image descriptions. InCVPR, pages 3128–3137. IEEE Computer Society, 2015. 2

    2015

  16. [24]

    Self-monitoring navigation agent via auxiliary progress estimation

    Chih-Yao Ma, Jiasen Lu, Zuxuan Wu, Ghassan Al- Regib, Zsolt Kira, Richard Socher, and Caiming Xiong. Self-monitoring navigation agent via auxiliary progress estimation. In Proceedings of the International Confer- ence on Learning Representations (ICLR), 2019. 7

    2019

  17. [25]

    The regretful agent: Heuristic- aided navigation through progress estimation

    Chih-Yao Ma, Zuxuan Wu, Ghassan AlRegib, Caiming Xiong, and Zsolt Kira. The regretful agent: Heuristic- aided navigation through progress estimation. 2019. 7

    2019

  18. [26]

    Walk the talk: Connecting language, knowl- edge, action in route instructions

    Matt MacMahon, Brian Stankiewicz, and Benjamin Kuipers. Walk the talk: Connecting language, knowl- edge, action in route instructions. In In Proc. of the Nat. Conf. on Artificial Intelligence (AAAI, pages 1475– 1482, 2006. 1

    2006

  19. [27]

    Yuille, and Kevin Murphy

    Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L. Yuille, and Kevin Murphy. Gener- ation and comprehension of unambiguous object de- scriptions. In CVPR, pages 11–20. IEEE Computer Society, 2016. 2

    2016

  20. [28]

    Grounded language learning: Where robotics and NLP meet

    Cynthia Matuszek. Grounded language learning: Where robotics and NLP meet. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18, pages 5687–5691. In- ternational Joint Conferences on Artificial Intelligence Organization, 7 2018. 1

    2018

  21. [29]

    Learning to nav- igate in cities without a map

    Piotr Mirowski, Matt Grimes, Mateusz Malinowski, Karl Moritz Hermann, Keith Anderson, Denis Teplyashin, Karen Simonyan, Koray Kavukcuoglu, An- drew Zisserman, and Raia Hadsell. Learning to nav- igate in cities without a map. In S. Bengio, H. Wal- lach, H. Larochelle, K. Grauma...

    2018

  22. [30]

    GloVe: Global vectors for word represen- tation

    Jeffrey Pennington, Richard Socher, and Christopher Manning. GloVe: Global vectors for word represen- tation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543. Association for Compu- tational Linguistics, 2014. 1, 3

    2014

  23. [31]

    Berg, and Li Fei-Fei

    Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexan- der C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (I...

    2015

  24. [32]

    Mike Schuster and Kuldip K. Paliwal. Bidirectional recurrent neural networks. IEEE Trans. Signal Process- ing, 45:2673–2681, 1997. 3

    1997

  25. [33]

    A survey of available corpora for building data-driven dialogue systems: The journal version

    Iulian Vlad Serban, Ryan Lowe, Peter Henderson, Lau- rent Charlin, and Joelle Pineau. A survey of available corpora for building data-driven dialogue systems: The journal version. D&D, 9(1):1–49, 2018. 3

    2018

  26. [34]

    Learning to navigate unseen environments: Back translation with environmental dropout

    Hao Tan, Licheng Yu, and Mohit Bansal. Learning to navigate unseen environments: Back translation with environmental dropout. In Proceedings of the 2019 Conference of the North American Chapter of the As- sociation for Computational Linguistics: Human Lan- guage Technologies, ...

    2019

  27. [35]

    Visual represen- tations for semantic target driven navigation

    Alexander Toshev, Arsalan Mousavian, James David- son, Jana Kosecka, and Marek Fiser. Visual represen- tations for semantic target driven navigation. 2018. 2

    2018

  28. [36]

    Rep- resentation learning with contrastive predictive coding

    A¨aron van den Oord, Yazhe Li, and Oriol Vinyals. Rep- resentation learning with contrastive predictive coding. CoRR, abs/1807.03748, 2018. 2, 3, 4

    2018 arXiv

  29. [37]

    Show and tell: A neural image caption generator

    Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: A neural image caption generator. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3156–3164,

    2015

  30. [38]

    Video captioning via hierar- chical reinforcement learning

    Xin Wang, Wenhu Chen, Jiawei Wu, Yuan-Fang Wang, and William Yang Wang. Video captioning via hierar- chical reinforcement learning. 2018 IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 4213–4222, 2018. 2

    2018

  31. [39]

    Reinforced cross-modal match- ing and self-supervised imitation learning for vision- language navigation

    Xin Wang, Qiuyuan Huang, AsliC ¸elikyilmaz, Jianfeng Gao, Dinghan Shen, Yuan-Fang Wang, William Yang Wang, and Lei Zhang. Reinforced cross-modal match- ing and self-supervised imitation learning for vision- language navigation. CoRR, abs/1811.10092, 2018. 1, 2, 4, 5, 7, 8

    2018 arXiv

  32. [40]

    Look before you leap: Bridg- ing model-free and model-based reinforcement learn- ing for planned-ahead vision-and-language navigation

    Xin Wang, Wenhan Xiong, Hongmin Wang, and William Yang Wang. Look before you leap: Bridg- ing model-free and model-based reinforcement learn- ing for planned-ahead vision-and-language navigation. In Vittorio Ferrari, Martial Hebert, Cristian Sminchis- escu, and Yair Weiss, edi...

    2018

  33. [41]

    Williams

    Ronald J. Williams. Simple statistical gradient- following algorithms for connectionist reinforcement learning. Machine Learning, 8(3):229–256, May 1992. 1, 4, 5

    1992

  34. [42]

    Courville, Ruslan Salakhutdinov, Richard S

    Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C. Courville, Ruslan Salakhutdinov, Richard S. Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In Proceedings of the 32nd International Conference on Machine Learning,...

    2015

  35. [43]

    Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, and Alexander J. Smola. Stacked attention networks for image question answering. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 21–29, 2016. 2

    2016

  36. [44]

    Video paragraph captioning using hierarchical recurrent neural networks

    Haonan Yu, Jiang Wang, Zhiheng Huang, Yi Yang, and Wei Xu. Video paragraph captioning using hierarchical recurrent neural networks. pages 4584–4593, 06 2016. 2

    2016

  37. [45]

    Zeiler and Rob Fergus

    Matthew D. Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In Computer Vi- sion - ECCV 2014 - 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part I, pages 818–833, 2014. 2

    2014

  38. [46]

    Lim, Abhinav Gupta, Li Fei-Fei, and Ali Farhadi

    Yuke Zhu, Roozbeh Mottaghi, Eric Kolve, Joseph J. Lim, Abhinav Gupta, Li Fei-Fei, and Ali Farhadi. Target-driven visual navigation in indoor scenes using deep reinforcement learning. In 2017 IEEE Interna- tional Conference on Robotics and Automation, ICRA 2017, Singapore, Sing...

    2017

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.