Walking with MIND: Mental Imagery eNhanceD Embodied QA

Fei Wu; Juncheng Li; Siliang Tang; Yueting Zhuang

REVIEW 4 major objections 5 minor 36 references

Walking with MIND: Mental Imagery eNhanceD Embodied QA

T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A mental imagery module that predicts future views as short-term subgoals improves an embodied question-answering agent's navigation and answer accuracy.

desk verdict A clearly written, honestly modest EmbodiedQA paper whose headline mechanism—mental images as subgoals—is plausible but never directly validated; worth refereeing, but the central claim needs stronger evidence. read the letter →

arxiv 1908.01482 v1 pith:ACI7VRQR submitted 2019-08-05 cs.CV cs.AI

Juncheng Li , Siliang Tang , Fei Wu , Yueting Zhuang This is my paper · ORCID

classification cs.CVcs.AI

keywords EmbodiedQuestionAnsweringMentalimagerybeta-VAEMixturedensitynetworkShort-termsubgoalsDeepreinforcementlearningVisualnavigation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that an embodied question-answering agent can better navigate and answer by explicitly generating mental images of what it would see after future actions. The proposed MIND module learns a compressed latent representation of egocentric views with a $\beta$-VAE, then an imagery model predicts the next latent state (and decodes it into a mental image) given the current latent and a candidate action. In the reinforcement learning framework, these mental images are treated as short-term subgoals, and a planned reward penalizes any mental image that does not increase the question-answering model's probability of the correct answer. The central claim is that this mental-imagery mechanism improves planning efficiency, generalization to unseen houses, and interpretability compared with agents that act directly from primitive actions, and the experiments on the EQA benchmark report larger navigation gains and higher answer accuracy.

What carries the argument

The central machinery is the MIND module: a $\beta$-VAE mental autoencoder whose decoder turns a sampled latent code into an interpretable mental image, paired with an imagery model—an LSTM whose output layer is a Mixture Density Network—that predicts the next latent mental representation $m'_{t+1}$ from the current latent, the agent's action, and the LSTM hidden state. The predicted latent is decoded into a mental image that is treated as a short-term subgoal; a planned reward in the A3C objective measures whether including that mental image increases the question-answering model's probability of the correct answer, thereby coupling the world model to the task objective.

What would settle it

Measure the imagery model's prediction error on held-out EQA trajectories: decode predicted latent codes at 1, 3, and 5 steps ahead and compare the resulting mental images with the actual frames the agent observes; if the error grows sharply and the images diverge within a few steps, the proposed subgoal mechanism is not supported.

Watch

Extended reading notes

Core claim

The core discovery is that treating predicted future observations as short-term subgoals improves embodied planning without needing annotated subgoal sequences. The MIND module first trains a mental autoencoder ($\beta$-VAE) to compress RGB frames into a disentangled latent code, then trains an imagery model (LSTM with a Mixture Density Network output) to predict the next latent code for each action; decoding this prediction yields a mental image. In the A3C fine-tuning stage, a planned reward compares the answer probability computed with and without the current mental image and rewards imagery that raises the correct answer's probability. The paper reports that MIND(BC+A3C) achieves better $d_\Delta$ at T-10 and T-30 and better QA accuracy at all tested distances compared with PACMAN and NMC, and that the planned reward contributes more to performance than the progressive distance reward.

Load-bearing premise

The entire benefit rests on the imagined future views being accurate enough to faithfully represent what the agent would actually see after a few actions; if those images are blurry or wrong, subgoal planning and the planned reward lose their grounding.

Editorial extensions

If this is right

The MIND agent can plan in terms of learned subgoals without needing externally annotated subgoal sequences, unlike NMC which requires such annotations.
Because the imagery model is pretrained on expert demonstrations and models only environment dynamics, it can transfer to new scenes with the same dynamics, improving sample efficiency when few demonstrations are available.
The generated mental images give a human-readable visualization of the agent's short-term intentions, enabling real-time behavioral interpretation and potential human correction.
The planned reward mechanism is a general template: any embodied task with a differentiable or learnable evaluator of progress can use imagined future states to shape its policy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the mental-image fidelity is the real cause of the reported gains, then measuring prediction error on held-out trajectories would let practitioners predict how the benefit scales with horizon; the paper does not report such a measurement.
The planned reward could be applied as a self-supervised signal to train the imagery model itself, using decoded imagined images to query the answer model and improve prediction beyond the expert-demonstration pretraining.
The interpretability claim suggests a direct extension: showing the mental images to a human operator before execution could support human-in-the-loop correction in real navigation tasks.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Here's my take on arXiv:1908.01482. It's a well-written, modest EmbodiedQA paper. The MIND module—beta-VAE encoder, LSTM-MDN prediction model, and a planned reward that treats predicted frames as subgoals—is a new combination for this task, and the authors describe it clearly. The ablation shows the planned reward contributes, and the few-shot generalization experiment is a nice addition. That part deserves credit.

The soft spots are real and concentrated around the central claim. The paper never quantitatively validates the imagery model: no reconstruction or prediction error, no horizon analysis, no disentanglement metric. That makes it hard to attribute the gains to accurate mental images rather than to the extra LSTM features or the answer-compatibility reward. The planned reward in Eq. 8 directly rewards images that increase the QA model's probability of the ground-truth answer, so task-relatedness is partially built into the objective. The stress-test note is right: the mental-image-as-subgoal mechanism is not actually demonstrated.

The experimental evidence is also thinner than the text suggests. No error bars or significance tests; the margins are small, and at T-50 NMC beats MIND on navigation (1.70 vs 1.65). The Blindfold baseline shows that dataset bias explains a large share of QA accuracy, and the authors acknowledge Wu et al.'s strong simple baseline but do not compare against it. The horizon mismatch between one-step training and multi-step unrolling is unexamined, and code is not released. These are not fatal flaws—the paper is still a reasonable empirical contribution—but they do prevent acceptance as-is.

Who is this for? Researchers working on embodied QA, vision-and-language navigation, or world-model-based RL. They will get a clear, readable introduction to one way of adding a predictive module to a navigator, plus some useful ablation results. I would not cite it as evidence for the mental-imagery mechanism, but I would cite it as an early attempt at subgoal-style planning with predicted latents.

My recommendation: send it to peer review. The core question—does a task-conditioned predictive model help embodied agents?—is worth serious attention, and the paper is honest about some of its limitations. I would ask for code, error bars, a comparison to Wu et al., and some direct evaluation of the imagery model (even a few reconstruction examples against ground truth). With those, it could be a solid contribution.

Referee Report

4 major / 5 minor

Summary. The paper proposes a Mental Imagery eNhanceD (MIND) module for EmbodiedQA, consisting of a beta-VAE-based mental autoencoder and an LSTM+MDN imagery model that predicts the agent's future mental representation after an action. The predicted mental image is decoded and fed to a PACMAN-style planner, and the agent is trained with behavior cloning followed by A3C using three rewards: final, progressive, and planned. The central claim is that mental images act as reusable short-term subgoals, yielding better planning efficiency, generalization, and interpretability. The main experiments compare MIND(BC+A3C) with PACMAN, NMC, and a Blindfold baseline on the EQA test set, reporting navigation dDelta and QA accuracy at three initial distances, plus ablations and a generalization study.

Significance. If fully validated, the idea of modeling mental imagery as a latent-space future prediction and using it to define subgoals is valuable and timely: it offers a path to more interpretable embodied agents without manually annotated subgoals. The paper is creditably explicit about its module architecture and includes ablation and generalization experiments. However, the empirical support is currently incomplete: the imagery model's prediction quality is never measured, the planned reward directly optimizes task-relatedness and therefore cannot serve as evidence for predictive accuracy, and the reported improvements lack error bars and significance tests. These are load-bearing gaps because the paper's interpretation of mental images as accurate short-term subgoals, rather than merely useful policy biases, is not supported by the present evidence.

major comments (4)

[Section 3.3, Eq. (2), Fig. 4] The imagery model is defined in Eq. (2) and Fig. 3 as a one-step predictor of m'_{t+1} given m_t and a_t, but Section 3.3 states that m'_{t+1} is 'a further outcome of several actions' and Fig. 4 unrolls the model over multiple planner steps. The number of actions in this horizon is never specified, and no training loss is given for multi-step unrolling. If the model is trained one-step and unrolled several steps at decision time, prediction error accumulation is unexamined. The authors should specify the prediction horizon k, describe how multi-step training is performed (if any), and report quantitative prediction quality (e.g., MSE or SSIM against the actual observations at the corresponding future states) as a function of the horizon.
[Section 4.3, Eq. (8)] The planned reward rm in Eq. (8) is defined as the improvement in Pa(o*|..., M_t) over Pa(o*|...), where o* is the ground-truth answer. Because the RL objective maximizes this reward, the MIND module is explicitly trained to produce any mental image that increases the QA model's probability of the correct answer. Consequently, the paper's later claim that the generated mental images are 'task-related' and 'objective-related' is partly enforced by the reward definition and is not independent evidence that M_t predicts the actual future observation. The case study in Fig. 7 does not compare the mental images with the real observations at the corresponding future positions. The authors should provide such comparisons, both at the pixel level and at the semantic level, to support the central interpretation of mental images as short-term subgoals.
[Section 5.4, Table 1] Table 1 reports single-run results without error bars, confidence intervals, or significance tests. Several of the claimed improvements over the strongest baselines are very small: for QA accuracy, 54.83% vs 53.58% at T-10, 46.71% vs 46.21% at T-30, and 44.56% vs 44.32% at T-50 against NMC(BC+A3C); for navigation dDelta at T-50, MIND's 1.65 is lower than NMC's 1.70, which the text acknowledges. The claim that MIND is superior across the board is therefore not statistically established. The authors should run at least five independent seeds, report means and standard deviations, and include a significance test or an effect-size measure.
[Section 5.4, Generalizability & Convergence Speed] The generalization experiment compares MIND with PACMAN when both are trained on subsets of the validation data, but the MIND module is pretrained on the full training set, whereas PACMAN receives no comparable pretrained environment model or observation encoder. This is a confound: the improved performance at small data sizes shown in Fig. 8 could reflect the additional pretraining data rather than the proposed imagery mechanism. A matched comparison should pretrain an equally expressive module for PACMAN, or train MIND without its pretrained module, to isolate the effect of the MIND idea.

minor comments (5)

[Section 5.3] Key hyperparameters are missing: the beta-VAE coefficient beta, the planned-reward weight lambda_f, and the imagery prediction horizon k are not reported. Without these values the experiments are not reproducible.
[Section 3.1, Eq. (1)] The notation is inconsistent: the text says the encoder outputs vectors mu and delta and the latent is sampled from N(mu, delta), but the equation and Fig. 2 suggest a diagonal Gaussian with parameters mu and sigma. Please use consistent notation, e.g., N(mu, sigma^2 I) with sigma denoting the standard deviation.
[Section 4.3, Eq. (8)] The index notation in Eq. (8) is confusing: the text refers to 'the last four frames I^0_t, ..., I^{-4}_{t-1}' and 'the last five frames,' but the expression I^{-3}_{t-1} is used for the third-last frame. Please clarify the time indexing so the reader can determine exactly which frames are used in each probability term.
[Section 5.4, Case study] The text says 'From the image (d) in Figure 5' but the referenced case-study images appear in Figure 7, not Figure 5. Please correct the cross-reference.
[Throughout] There are many typographical and encoding errors: 'signiïňĄcantly', 'desigh', 'eïňĂect', 'thress agents', 'a few trails' (should be 'trials'), 'fro mObservations' in the reference, and several garbled Unicode quotes. A careful proofreading pass is needed.

Circularity Check

1 steps flagged · score 4.0 of 10

Planned reward makes 'task-related mental imagery' true by construction, but end-to-end performance gains are independently measured.

self definitional [Section 4.3, Eq. 8 and following text]
"Let Pa(o∗|...) denote the probability of the correct answer produced by the question-answering model, and o∗ is the correct answer among 172 candidates. The planned reward is written as: rm(st , at) = Pa(o∗|I−3 t−1, ..., I 0 t , Mt)− Pa(o∗|I−4 t−1, ..., I 0 t) (8) ... With the mental image, if the probability of the correct answer increases, it means that our MIND agent forms a task-related and objective-related short-term subgoals."

Eq. 8 conditions on the ground-truth answer o* and defines r_m as the increase in the QA model's probability of o* when the mental image M_t is appended to the last four frames. The RL objective (Eq. 10) maximizes total reward including r_m, so after A3C fine-tuning any increase in 'task-relatedness' of M_t is enforced by the training signal, not observed independently. The later claims that planned reward 'encourages our MIND module to generate more task-related imagery' and that mental images are short-term subgoals therefore restate the optimization objective. Because M_t is never compared to the actual future observation, the reward can be satisfied by any image that nudges the pretrained QA model toward o*, whether or not it is a predictive subgoal.

full rationale

The central end-to-end results (dDelta and QA accuracy in Table 1, and the few-shot generalization curves) are genuine empirical comparisons against PACMAN and NMC; those numbers are not produced by the planned-reward definition. No load-bearing self-citation or imported uniqueness theorem is present. The main circular element is the paper's interpretation of mental images as task-related short-term subgoals: that property is built into Eq. 8 by rewarding increases in P_a(o*|..., M_t), so the conclusions drawn in Section 5.4 ('the planned reward encourages ... more task-related imagery', and the gain from r_m) are partly true by construction. The paper also provides no reconstruction/prediction error or horizon analysis for the imagery model, but that is a missing-validation issue rather than a circular-derivation issue. On balance, one qualitative claim is enforced by the reward definition, while the main performance claims retain independent content, giving a moderate score.

Assumptions & free parameters 5 free parameters · 5 assumptions · 3 invented entities

The central method rests on standard deep learning machinery (VAE, LSTM, A3C) plus domain assumptions that expert demonstrations and the QA model's probabilities are reliable training signals. The main hand-chosen quantities are the beta-VAE beta, latent dimension, MDN components, imagery horizon, and final reward weight. The invented constructs are the mental representation, mental image, and planned reward; none has independent evidence outside the paper's own framework.

free parameters (5)

beta-VAE coefficient beta = not reported
Controls the strength of disentanglement in the mental encoder; the paper claims disentangled factors such as room direction and scale but never reports beta, so the interpretability of mental representations is not reproducible.
Imagery prediction horizon k = not reported
Section 3.3 says the model predicts 'a further outcome of several actions' instead of one atomic action, but no k is given, and Eq. 8 does not specify how many steps ahead the mental image is predicted.
Number of Gaussians in MDN and LSTM hidden units = 5 Gaussians, 512 hidden units
Reported as selected after exploring several structures; these hand-tuned hyperparameters affect prediction quality and are part of the method's configuration.
lambda_f in final reward = not reported
Balances correctness and navigation efficiency in Eq. 6; no value is given, so the reward used in training is not fully specified.
Latent dimension of beta-VAE = 128
The latent space dimension is set to 128; the central claim that mental representations are useful for planning depends on this choice.

assumptions (5)

domain assumption A beta-VAE with a standard Gaussian prior yields a disentangled latent space where dimensions correspond to interpretable generative factors.
Section 3.1 claims disentanglement (room direction, scale) without quantitative evidence or comparison to other VAE variants.
domain assumption Expert shortest-path demonstrations provide sufficient coverage to pretrain a mental encoder and imagery model that transfer to new environments.
Section 3.3 trains MIND on these trajectories; if the demonstrations are unrepresentative, the learned dynamics will not transfer.
domain assumption The question-answering model's probability for the correct answer is a reliable training signal for subgoal usefulness.
Eq. 8 defines the planned reward as the change in Pa(o*|...) when the mental image is included, assuming the QA model is well calibrated enough for reward shaping.
domain assumption The House3D simulator environment dynamics are deterministic and consistent with the imagery model's sequence model.
Section 4 assumes primitive actions such as forward and turn have fixed, deterministic effects, which the imagery model learns from trajectories.
standard math Standard deep RL and VAE machinery (reparameterization, A3C, GAE) is valid and correctly implemented.
Used without proof in Sections 3 and 4 as background methodology.

invented entities (3)

Mental representation m_t sampled by the beta-VAE encoder
purpose: Compressed latent state of the observation used by the imagery model and planner.
No external measurement validates that the latent dimensions correspond to the claimed generative factors; only reconstructions are shown.
Mental image M_t produced by the decoder
purpose: Predicted future frame used as a short-term subgoal and for interpretability.
The paper labels these images as subgoals and intentions, but that interpretation is only illustrated in a case study, not measured.
Planned reward r_m
purpose: Training signal that rewards mental images that raise the QA model's probability of the correct answer.
This objective is defined by the authors using ground-truth answers; it enforces task-relatedness by construction.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Walking with MIND: Mental Imagery eNhanceD Embodied QA." pith.science (2026). https://pith.science/paper/ACI7VRQR

@misc{pith2026190801482,
  author       = {Pith},
  title        = {Pith review of: Walking with MIND: Mental Imagery eNhanceD Embodied QA},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ACI7VRQR}},
  note         = {Machine review of arXiv:1908.01482}
}

read the original abstract

The EmbodiedQA is a task of training an embodied agent by intelligently navigating in a simulated environment and gathering visual information to answer questions. Existing approaches fail to explicitly model the mental imagery function of the agent, while the mental imagery is crucial to embodied cognition, and has a close relation to many high-level meta-skills such as generalization and interpretation. In this paper, we propose a novel Mental Imagery eNhanceD (MIND) module for the embodied agent, as well as a relevant deep reinforcement framework for training. The MIND module can not only model the dynamics of the environment (e.g. 'what might happen if the agent passes through a door') but also help the agent to create a better understanding of the environment (e.g. 'The refrigerator is usually in the kitchen'). Such knowledge makes the agent a faster and better learner in locating a feasible policy with only a few trails. Furthermore, the MIND module can generate mental images that are treated as short-term subgoals by our proposed deep reinforcement framework. These mental images facilitate policy learning since short-term subgoals are easy to achieve and reusable. This yields better planning efficiency than other algorithms that learn a policy directly from primitive actions. Finally, the mental images visualize the agent's intentions in a way that human can understand, and this endows our agent's actions with more interpretability. The experimental results and further analysis prove that the agent with the MIND module is superior to its counterparts not only in EQA performance but in many other aspects such as route planning, behavioral interpretation, and the ability to generalize from a few examples.

Figures

Figures reproduced from arXiv: 1908.01482 by the authors.

**Figure 1.** An example of Embodied QA task. KEYWORDS Embodied Question-Answering; Cross-Media; Multi-Modal Understanding; Vision-and-Language Navigation; ACM Reference Format: Juncheng Li, Siliang Tang, Fei Wu, and Yueting Zhuang. 2019. Walking with MIND: Mental Imagery eNhanceD Embodied QA. In Proceedings of the 27th ACM International Conference on Multimedia (MM ’19), October 21–25, 2019, Nice, France. ACM, New York, NY, USA… view at source ↗

**Figure 2.** The encoder outputs low dimension vectors [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** An example of the MIND Module. Mental Encoder [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Figures from the paper (5 more)

**Figure 4.** Figure 4: The overview of the MIND agent. Before deciding an action to execute, it predicts some short-term subgoals in the [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

**Figure 5.** Figure 5: Results of MIND agent with different reward. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

**Figure 6.** Figure 6: Learning curves of MIND and PACMAN agent. [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]

**Figure 7.** Figure 7: Example trajectories executed by PACMAN, MIND [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]

**Figure 8.** Figure 8: The navigation performance d∆ and answering accuracy at T−10,T−30,T−50. Planning Efficiency & Behavioral Interpretation: To demonstrate our method’s superior performance on route planning and behavioral interpretation, we carry out a case study. As shown in [PITH_FU…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

36 extracted references · 22 canonical work pages

[1]

Ankesh Anand, Eugene Belilovsky, Kyle Kastner, Hugo Larochelle, and Aaron Courville. 2018. Blindfold baselines for embodied qa. arXiv preprint arXiv:1811.05013 (2018)

arXiv 2018
[2]

Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sun- derhauf, Ian Reid, Stephen Gould, and Anton van den Hengel. 2018. Vision-and- Language Navigation: Interpreting Visually-Grounded Navigation Instructions in Real Environments. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (Jun 2018). https://doi.org/10.1109/cvpr...

arXiv 2018
[3]

Jacob Andreas, Dan Klein, and Sergey Levine. 2017. Modular multitask rein- forcement learning with policy sketches. In Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org, 166–175

work page 2017
[4]

1994.Mixture density networks

Christopher M Bishop. 1994.Mixture density networks. Technical Report. Citeseer

work page 1994
[5]

Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guil- laume Desjardins, and Alexander Lerchner

Christopher P. Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guil- laume Desjardins, and Alexander Lerchner. 2018. Understanding disentangling in Îš-VAE. arXiv:cs.LG/1804.03599

arXiv 2018
[6]

Devendra Singh Chaplot, Kanthashree Mysore Sathyendra, Rama Kumar Pa- sumarthi, Dheeraj Rajagopal, and Ruslan Salakhutdinov. 2017. Gated-Attention Architectures for Task-Oriented Language Grounding. arXiv:cs.LG/1706.07230

work page Pith review arXiv 2017
[7]

Chen and Raymond J

David L. Chen and Raymond J. Mooney. 2011. Learning to Interpret Natural Language Navigation Instructions fro mObservations. In Proceedings of the 25th AAAI Conference on Artificial Intelligence (AAAI-2011) . San Francisco, CA, USA

work page 2011
[8]

Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. 2018. Embodied Question Answering. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

work page 2018

Show all 36 references

[9]

Abhishek Das, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra
[10]

Ronald A Finke. 1989. Principles of mental imagery. The MIT Press

1989
[11]

Daniel Gordon, Aniruddha Kembhavi, Mohammad Rastegari, Joseph Redmon, Dieter Fox, and Ali Farhadi. 2018. Iqa: Visual question answering in interactive environments. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4089–4098

2018
[12]

David Ha and JÃĳrgen Schmidhuber. 2018. World Models. arXiv:cs.LG/1803.10122

2018 arXiv
[13]

Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. 2017. beta-vae: Learning basic visual concepts with a constrained variational framework. In International Conference on Learning Representation...

2017
[14]

Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory.Neural computation 9, 8 (1997), 1735–1780

1997
[15]

Philip N Johnson-Laird. 1995. Mental models, deductive reasoning, and the brain. The cognitive neurosciences 65 (1995), 999–1008

1995
[16]

Natalie Jones, Helen Ross, Timothy Lynam, Pascal Perez, and Anne Leitch. 2011. Mental models: an interdisciplinary synthesis of theory and methods. (2011)

2011
[17]

Diederik P Kingma and Max Welling. 2013. Auto-Encoding Variational Bayes. arXiv:cs.LG/1312.6114

2013 arXiv
[18]

Eric Kolve, Roozbeh Mottaghi, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. 2017. Ai2-thor: An interactive 3d environment for visual ai. arXiv preprint arXiv:1712.05474 (2017)

2017 arXiv
[19]

SM Kosslyn and Zenon Pylyshyn. 1994. Image and brain: The resolution of the imagery debate. Nature 372, 6503 (1994), 289–289

1994
[20]

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classifica- tion with deep convolutional neural networks. In Advances in neural information processing systems. 1097–1105

2012
[21]

Matt Macmahon, Brian Stankiewicz, and Benjamin Kuipers. 2006. Walk the Talk: Connecting Language, Knowledge, Action in Route Instructions. In In Proc. of the Nat. Conf. on Artificial Intelligence (AAAI . 1475–1482

2006
[22]

Hongyuan Mei, Mohit Bansal, and Matthew R. Walter. 2015. Listen, Attend, and Walk: Neural Mapping of Navigational Instructions to Action Sequences. arXiv:cs.CL/1506.04089

2015 arXiv
[23]

Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Tim- othy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. 2016. Asyn- chronous methods for deep reinforcement learning. In International conference on machine learning. 1928–1937

2016
[24]

Junhyuk Oh, Satinder Singh, Honglak Lee, and Pushmeet Kohli. 2017. Zero-shot task generalization with multi-task deep reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning-Volume 70 . JMLR. org, 2661–2670

2017
[25]

John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel
[26]

Chen Tessler, Shahar Givony, Tom Zahavy, Daniel J Mankowitz, and Shie Mannor
[27]

Xin Wang, Qiuyuan Huang, Asli Celikyilmaz, Jianfeng Gao, Dinghan Shen, Yuan- Fang Wang, William Yang Wang, and Lei Zhang. 2018. Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Naviga- tion. arXiv preprint arXiv:1811.10092 (2018)

2018 arXiv
[28]

Ronald J Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning 8, 3-4 (1992), 229–256

1992
[29]

Margaret Wilson. 2002. Six views of embodied cognition. Psychonomic bulletin & review 9, 4 (2002), 625–636

2002
[30]

Terry Winograd. 1971. Procedures as a representation for data in a computer pro- gram for understanding natural language . Technical Report. MASSACHUSETTS INST OF TECH CAMBRIDGE PROJECT MAC

1971
[31]

Yu Wu, Lu Jiang, and Yi Yang. 2019. Revisiting EmbodiedQA: A Simple Baseline and Beyond. arXiv:cs.CV/1904.04166

2019 arXiv
[32]

Yi Wu, Yuxin Wu, Georgia Gkioxari, and Yuandong Tian. 2018. Building General- izable Agents with a Realistic and Rich 3D Environment. arXiv:cs.LG/1801.02209

2018 arXiv
[33]

Edward N Zalta, Uri Nodelman, Colin Allen, and John Perry. 2003. Stanford encyclopedia of philosophy

2003
[2015]

arXiv preprint arXiv:1506.02438 (2015)

High-dimensional continuous control using generalized advantage estima- tion. arXiv preprint arXiv:1506.02438 (2015)

2015 arXiv
[2017]

In Thirty- First AAAI Conference on Artificial Intelligence

A deep hierarchical approach to lifelong learning in minecraft. In Thirty- First AAAI Conference on Artificial Intelligence
[2018]

arXiv preprint arXiv:1810.11181 (2018)

Neural modular control for embodied question answering. arXiv preprint arXiv:1810.11181 (2018)

2018 arXiv

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

Ankesh Anand, Eugene Belilovsky, Kyle Kastner, Hugo Larochelle, and Aaron Courville. 2018. Blindfold baselines for embodied qa. arXiv preprint arXiv:1811.05013 (2018)

arXiv 2018

[2] [2]

Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sun- derhauf, Ian Reid, Stephen Gould, and Anton van den Hengel. 2018. Vision-and- Language Navigation: Interpreting Visually-Grounded Navigation Instructions in Real Environments. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (Jun 2018). https://doi.org/10.1109/cvpr...

arXiv 2018

[3] [3]

Jacob Andreas, Dan Klein, and Sergey Levine. 2017. Modular multitask rein- forcement learning with policy sketches. In Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org, 166–175

work page 2017

[4] [4]

1994.Mixture density networks

Christopher M Bishop. 1994.Mixture density networks. Technical Report. Citeseer

work page 1994

[5] [5]

Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guil- laume Desjardins, and Alexander Lerchner

Christopher P. Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guil- laume Desjardins, and Alexander Lerchner. 2018. Understanding disentangling in Îš-VAE. arXiv:cs.LG/1804.03599

arXiv 2018

[6] [6]

Devendra Singh Chaplot, Kanthashree Mysore Sathyendra, Rama Kumar Pa- sumarthi, Dheeraj Rajagopal, and Ruslan Salakhutdinov. 2017. Gated-Attention Architectures for Task-Oriented Language Grounding. arXiv:cs.LG/1706.07230

work page Pith review arXiv 2017

[7] [7]

Chen and Raymond J

David L. Chen and Raymond J. Mooney. 2011. Learning to Interpret Natural Language Navigation Instructions fro mObservations. In Proceedings of the 25th AAAI Conference on Artificial Intelligence (AAAI-2011) . San Francisco, CA, USA

work page 2011

[8] [8]

Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. 2018. Embodied Question Answering. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

work page 2018