Pith. sign in

REVIEW 4 major objections 5 minor 37 references

What to Do Next? Memorizing skills from Egocentric Instructional Video

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read TAMFormer claims that encoding past successes as a topological affordance memory lets an egocentric agent decide what to do next, and that a goal-association check plus a replan step restores planning when actions deviate.

desk verdict New task and a plausible memory-based planner, but the headline robustness result rests on an under-specified replanning algorithm. read the letter →

arxiv 2507.02997 v1 pith:ATJCPBP5 submitted 2025-07-01 cs.LG cs.AIcs.CV

Jing Bi , Chenliang Xu This is my paper
classification cs.LGcs.AIcs.CV
keywords interactiveactionplanningegocentricinstructionalvideotopologicalaffordancememorylearningcontrastivetransformergenerationreplanningimitation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is trying to establish that an agent can learn high-level household task planning from egocentric instructional videos by memorizing which actions past successful demonstrations afford in each situation, rather than by modeling rewards or full scene graphs. It introduces Interactive Action Planning, a task setting in which the agent is trained offline on expert demonstrations but evaluated online in an interactive simulator that gives visual feedback only after an action is executed. The central claim is that the proposed Topological Affordance Memory, combined with a transformer action generator, lets the agent retrieve relevant past experiences and keep planning even when executed actions deviate from the plan. If this holds, visual task planning becomes more robust in realistic settings where mistakes happen, and can be learned without hand-designed rewards or complete state information.

What carries the argument

The load-bearing object is the Topological Affordance Memory (TAM), a graph of learned nodes built from expert demonstrations, where each node's key comes from a localization network trained to tell whether two frames are temporally close and its values come from two learned functions: an affordance projection trained with supervised InfoNCE contrastive loss to pull together observations that afford the same action, and a goal-association discriminator trained with binary cross-entropy to judge whether two nodes serve the same stated goal. The action generator is a transformer decoder that attends to retrieved memory nodes, conditioned on goal and action history. The replanning algorithm treats the localized memory node as an input feature and iteratively perturbs it, using the gradient of a loss with respect to the node, until the goal-association output with the previous node passes a threshold, yielding an alternative memory to drive subsequent action prediction.

What would settle it

Take the trained model and force a sequence of random actions that moves the agent into an out-of-distribution state of the simulator, then compare plan quality with replanning enabled versus disabled; if the goal-association output on such states is near chance or the replanned actions do not improve goal completion, the recovery claim collapses.

Watch

Extended reading notes

Core claim

The paper's central discovery is that encoding past demonstrations as a graph whose nodes associate visual observations with environment affordances, and whose keys are learned by a temporal-closeness localization network, gives a planning agent a stable substrate for deciding what to do next. On top of this memory, the paper builds an action generator that retrieves nearby memories and decodes the next action with a transformer conditioned on the goal and action history. The paper further claims that a goal-association discriminator, checked against the localized memory node, can detect when execution has drifted, and that an iterative adversarial-style update of the localized node—treated like auto-correcting a sentence—produces a valid alternative plan. Experiments in a photorealistic interactive household simulator are presented as evidence that this combined system outperforms prior planning approaches and, unlike them, remains effective when actions are randomly permuted.

Load-bearing premise

The plan's recovery mechanism assumes that the goal-association discriminator, trained only on clean expert demonstrations, still judges correctly when random action permutations push the agent into states it has never seen; if that assumption fails, replanning can retrieve irrelevant memories and hurt performance.

Editorial extensions

If this is right

  • Storing successful experiences as affordance-tagged graph nodes should let a planner transfer knowledge across episodes that share the same scene affordances, even when the exact trajectories differ.
  • A learned localization network should retrieve useful memories under visual variation better than raw pixel similarity, because pixel distances are too noisy for distinguishing similar-looking locations.
  • Goal-association checking and replanning should let the agent recover from execution errors, and removing either component should degrade interactive performance.
  • Planning with a transformer decoder over retrieved memories should produce more coherent multi-step plans than classifying each next action independently from the current affordance.
  • The approach should work with egocentric partial observations, without requiring full state or a dense reward signal.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not test what happens when expert demonstrations contain mistakes; an untested extension is to add a 'bad memory' of failed attempts so the planner can avoid undesirable actions, which the authors suggest but do not implement.
  • Because the localization key is trained only on temporal closeness, its notion of similarity may be tied to visual appearance; if so, the memory should transfer across homes that look alike but could fail on radically different layouts, a testable hypothesis the paper does not pursue.
  • The replan step is essentially an adversarial perturbation of the memory node, so if the goal discriminator is not calibrated on out-of-distribution observations, replanning could steer the agent toward irrelevant rather than corrective memories; robustness under attack is only as reliable as the discriminator's extrapolation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a new task, interactive action planning, in which an agent must produce high-level action sequences from egocentric observations in a simulated environment, and introduces TAMFormer, a model combining a Topological Affordance Memory (TAM) with a transformer-based action generator. The memory is built from expert demonstrations by learning affordance representations (supervised contrastive learning), a goal-association discriminator, and a self-supervised localization network. For planning, the model retrieves relevant memory nodes with the localization network, feeds them to a transformer decoder, and uses a replanning algorithm intended to recover from action deviations. Experiments in VirtualHome compare TAMFormer to several baselines under pure-text, static visual, interactive, and interactive-attack settings, and ablations study the contributions of replanning, learned localization, the transformer decoder, and goal conditioning. The paper reports substantial gains in LCS, executability, and graph-F1 metrics, and claims robustness to random action permutations.

Significance. If the central claims are reproducible, the paper makes a useful contribution by introducing a memory-augmented planning formulation that avoids online reinforcement learning and by proposing an interactive evaluation protocol that is more demanding than static procedure-planning benchmarks. The idea of storing environment affordances in a topological memory and using a goal-association discriminator for deviation detection is interesting, and the reported interactive results are considerably above the examined baselines. The paper also explicitly acknowledges a limitation (reliance on flawless demonstrations), which is appropriate. However, the significance is currently undercut by reproducibility gaps: the replanning algorithm is under-specified, the attack protocol is not precisely defined, and no statistical variability is reported. Because the robustness claim is one of the paper's headline contributions, these gaps prevent the reader from verifying the core mechanism.

major comments (4)
  1. [Sec. 3.2, Algorithm 1] Algorithm 1 is not a well-defined optimization procedure. The input list includes an "objective function Loss," but Loss is never defined in the paper; the surrounding text mentions a "modified cost function" but does not specify it. Line 5 updates the localized node using the gradient sign of Loss with respect to n, yet it is unclear whether n is a node index in a discrete graph or a continuous latent vector, and no justification is given for taking a continuous gradient step on a discrete memory node. The notation is also inconsistent: Section 3.1 defines the goal association function as P_sigma, while Algorithm 1 and the Figure 3 caption use P_mu. Because the ablation "w/o replan" drops attack LCS from 64% to 37% (Table 2), the reported robustness improvement is attributed to this underspecified mechanism. Please provide a concrete definition of Loss, clarify the representation of memory nodes, specify the threshold T, step size alpha, and trial limit K, and use a consistent symbol for the goal association network.
  2. [Sec. 3.1, TAM construction] The construction of the Topological Affordance Memory is not fully specified. The text states that keys are learned representations from the localization network and values are learned by P_mu and P_sigma, but it does not describe how nodes are created from demonstrations, how many nodes result, how edges (if any) are formed, how duplicate or near-duplicate observations are merged, or how the K nearest neighbors for retrieval are selected in the cross-attention module. These details are essential for reproducing the method and for interpreting the t-SNE visualization in Figure 5. Please provide a precise memory-construction procedure, including node initialization, update rules, and graph connectivity.
  3. [Sec. 4.1, Visual interactive attack] The attack protocol is described only as "we will randomly permute the predicted action" with no information about the probability of permutation, the number of permuted actions per episode, whether the same action can be chosen, or the random seed. The reported attack robustness values in Table 2 and the ablation "w/o replan" cannot be meaningfully compared across variants or reproduced by other groups without this information. Please specify the permutation distribution and schedule used in the experiments.
  4. [Tables 1, 2, and 3] No error bars, confidence intervals, or multiple-seed results are reported anywhere in the experimental section. All metrics are given as single numbers, which is concerning given that the differences between TAMFormer and some baselines are large (e.g., 72% vs. 31% LCS in the interactive setting, Table 1) but could still be less meaningful if the evaluation has high variance or if the test set is small. Please report the number of seeds, standard deviations, and the number of evaluation episodes per setting.
minor comments (5)
  1. [Sec. 4.5] The text in Section 4.5 says "Table 3 shows the performance of the full model for reference, as well as the results of each variation we tested," but the preceding Table 3 is specifically captioned for the Vis-Interact setting. Since decriptions for both the Vis-Interact and Vis-Inter-Attack ablations follow, the references to Table 2 and Table 3 should be disentangled for clarity.
  2. [Algorithm 1] In Algorithm 1, the loop counter is written as "trail" and "trial" inconsistently; please use a single spelling (likely "trial"), and define the stopping condition precisely.
  3. [Sec. 4.5, naive goal] The description of the "naive goal" ablation is ambiguous: the first sentence says the goal association function is trained with an additional goal as input, and the next says one variant removes the dependence on the goal. Please state explicitly which variant is used for the "naive goal" row in Tables 2 and 3.
  4. [Throughout] There are several typographical errors that should be corrected: "sorrounding" (Introduction), "whic" (Section 4.1), "groud-truth" (Section 4.4), "donates" (Section 4.2), and inconsistent capitalization of "R2+1D18" (Section 3.1).
  5. [Figure 3 caption] The Figure 3 caption uses P_mu to denote the goal association function, while Section 3.1 defines it as P_sigma; please make the notation uniform.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central planning result is an external benchmark on held-out apartments; the under-specified replan loss is a correctness issue, not a circular reduction.

full rationale

No circular step meets the quoted-reduction bar. The memory, affordance, localization, and goal-association modules are trained on expert demonstrations, but the reported LCS, Executability, and graph-F1 results are measured in VirtualHome on 10 unseen apartments (Sec. 4: 'we randomly spawned the character into four locations for each program and collected the videos... generated 10 unseen apartment for the test set'), under interactive execution and random action permutation, so the headline result is not a restatement of fitted parameters. The only same-author citation is [4] (Ext-Mgail), used as a baseline to compare against rather than to justify a design premise or import a uniqueness theorem; it is not load-bearing. Algorithm 1's undefined 'Loss' and the P_mu/P_sigma symbol mismatch (Sec. 3.1 vs Algorithm 1) are reproducibility/soundness defects, but they do not make any predicted metric equal to an input by construction. The paper also explicitly acknowledges its main limitation in Sec. 5 ('a major limitation of our method is the requirement for successful experiences'), which further shows the claims are not presented as self-justifying. Therefore the circularity score is 0.

Assumptions & free parameters 6 free parameters · 3 assumptions · 1 invented entities

The central claim rests on the quality of expert demonstrations, the reliability of the temporal-closeness proxy for localization, and the out-of-distribution validity of the goal-association discriminator used in replanning. The paper provides no error bars, no code, and no analysis of these assumptions.

free parameters (6)
  • InfoNCE temperature tau = not reported
    Temperature in affordance contrastive loss (Eq. 1); hand chosen, affects representation sharpness.
  • replan threshold T = not reported
    Threshold for goal association score in Algorithm 1; controls when replanning triggers.
  • replan step size alpha = not reported
    Gradient step size in adversarial update, Algorithm 1.
  • trial limit K = not reported
    Maximum iterations in replanning loop, Algorithm 1.
  • frames per observation f_k = 5
    Number of starting and ending frames used as observation (Sec. 3).
  • retrieval neighbors K = not reported
    Number of nearest memory nodes retrieved during planning; never specified in the paper.
assumptions (3)
  • domain assumption VirtualHome expert demonstrations are successful and representative of the test tasks
    The memory is built solely from expert demos; flawed demos would poison the memory (paper admits this limitation in Sec. 5).
  • domain assumption Temporal closeness (adjacent frames) is a valid proxy for spatial or situational proximity in the latent space
    Localization network L is trained to predict temporal adjacency; used as key for memory retrieval.
  • domain assumption The goal-association discriminator trained on demonstration states remains valid on out-of-distribution states reached after action deviations
    Replanning relies on P_sigma scores for attacked states; never validated on such states.
invented entities (1)
  • Topological Affordance Memory (TAM)
    purpose: Stores structured expert experience associating visual observations with environment affordances and goal associations for planning
    A new algorithmic construct; its effectiveness is only evidenced by in-paper experiments on VirtualHome. No external falsifiable handle beyond the paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of What to Do Next? Memorizing skills from Egocentric Instructional Video." pith.science (2026). https://pith.science/paper/ATJCPBP5

@misc{pith2026250702997,
  author       = {Pith},
  title        = {Pith review of: What to Do Next? Memorizing skills from Egocentric Instructional Video},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ATJCPBP5}},
  note         = {Machine review of arXiv:2507.02997}
}
read the original abstract

Learning to perform activities through demonstration requires extracting meaningful information about the environment from observations. In this research, we investigate the challenge of planning high-level goal-oriented actions in a simulation setting from an egocentric perspective. We present a novel task, interactive action planning, and propose an approach that combines topological affordance memory with transformer architecture. The process of memorizing the environment's structure through extracting affordances facilitates selecting appropriate actions based on the context. Moreover, the memory model allows us to detect action deviations while accomplishing specific objectives. To assess the method's versatility, we evaluate it in a realistic interactive simulation environment. Our experimental results demonstrate that the proposed approach learns meaningful representations, resulting in improved performance and robust when action deviations occur.

Figures

Figures reproduced from arXiv: 2507.02997 by the authors.

Figure 1
Figure 1. The problem formulation: Given the description of the goal gi set up table and initial observation, the agent should need to plan a sequence of actions to achieve the task. When executing an action a i t , the agent will receive a stack of [o s t , oe t ] as the observation, where o s t , oe t represent the starting/ending point of the action. Different trajectories may share the same scene; thus, understanding the … view at source ↗
Figure 2
Figure 2. Topological Affordance Memory: We embed the visual observation [o s t , oe t ] as v i a , vj a representing the affordance i, j. The two encoded feature will be projected to latent space for further contrastive learning. The goal association function conditioned on given goal g and tries to discriminate whether two features are associated with g. The affordance function learns to assign high value for pair of featur… view at source ↗
Figure 3
Figure 3. The Planning Module: Starting from the initial observation frame, we first use the localization step to identify the current situation in the learned memory. The darkness of the arrow in the figure represents the similarity between the frame representation vector and the key of the memory. Once we retrieve a similar situation from the memory, we use the goal association function Pµ to check whether the current progr… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Dataset collection: Each task is formed as a se￾quence of high-level actions. We executed these actions in a simulated environment with built-in characters, capturing a video that serves as the expert demonstration. During eval￾uation, we randomly spawn the character i…
Figure 5
Figure 5. Figure 5: Visualization of memory retrieve: We used t-SNE on the learned memory and colorized them based on the environment location. The red dot represents the activated memory location corresponding to the current observation. space, inspired by [29], we downsampled images to …

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

37 extracted references · 30 canonical work pages

  1. [1]

    Pddl— the planning domain definition language

    Constructions Aeronautiques, Adele Howe, Craig Knoblock, ISI Drew McDermott, Ashwin Ram, Manuela Veloso, Daniel Weld, David Wilkins SRI, Anthony Barrett, Dave Christian- son, et al. Pddl— the planning domain definition language. Technical Report, Tech. Rep., 1998. 1

    work page 1998

  2. [2]

    Psychology press, 2014

    John R Anderson and Gordon H Bower.Human associative memory. Psychology press, 2014. 4

    work page 2014

  3. [3]

    Vision-and-language navigation: In- terpreting visually-grounded navigation instructions in real environments

    Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko S ¨underhauf, Ian Reid, Stephen Gould, and Anton Van Den Hengel. Vision-and-language navigation: In- terpreting visually-grounded navigation instructions in real environments. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 3674–3683,

  4. [4]

    Procedure planning in instructional videos via contextual modeling and model- based policy learning

    Jing Bi, Jiebo Luo, and Chenliang Xu. Procedure planning in instructional videos via contextual modeling and model- based policy learning. InProceedings of the IEEE/CVF In- ternational Conference on Computer Vision, pages 15611– 15620, 2021. 1, 6

    work page 2021

  5. [5]

    Intentnet: Learning to predict intention from raw sensor data

    Sergio Casas, Wenjie Luo, and Raquel Urtasun. Intentnet: Learning to predict intention from raw sensor data. InCon- ference on Robot Learning, pages 947–956, 2018. 2

    work page 2018

  6. [6]

    Procedure planning in instructional videos

    Chien-Yi Chang, De-An Huang, Danfei Xu, Ehsan Adeli, Li Fei-Fei, and Juan Carlos Niebles. Procedure planning in instructional videos. InProceedings of the European Con- ference on Computer Vision (ECCV), 2020. 1, 6

    work page 2020

  7. [7]

    Ryoo, and Franc ¸ois Br´emond

    Rui Dai, Srijan Das, Kumara Kahatapitiya, Michael S. Ryoo, and Franc ¸ois Br´emond. Ms-tct: Multi-scale temporal con- vtransformer for action detection. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 20041–20051, June 2022. 1

    work page 2022

  8. [8]

    Embodied question answer- ing

    Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. Embodied question answer- ing. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 1–10, 2018. 3

    work page 2018

Show all 37 references
  1. [9]

    Revisiting skeleton-based action recognition

    Haodong Duan, Yue Zhao, Kai Chen, Dahua Lin, and Bo Dai. Revisiting skeleton-based action recognition. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2969–2978, 2022. 1

    2022

  2. [10]

    What do navigation agents learn about their environment? InProceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 10276–10285, 2022

    Kshitij Dwivedi, Gemma Roig, Aniruddha Kembhavi, and Roozbeh Mottaghi. What do navigation agents learn about their environment? InProceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 10276–10285, 2022. 3

    2022

  3. [11]

    Sparse graphical memory for robust planning.Advances in Neural Information Pro- cessing Systems, 33:5251–5262, 2020

    Scott Emmons, Ajay Jain, Misha Laskin, Thanard Kurutach, Pieter Abbeel, and Deepak Pathak. Sparse graphical memory for robust planning.Advances in Neural Information Pro- cessing Systems, 33:5251–5262, 2020. 3

    2020

  4. [12]

    Search on the replay buffer: Bridging planning and rein- forcement learning.Advances in Neural Information Pro- cessing Systems, 32, 2019

    Ben Eysenbach, Russ R Salakhutdinov, and Sergey Levine. Search on the replay buffer: Bridging planning and rein- forcement learning.Advances in Neural Information Pro- cessing Systems, 32, 2019. 3

    2019

  5. [13]

    Maria Fox and Derek Long. Pddl2. 1: An extension to pddl for expressing temporal planning domains.Journal of artifi- cial intelligence research, 20:61–124, 2003. 1

    2003

  6. [14]

    Cognitive mapping and plan- ning for visual navigation

    Saurabh Gupta, James Davidson, Sergey Levine, Rahul Suk- thankar, and Jitendra Malik. Cognitive mapping and plan- ning for visual navigation. InProceedings of the IEEE con- ference on computer vision and pattern recognition, pages 2616–2625, 2017. 1

    2017

  7. [15]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. InProceed- ings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016. 5

    2016

  8. [16]

    Supervised contrastive learning

    Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors,Advances in Neural Information Proces...

    2020

  9. [17]

    Springer Science & Business Media,

    Teuvo Kohonen.Associative memory: A system-theoretical approach, volume 17. Springer Science & Business Media,

  10. [18]

    Ai2-thor: An interactive 3d environment for visual ai.arXiv preprint arXiv:1712.05474,

    Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Daniel Gordon, Yuke Zhu, Ab- hinav Gupta, and Ali Farhadi. Ai2-thor: An interactive 3d environment for visual ai.arXiv preprint arXiv:1712.05474,

    arXiv

  11. [19]

    Ad- versarial examples in the physical world

    Alexey Kurakin, Ian J Goodfellow, and Samy Bengio. Ad- versarial examples in the physical world. InArtificial in- telligence safety and security, pages 99–112. Chapman and Hall/CRC, 2018. 5

    2018

  12. [20]

    Deep learning.nature, 521(7553):436–444, 2015

    Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning.nature, 521(7553):436–444, 2015. 1

    2015

  13. [21]

    Offline reinforcement learning: Tutorial, review, and perspectives on open problems.arXiv preprint arXiv:2005.01643, 2020

    Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems.arXiv preprint arXiv:2005.01643, 2020. 3

    2005 arXiv

  14. [22]

    Synthesizing environment-aware activities via activity sketches

    Yuan-Hong Liao, Xavier Puig, Marko Boben, Antonio Tor- ralba, and Sanja Fidler. Synthesizing environment-aware activities via activity sketches. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6291–6299, 2019. 4

    2019

  15. [23]

    Hallucinative topological memory for zero- shot visual planning

    Kara Liu, Thanard Kurutach, Christine Tung, Pieter Abbeel, and Aviv Tamar. Hallucinative topological memory for zero- shot visual planning. InInternational Conference on Ma- chine Learning, pages 6259–6270. PMLR, 2020. 3

    2020

  16. [24]

    The ontogeny of human memory: A cog- nitive neuroscience perspective.Developmental psychology, 31(5):723, 1995

    Charles A Nelson. The ontogeny of human memory: A cog- nitive neuroscience perspective.Developmental psychology, 31(5):723, 1995. 4

    1995

  17. [25]

    Repre- sentation learning with contrastive predictive coding.arXiv preprint arXiv:1807.03748, 2018

    Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Repre- sentation learning with contrastive predictive coding.arXiv preprint arXiv:1807.03748, 2018. 3

    2018 arXiv

  18. [26]

    Object-relation reasoning graph for action recognition

    Yangjun Ou, Li Mi, and Zhenzhong Chen. Object-relation reasoning graph for action recognition. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 20133–20142, June 2022. 1

    2022

  19. [27]

    Neural episodic con- trol

    Alexander Pritzel, Benigno Uria, Sriram Srinivasan, Adria Puigdomenech Badia, Oriol Vinyals, Demis Hassabis, Daan Wierstra, and Charles Blundell. Neural episodic con- trol. InInternational Conference on Machine Learning, pages 2827–2836. PMLR, 2017. 2, 3

    2017

  20. [28]

    Virtualhome: Simulating household activities via programs

    Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Torralba. Virtualhome: Simulating household activities via programs. InProceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8494–8502, 2018. 5

    2018

  21. [29]

    Semi-parametric topological memory for navigation

    Nikolay Savinov, Alexey Dosovitskiy, and Vladlen Koltun. Semi-parametric topological memory for navigation. InIn- ternational Conference on Learning Representations (ICLR),

  22. [30]

    Facenet: A unified embedding for face recognition and clus- tering

    Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clus- tering. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 815–823, 2015. 3

    2015

  23. [31]

    A closer look at spatiotemporal convolutions for action recognition

    Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. A closer look at spatiotemporal convolutions for action recognition. InProceedings of the IEEE conference on Computer Vision and Pattern Recogni- tion, pages 6450–6459, 2018. 4

    2018

  24. [32]

    Repre- sentation learning with contrastive predictive coding.arXiv e-prints, pages arXiv–1807, 2018

    Aaron Van den Oord, Yazhe Li, and Oriol Vinyals. Repre- sentation learning with contrastive predictive coding.arXiv e-prints, pages arXiv–1807, 2018. 4

    2018

  25. [33]

    Attention is all you need.Advances in neural information processing systems, 30, 2017

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need.Advances in neural information processing systems, 30, 2017. 5, 6

    2017

  26. [34]

    Jiaojiao Zhao, Yanyi Zhang, Xinyu Li, Hao Chen, Bing Shuai, Mingze Xu, Chunhui Liu, Kaustav Kundu, Yuanjun Xiong, Davide Modolo, Ivan Marsic, Cees G. M. Snoek, and Joseph Tighe. Tuber: Tubelet transformer for video ac- tion detection. InProceedings of the IEEE/CVF Conference o...

    2022

  27. [35]

    Visual semantic planning using deep successor representa- tions

    Yuke Zhu, Daniel Gordon, Eric Kolve, Dieter Fox, Li Fei- Fei, Abhinav Gupta, Roozbeh Mottaghi, and Ali Farhadi. Visual semantic planning using deep successor representa- tions. InProceedings of the IEEE international conference on computer vision, pages 483–492, 2017. 1

    2017

  28. [36]

    A comprehensive study of deep video action recognition.arXiv preprint arXiv:2012.06567, 2020

    Yi Zhu, Xinyu Li, Chunhui Liu, Mohammadreza Zolfaghari, Yuanjun Xiong, Chongruo Wu, Zhi Zhang, Joseph Tighe, R Manmatha, and Mu Li. A comprehensive study of deep video action recognition.arXiv preprint arXiv:2012.06567, 2020. 1

    2012 arXiv

  29. [37]

    Target-driven vi- sual navigation in indoor scenes using deep reinforcement learning

    Yuke Zhu, Roozbeh Mottaghi, Eric Kolve, Joseph J Lim, Ab- hinav Gupta, Li Fei-Fei, and Ali Farhadi. Target-driven vi- sual navigation in indoor scenes using deep reinforcement learning. In2017 IEEE international conference on robotics and automation (ICRA), pages 3357–3364. IE...

    2017

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.