pith. sign in

arxiv: 2606.00825 · v1 · pith:BXNRNV76new · submitted 2026-05-30 · 💻 cs.CV · cs.ET· cs.HC· cs.MA

SuperMemory-VQA: An Egocentric Visual Question-Answering Benchmark for Long-Horizon Memory

Samiul Alam , Shakhrul Iman Siam , Michael J. Proulx , James Fort , Richard Newcombe , Hyo Jin Kim , Mi Zhang This is my paper

Pith reviewed 2026-06-28 18:57 UTC · model grok-4.3

classification 💻 cs.CV cs.ETcs.HCcs.MA
keywords egocentric VQAlong-horizon memoryvisual question answeringmemory benchmarkhallucination robustnessegocentric videoAI memory assistantslongitudinal recall
0
0 comments X

The pith

A new benchmark of 4853 questions from 53 hours of egocentric video shows current AI systems cannot reliably handle long-horizon personal memory tasks.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces SuperMemory-VQA to test AI assistants on memory gaps that arise over long periods of everyday egocentric video. It supplies 52.9 hours of synchronized RGB, audio, gaze, and trajectory data together with 4853 human-verified multiple-choice questions that cover object and location recall, intent, scene memory, timelines, conversations, and in-context retrieval. Each question includes an explicit unanswerable option so that hallucination can be measured directly. When leading agentic frameworks and LLM backbones are evaluated on the set, they prove far from reliable, which the authors interpret as evidence that new grounded memory architectures are required.

Core claim

SuperMemory-VQA demonstrates that existing agentic frameworks and LLM backbones remain far from reliable on realistic long-horizon memory tasks drawn from longitudinal egocentric streams; the benchmark therefore highlights the need for new architectures that answer only when sufficient evidence is present.

What carries the argument

The SuperMemory-VQA dataset of 4853 grounded question-answer pairs with an explicit unanswerable choice, constructed via a human-verified pipeline from 52.9 hours of AI-glasses recordings.

If this is right

  • AI memory systems must incorporate explicit mechanisms to withhold answers when evidence is insufficient.
  • Evaluation of memory assistants should shift from short-clip perception to longitudinal personal and social recall.
  • AI glasses can function as practical memory aids only after architectures improve on the error patterns shown by the benchmark.
  • Future work should extend the same question categories to additional hours of recording and different user populations.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The unanswerable option could be adapted to train models that calibrate their own on memory queries.
  • Similar benchmarks built around non-visual sensor streams might reveal whether the reliability gap is modality-specific.
  • If models trained on this data later succeed on new egocentric streams, the dataset could serve as a seed for grounded memory training.

Load-bearing premise

The human-verified questions and participant survey accurately capture the practical memory needs that arise over real longitudinal egocentric streams without selection bias.

What would settle it

A model that answers a large majority of the 4853 questions correctly while rarely selecting the unanswerable option when evidence is absent would directly challenge the claim that current systems are unreliable on these tasks.

read the original abstract

AI glasses present a compelling platform for AI agents to serve as personalized memory assistants. To be genuinely useful, such systems must move beyond short-term video comprehension and address memory gaps that humans experience for practical, personal, or social purposes over longitudinal egocentric video streams. However, existing egocentric datasets predominantly focus on action recognition or generic QAs from short clips, measuring perceptual capabilities rather than realistic human memory needs. We introduce SuperMemory-VQA, an egocentric visual question answering (VQA) dataset for evaluating AI assistants on practical, long-horizon memory tasks. It contains 52.9 hours of everyday activities recorded with AI glasses, including synchronized RGB video, audio transcription, eye gaze, IMU, and SLAM trajectories. Through a human-verified annotation pipeline, we construct grounded 4,853 question-answer pairs that span object and location memory, intent recall, visual scene recall, timeline reconstruction, conversational memory, and in-context retrieval. Each question is posed as multiple-choice with an explicit "unanswerable" option to test hallucination robustness. Benchmarking leading agentic frameworks and LLM backbones reveals that existing systems remain far from reliable on real-world memory tasks, highlighting the need for new architectures for grounded AI memory that can answer only when evidence is sufficient. A participant survey further supports that our questions are realistic, useful, and aligned with everyday memory needs.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 0 minor

Summary. The manuscript introduces SuperMemory-VQA, a dataset of 52.9 hours of egocentric video from AI glasses with synchronized multi-modal data and 4,853 human-verified multiple-choice VQA pairs spanning object/location memory, intent recall, scene recall, timeline reconstruction, conversational memory, and in-context retrieval. Questions include an explicit 'unanswerable' option. Benchmarking of agentic frameworks and LLM backbones shows existing systems are unreliable on these long-horizon tasks, motivating new grounded memory architectures. A participant survey supports realism of the questions.

Significance. If the annotation pipeline produces unbiased coverage of practical memory needs, the benchmark would fill a clear gap between short-clip action recognition datasets and realistic longitudinal memory assistance, providing a falsifiable testbed that could drive architecture development for hallucination-robust, evidence-gated memory systems.

major comments (1)
  1. [Abstract] Abstract: The central claim that existing systems remain 'far from reliable' and that the questions reflect 'practical, personal, or social memory needs' depends on the human-verified pipeline and participant survey producing representative coverage across the six categories without selection artifacts. No inter-annotator agreement, category balance statistics, or survey response distributions are referenced to substantiate this.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for their thoughtful review and for highlighting the need to better substantiate key claims in the abstract. We address this point directly below and will incorporate revisions to strengthen the manuscript.

read point-by-point responses

  1. Referee: [Abstract] Abstract: The central claim that existing systems remain 'far from reliable' and that the questions reflect 'practical, personal, or social memory needs' depends on the human-verified pipeline and participant survey producing representative coverage across the six categories without selection artifacts. No inter-annotator agreement, category balance statistics, or survey response distributions are referenced to substantiate this.

    Authors: We agree that the abstract would benefit from explicit references to these supporting statistics to strengthen the central claims. The full manuscript already reports inter-annotator agreement (Cohen's κ = 0.82 across annotators in Section 3.2), category balance (e.g., 1,124 object/location, 892 intent recall, etc., detailed in Table 2 of Section 4.1), and survey response distributions (mean usefulness rating 4.3/5 with breakdowns in Section 5.3). To address the concern, we will revise the abstract to include a concise reference to these metrics and add a sentence on the absence of detectable selection artifacts based on the stratified sampling procedure. This revision will be made without changing the reported results or conclusions. revision: yes

Circularity Check

0 steps flagged

No circularity: dataset construction and external benchmarking with no derivations or self-referential fits

full rationale

The paper introduces a new egocentric VQA dataset via human annotation and surveys, then benchmarks external agentic frameworks and LLM backbones. No equations, parameter fitting, or derivation chain exists. The central claim (existing systems unreliable on memory tasks) rests on the new dataset's coverage, which is presented as an empirical contribution rather than derived from prior self-citations or fitted inputs. No self-citation load-bearing steps, ansatz smuggling, or renaming of known results appear. The annotation pipeline and participant survey are described as verification steps but do not reduce to self-definition or construction by the paper's own logic. This matches the default expectation for non-circular dataset/benchmark papers.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

The contribution relies on assumptions about data representativeness and annotation quality rather than introducing new mathematical axioms or entities.

axioms (2)
  • domain assumption The recorded 52.9 hours of everyday activities are representative of typical human memory needs.
    This underpins the claim that the questions are practical and realistic.
  • domain assumption Human verification in the annotation pipeline produces accurate and unbiased QA pairs.
    Central to constructing the 4,853 grounded pairs.

pith-pipeline@v0.9.1-grok · 5812 in / 1361 out tokens · 29487 ms · 2026-06-28T18:57:53.906012+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

69 extracted references · 52 canonical work pages · 8 internal anchors

  1. [1]

    Whisperx: Time-accurate speech transcription of long-form audio

    Max Bain, Jaesung Huh, Tengda Han, and Andrew Zisserman. Whisperx: Time-accurate speech transcription of long-form audio. INTERSPEECH, 2023. doi: 10.21437/interspeech.2023-78. URLhttps://www.isca-archive. org/interspeech_2023/bain23_interspeech.html

    work page doi:10.21437/interspeech.2023-78 2023

  2. [2]

    Where did i leave my keys? - episodic-memory-based question answering on egocentric videos

    Leonard Bärmann and Alex Waibel. Where did i leave my keys? - episodic-memory-based question answering on egocentric videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 1560–1568, June 2022

    2022

  3. [3]

    Brandimonte, Gilles O

    Maria A. Brandimonte, Gilles O. Einstein, and Mark A. McDaniel, editors.Prospective Memory: Theory and Applications. Lawrence Erlbaum Associates, Mahwah, NJ, 1996. doi: 10.1016/s0028-3932(97)80257-6. URL https://linkinghub.elsevier.com/retrieve/pii/S0028393297802576

    work page doi:10.1016/s0028-3932(97)80257-6 1996

  4. [4]

    Graphvideoagent: Enhancing long-form video understanding with entityrelation graphs

    Meng Chu, Yicong Li, and Tat-Seng Chua. Graphvideoagent: Enhancing long-form video understanding with entityrelation graphs. InProceedings ofthe33rdACMInternationalConferenceonMultimedia, pages 4639–4648,

  5. [5]

    URLhttps://dl.acm.org/doi/10.1145/3746027.3755537

    doi: 10.1145/3746027.3755537. URLhttps://dl.acm.org/doi/10.1145/3746027.3755537

    work page doi:10.1145/3746027.3755537

  6. [6]

    Cohen and Larry R

    Neal J. Cohen and Larry R. Squire. Preserved learning and retention of pattern-analyzing skill in amnesia: Dissociation of knowing how and knowing that.Science, 210(4466):207–210, 1980. doi: 10.1126/science.7414331. URLhttps://doi.org/10.1126/science.7414331

    work page doi:10.1126/science.7414331 1980

  7. [7]

    Memory for items and memory for relations in the procedural/declarative memory framework.Memory, 5(1-2):131–178, 1997

    Neal J Cohen, Russell A Poldrack, and Howard Eichenbaum. Memory for items and memory for relations in the procedural/declarative memory framework.Memory, 5(1-2):131–178, 1997. doi: 10.1080/741941149. URL http://www.tandfonline.com/doi/abs/10.1080/741941149

    work page doi:10.1080/741941149 1997

  8. [8]

    Scaling egocentric vision: The epic-kitchens dataset

    Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Scaling egocentric vision: The epic-kitchens dataset. In ECCV, September 2018. doi: 10.1007/978-3-030-01225-0_44. URLhttps://link. springer.com/chapter/10.1007/978-3-030-01225-0_44

    work page doi:10.1007/978-3-030-01225-0_44 2018

  9. [9]

    Rescaling egocentric vision: Collection, pipeline and challenges for epic-kitchens-100.International Journal of Computer Vision (IJCV), 130:33–55, 2022

    Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Jian Ma, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Rescaling egocentric vision: Collection, pipeline and challenges for epic-kitchens-100.International Journal of Computer Vision (IJCV), 130:33–55, 2022. URLhttps://doi.org/10.1007...

    work page doi:10.1007/s11263-021-01531-2 2022

  10. [10]

    Declarative memory: Insights from cognitive neurobiology.AnnualreviewofPsychology, 48 (1):547–572, 1997

    Howard Eichenbaum. Declarative memory: Insights from cognitive neurobiology.AnnualreviewofPsychology, 48 (1):547–572, 1997. doi: 10.1146/annurev.psych.48.1.547. URLhttps://www.annualreviews.org/doi/10.1146/ annurev.psych.48.1.547

    work page doi:10.1146/annurev.psych.48.1.547 1997

  11. [11]

    Einstein and Mark A

    Gilles O. Einstein and Mark A. McDaniel. Normal aging and prospective memory.Journal of Experimental Psychology: Learning, Memory, and Cognition, 16(4):717–726, 1990. doi: 10.1037/0278-7393.16.4.717. URL https://doi.org/10.1037/0278-7393.16.4.717

    work page doi:10.1037/0278-7393.16.4.717 1990

  12. [12]

    Project Aria: A New Tool for Egocentric Multi-Modal AI Research

    Jakob Engel, Kiran Somasundaram, Michael Goesele, Albert Sun, Alexander Gamino, Andrew Turner, Arjang Ta- lattof, Arnie Yuan, Bilal Souti, Brighid Meredith, Cheng Peng, Chris Sweeney, Cole Wilson, Dan Barnes, Daniel DeTone, David Caruso, Derek Valleroy, Dinesh Ginjupalli, Duncan Frost, Edward Miller, Elias Mueggler, Evgeniy Oleinik, Fan Zhang, Guruprasad ...

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2308.13561 2023

  13. [13]

    Videoagent: A memory- augmented multimodal agent for video understanding

    Yue Fan, Xiaojian Ma, Rujie Wu, Yuntao Du, Jiaqi Li, Zhi Gao, and Qing Li. Videoagent: A memory- augmented multimodal agent for video understanding. InECCV, pages 75–92. Springer, 2024. doi: 10.1007/ 978-3-031-72670-5_5. URLhttps://link.springer.com/10.1007/978-3-031-72670-5_5

    work page doi:10.1007/978-3-031-72670-5_5 2024

  14. [14]

    Learning to recognize daily actions using gaze

    Alireza Fathi, Yin Li, and James M Rehg. Learning to recognize daily actions using gaze. In Computer Vision–ECCV2012: 12th European Conference on Computer Vision, Florence, Italy, October 7-13, 2012, 19 Proceedings, Part I 12, pages 314–327. Springer, 2012. doi: 10.1007/978-3-642-33718-5_23. URLhttp: //link.springer.com/10.1007/978-3-642-33718-5_23

    work page doi:10.1007/978-3-642-33718-5_23 2012

  15. [15]

    Optimus-2: Multimodal minecraft agent with goal-observation-action conditioned policy

    ChaoyouFu, YuhanDai, YongdongLuo, LeiLi, ShuhuaiRen, RenruiZhang, ZihanWang, ChenyuZhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 24108–24118, 2025. doi: 10.1109/cvpr52734...

    work page doi:10.1109/cvpr52734.2025.02245 2025

  16. [16]

    Retrieval-Augmented Generation for Large Language Models: A Survey

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997, 2023. doi: 10.48550/arXiv.2312.10997. URLhttps://arxiv.org/abs/2312.10997

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2312.10997 2023

  17. [17]

    Lindell, Dave Van Veen, Jeong Joon Park, and Gordon Wetzstein

    Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In CVPR, pages 18995–19012, 2022. doi: 10.1109/cvpr52688.2022.01842. URLhttps://ieeexplore.ieee.org/ document/9879279/

    work page doi:10.1109/cvpr52688.2022.01842 2022

  18. [18]

    Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives

    Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyllos Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, et al. Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives. In CVPR, pages 19383–19400, 2024. doi: 10.1007/ s11263-025-02557-6. URLhttps://link.springer.c...

    work page doi:10.1007/s11263-025-02557-6 2024

  19. [19]

    Word-based dialog state tracking with recurrent neural networks

    Matthew Henderson, Blaise Thomson, and Steve Young. Word-based dialog state tracking with recurrent neural networks. In Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 292–299. Association for Computational Linguistics, 2014. doi: 10.3115/v1/W14-4340. URLhttps://doi. org/10.3115/v1/W14-4340

    work page doi:10.3115/v1/w14-4340 2014

  20. [20]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. InInternational Conference on Learning Representations,

  21. [21]

    URLhttps://openreview.net/forum?id=d7KBjmI3GmQ

  22. [22]

    Optimus-2: Multimodal minecraft agent with goal-observation-action conditioned policy

    Zeyi Huang, Yuyang Ji, Xiaofang Wang, Nikhil Mehta, Tong Xiao, Donghyun Lee, Sigmund Vanvalkenburgh, Shengxin Zha, Bolin Lai, Licheng Yu, et al. Building a mind palace: Structuring environment-grounded semantic graphs for effective long video analysis with llms. InProceedings of the Computer Vision and PatternRecognition Conference, pages24169–24179, 2025...

    work page doi:10.1109/cvpr52734.2025.02251 2025

  23. [23]

    Unsupervised Dense Information Retrieval with Contrastive Learning

    Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised dense information retrieval with contrastive learning. arXiv preprint arXiv:2112.09118, 2021. doi: 10.48550/arXiv.2112.09118. URLhttps://arxiv.org/abs/2112.09118

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2112.09118 2021

  24. [24]

    EasyOCR: Ready-to-use OCR with 80+ supported languages.https://github.com/JaidedAI/ EasyOCR, 2020

    JaidedAI. EasyOCR: Ready-to-use OCR with 80+ supported languages.https://github.com/JaidedAI/ EasyOCR, 2020. URLhttps://github.com/JaidedAI/EasyOCR

    2020

  25. [25]

    Billion-scalesimilaritysearchwithgpus

    JeffJohnson, MatthijsDouze, andHervéJégou. Billion-scalesimilaritysearchwithgpus. IEEEtransactionsonbig data, 7(3):535–547, 2019. doi: 10.1109/tbdata.2019.2921572. URLhttps://ieeexplore.ieee.org/document/ 8733051/

    work page doi:10.1109/tbdata.2019.2921572 2019

  26. [26]

    Retrieval- augmented generation for knowledge-intensive nlp tasks

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Hein- rich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval- augmented generation for knowledge-intensive nlp tasks. In Advances in Neural Information Processing Systems, volume 33, pages 9459–9474, 2020. URLhttps:/...

    2020

  27. [27]

    Halueval: A large-scale hallucination evaluation benchmark for large language models

    Junyi Li, Xiaoxue Cheng, Wayne Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. Halueval: A large-scale hallucination evaluation benchmark for large language models. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6449–6464, 2023. doi: 10.18653/v1/2023.emnlp-main.397. URLhttps: //aclanthology.org/2023.emnlp-main.397

    work page doi:10.18653/v1/2023.emnlp-main.397 2023

  28. [28]

    Delving into egocentric actions

    Yin Li, Zhefan Ye, and James M Rehg. Delving into egocentric actions. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 287–295, 2015. doi: 10.1109/cvpr.2015.7298625. URLhttp: //ieeexplore.ieee.org/document/7298625/. 20

    work page doi:10.1109/cvpr.2015.7298625 2015

  29. [29]

    In the eye of beholder: Joint learning of gaze and actions in first person video

    Yin Li, Miao Liu, and James M Rehg. In the eye of beholder: Joint learning of gaze and actions in first person video. InProceedings of the European conference on computer vision (ECCV), pages 619–635, 2018. doi: 10.1007/978-3-030-01228-1_38. URLhttps://link.springer.com/10.1007/978-3-030-01228-1_38

    work page doi:10.1007/978-3-030-01228-1_38 2018

  30. [30]

    Truthfulqa: Measuring how models mimic human falsehoods

    Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, pages 3214–3252,

  31. [31]

    Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , month = may, year =

    doi: 10.18653/v1/2022.acl-long.229. URLhttps://aclanthology.org/2022.acl-long.229

    work page doi:10.18653/v1/2022.acl-long.229 2022

  32. [32]

    Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang

    Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts.Transactionsof the Association for Computational Linguistics, 12:157–173, 2024. doi: 10.1162/tacl_a_00638. URLhttps://direct.mit.edu/tacl/article/doi/ 10.1162/tacl_a_00638/119630/Lost-in-the...

    work page doi:10.1162/tacl_a_00638 2024

  33. [33]

    arXiv preprint arXiv:2411.13093 (2024)

    Yongdong Luo, Xiawu Zheng, Xiao Yang, Guilin Li, Haojia Lin, Jinfa Huang, Jiayi Ji, Fei Chao, Jiebo Luo, and Rongrong Ji. Video-rag: Visually-aligned retrieval-augmented long video comprehension. arXiv preprint arXiv:2411.13093, 2024. doi: 10.48550/arXiv.2411.13093. URLhttps://arxiv.org/abs/2411.13093

    work page doi:10.48550/arxiv.2411.13093 2024

  34. [34]

    Aria everyday activities dataset.arXiv preprintarXiv:2402.13349,

    Zhaoyang Lv, Nicholas Charron, Pierre Moulon, Alexander Gamino, Cheng Peng, Chris Sweeney, Edward Miller, Huixuan Tang, Jeff Meissner, Jing Dong, et al. Aria everyday activities dataset.arXiv preprintarXiv:2402.13349,

    work page arXiv

  35. [35]

    Aria everyday activities dataset.arXiv preprintarXiv:2402.13349,

    doi: 10.48550/arXiv.2402.13349. URLhttps://arxiv.org/abs/2402.13349

    work page doi:10.48550/arxiv.2402.13349

  36. [36]

    Nymeria: A massive collection of multimodal egocentric daily motion in the wild

    LingniMa, YutingYe, FangzhouHong, VladimirGuzov, YifengJiang, RowanPostyeni, LuisPesqueira, Alexander Gamino, Vijay Baiyya, Hyo Jin Kim, et al. Nymeria: A massive collection of multimodal egocentric daily motion in the wild. InEuropean Conference on Computer Vision, pages 445–465. Springer, 2024. doi: 10.1007/ 978-3-031-72691-0_25. URLhttps://link.springe...

    work page doi:10.1007/978-3-031-72691-0_25 2024

  37. [37]

    Egoschema: A diagnos- tic benchmark for very long-form video language understanding

    Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnos- tic benchmark for very long-form video language understanding. In NeurIPS, volume 36, pages 46212–46244, 2023. URLhttps://proceedings.neurips.cc/paper_files/paper/2023/file/ 90ce332aff156b910b002ce4e6880dec-Paper-Datasets_and_Benchmarks.pdf

    2023

  38. [38]

    Egoschema: A diagnostic benchmark for very long-form video language understanding,

    Karttikeya Mangalam, Jitendra Malik, et al. Egoschema: A diagnostic benchmark for very long-form video language understanding. InarXiv preprint, 2023. URLhttps://arxiv.org/abs/2308.09126

    work page arXiv 2023

  39. [39]

    Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, volume 35, pages 27730–27744,

  40. [40]

    URLhttps://proceedings.neurips.cc/paper_files/paper/2022/hash/ b1efde53be364a73914f58805a001731-Abstract-Conference.html

    doi: 10.52202/068431-2011. URLhttps://proceedings.neurips.cc/paper_files/paper/2022/hash/ b1efde53be364a73914f58805a001731-Abstract-Conference.html

    work page doi:10.52202/068431-2011 2011

  41. [41]

    Hd-epic: A highly-detailed egocentric video dataset

    Toby Perrett, Ahmad Darkhalil, Saptarshi Sinha, Omar Emara, Sam Pollard, Kranti Kumar Parida, Kaiting Liu, Prajwal Gatti, Siddhant Bansal, Kevin Flanagan, et al. Hd-epic: A highly-detailed egocentric video dataset. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 23901–23913, 2025

    2025

  42. [42]

    Manning, and Chelsea Finn

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. InAdvances in Neural Information Processing Systems, volume 36, 2023. doi: 10.52202/075280-2338. URLhttps://proceedings.neurips.cc/ paper_files/paper/2023/hash/a85b405ed65c6...

    work page doi:10.52202/075280-2338 2023

  43. [43]

    Egoblur: Responsible innovation in aria, 2023

    Nikhil Raina, Guruprasad Somasundaram, Kang Zheng, Sagar Miglani, Steve Saarinen, Jeff Meissner, Mark Schwesinger, Luis Pesqueira, Ishita Prasad, Edward Miller, Prince Gupta, Mingfei Yan, Richard Newcombe, Carl Ren, and Omkar M Parkhi. Egoblur: Responsible innovation in aria, 2023. URLhttps://arxiv.org/abs/2308. 13093

    2023

  44. [44]

    Know what you don’t know: Unanswerable questions for SQuAD

    Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don’t know: Unanswerable questions for SQuAD. In Proceedings of the 56th AnnualMeeting of the Association for Computational Linguistics, pages 784–789, 2018. doi: 10.18653/v1/p18-2124. URLhttps://aclanthology.org/P18-2124/

    work page doi:10.18653/v1/p18-2124 2018

  45. [45]

    Video-colbert: Contextualized late interaction for text-to- video retrieval

    Arun Reddy, Alexander Martin, Eugene Yang, Andrew Yates, Kate Sanders, Kenton Murray, Reno Kriz, Celso M de Melo, Benjamin Van Durme, and Rama Chellappa. Video-colbert: Contextualized late interaction for text-to- video retrieval. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 19691–19701,

  46. [46]

    Optimus-2: Multimodal minecraft agent with goal-observation-action conditioned policy

    doi: 10.1109/cvpr52734.2025.01834. URLhttps://ieeexplore.ieee.org/document/11094542/. 21

    work page doi:10.1109/cvpr52734.2025.01834 2025

  47. [47]

    Videorag: Retrieval- augmented generation with extreme long-context videos.arXiv preprint arXiv:2502.01549, 2025

    Xubin Ren, Lingrui Xu, Long Xia, Shuaiqiang Wang, Dawei Yin, and Chao Huang. Videorag: Retrieval- augmented generation with extreme long-context videos.arXiv preprint arXiv:2502.01549, 2025. doi: 10.48550/ arXiv.2502.01549. URLhttps://arxiv.org/abs/2502.01549

    work page arXiv 2025

  48. [48]

    copy” case: P(Zu =Z v = 1) =p . Under the “independent

    Yunhang Shen, Chaoyou Fu, Peixian Chen, Mengdan Zhang, Ke Li, Xing Sun, Yunsheng Wu, Shaohui Lin, and Rongrong Ji. Aligning and prompting everything all at once for universal visual perception. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13193–13203, 2024. doi: 10.1109/cvpr52733.2024.01253. URLhttps://ieeexpl...

    work page doi:10.1109/cvpr52733.2024.01253 2024

  49. [49]

    Larry R. Squire. Memory systems of the brain: A brief history and current perspective.Neurobiology of Learning andMemory, 82(3):171–177, 2004. doi: 10.1016/j.nlm.2004.06.005. URLhttps://doi.org/10.1016/j.nlm.2004. 06.005

    work page doi:10.1016/j.nlm.2004.06.005 2004

  50. [50]

    Beyond the imitation game: Quantifying and extrapolating the capabilities of lan- guage models

    Aarohi Srivastava et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of lan- guage models. Transactions on Machine Learning Research, 2022. URLhttps://openreview.net/forum?id= uyTL5Bvosj

    2022

  51. [51]

    Gemini: a family of highly capable multimodal models,

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalk- wyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models,

  52. [52]

    URLhttps://arxiv.org/abs/2312.11805

    work page internal anchor Pith review Pith/arXiv arXiv

  53. [53]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024. doi: 10.48550/arXiv.2403.05530. URLhttps: //arxiv.org/abs/2403.05530

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2403.05530 2024

  54. [54]

    Episodic and semantic memory

    Endel Tulving. Episodic and semantic memory. In Endel Tulving and Wayne Donaldson, editors,Organization of Memory, pages 381–403. Academic Press, 1972. doi: 10.4135/9781446212967.n15. URLhttps://sk.sagepub. com/books/cognitive-psychology/n15.xml

    work page doi:10.4135/9781446212967.n15 1972

  55. [55]

    Episodic memory: From mind to brain.Annual Review of Psychology, 53(1):1–25, 2002

    Endel Tulving. Episodic memory: From mind to brain.Annual Review of Psychology, 53(1):1–25, 2002. doi: 10.1146/annurev.psych.53.100901.135114. URLhttps://doi.org/10.1146/annurev.psych.53.100901.135114

    work page doi:10.1146/annurev.psych.53.100901.135114 2002

  56. [56]

    2025.doi: 10.48550/arXiv.2506

    David Wan, Han Wang, Elias Stengel-Eskin, Jaemin Cho, and Mohit Bansal. Clamr: Contextualized late- interaction for multimodal content retrieval.arXiv preprint arXiv:2506.06144, 2025. doi: 10.48550/arXiv.2506. 06144. URLhttps://arxiv.org/abs/2506.06144

    work page doi:10.48550/arxiv.2506 2025

  57. [57]

    Videoagent: Long-form video understanding with large language model as agent

    Xiaohan Wang, Yuhui Zhang, Orr Zohar, and Serena Yeung-Levy. Videoagent: Long-form video understanding with large language model as agent. InECCV, pages 58–76. Springer, 2024. doi: 10.1007/978-3-031-72989-8_4. URLhttps://link.springer.com/10.1007/978-3-031-72989-8_4

    work page doi:10.1007/978-3-031-72989-8_4 2024

  58. [58]

    Teaching CLIP to count to ten

    Xin Wang, Taein Kwon, Mahdi Rad, Bowen Pan, Ishani Chakraborty, Sean Andrist, Dan Bohus, Ashley Feniello, Bugra Tekin, Felipe Vieira Frujeri, Neel Joshi, and Marc Pollefeys. Holoassist: An egocentric human interaction dataset for interactive ai assistants in the real world. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 2...

    work page doi:10.1109/iccv51070.2023.01854 2023

  59. [59]

    Smith, Daniel Khashabi, and Hannaneh Hajishirzi

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume1: Long Papers), pages 13484–13508, Toronto, Canada, 2023. Association fo...

    work page doi:10.18653/v1/2023.acl-long.754 2023

  60. [60]

    Efficient Guided Generation for Large Language Models

    BrandonT.WillardandRémiLouf. Efficientguidedgenerationforlargelanguagemodels. CoRR,abs/2307.09702,

    work page internal anchor Pith review Pith/arXiv arXiv

  61. [61]

    Efficient Guided Generation for Large Language Models

    doi: 10.48550/arXiv.2307.09702. URLhttps://arxiv.org/abs/2307.09702

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2307.09702

  62. [62]

    Transferable multi-domain state generator for task-oriented dialogue systems

    Chien-Sheng Wu, Andrea Madotto, Ehsan Hosseini-Asl, Caiming Xiong, Richard Socher, and Pascale Fung. Transferable multi-domain state generator for task-oriented dialogue systems. InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 808–819. Association for Computational Lin- guistics, 2019. doi: 10.18653/v1/P19-...

    work page doi:10.18653/v1/p19-1078 2019

  63. [63]

    Efficient streaming language models with attention sinks

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. InThe TwelfthInternational Conference on Learning Representations, 2024. URLhttps: //openreview.net/forum?id=NG7sS51zVF. 22

    2024

  64. [64]

    Adavideorag: Omni- contextual adaptive retrieval-augmented efficient long video understanding

    Zhucun Xue, Jiangning Zhang, Xurong Xie, Yong Liu, Xiangtai Li, Dacheng Tao, et al. Adavideorag: Omni- contextual adaptive retrieval-augmented efficient long video understanding. InNeurIPS, 2025. doi: 10.48550/ arXiv.2506.13589. URLhttps://openreview.net/forum?id=FDAI0PY9Qp

    work page arXiv 2025

  65. [65]

    Qwen3 Technical Report

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  66. [66]

    Reading recognition in the wild

    Charig Yang, Samiul Alam, Shakhrul Iman Siam, Michael J Proulx, Lambert Mathias, Kiran Somasundaram, Luis Pesqueira, James Fort, Sheroze Sheriffdeen, Omkar Parkhi, Yuheng Ren, Mi Zhang, Yuning Chai, Richard Newcombe, and Hyo Jin Kim. Reading recognition in the wild. InAdvances in Neural Information Processing Systems, 2025. URLhttps://nips.cc/virtual/2025...

    2025

  67. [67]

    Optimus-2: Multimodal minecraft agent with goal-observation-action conditioned policy

    Jingkang Yang, Shuai Liu, Hongming Guo, Yuhao Dong, Xiamengwei Zhang, Sicheng Zhang, Pengyun Wang, Zitang Zhou, Binzhu Xie, Ziyue Wang, et al. Egolife: Towards egocentric life assistant. InCVPR, pages 28885– 28900, 2025. doi: 10.1109/cvpr52734.2025.02690. URLhttps://ieeexplore.ieee.org/document/11095171/

    work page doi:10.1109/cvpr52734.2025.02690 2025

  68. [68]

    Memory-enhanced retrieval augmentation for long video understanding.arXiv preprint arXiv:2503.09149, 2025

    Huaying Yuan, Zheng Liu, Minghao Qin, Hongjin Qian, Yan Shu, Zhicheng Dou, Ji-Rong Wen, and Nicu Sebe. Memory-enhanced retrieval augmentation for long video understanding.arXiv preprint arXiv:2503.09149, 2025. doi: 10.48550/arXiv.2503.09149. URLhttps://arxiv.org/abs/2503.09149

    work page doi:10.48550/arxiv.2503.09149 2025

  69. [69]

    I cannot find the black scissors. Where did I leave them last?

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judg- ing llm-as-a-judge with mt-bench and chatbot arena. In Advances in Neural Information Processing Systems, volume 36, pages 46595–46623, 2023. URLhttps://papers.nips.cc/pap...

    2023