How Far Can Off-the-Shelf Multimodal Large Language Models Go in Online Episodic Memory Question Answering?

Antonino Furnari; Giovanni Maria Farinella; Giuseppe Lando; Rosario Forte

REVIEW 3 major objections 5 minor 14 references

How Far Can Off-the-Shelf Multimodal Large Language Models Go in Online Episodic Memory Question Answering?

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A training-free two-stage pipeline matches dedicated online video-question-answering systems while storing only kilobytes of text per minute.

desk verdict A useful training-free baseline for OEM-VQA, but the headline 'matches SOTA' needs a like-for-like split and error bars before I'd trust the exact number. read the letter →

arxiv 2506.16450 v1 pith:2NHOXSZA submitted 2025-06-19 cs.CV

Giuseppe Lando , Rosario Forte , Giovanni Maria Farinella , Antonino Furnari This is my paper

classification cs.CV

keywords onlinevideoquestionansweringegocentricepisodicmemorymultimodallargelanguagemodelstextualzero-shotpipelineQAEgo4Defficiency

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether off-the-shelf multimodal large language models, used with no task-specific training, can handle online episodic-memory video question answering. It answers yes: a two-stage pipeline turns a streaming egocentric video into a few kilobytes of first-person text per minute, then lets a text-only reasoning model answer four-choice questions, reaching 56.0% accuracy on the QAEgo4D-Closed benchmark. That matches the best dedicated system in the comparison while storing roughly 100,000 times less auxiliary memory. The paper also shows which design choices matter: clip length, question templates in the captioning prompt, and chain-of-thought reasoning in the answerer.

What carries the argument

The load-bearing mechanism is the textual memory: instead of storing video tokens or key-value caches, the pipeline lets an MLLM describe each non-overlapping clip (30 s for Gemini, 15 s for LLaVaOneVision) in first-person narrative, concatenates these descriptions into a persistent human-readable memory, and at query time feeds the whole memory plus the four options to a text-only LLM. The prompt to the descriptor is engineered with question templates that act as soft supervision, and the paper finds that feeding the previous caption back as context hurts accuracy, which it attributes to hallucination snowballing.

What would settle it

Run the Gemini-based pipeline and the ReKV-LLaVaOneVision 7B baseline on the identical QAEgo4D-Closed split, using the same answer-selection procedure; if the 56.0% versus 56.0% equality widens outside sampling noise, the paper's central equality claim fails.

Watch

Extended reading notes

Core claim

The central claim is that a training-free pipeline can match state-of-the-art OEM-VQA performance: a descriptor MLLM (Gemini 2.0 Flash Thinking for the best configuration) generates a compact textual memory of about 3.6 kB per video minute, and a separate LLM reasoner (DeepSeek R1) selects answers from it, attaining 56.0% accuracy on QAEgo4D-Closed. The paper further claims that using LLaVaOneVision 7B as the descriptor reaches 51.88%, and that the memory cost of the approach is orders of magnitude below existing methods while accuracy remains on par.

Load-bearing premise

The headline match with dedicated systems assumes every accuracy number in Table 1 comes from the same QAEgo4D-Closed split and protocol, but the paper never states its split and Figure 4 shows the validation set.

Editorial extensions

If this is right

If the claim holds, OEM-VQA does not require training a dedicated video model; any capable descriptor MLLM and reasoner LLM can be combined in a few lines of prompting.
Auxiliary memory for long egocentric video can drop from hundreds of megabytes per minute to a few kilobytes per minute, making on-device or long-horizon deployment plausible.
Because the memory is plain text, failures become inspectable: a wrong answer can be traced to a missing or vague caption rather than to an opaque encoder.
Lightweight variants still work, with Gemini Flash Lite dropping only 0.4 points and small reasoners costing around 10 points, so the accuracy-memory trade-off is tunable.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the memory is the bottleneck, then retrieval-based memory selection or adaptive clip segmentation should improve accuracy on long videos, a direction the authors mention in their outlook.
The negative effect of previous-caption context is a concrete, testable instance of hallucination snowballing in long-form captioning: injecting a single wrong fact into an earlier caption should measurably corrupt the answer.
Because descriptor prompts with question templates help so strongly, the same pipeline could be steered toward other recall tasks, such as object locations, counts, or names, simply by changing the template list.
The paper leaves open whether the 56.0% match is measured against the same data split as the compared systems; a shared evaluation would make the equality claim exact.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is a sensible, clearly-written engineering study. The idea is simple—stream the video, caption chunks with an off-the-shelf MLLM, answer questions from the text via an LLM—and the paper shows it gets surprisingly far: 56% on QAEgo4D-Closed with ~3.6 kB/min, compared to 56% for ReKV-LLaVaOneVision 7B at ~330 MB/min. The memory reduction is real and practically interesting. The ablations are the actual contribution: removing the previous caption as context consistently helps both descriptor models, which they attribute to hallucination snowballing, and adding question templates gives a large boost. Those findings are worth knowing.

The soft spots are real. The paper never says which split of QAEgo4D-Closed Table 1 is on. Figure 4 says validation, and the ablations used to pick the best config are on that same set. Published SOTA numbers for ReKV and GroundVQA are typically on the test split. If that's the case here, the 56.0 vs 56.0 equality is not like-for-like, and the best-config-on-validation selection adds optimistic bias. There are also no error bars, no number of questions, no mention of repeated runs, so even within a split the match could be noise. The memory metric is inconsistently defined: the LLaVaOneVision config captions every 15s but only keeps one caption per 30s at query time, so the reported '5.05 KB/min' is the query-time footprint, not the stored memory. If you count all generated captions, it's larger. And since Gemini 2.0 Flash Thinking is closed and no code is provided, the exact numbers aren't reproducible.

None of this kills the central idea—a training-free textual-memory pipeline is clearly viable and the ablations are informative. But the paper's headline quantitative claim is fragile. A serious revision should state the split, report variance, align the memory metric with what is actually stored, and ideally release code and prompts.

I'd send it to review—the question is relevant, the results are directionally sound, and the ablation findings are useful. But the referees should push hard on the split and the error bars.

Referee Report

3 major / 5 minor

Summary. The paper proposes a training-free pipeline for online episodic-memory video question answering (OEM-VQA). A video is segmented into clips, each described by a multimodal LLM ('descriptor'), the resulting text is concatenated into a compact memory, and a text-only LLM ('reasoner') answers multiple-choice questions by reading that memory. The method is evaluated on QAEgo4D-Closed, where the best configuration, using Gemini 2.0 Flash Thinking as descriptor and DeepSeek R1 as reasoner, is reported to reach 56.0% accuracy with 3.57 kB of stored memory per video minute, matching the accuracy of ReKV-LLaVaOneVision 7B while being orders of magnitude more memory-efficient. Extensive ablations vary the descriptor model, clip length, inclusion of previous captions, question templates, and reasoner model.

Significance. If the headline quantitative claim holds, this is a valuable result: it would show that off-the-shelf multimodal LLMs can perform online episodic-memory video QA without any task-specific training, and that a few kilobytes of textual memory per minute can rival dedicated systems that store hundreds of megabytes. The ablation study is also useful, particularly the finding that removing the previous caption from the descriptor prompt improves accuracy, and that question templates give a large boost for the smaller descriptor model. However, the central claim of matching state-of-the-art performance is currently under-supported because the evaluation split is not stated in the comparison table, the best configuration is selected on the same data used for the headline number, and no uncertainty estimates are provided.

major comments (3)

[Section 4.1, Table 1] The evaluation split used for the reported accuracies is not stated. Figure 4 explicitly labels its qualitative examples as coming from the validation set, while the published SOTA numbers for ReKV and GroundVQA are typically reported on the official test split. If Table 1 compares the proposed method's validation-set accuracy with prior methods' test-set accuracy, the claimed 56.0% vs. 56.0% equality with ReKV-LLaVaOneVision 7B is not established. Please state the exact split used for every number in Table 1 and confirm that the compared methods were evaluated under the same protocol and split.
[Section 4.2, Tables 2--5] The best configuration, which produces the headline 56.0% accuracy, was selected by varying the descriptor model, clip length, previous-caption context, question templates, and reasoner model on the same benchmark set used for the final comparison. This is model selection on the evaluation set, and the reported number is the maximum over many configurations; it is therefore optimistically biased relative to a configuration chosen without access to the test labels. The authors should either report performance of the selected configuration on a held-out test split (with the ablations performed on a separate validation split) or clearly state that the reported number is a validation-set result obtained after model selection, and adjust the comparison accordingly.
[Section 4.1, Table 1] No error bars, confidence intervals, or number of test questions are reported for any of the accuracies. A single run of 56.0% vs. 56.0% does not establish that the proposed method matches ReKV, because the difference could be within sampling noise. Please report the number of questions evaluated, the standard deviation over repeated runs if feasible, or a bootstrap confidence interval for the key comparisons so that the matching claim is statistically meaningful.

minor comments (5)

[Abstract] The sentence 'Extensive ablations provides insights' has a subject-verb agreement error; it should read 'Extensive ablations provide insights'.
[Table 1] The method name 'RekV-LLava1v 0.5' appears to be a typo; it should likely read 'ReKV-LLaVaOneVision 0.5B' to match the rest of the paper.
[Section 4.2, Table 5 discussion] The sentence comparing DeepSeek R1 and DeepSeek V3 states '51.88% vs 57.69% in the case of LLaVaOneVision', but Table 5 reports 51.88% for DeepSeek R1 and 47.69% for DeepSeek V3 for that descriptor. The '57.69%' appears to be a typo and should be corrected to '47.69%'.
[Figure 4 and Table 1] Figure 4 states its examples are from the validation set, but Table 1 does not indicate the split. If the final comparisons are indeed on the validation set, this should be stated explicitly in the table caption and in the abstract to avoid misleading readers.
[General] The paper interchangeably uses 'MLLM' and 'VLLM' (e.g., 'Video Language Models (MLLMs)' in Section 2); please unify the terminology and expand all abbreviations consistently.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper is an empirical benchmark study with no fitted parameters, no self-citation chain, and no prediction that reduces to its inputs by construction.

full rationale

The paper's central claim is an empirical comparison: an off-the-shelf MLLM-based descriptor compresses video into textual memory and an LLM reasoner answers multiple-choice questions, achieving 56.0% accuracy on QAEgo4D-Closed. There is no derivation chain that could be circular. The memory is simply defined as the concatenation of clip-level descriptions (Eq. 1), and the answer is produced by a reasoner module over that memory (a = Q(m, q)). No parameter is fitted to the benchmark and then renamed as a prediction; the ablations select a configuration from measured validation accuracy, which is model selection, not circular reasoning. The comparison with ReKV and GroundVQA relies on externally reported numbers, not on the authors' own prior work, and no uniqueness theorem or ansatz is imported from self-citations. The potential concern that Table 1 may mix validation- and test-split results, or that the best configuration is chosen on the same validation set used for the headline number, is a correctness/rigor issue about evaluation protocol, not a circularity in the sense of a claim being equivalent to its inputs by definition. The paper is self-contained as an empirical study, and no step reduces to its own inputs.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The paper's claim is empirical, so the ledger consists of evaluation choices and benchmark assumptions rather than mathematical axioms. No new entities are postulated, and the free parameters are configuration choices selected on validation data.

free parameters (6)

descriptor_model = Gemini 2.0 Flash Thinking (vs LLaVaOneVision 7B)
Model family chosen after ablation; directly affects accuracy and memory size.
clip_length = 30 s for Gemini, 15 s for LLaVaOneVision
Ablated over {30, 15, 5} seconds; best per model selected on validation, controlling memory granularity and accuracy.
include_previous_description = False
Ablated; removing context improved accuracy from 51.6% to 56.0% (Gemini) and from 47.9% to 51.9% (LLaVa).
include_question_templates = True
Ablated; templates improved accuracy from 50.8% to 56.0% (Gemini) and from 32.2% to 51.9% (LLaVa).
reasoner_model = DeepSeek R1
Ablated across four LLMs; DeepSeek R1 selected for best accuracy.
query-time caption downsampling interval = one caption per 30 s
When clip length is 15 s or 5 s, the context becomes too large, so only one caption every 30 s is kept at query time; this changes the effective memory size and is not separately ablated.

assumptions (4)

domain assumption QAEgo4D-Closed multiple-choice accuracy is a valid measure of episodic-memory question answering ability.
Used as the sole evaluation benchmark in Section 4; no other metric or task is considered.
domain assumption Comparing methods by auxiliary memory size in KB/MB per minute is a fair efficiency measure, despite different ingestion computations.
Section 4.1 compares text memory to KV-cache sizes without accounting for the compute cost of generating descriptions or the different roles of the stored data.
domain assumption The off-the-shelf MLLMs have not memorized QAEgo4D-Closed through pretraining.
Implicit in the 'off-the-shelf zero-shot' framing; closed-source APIs make contamination untestable, especially for Gemini.
domain assumption Textual captions preserve enough information to answer the multiple-choice questions.
The entire pipeline depends on the descriptor's lossy text summaries; the authors concede in Section 4.4 that missing details cause errors.

how reviews work

0 comments

Cite this review

Pith. "Pith review of How Far Can Off-the-Shelf Multimodal Large Language Models Go in Online Episodic Memory Question Answering?." pith.science (2026). https://pith.science/paper/2NHOXSZA

@misc{pith2026250616450,
  author       = {Pith},
  title        = {Pith review of: How Far Can Off-the-Shelf Multimodal Large Language Models Go in Online Episodic Memory Question Answering?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2NHOXSZA}},
  note         = {Machine review of arXiv:2506.16450}
}

read the original abstract

We investigate whether off-the-shelf Multimodal Large Language Models (MLLMs) can tackle Online Episodic-Memory Video Question Answering (OEM-VQA) without additional training. Our pipeline converts a streaming egocentric video into a lightweight textual memory, only a few kilobytes per minute, via an MLLM descriptor module, and answers multiple-choice questions by querying this memory with an LLM reasoner module. On the QAEgo4D-Closed benchmark, our best configuration attains 56.0% accuracy with 3.6 kB per minute storage, matching the performance of dedicated state-of-the-art systems while being 10**4/10**5 times more memory-efficient. Extensive ablations provides insights into the role of each component and design choice, and highlight directions of improvement for future research.

Figures

Figures reproduced from arXiv: 2506.16450 by the authors.

**Figure 1.** Offline vs. online episodic-memory QA. In the offline setting (left), the model re-processes the full video whenever it has to answer a new question. In this setting, the video itself acts as a form of high-dimensional memory and video processing happens with prior knowledge of the query. In online settings (right), video is processed in a streaming fashion to build a compact memory, which is later referenced to ans… view at source ↗

**Figure 2.** Method Overview The input egocentric video is divided into non-overlapping clips (c1, c2, . . . , cK), each processed independently by a Video LLM Descriptor to generate a textual description dk. These descriptions are concatenated to form a compact textual memory M, which is persistent and independent. At query time, a natural language question is combined with the accumulated memory and passed to an LLM Reasoner,… view at source ↗

**Figure 3.** Our Descriptor prompt (left) follows a rigid structure consisting of five main sections: • Task description; • Detailed instructions; • Question templates; • In-context examples; and • Previous descriptions. • Reasoner prompt example (right). without retaining raw visual data (see [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Qualitative examples from the validation set. Left: Video clips are processed by the descriptor into textual memory entries. Centre: two multiple-choice questions are presented to the reasoner, with a ✔ on the correct answer and a Æ on the answer chosen by the reasoner…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

14 extracted references · 6 canonical work pages

[1]

In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)

Bärmann, L., Waibel, A.: Where did i leave my keys? — episodic-memory-based question answering on egocentric videos. In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). pp. 1559–1567 (2022).https://doi.org/10.1109/CVPRW56347.2022.00162

arXiv 2022
[2]

In: CVPR (2024)

Di, S., Xie, W.: Grounded question-answering in long egocentric videos. In: CVPR (2024)

work page 2024
[3]

Di, S., Yu, Z., Zhang, G., Li, H., Zhong, T., Cheng, H., Li, B., He, W., Shu, F., Jiang, H.: Streaming video question-answering with in-context video kv-cache retrieval (2025),https://arxiv.org/abs/2503.00540

arXiv 2025
[4]

Grauman, K., et al.: Ego4d: Around the world in 3,000 hours of egocentric video (2022),https://arxiv.org/abs/2110.07058

arXiv 2022
[5]

arXiv preprint arXiv:2408.03326 (2024)

Li, B., Zhang, Y., Guo, D., Zhang, R., Li, F., Zhang, H., Zhang, K., Li, Y., Liu, Z., Li, C.: Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326 (2024)

arXiv 2024
[6]

Patel, A., Chitalia, V., Yang, Y.: Advancing egocentric video question answering with multimodal large language models (2025),https://arxiv.org/abs/2504. 04550

work page 2025
[7]

Qwen, :, et al.: Qwen2.5 technical report (2025),https://arxiv.org/abs/2412. 15115

work page 2025
[8]

In: Ranzato, M., Beygelz- imer, A., Dauphin, Y., Liang, P., Vaughan, J.W

Ryoo, M., Piergiovanni, A., Arnab, A., Dehghani, M., Angelova, A.: Token- learner: Adaptive space-time tokenization for videos. In: Ranzato, M., Beygelz- imer, A., Dauphin, Y., Liang, P., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems. vol. 34, pp. 12786–12797. Curran Associates, Inc. (2021),https://proceedings.neurips.cc/paper_fil...

work page 2021

Show all 14 references

[9]

Team, G., et al.: Gemini: A family of highly capable multimodal models (2024), https://arxiv.org/abs/2312.11805

2024 arXiv
[10]

Organization of memory1(381- 403), 1 (1972)

Tulving, E., et al.: Episodic and semantic memory. Organization of memory1(381- 403), 1 (1972)

1972
[11]

Wang, Y., Li, K., Li, X., Yu, J., He, Y., Wang, C., Chen, G., Pei, B., Yan, Z., Zheng, R., Xu, J., Wang, Z., Shi, Y., Jiang, T., Li, S., Zhang, H., Huang, Y., Qiao, Y., Wang, Y., Wang, L.: Internvideo2: Scaling foundation models for multimodal video understanding (2024),https:...

2024 arXiv
[12]

Yang, J., Liu, S., Guo, H., Dong, Y., Zhang, X., Zhang, S., Wang, P., Zhou, Z., Xie, B., Wang, Z., Ouyang, B., Lin, Z., Cominelli, M., Cai, Z., Zhang, Y., Zhang, P., Hong, F., Widmer, J., Gringoli, F., Yang, L., Li, B., Liu, Z.: Egolife: Towards egocentric life assistant (2025...

2025
[13]

Zhang, B., Li, K., Cheng, Z., Hu, Z., Yuan, Y., Chen, G., Leng, S., Jiang, Y., Zhang, H., Li, X., Jin, P., Zhang, W., Wang, F., Bing, L., Zhao, D.: Videollama 3: Frontier multimodal foundation models for image and video understanding (2025), https://arxiv.org/abs/2501.13106

2025 arXiv
[14]

In: Forty-first International Conference on Machine Learning (2024),https://openreview.net/forum?id=FPlaQyAGHu

Zhang, M., Press, O., Merrill, W., Liu, A., Smith, N.A.: How language model hallucinations can snowball. In: Forty-first International Conference on Machine Learning (2024),https://openreview.net/forum?id=FPlaQyAGHu

2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)

Bärmann, L., Waibel, A.: Where did i leave my keys? — episodic-memory-based question answering on egocentric videos. In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). pp. 1559–1567 (2022).https://doi.org/10.1109/CVPRW56347.2022.00162

arXiv 2022

[2] [2]

In: CVPR (2024)

Di, S., Xie, W.: Grounded question-answering in long egocentric videos. In: CVPR (2024)

work page 2024

[3] [3]

Di, S., Yu, Z., Zhang, G., Li, H., Zhong, T., Cheng, H., Li, B., He, W., Shu, F., Jiang, H.: Streaming video question-answering with in-context video kv-cache retrieval (2025),https://arxiv.org/abs/2503.00540

arXiv 2025

[4] [4]

Grauman, K., et al.: Ego4d: Around the world in 3,000 hours of egocentric video (2022),https://arxiv.org/abs/2110.07058

arXiv 2022

[5] [5]

arXiv preprint arXiv:2408.03326 (2024)

Li, B., Zhang, Y., Guo, D., Zhang, R., Li, F., Zhang, H., Zhang, K., Li, Y., Liu, Z., Li, C.: Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326 (2024)

arXiv 2024

[6] [6]

Patel, A., Chitalia, V., Yang, Y.: Advancing egocentric video question answering with multimodal large language models (2025),https://arxiv.org/abs/2504. 04550

work page 2025

[7] [7]

Qwen, :, et al.: Qwen2.5 technical report (2025),https://arxiv.org/abs/2412. 15115

work page 2025

[8] [8]

In: Ranzato, M., Beygelz- imer, A., Dauphin, Y., Liang, P., Vaughan, J.W

Ryoo, M., Piergiovanni, A., Arnab, A., Dehghani, M., Angelova, A.: Token- learner: Adaptive space-time tokenization for videos. In: Ranzato, M., Beygelz- imer, A., Dauphin, Y., Liang, P., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems. vol. 34, pp. 12786–12797. Curran Associates, Inc. (2021),https://proceedings.neurips.cc/paper_fil...

work page 2021