{"id":"6e64a231-06d2-4098-b626-3f9be8f1d646","arxiv_id":"2601.18157","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A planning agent that searches a time-annotated entity graph plus visual and audio retrieval beats prior methods on week-long egocentric video QA.","lead":"EGAgent builds a time-stamped map of people, places, and objects from week-long glasses video, then uses a planning agent to search it along with frames and transcripts, reaching 57.5% on EgoLifeQA. It is a step toward always-on personal assistants that can answer detailed questions about days of your life.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Entity-graph ablation is confounded: adding EG also adds the fused captions (C) used to build it, so the RelationMap gains may be from extra context, not graph structure.","rationale":"The paper is well-engineered and the empirical gains are plausible, but the central attribution to entity graph reasoning is underdetermined by the current ablations. The least secure condition is not graph extraction accuracy per se but the experimental isolation of the graph contribution: the EG+F+T condition introduces both the graph tool and the fused captions used to build it, while the F+T baseline lacks the captions. This missing control is directly testable and would settle whether the RelationMap/TaskMaster gains come from structured graph reasoning or simply from exposing an LLM agent to richer fused-caption context. The reader's concern about upstream perception/transcript quality is valid and related, but it is secondary: even a perfect graph would not validate the causal claim without the F+T+C control. Since the paper already carries a conditional verdict and this concern strengthens the need for that condition rather than overturning the reported result, no verdict change is required.","tokens_in":29112,"tokens_out":14839,"duration_ms":168158,"concrete_test":"Add a control tool: replace the Entity Graph Search tool with a BM25 or embedding-based search over the same fused captions (C) used for graph construction, keeping visual and transcript tools, planner, analyzer, VQA, and token budget fixed. Run on all 500 EgoLifeQA MCQs with GPT-4.1 and Gemini 2.5 Pro backbones. If this caption-search control reaches ~50.7% (GPT-4.1) or ~57.5% (Gemini) and matches the RelationMap gain, the graph structure is not the active ingredient; if it falls materially short, the graph's structured retrieval is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table 5 isolates the entity graph by comparing EGAgent GPT-4.1 (F+T) (48.6 avg; RelationMap 40.0) with EGAgent GPT-4.1 (EG+F+T) (50.7 avg; RelationMap 53.6). But in the EG+F+T condition, the graph is extracted from fused captions plus transcripts (Appendix E), and the F+T baseline has no access to those fused captions at all. The condition therefore adds two things at once: the entity-graph search tool and the caption-derived content C. The same conflation affects the headline comparison in Table 1: EGAgent Gemini (EG+F+T) is compared to uniform-sampling Gemini (F,T), so the +10.7 attributed to graph reasoning also includes the addition of C. There is no control condition (F+T+C without the graph). Consequently the central claim that the entity scene graph—rather than the extra fused-caption information—drives the RelationMap/TaskMaster gains is not yet established. The sparse graph audit (100/13,968 edges) is a secondary issue; the missing control is more directly load-bearing because it determines whether the mechanism is graph structure at all.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes EGAgent, an agentic framework for question answering over very long (week-scale) egocentric video. The system first builds a temporally annotated entity scene graph from fused visual captions and audio transcripts, then uses a planning agent to decompose a query into subtasks and to call three retrieval tools: visual search over frame embeddings, audio transcript search, and SQL-style entity graph search. Retrieved evidence is distilled by an analyzer into a working memory that a VQA agent uses to produce the final answer. On EgoLifeQA, EGAgent with a Gemini 2.5 Pro backbone achieves 57.5% MCQ accuracy, a 20.6-point gain over EgoButler (Gemini 1.5 Pro) and 10.7 points over uniform-sampling Gemini 2.5 Pro; on Video-MME (Long) it reports 74.1%, behind native Gemini 2.5 Pro (82.0%) but ahead of other agentic baselines. The paper attributes the EgoLifeQA gains primarily to entity graph reasoning, especially on RelationMap and TaskMaster categories.","tokens_in":29469,"tokens_out":9228,"duration_ms":100938,"significance":"If the central claim is established, the paper makes a useful empirical contribution: it demonstrates that a structured, temporally localized entity graph can support multi-hop, cross-modal reasoning over week-long egocentric video, and it provides an extensive set of ablations, oracle upper bounds, recall analysis, and latency/memory measurements. The public code release, clear method description, and detailed appendices are strengths. However, the headline attribution of the gains to entity graph structure is currently under-supported because the graph is confounded with the addition of caption-derived content, and the reported numbers come from a single run without variance estimates. These issues are fixable with additional control experiments and uncertainty reporting, so the work is promising but not yet ready for acceptance.","major_comments":[{"comment":"The central claim that entity graph reasoning drives the RelationMap and TaskMaster gains is confounded. Table 5 compares EGAgent GPT-4.1 (F+T) with EGAgent GPT-4.1 (EG+F+T): in the latter condition the graph is extracted from fused captions and transcripts (C+T), so the comparison adds both the entity-graph search tool and the caption-derived content C. The same conflation appears in the headline Table 1 comparison (Gemini 2.5 Pro uniform F,T vs EGAgent Gemini EG+F+T). There is no control condition (F+T+C without the graph) that isolates the graph structure from the additional fused-caption information. Please add such a control — e.g., let the agent access the fused captions via a caption search tool or include them in the transcript tool — to demonstrate that the gains come from the entity graph itself rather than from the extra information content.","section":"Section 4.3 / Table 5 / Appendix E"},{"comment":"All accuracies are reported from a single run, as acknowledged in Appendix E. For a benchmark paper whose main result is a 20.6-point SOTA gap over a strong baseline, point estimates with no variance are not sufficient, especially because LLM-based agent pipelines can have run-to-run variability despite temperature-0 settings. Please report at least 3 runs for the main conditions in Tables 1 and 5 (or provide bootstrap confidence intervals over the 500 EgoLifeQA questions), and state the random seed or API version used. This is important for assessing whether the reported gains are statistically reliable.","section":"Appendix E (Implementation Details)"},{"comment":"The entity graph quality audit covers only 100 of the 13,968 extracted relationships (94% accuracy). Since the paper explicitly acknowledges that graph construction depends on upstream perception and language models, a 100-edge sample gives a wide confidence interval and does not characterize failures across relation types or temporal intervals. Please either enlarge the audit, report a confidence interval, or provide a breakdown of error types (e.g., wrong entity, wrong relation, wrong timestamp). This is secondary to the confound in the ablation, but it matters for the claimed robustness of the graph-based mechanism.","section":"Appendix D.5 / Section 6"}],"minor_comments":[{"comment":"The column labeled 'A verage' appears to be an overall accuracy that is not the simple mean of the five category accuracies. For example, EGAgent GPT-4.1 (F+T) reports 48.6 while the five categories listed average to about 50.8; EGAgent GPT-4.1 (EG+F+T) reports 50.7 while the categories average to about 53.8. Please clarify whether 'average' is a weighted overall accuracy, and if so, report per-category question counts. This will let readers verify the table.","section":"Tables 1 and 5"},{"comment":"The relation types are written with hyphens in Equation (3) (talks-to, interacts-with) but as uppercase underscores elsewhere (TALKS_TO, INTERACTS_WITH). Please make the notation consistent.","section":"Section 3.2, Equation (3)"},{"comment":"The caption says the current state-of-the-art is bolded and the second-highest is underlined, but the formatting is not visible in the text. If the camera-ready version uses actual bolding/underlining, this is fine; otherwise please state the ranking explicitly (e.g., Gemini 2.5 Pro is first, EGAgent Gemini 2.5 Pro is second).","section":"Table 2 caption"},{"comment":"There is an apparent stray annotation in the caption ('D: se other diagram to reflect that this is a set of doc') that looks like an editing note rather than intended text. Please remove or correct it.","section":"Figure 3 caption"},{"comment":"The text says 'when we add the entity graph search tool T oolaud', but T oolaud is the audio transcript tool; the sentence should refer to the entity graph tool T ooleg. Please fix the notation.","section":"Appendix D.1, final paragraph"}],"recommendation":"major_revision","confidential_remarks":"The paper is well-engineered and the empirical results are potentially interesting, but the key mechanism claim — that the entity graph itself, rather than the extra caption-derived content, causes the EgoLifeQA gains — is not yet established by the provided ablations. The missing control is a straightforward fix, and the single-run issue is also addressable. If the authors add the control and uncertainty estimates, the paper could be a solid contribution to long-horizon egocentric video understanding. I would not reject; the issues are within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know about this paper if you care about very-long egocentric video QA. EGAgent builds a temporally-annotated entity scene graph, exposes it to a planning agent through SQL search with a strict-to-relaxed relaxation strategy, and combines that with visual and transcript tools. On EgoLifeQA it gets 57.5% with a Gemini 2.5 Pro backbone, a 20.6-point jump over the prior SOTA, and the per-category gains are largest exactly where the paper says they should be: RelationMap and TaskMaster. That is a real, practical result.\n\nWhat is genuinely new: the combination of time-stamped edges on an entity graph, SQL-based agentic querying, and a relaxation hierarchy that recovers from noisy extraction. The paper also does more homework than most: oracle search experiments, retrieval recall at multiple temporal windows, per-tool ablations, latency/memory tables, and public code. The limitations section explicitly flags dependence on upstream perception and manual diarization, which is honest.\n\nSoft spots, in proportion. First, all numbers come from a single run, with no variance estimates; the paper says this in Appendix E, and for agentic systems it is understandable, but it still limits confidence in the 2-3 point margins. Second, the entity graph audit covers only 100 of 13,968 relations, so the 94% accuracy claim is a weak reed. Third — the point that matters most — Table 5's comparison of EGAgent (F+T) vs (EG+F+T) adds two things at once: the graph search tool and the fused captions used to build it. The RelationMap jump from 40.0 to 53.6 is therefore not cleanly attributable to graph structure alone. That said, Table 3 shows that switching the graph-extraction input from transcripts to fused captions only buys about 1.5 points for GPT-4.1, while the RelationMap gain in Table 5 is about 13.6 points. So the graph itself is almost certainly carrying the load, but a control condition (F+T+C without the graph) would settle it. The relation schema being tuned on EgoLife-like data is a lesser concern; the authors hand-wave that it is prompt-defined and domain-agnostic, which is plausible but untested.\n\nBottom line: this is a solid, well-engineered systems paper with a useful new architecture and evidence that is mostly on point. It deserves a serious referee. I would ask the authors to add a clean control, report variance or at least multiple runs, and expand the graph audit sample. If those land, this is a clear accept.","headline":"A genuinely useful agentic framework for week-scale egocentric video, with strong EgoLifeQA gains and honest ablations; the main caveat is that one central ablation conflates the entity graph with the extra fused captions, though the graph is still very likely doing real work.","tokens_in":29917,"tokens_out":2107,"would_cite":true,"duration_ms":23176,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper seeks to establish that a planning agent with a time-stamped entity scene graph, queried through SQL alongside visual and audio search, enables compositional multi-hop reasoning over week-long egocentric video—beating uniform-samp","keywords":["very long video understanding","entity scene graph","egocentric video","agentic retrieval","multi-hop reasoning","temporal localization","EgoLifeQA","long-context video question answering"],"falsifier":"Run EGAgent on EgoLifeQA with the entity-graph tool disabled but visual and transcript search intact; if accuracy stays near 57.5% instead of falling to roughly the 48.6% reported for the no-graph variant, the graph is not doing the causal work. Alternatively, replace EgoLife's manual diarization with an off-the-shelf diarizer and observe whether RelationMap and TaskMaster scores drop sharply, or enlarge the manual audit of graph edges to all 13,968 and see whether the 94% accuracy rate survives.","tokens_in":29062,"feed_emoji":"🎥","tokens_out":6838,"duration_ms":76876,"temperature":0.7,"pith_summary":"The paper is trying to establish that very long video understanding, especially week-long egocentric footage from always-on wearable cameras, requires an explicit time-aware entity graph rather than just larger context windows or better frame retrieval. It proposes EGAgent, an agent that decomposes each query into sub-tasks, searches three sources—visual frames, transcripts, and the entity graph—and accumulates evidence before producing an answer. The entity graph is built by an LLM from fused visual captions and speaker-labeled transcripts, with nodes for people, places, and objects and edges for four relation types, each annotated with start and end times. On EgoLifeQA, EGAgent reaches 57.5% multiple-choice accuracy, up from 36.9% for the previous best EgoButler and 46.8% for uniform-sampling Gemini 2.5 Pro, with the largest gains on RelationMap and TaskMaster categories that require multi-hop relational reasoning. A sympathetic reader would care because this is the operating regime of future personal assistants: persistent, cross-day memory over continuous video from a first-person perspective.","feed_headline":"Entity graphs push week-long video question answering to 57.5%","feed_subtitle":"Searching a timestamped graph of people, places, and objects beats the prior best by 20.6 points on EgoLifeQA.","key_machinery":"The load-bearing object is the time-annotated entity scene graph G=(V,E). Nodes are entities typed as person, object, or location; edges are relations drawn from a fixed four-type schema (talks-to, interacts-with, mentions, uses), each stored as a tuple (source, target, relation, start time, end time, supporting text snippet) in a SQLite table. The graph is built incrementally by an LLM extractor over 30-second fused captions and transcripts, and it preserves identity and temporal ordering across days—information that flat retrieval loses. The second mechanism is the planner's strict-to-relaxed SQL query strategy, which broadens time windows, entity names, and relation types in stages, so no","core_discovery":"The paper's central claim is that compositional, multi-hop questions about week-long video—\"Who was with us the last time we took the car?\" or \"Who usually speaks least in group discussions?\"—cannot be answered by flat retrieval over frames and captions, because entity identities and relationships dissolve across days. EGAgent therefore extracts an entity scene graph from fused visual captions and diarized transcripts, with nodes typed as person, object, or location, and edges constrained to four relation types (talks-to, interacts-with, mentions, uses), each edge carrying a start/end time and the supporting snippet. At inference, a planning agent generates sub-tasks, issues strict-to-relaxe","pith_inferences":["If the entity graph is intended as persistent memory for wearable assistants, one testable extension is to log query patterns over graph edges and measure whether repeated personal questions can be answered without re-encoding raw video frames.","The paper's fixed four-relation schema trades granularity for robustness; an editorially suggested stress test is to re-tune the prompt schema for other domains (offices, kitchens, hospitals) and measure whether a finer relation taxonomy improves or fragments retrieval recall.","The reliance on manually diarized transcripts suggests a concrete external test: substitute off-the-shelf speaker diarization and watch whether RelationMap and TaskMaster scores collapse—if they do, the bottleneck shifts from reasoning to perception.","Because each graph edge carries a supporting text snippet, the system could be extended to provide evidence citations for its answers, enabling users to verify longitudinal claims by jumping directly to the relevant moment."],"forward_implications":["Entity graph reasoning lifts the multi-hop relational categories by roughly 20 percentage points over prior state of the art and by 17–21 points over a strong uniform-sampling baseline, suggesting structured representations matter more than raw context-window size.","Combined visual, audio, and graph search achieves 0.884 recall at a 10-second temporal window on EgoLifeQA, showing that the agent's overall temporal localization is strong, not just its retrieval of content.","The entity graph adds only about 12.8% inference overhead while occupying about 2 MB of storage for 52 hours of video, making it a lightweight persistent index for always-on assistants.","On Video-MME (Long), EGAgent matches AdaVideoRAG with over 10x fewer frames, indicating that graph-based retrieval is efficient even in the 30–60-minute regime.","Oracle experiments with perfect temporal localization saturate at 68.7% accuracy with GPT-4.1 and 57.9% with transcript-only search, implying that retrieval is not the only bottleneck—multimodal reasoning over retrieved evidence also limits performance."],"fun_headline_variants":["Entity scene graphs answer week-long video questions","Graph agent solves multi-hop Q&A over days of video","Week-long video reasoning via entity graph planning","Entity graphs boost EgoLifeQA Q&A to 57.5%","Graph-based agent recalls people, places across days"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The whole system leans on the LLM that builds the entity graph: if it misses, mislabels, or mis-times relationships between people, places, and objects, the multi-hop reasoning that drives the EgoLifeQA gains has nothing to reason over—and the authors audit only 100 of 13,968 extracted relationships (94% accurate) while depending on manually diarized transcripts.","fun_headline_variants_meta":{"raw":{"variants":["Entity scene graphs answer week-long video questions","Graph agent solves multi-hop Q&A over days of video","Week-long video reasoning via entity graph planning","Entity graphs boost EgoLifeQA Q&A to 57.5%","Graph-based agent recalls people, places across days"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000186,"raw_usage":{"total_tokens":1170,"prompt_tokens":762,"completion_tokens":408,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":506,"completion_tokens_details":{"reasoning_tokens":331}},"tokens_in":506,"tokens_out":408,"duration_ms":5371,"temperature":1.0,"reasoning_tokens":331,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T08:02:57.793811+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run EGAgent on EgoLifeQA with the entity-graph tool disabled but visual and transcript search intact; if accuracy stays near 57.5% instead of falling to roughly the 48.6% reported for the no-graph variant, the graph is not doing the causal work. Alternatively, replace EgoLife's manual diarization with an off-the-shelf diarizer and observe whether RelationMap and TaskMaster scores drop sharply, or enlarge the manual audit of graph edges to all 13,968 and see whether the 94% accuracy rate survives.","supporting_citations":[],"review_version":1}