{"id":"b81f742b-16ca-4821-abee-4538e6890c3a","arxiv_id":"2505.24690","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Hier-EgoPack learns reusable task prototypes from support tasks and uses them to improve novel task learning across egocentric video benchmarks.","lead":"The paper introduces Hier-EgoPack, a unified framework that learns reusable task prototypes from multiple egocentric video tasks and uses them to help learn new tasks. It reports small but consistent improvements over prior baselines on five Ego4D benchmarks, suggesting that task-specific perspectives can be carried across video understanding tasks of different temporal scales.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported gains may stem from the added k-NN refinement and fusion modules rather than from the learned prototype content; Section 3.3 and Table 1 lack a control that ablates the information carried by the prototypes.","rationale":"The reader identifies the AR-only construction of prototypes as the weakest assumption. I read that as a plausible concern, but it is not the most load-bearing one. The paper's central claim is that the particular learned task perspectives ('reusable concepts') improve novel task learning. To support that claim, it is necessary to show that the information in the prototypes is causally responsible for the improvements. The experimental design in Table 1 compares the full Hier-EgoPack to a Single Task model that lacks the k-NN retrieval, SAGE refinement, and fusion modules entirely. This confound means any of those additions could explain the gains. The paper does provide Figure 3 as an analysis of activation consensus, but that is post-hoc and does not establish a causal role for prototype content. A random-prototype control is the standard way to isolate that role; its absence is a concrete gap. The concern is aggravated by the small effect sizes and the absence of error bars: if the effect is within noise, the finding vanishes. The reader's AR-transfer concern is a downstream question: even if we accept that prototypes matter, we can ask whether AR is the right source. But that question is secondary to whether the prototype content matters at all. The proposed concrete test settles the primary question directly. Since the paper otherwise reports a consistent pattern of modest improvements and makes code available, the right outcome is still conditional acceptance, pending the control experiment. Hence I recommend keeping the reader's CONDITIONAL verdict, i.e., UNCHANGED.","tokens_in":5403,"tokens_out":7017,"duration_ms":80926,"concrete_test":"Run the Hier-EgoPack pipeline from Section 3.3 on Ego4D with the learned prototypes Pk replaced by (a) randomly initialized frozen features of the same dimension and (b) prototypes built from the same features but with action labels randomly shuffled, keeping all other components (k-NN, SAGE refinement, fusion) and hyperparameters identical. Compute AR Top-1, OSCC accuracy, and MQ mAP for these controls and for the reported Hier-EgoPack, each over at least 3 random seeds with mean and standard deviation. If the control numbers are within one standard deviation of the reported Hier-EgoPack numbers, or remain above the Single Task baseline to the same extent, then the improvement is not due to the prototype content, and the central claim is unsupported. Also report the same metrics for Single Task to establish the noise floor.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that carrying learned task-specific prototypes improves novel task learning. The sole quantitative evidence (Table 1) compares Hier-EgoPack, which adds frozen prototypes, k-NN retrieval, SAGE graph refinement, and fusion (Section 3.3), to a Single Task baseline that uses the same hierarchical TDGC backbone but none of that machinery. Therefore the reported gains (AR verb 27.30 vs 26.93, OSCC 75.60 vs 75.22, MQ mAP 21.0 vs 20.2) are jointly attributable to all added components. The paper does not include a control that removes or degenerates the information content of the prototypes while keeping the rest of the pipeline intact. No random-prototype baseline, no shuffled-label prototype baseline, and no 'single global prototype' comparison is reported. Without such a control, the paper's central conceptual contribution, that the specific learned reusable concepts are what helps, is underdetermined. A further aggravating factor is that the absolute improvements are small (0.2-0.4 points), and no multiple-seed statistics or error bars are given, so it is conceivable the entire effect is noise. The most load-bearing concern is therefore not whether AR annotations are the ideal common thread (the reader's candidate) but whether the prototype content matters at all: if random prototypes give the same numbers, the method's success is due to the extra refinement architecture, not to reusable concepts.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Hier-EgoPack, a framework for egocentric video understanding that learns task-specific prototypes from multiple support tasks during multi-task pretraining and reuses them when learning a novel task. The architecture is a hierarchical temporal graph backbone with Temporal Distance Gated Convolution, task-specific projection necks and heads, and a k-NN/SAGE-based refinement that queries frozen prototypes. Experiments on Ego4D benchmarks report small improvements over single-task baselines (e.g., AR verb Top-1 27.30 vs 26.93, OSCC 75.60 vs 75.22, MQ mAP 21.0 vs 20.2) and a qualitative activation-consensus analysis.","tokens_in":5693,"tokens_out":4114,"duration_ms":49157,"significance":"If the claimed improvements are robust, the idea of carrying frozen task perspectives as a reusable backpack is a useful contribution to egocentric video understanding, and the unified architecture is interesting. However, the empirical support is currently insufficient because the reported gains are small, unaccompanied by uncertainty estimates, and not decomposed into contributions of prototype content versus the extra refinement machinery. The activation consensus analysis is qualitative and does not establish the causal role of the learned prototypes. The code release is a strength, as it will support reproducibility and further analysis.","major_comments":[{"comment":"The central claim that learned task-specific prototypes improve novel task learning is underdetermined because no control ablates the content of the prototypes. Hier-EgoPack differs from the Single Task baseline not only by adding the frozen prototype set but also by adding k-NN retrieval, SAGE graph refinement, and fusion; a random-prototype, shuffled-label, or single-global-prototype baseline would isolate whether the specific learned representations matter. Without such a control, the reported gains (e.g., AR verb 27.30 vs 26.93, OSCC 75.60 vs 75.22, MQ mAP 21.0 vs 20.2) may be attributable entirely to the extra refinement architecture rather than to reusable concepts.","section":"Section 3.3 and Table 1"},{"comment":"No measure of variance is reported. The differences between Hier-EgoPack and the Single Task baseline are 0.2–0.4 percentage points for the headline metrics, so without multiple seeds or significance tests the improvements may be within run-to-run noise.","section":"Table 1"},{"comment":"The decision to build prototypes exclusively from action recognition annotations is motivated by the claim that actions are the 'common thread behind the different tasks,' but this assumption is never tested. Please compare with prototypes constructed from other support tasks or from all available support data, and with a condition that keeps the same interaction machinery but uses a degenerate prototype set, to show that the AR-derived prototypes are the appropriate carrier of transfer.","section":"Section 3.2"},{"comment":"The activation consensus analysis is presented as evidence that different tasks activate diverse prototypes, but the text draws conclusions about effectiveness ('low average activations consensus and high diversity... shows how Hier-EgoPack is effectively integrating different perspectives') without tying this qualitative measure to any quantitative outcome. Reporting the correlation between consensus and task performance, or comparing consensus under controlled prototype manipulations, would make the analysis informative.","section":"Section 4 and Figure 3"}],"minor_comments":[{"comment":"The header 'AR Top-1 (%) Verb Noun' is ambiguous; clarify that Verb and Noun are separate columns, and similarly for OSCC Acc., and mark which columns are lower-better (LTA ED, PNR Err).","section":"Table 1"},{"comment":"The caption uses 'Activations consensus' and the matrices are asymmetric, but the text does not explain the asymmetry; also define the color scale.","section":"Figure 3"},{"comment":"The number of SAGE layers M and the fusion coefficients (if any) are not specified; report these hyperparameters in the experimental section.","section":"Section 3.3"},{"comment":"The values of the temporal edge threshold τ and the number of stages L per task are not reported, though L is said to be task-dependent; include this information to support reproducibility.","section":"Section 3.1"},{"comment":"The reference list is incomplete: reference [2] is missing author names, and the institutional affiliation footer is absent.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper extends the authors' own EgoPack [6] with a hierarchical temporal backbone and TDGC; while this is a legitimate incremental direction, the experiments do not isolate the novelty. The small absolute gains and the missing prototype-content ablation suggest that the manuscript needs substantial additional experiments before the central claim is supported. Please also check whether the support-task training videos and novel-task test videos on Ego4D share footage, since such overlap would inflate transfer results."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Hier-EgoPack is a solid, incremental extension of EgoPack. The new bits are a hierarchical temporal backbone with TDGC layers and a prototype-refinement step that fuses task perspectives via k-NN and SAGE. The paper is clearly written, the code is released, and the experiments cover five Ego4D tasks with consistent, though modest, gains over the single-task baseline.\n\nWhat it does well: the architecture is sensible for mixing fine-grained and long-range tasks, the TDGC layer is a simple way to encode relative temporal order, and the prototype-fusion idea is a natural continuation of the EgoPack program. Reporting results on five benchmarks at different temporal granularities is useful for the egocentric-vision community.\n\nThe main soft spot is that the central claim—that the learned prototype content is what helps—is not actually isolated. It compares Hier-EgoPack (backbone + prototypes + k-NN + SAGE refinement + fusion) against Single Task (backbone alone). A random-prototype or shuffled-label-prototype control would be needed to show the reused concepts carry the signal, rather than the extra refinement machinery. Without that, the 0.3–0.8 point improvements are plausibly attributable to the added parameters and fusion, and the paper overstates the role of 'reusable concepts.' The lack of error bars or multiple seeds makes it worse.\n\nAlso worth flagging, as a minor point: the prototypes are built from AR annotations, and the Ego4D support and novel tasks come from the same underlying video corpus. The paper doesn't discuss potential train/eval overlap with the support task's data. This may be harmless in practice, but it deserves a sentence.\n\nThe activation-consensus analysis is qualitative and doesn't directly support the quantitative gains. The 'paradigm shift' language in the intro is overblown, but that's rhetoric, not substance.\n\nDespite the missing ablations, the work is coherent and the extension is legitimate. I'd send it to review with a request for prototype-content ablations and multi-seed numbers. The audience is researchers working on egocentric video understanding or multi-task learning; for them it's a worthwhile read.","headline":"A useful incremental extension of EgoPack, but the central claim that learned prototype content drives the gains is undercut by missing ablations.","tokens_in":6182,"tokens_out":2282,"would_cite":false,"duration_ms":25355,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Hier-EgoPack claims that a frozen collection of task-specific prototypes can be carried as a backpack to improve learning of novel egocentric video tasks, with gains reported on five Ego4D benchmarks.","keywords":["egocentric video understanding","cross-task knowledge transfer","concept learning","prototype learning","temporal graph neural networks","Ego4D","action recognition","multi-task learning"],"falsifier":"An experiment that removes the action-recognition prototypes from the backpack, or replaces them with prototypes built from a different task's labels (e.g., Object State Change), and then checks whether the reported AR, OSCC, and MQ gains vanish would settle whether action labels are the load-bearing common thread.","tokens_in":5234,"feed_emoji":"🎒","tokens_out":7645,"duration_ms":77556,"temperature":0.7,"pith_summary":"This paper aims to establish that egocentric video understanding systems can abstract learned knowledge into a reusable 'backpack' of concepts and use it to learn new video tasks more effectively than solving each task from scratch. The proposed Hier-EgoPack represents a video as a temporal graph, shares a hierarchical backbone across tasks, and stores each support task's perspective as a set of frozen prototypes built from action-recognition annotations. At novel-task time, features are refined by k-NN retrieval and message passing over those prototypes, letting each old task contribute its perspective to the new prediction. On Ego4D the paper reports gains over single-task, multitask, EgoPack, and EgoT2 baselines—for example AR verb Top-1 rises from 26.93 to 27.30 and Moment Queries mAP from 20.2 to 21.0—suggesting cross-task knowledge transfer can work across very different temporal scales.","feed_headline":"A backpack of reusable concepts lifts five egocentric video tasks","feed_subtitle":"Novel tasks retrieve the frozen prototypes by k-NN and beat single-task and multitask baselines.","key_machinery":"The load-bearing object is the backpack, a set of frozen task-specific prototypes $P_k = \\{p_k^0, \\dots, p_k^P\\} \\subset \\mathbb{R}^{D}$, where each prototype is the aggregate of a unique verb-noun action label as seen by task $k$'s projection neck. The novel task's features act as queries; k-NN search selects neighboring prototypes and SAGE-convolution message passing refines the query features without updating the prototypes, preserving each task's original perspective. The second mechanism is the Temporal Distance Gated Convolution (TDGC) layer, which re-weights each neighbor's contribution by the sign and magnitude of the timestamp difference, letting the shared hierarchical backbone capture relative past/future order across temporal scales.","core_discovery":"The central claim is that knowledge from a set of support tasks can be abstracted into a compact, frozen backpack of task-specific prototypes and then exploited to learn a novel task better than training that task alone. Each prototype aggregates the features of one verb-noun action label from one task's perspective, produced by forwarding action-annotated videos through a shared temporal graph backbone and projecting them through that task's neck. At novel-task time the model treats its own features as queries, retrieves the closest prototypes via k-NN, and refines the features with SAGE-convolution message passing while leaving the prototypes untouched, so each support task effectively votes on the new prediction. The paper reports consistent improvements over single-task, multitask, EgoPack, and EgoT2 baselines on five Ego4D benchmarks, including AR verb Top-1 accuracy of 27.30 versus 26.93 for single-task training and Moment Queries mAP of 21.0 versus 20.2.","pith_inferences":["A testable extension the paper does not pursue is to build the backpack from a non-action task, such as Object State Change labels; if the transfer gains persist, the 'actions as common thread' assumption is not the only source of reusable knowledge.","The reported smaller gains on long-horizon tasks (LTA, MQ) suggest the backpack may transfer best when the novel task shares segment-level alignment with action recognition; an editor would infer that aligning prototypes to different temporal scales is the next bottleneck to attack.","Because the prototypes are frozen, the framework currently treats knowledge as static; an extension would be to update or grow the backpack as each novel task is learned, converting the method into a lifelong skill-acquisition system."],"forward_implications":["If the reported gains hold, an egocentric agent can be pre-trained once on a set of support tasks and then learn a new task without access to the old labels, using the frozen prototypes as priors.","Because the prototypes are frozen and only the novel task's neck and head are trained, the scheme avoids catastrophic forgetting and task interference that hard-parameter-sharing multitask learning suffers from.","The unified graph backbone with TDGC layers means the same architecture can handle both fine-grained tasks (AR, OSCC) and long-range tasks (LTA, MQ), so positive transfer can occur across different temporal scales.","The activation-consensus analysis indicates that different support tasks activate different prototypes for the same sample, so the gain comes from diverse task perspectives rather than a single shared representation."],"supporting_citations":[{"why":"Supplies the Ego4D dataset, the five task benchmarks, and the baseline scores that Hier-EgoPack compares against.","marker":"[3]"},{"why":"Provides the pretrained EgoVLP encoder that produces the initial segment features used as graph node inputs.","marker":"[5]"},{"why":"Introduces the EgoPack backpack-of-prototypes framework that Hier-EgoPack extends with hierarchical temporal reasoning.","marker":"[6]"},{"why":"Provides the EgoT2 task-translation baseline that Hier-EgoPack must beat in cross-task transfer.","marker":"[9]"}],"fun_headline_variants":["Backpack of reusable prototypes boosts five egocentric tasks","Portable skill backpack lifts novel task performance","Frozen prototypes as a queryable skill backpack","k-NN skill retrieval improves five Ego4D benchmarks","From support tasks to a reusable concept backpack"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The backpack is built exclusively from action-recognition annotations, so the whole transfer mechanism rests on the assumption that human action categories are the common thread linking all egocentric video tasks.","fun_headline_variants_meta":{"raw":{"variants":["Backpack of reusable prototypes boosts five egocentric tasks","Portable skill backpack lifts novel task performance","Frozen prototypes as a queryable skill backpack","k-NN skill retrieval improves five Ego4D benchmarks","From support tasks to a reusable concept backpack"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000208,"raw_usage":{"total_tokens":1355,"prompt_tokens":846,"completion_tokens":509,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":462,"completion_tokens_details":{"reasoning_tokens":436}},"tokens_in":462,"tokens_out":509,"duration_ms":5896,"temperature":1.0,"reasoning_tokens":436,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:14:40.445044+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"An experiment that removes the action-recognition prototypes from the backpack, or replaces them with prototypes built from a different task's labels (e.g., Object State Change), and then checks whether the reported AR, OSCC, and MQ gains vanish would settle whether action labels are the load-bearing common thread.","supporting_citations":[{"cited_title":"Understanding Video Transformers via Universal Concept Discovery","cited_arxiv_id":"2401.10831","evidence_quote":"Supplies the Ego4D dataset, the five task benchmarks, and the baseline scores that Hier-EgoPack compares against."},{"cited_title":"Egocentric video-language pretraining","cited_arxiv_id":null,"evidence_quote":"Introduces the EgoPack backpack-of-prototypes framework that Hier-EgoPack extends with hierarchical temporal reasoning."}],"review_version":1}