{"id":"7d110298-5359-41e8-a253-6ada045eb0aa","arxiv_id":"2506.02382","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A multi-modal action anticipation model with a clustering-based fine-grained text generator and temporal consistency loss reports state-of-the-art gains, though key claims are not fully supported by the provided evidence.","lead":"This paper introduces m&m-Ant, an action anticipation model that combines video features with fine-grained labels generated from the video itself, alongside a temporal consistency loss. It reports state-of-the-art accuracy gains on Breakfast, 50 Salads, and DARai, but some ablation results contradict the claimed consistency of the improvement.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper never defines the loss used to train the final action-anticipation module, so the reported 3.08% SOTA gain is not reproducible as written; internal ablation contradictions further undermine the consistency claim.","rationale":"The reader's verdict is REJECT, and I agree with that outcome, but my most load-bearing concern is not the one the reader singled out as the weakest assumption. The reader focused on whether the internally generated 'fine-grained text' is genuinely complementary to the video features and coarse labels. That is a real concern: because the text is produced by clustering video features and classifying with dataset labels, it may not add independent information, and the multi-modal framing could reduce to auxiliary supervision. However, there is an even more basic problem that blocks evaluation of the entire claim. The paper never specifies the training objective for the final action-anticipation module. Section 3.4 defines only the forward pass, Section 3.2 defines only the text-generator loss, and Section 4.1 lists optimizer settings without any loss. This means the architecture described in the paper cannot be trained to maximize anticipation accuracy; the reported numbers are therefore not reproducible from the text alone. If the released code contains an explicit anticipation loss, this objection can be resolved, which is why a concrete code inspection is the right test. The ablation inconsistency in Table 3 (50 Salads, observation rate 0.3: uni-modal 90.92 vs multi-modal 84.55) and Table 4 (87.91 vs 84.55 at the same setting) is independent evidence that the paper's claim of consistent gains is not supported by its own tables. Because these issues directly undermine the central claim and the reader already recommended REJECT, no verdict adjustment is needed. My agreement is partial rather than full because I would place the decisive weight on the missing anticipation objective and the ablation contradictions rather than on the complementarity of the internally generated text.","tokens_in":9105,"tokens_out":5154,"duration_ms":51743,"concrete_test":"Inspect the released repository (olivesgatech/mM-ant), locate the training script, and identify the exact loss minimized for the multi-modal action anticipation module. If no explicit anticipation loss (e.g., cross-entropy between predicted future labels and ground-truth future labels) is present, the method as written is untrainable and the reported gains are unsupported. If such a loss exists, re-run the 50 Salads experiment at observation rate 0.3 with the released code and fixed seeds, and verify whether multi-modal beats uni-modal as Table 3 claims or whether the printed table values reproduce; if they do not, the central consistency claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that m&m-Ant improves state-of-the-art anticipation accuracy by an average of 3.08% through multi-modal and multi-level modeling—requires that the proposed architecture can actually be trained to anticipate. The paper never specifies the training objective for either the video segmentation module or the multi-modal action anticipation module. Section 3.2 defines a loss only for the fine-grained text generator, L_total = L_ce + L_tcl (Eq. 3). Section 3.4 describes the forward computations Hinput = Concat(Hvideo, Hfine-grained), MHSA, and MHCS, but gives no objective over future action labels. Section 4.1 states only that the model is 'trained for 60 epochs using AdamW optimizer' with certain hyperparameters; no cross-entropy loss, sequence loss, or total loss for anticipation is provided. Without a specified anticipation loss, the model cannot be trained as described, and the reported accuracy gains cannot be attributed to the proposed modules. This is not a cosmetic omission: the final fusion module is the component that is supposed to make the multi-modal and hierarchical design effective. The ablation tables corroborate the problem. Table 3 reports that on 50 Salads at observation rate 0.3, the uni-modal baseline scores 90.92 while the multi-modal model scores 84.55—the opposite of the text's claim that multi-modal 'consistently outperforms the uni-modal counterpart across all observation rates.' Table 4 similarly reports w/o multi-level 87.91 versus w/ multi-level 84.55 at the same setting. These internal contradictions mean the paper's own experimental support does not establish the central claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes m&m-Ant, a multi-modal and multi-level framework for long-term action anticipation. It combines visual features with a generated fine-grained text modality and coarse segmentation labels, using a video encoder, a video segmentation module, a fine-grained text generator trained with a temporal consistency loss, and a multi-modal attention module. The claimed contribution is a 3.08% average accuracy improvement over state-of-the-art on Breakfast, 50 Salads, and DARai. The paper includes ablation studies on the effect of the multi-modal fusion and the multi-level (hierarchical) modeling.","tokens_in":9437,"tokens_out":4207,"duration_ms":36125,"significance":"If the results hold, the idea of generating fine-grained textual semantics from visual features and fusing them with video and coarse labels in a cross-attention architecture is a plausible and potentially useful direction for long-term action anticipation. The paper provides a public code link and evaluates on three datasets, including a recently introduced DARai dataset. However, the manuscript as written has major reproducibility gaps (no anticipation loss is specified) and internal inconsistencies in the ablation tables that directly contradict the stated claims, so the empirical evidence cannot currently be accepted at face value.","major_comments":[{"comment":"No training objective is defined for the action anticipation module or the video segmentation module. Equation (3) defines a loss only for the fine-grained text generator (L_total = L_ce + L_tcl). Section 3.4 describes the forward computations of the multi-modal action anticipation module (H_input, MHSA, MHCS) but gives no loss over future action labels, and Section 4.1 lists only optimizer settings (AdamW, 60 epochs, learning rate 1e-3). Without an anticipation loss, the model cannot be trained as described, and the reported accuracy gains cannot be reproduced or attributed to the proposed architecture.","section":"Sections 3.2-3.4 and 4.1"},{"comment":"The text claims that the multi-modal approach 'consistently outperforms the uni-modal counterpart across all observation rates,' but Table 3 shows the opposite on 50 Salads at observation rate 0.3: uni-modal 90.92 vs. multi-modal 84.55, a reversal of 6.37 points. Table 4 similarly shows w/o multi-level 87.91 vs. w/ multi-level 84.55 at the same setting. The non-monotonic pattern in the uni-modal column (90.92 at 0.3 dropping to 73.92 at 0.4) also suggests a possible mislabeling or evaluation error. These contradictions undermine the central consistency claim and must be resolved, either by correcting the tables/text or by rerunning the experiments.","section":"Tables 3 and 4 and Section 4.2"},{"comment":"The textual modality is generated entirely from the same video features and dataset labels, with no external text source, recipe input, or language model. The fine-grained text generator first clusters video features and then classifies them into fine-grained labels. The claimed multi-modal complementarity is therefore not independently grounded: the 'text' may be just a re-encoding of the visual features. The paper should include a control experiment that replaces the fine-grained text generator with an equally expressive visual feature transformation (e.g., an MLP on the same video features) or replaces the text embeddings with one-hot fine-grained labels, to show that the improvement is due to cross-modal fusion rather than added parameters or label conditioning.","section":"Section 3.2 and Figure 2"},{"comment":"The paper states that results are averaged over 3 runs with fixed seeds 1, 10, 13452, but no standard deviations or confidence intervals are reported in any table. Several reported differences are small (e.g., DARai, alpha=0.2, beta=0.1: FUTR 25.05 vs. m&m-Ant 25.75), so without error bars the central claim of a 3.08% average improvement cannot be assessed statistically. Reporting variance or per-seed results is necessary to establish that the gains are not noise.","section":"Section 4.1 and Tables 1-4"}],"minor_comments":[{"comment":"The inter-cluster separation loss L_inter = sum_{k,k' != k} 1/||mu_k - mu_k'|| is unbounded as cluster centroids approach each other and its scale depends on the number of clusters; the paper does not discuss the choice of lambda_1 and lambda_2 or their sensitivity.","section":"Section 3.2, Eq. (2)"},{"comment":"The clustering procedure is underspecified: the paper says frames of the same class within a continuous temporal interval are grouped into a cluster, but it does not state how intervals are determined (e.g., connected components), how many clusters are used, or what happens when the same class appears in disjoint intervals.","section":"Section 3.2"},{"comment":"The entry '25.7024.24' lacks a separating space and should read '25.70 24.24'; the same formatting problem appears in the row labels of Tables 3 and 4 (e.g., 'Multi-modal30.05').","section":"Table 2"},{"comment":"The phrase 'average performance across 3 number of iteration' is a typo; it should read '3 iterations' or '3 runs'.","section":"Section 4.1"},{"comment":"The paper does not specify how the reported 3.08% average improvement is computed across datasets and evaluation settings; the aggregation rule should be stated explicitly.","section":"Section 4.2 and Table 1"},{"comment":"On 50 Salads at alpha=0.3, beta=0.1, both FUTR and m&m-Ant are reported at 100.00 accuracy, which is suspicious and suggests a saturated or degenerate evaluation setting; this should be discussed.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The DARai dataset is introduced in a companion paper by the same group (reference [14]), and the SOTA comparison on DARai is limited to two baselines (FUTR and AFFT). This does not invalidate the experiments, but the 'new benchmark' claim in the abstract should be tempered, and the editor may want to consider whether the in-house dataset evaluation is sufficiently broad for the journal's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear [colleague],\n\nYou should know two things about this one. The central claim is that a multi-modal, hierarchical model beats FUTR by 3.08% average accuracy on Breakfast, 50 Salads, and DARai. The paper never defines the loss used to train the final anticipation module. Section 3.2 gives a loss for the fine-grained text generator, Eq. (3), and Section 3.4 describes forward computations, but there is no objective over future action labels anywhere. That makes the reported numbers unreproducible as written.\n\nSecond, the paper's own ablation tables contradict the claim that multi-modal modeling 'consistently outperforms' the uni-modal baseline. In Table 3, on 50 Salads at observation rate 0.3, the uni-modal model scores 90.92 and the multi-modal model scores 84.55. Table 4 shows the same drop when adding multi-level modeling. Those are not small differences, and they directly undermine the 'consistently' wording.\n\nWhat is genuinely new here is the coupling of a clustering-based fine-grained label generator with a temporal consistency loss, and then feeding those generated labels as a second modality into a transformer for anticipation. The components exist elsewhere, but this particular combination is not something I've seen described. The temporal consistency loss—intra-cluster MSE plus inter-cluster separation—is sensible and could be useful for other label-generation tasks. The architecture is clearly illustrated, and the authors provide a code link.\n\nThe softer spots: calling this 'multi-modal' is generous, because the fine-grained text is generated from the same video features and ground-truth labels, not from any external text source. It is an auxiliary supervision signal, not an independent modality. The text encoder itself is never specified, and no error bars are reported, which is minor but worth noting.\n\nMy take: this paper deserves a serious referee, not a desk reject. The missing anticipation loss and the ablation contradictions are fixable in revision, and the underlying idea is worth testing. For someone working on long-term action anticipation, this is a promising direction, but the supporting evidence does not establish the headline gain. If you send it out, ask the authors to specify the training objective precisely, reconcile the ablation numbers, and tone down the multi-modal framing.","headline":"The paper omits the main training loss and its ablations contradict the headline 3.08% gain; the underlying idea is plausible enough to warrant peer review with clear demands for revision.","tokens_in":9979,"tokens_out":8932,"would_cite":false,"duration_ms":65609,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Self-generated fine-grained text lifts action anticipation by 3.08%.","keywords":["action anticipation","multi-modal learning","fine-grained text generation","temporal consistency loss","hierarchical semantic modeling","long-term anticipation","video understanding"],"falsifier":"Train m&m-Ant with the fine-grained text generator replaced by a fixed, untrained random projection of the same video features (or by the coarse labels copied verbatim) while keeping all other components and parameter counts identical; if the 3.08% average improvement over FUTR persists, the reported gain is not caused by the learned fine-grained textual semantics. Alternatively, measure the conditional mutual information between generated text and future action labels given observed video features; near-zero values would indicate the text carries no anticipatory information beyond the video.","tokens_in":8927,"feed_emoji":"🎬","tokens_out":7470,"duration_ms":58474,"temperature":0.7,"pith_summary":"The paper sets out to show that action anticipation from partially observed video improves when the model is given a second, textual modality alongside the visual stream, and that this text does not have to come from an external source. The proposed m&m-Ant method generates fine-grained textual labels from the video features themselves, using a temporal consistency loss to separate repeated occurrences of the same coarse action into distinct sub-action clusters. These fine-grained texts are concatenated with video embeddings and then aligned with coarse segmentation labels through cross-attention, forming a hierarchical multi-modal representation. On Breakfast, 50 Salads, and DARai, the authors report state-of-the-art accuracy with an average gain of 3.08% over prior methods, with the largest improvements on dense datasets and at low observation rates. A sympathetic reader would take the paper as establishing that a self-produced textual modality plus hierarchical semantic modeling is a practical and effective route to better anticipation without external text resources.","feed_headline":"Self-generated text lifts action anticipation by 3.08%","feed_subtitle":"Pairing video features with internally generated fine-grained text beats prior models on Breakfast, 50 Salads, and DARai.","key_machinery":"The load-bearing mechanism is the fine-grained text generator and its temporal consistency loss. Frames from the same action class that lie within a continuous temporal interval are grouped into a cluster, and the loss $L_{\\text{tcl}} = \\lambda_1 L_{\\text{intra}} + \\lambda_2 L_{\\text{inter}}$ is applied, where $L_{\\text{intra}} = \\sum_k \\sum_{x_i \\in X^{(k)}} \\lVert x_i - \\mu_k \\rVert^2$ enforces cohesion toward the cluster centroid $\\mu_k$ and $L_{\\text{inter}} = \\sum_{k, k' \\neq k} 1/\\lVert \\mu_k - \\mu_{k'} \\rVert$ penalizes clusters that are close, so repeated instances of the same nominal action separate into distinct clusters. The cluster assignments are rendered as fine-grained text tokens, concatenated with video features, and aligned with coarse segmentation labels through multi-head cross-attention in the anticipation module. This chain from raw features to clustered text to cross-modal attention is what carries the claimed accuracy improvement.","core_discovery":"The central claim is that a video-only anticipation system can be turned into a genuinely multi-modal system by generating fine-grained textual labels from the video's own features, and that these labels carry predictive signal beyond both raw visual features and coarse segmentation labels. Concretely, m&m-Ant concatenates sampled video embeddings with the generated fine-grained text, processes the concatenation through multi-head self-attention, and then injects coarse segmentation labels as keys and values in a multi-head cross-attention layer to encode high-level semantic structure. The fine-grained text generator is trained with a cross-entropy term plus a temporal consistency loss: an intra-cluster term pulls frames of the same action inside a continuous temporal interval toward their cluster centroid, while an inter-cluster term pushes apart clusters that are temporally distant even when they share the same class label. The paper argues that this recovers sub-action flows that frame-level classification ignores, and presents experiments on three datasets as evidence that the recovered signal is enough to beat the FUTR, GTD, and AFFT baselines. If the claim is right, the implication is that the bottleneck in long-term action anticipation is not the absence of external linguistic knowledge but the model's ability to organize observed video into semantically meaningful temporal units.","pith_inferences":["Because the 'text' modality is generated from the same video features, part of the gain may come from added model capacity or a regularizing auxiliary task; a matched-capacity single-modality baseline with the same parameter count would separate those effects.","The temporal consistency loss is a general recipe for sequence labeling: any task where the same nominal class recurs at widely separated times could benefit from the same intra-cluster/inter-cluster separation objective.","A natural next experiment is to add a genuine external text source, such as recipe instructions or narrated transcripts; if accuracy rises further, it would confirm that semantic text, not just the generator, is driving the improvement.","The inter-cluster term uses an inverse distance $1/\\lVert \\mu_k - \\mu_{k'} \\rVert$, which diverges as clusters coincide; a bounded or capped repulsion term would make training more stable and would test how much of the reported gain depends on this particular formulation."],"forward_implications":["At low observation rates (for example $\\alpha = 0.1$), where visual input is scarce, the fine-grained text lets the model exploit subtle cues that a visual-only model misses, yielding larger relative gains.","On dense, transition-heavy datasets such as 50 Salads, the improvement over the FUTR baseline is larger than on sparser datasets, because the fine-grained text captures sub-action nuances and frequent transitions.","Ablations in the paper show that the multi-modal variant beats a uni-modal variant at every observation rate, and that adding fine-grained text on top of coarse labels further improves accuracy, especially when little of the video has been observed.","On the untrimmed, hierarchically labeled DARai dataset, the method stays ahead of FUTR at every reported $\\alpha$ and $\\beta$ setting, with a maximum reported gain of 2.27%."],"supporting_citations":[{"why":"FUTR is the state-of-the-art baseline that m&m-Ant must beat on all three datasets.","marker":"[2]"},{"why":"Defines the long-term action anticipation protocol and provides the RNN/CNN baselines used for comparison.","marker":"[7]"},{"why":"Breakfast dataset, one of the three benchmarks used for evaluation.","marker":"[12]"},{"why":"50 Salads dataset, the dense-action benchmark where the largest gains are reported.","marker":"[13]"},{"why":"DARai dataset, the untrimmed hierarchical benchmark used for evaluation.","marker":"[14]"},{"why":"Supplies the observation of fine-grained groupings in representation space that motivates the fine-grained text generator.","marker":"[24]"},{"why":"Gated temporal diffusion baseline compared on Breakfast.","marker":"[25]"},{"why":"Anticipative feature fusion transformer baseline compared on DARai.","marker":"[26]"}],"fun_headline_variants":["Self-generated labels boost action anticipation by 3%","Video plus self-made text predicts actions better","Generated text from video improves action anticipation","Multi-level multi-modal anticipation with self-generated text","Video-only model turns to text, gains 3% on action anticipation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that the fine-grained text it generates from video features is a genuinely new semantic signal, not just a re-encoding of information the video features and coarse labels already contain; if that assumption fails, the multi-modal architecture gains nothing over a single-modality model with more parameters.","fun_headline_variants_meta":{"raw":{"variants":["Self-generated labels boost action anticipation by 3%","Video plus self-made text predicts actions better","Generated text from video improves action anticipation","Multi-level multi-modal anticipation with self-generated text","Video-only model turns to text, gains 3% on action anticipation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00054,"raw_usage":{"total_tokens":2629,"prompt_tokens":1021,"completion_tokens":1608,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":637,"completion_tokens_details":{"reasoning_tokens":1534}},"tokens_in":637,"tokens_out":1608,"duration_ms":11039,"temperature":1.0,"reasoning_tokens":1534,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:24:52.168403+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train m&m-Ant with the fine-grained text generator replaced by a fixed, untrained random projection of the same video features (or by the coarse labels copied verbatim) while keeping all other components and parameter counts identical; if the 3.08% average improvement over FUTR persists, the reported gain is not caused by the learned fine-grained textual semantics. Alternatively, measure the conditional mutual information between generated text and future action labels given observed video features; near-zero values would indicate the text carries no anticipatory information beyond the video.","supporting_citations":[{"cited_title":"Long-term Action Anticipation Action anticipation emphasizes predicting potential upcom- ing actions before they are observed","cited_arxiv_id":null,"evidence_quote":"FUTR is the state-of-the-art baseline that m&m-Ant must beat on all three datasets."},{"cited_title":"Future transformer for long-term action anticipation,","cited_arxiv_id":null,"evidence_quote":"Defines the long-term action anticipation protocol and provides the RNN/CNN baselines used for comparison."},{"cited_title":"When will you do what?-anticipating temporal occur- rences of activities,","cited_arxiv_id":null,"evidence_quote":"Breakfast dataset, one of the three benchmarks used for evaluation."},{"cited_title":"Time- conditioned action anticipation in one shot,","cited_arxiv_id":null,"evidence_quote":"50 Salads dataset, the dense-action benchmark where the largest gains are reported."},{"cited_title":"Temporal aggregate representations for long-range video understanding,","cited_arxiv_id":null,"evidence_quote":"DARai dataset, the untrimmed hierarchical benchmark used for evaluation."},{"cited_title":"Weakly supervised action learning with rnn based fine- to-coarse modeling,","cited_arxiv_id":null,"evidence_quote":"Supplies the observation of fine-grained groupings in representation space that motivates the fine-grained text generator."},{"cited_title":"Ms-tcn: Multi-stage temporal convolutional network for action segmenta- tion,","cited_arxiv_id":null,"evidence_quote":"Gated temporal diffusion baseline compared on Breakfast."},{"cited_title":"Leveraging uncertainty to rethink loss functions and evaluation measures for egocentric action anticipation,","cited_arxiv_id":null,"evidence_quote":"Anticipative feature fusion transformer baseline compared on DARai."}],"review_version":1}