{"id":"b2d541b9-df86-4ff5-9a95-3cecaf318d7d","arxiv_id":"2501.08837","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A bidirectional Mamba layer used as the diffusion generator improves accuracy and drastically cuts inference time for stochastic long-term action anticipation.","lead":"MANTA replaces the dilated-gated-convolution generator of the GTDA diffusion model with a bidirectional Mamba layer, giving a global temporal receptive field at linear cost. On Breakfast, 50Salads and Assembly101 it reports state-of-the-art stochastic action anticipation scores while running up to 65x faster at inference.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The observed-to-future information flow through the fixed-size S6 hidden state is the unverified load-bearing assumption; the backward scan does not supply observed context to future positions.","rationale":"The paper's central claim has two parts: state-of-the-art accuracy on three datasets and large efficiency gains. The efficiency part is well supported by Table 6, including a 65.3x inference speedup with code released, and the architecture is a plausible, well-ablated contribution. I do not see an internal inconsistency or a methodological error that invalidates the comparison. The correctness risk is concentrated in the long-range modelling premise: for the diffusion generator to denoise the masked future region, the network must move information from the observed block (where X is nonzero) across up to F masked positions. In a bidirectional S6 block, the backward path is reversed, so a future position's backward scan includes only later future positions; the forward path is the sole conduit for observed conditioning. That conduit is a fixed-size recurrent state, whose effective capacity over very long horizons is not measured. The existing long-sequence evidence is indirect: Table 3 shows bidirectionality helps, and Fig. 5 shows a Breakfast duration grouping, but neither isolates the observed-to-future propagation nor covers Assembly101's 25-minute videos. The proposed concrete test (an effective-receptive-field probe plus a comparison curve) would directly settle whether the theoretical global receptive field is realized. If the probe shows long-range influence, the concern is resolved; if not, the SOTA claim for very long sequences would need qualification. I therefore keep the reader's CONDITIONAL verdict: accept with a request for this analysis.","tokens_in":17935,"tokens_out":5715,"duration_ms":66328,"concrete_test":"Using the released checkpoint, compute an effective-receptive-field probe on Assembly101: set the observed conditioning block to zero except for a one-hot perturbation at the last observed frame, and measure the L2 change in the denoised future logits at each future offset k=1..F relative to the unperturbed conditioning. If the influence decays to noise before k=F, MANTA's forward S6 path does not actually deliver observed context across the full anticipation interval, undercutting the 'global receptive field' rationale for the SOTA long-sequence claim. Also report the same curve for GTDA to quantify relative long-range propagation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's strongest claim is that MANTA achieves state-of-the-art long-term anticipation by replacing GTDA's multi-stage dilated convolutions with a single stage of 15 Mamba blocks. The load-bearing premise is that the fixed-size S6 hidden state carries the observed visual conditioning across the entire masked future interval. In this architecture, the conditioning vector X (Eq. 3) is nonzero only for the P observed positions; for future positions, the forward S6 scan is the only path through which observed X can reach the masked region. The backward scan of the BSSL (Eqs. 11-14) processes the reversed sequence, so at a future position j it sees only inputs to the right of j (later future positions), never the observed block; it provides future context but does not directly propagate observed information into the future. Thus the bidirectional ablation (Table 3) does not validate long-range observed-to-future flow. The paper's direct evidence for long-sequence ability is aggregate accuracy plus Fig. 5, which is confined to Breakfast (max 10.8 minutes, β=0.2) and does not isolate far-future positions. Assembly101, with anticipation intervals up to 12.5 minutes, is not analyzed by horizon or distance from the observation boundary. Without an effective-receptive-field or information-flow measurement, the 'global receptive field' claim rests on a theoretical property of S6 rather than demonstrated effective behaviour.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MANTA, a diffusion-based stochastic long-term dense action anticipation model that replaces GTDA's multi-stage gated temporal convolutional generator with a single-stage stack of 15 bidirectional Mamba (S6) blocks. The core idea is that the selective state-space layer provides a global receptive field with linear complexity, allowing observed visual features to propagate into the zero-masked future region and improving both accuracy and efficiency. Experiments are conducted on Breakfast, 50Salads, and Assembly101 using the standard alpha/beta protocol, with Mean and Top-1 MoC metrics. The authors report state-of-the-art accuracy on the three datasets, with the exception of Assembly101 Top-1 at beta=0.1, and large speedups over GTDA (65.3x inference, 6.6x training). Ablations study the contributions of selectivity, bidirectionality, block structure, and block count, and supplementary material provides additional ablations and standard deviations for Breakfast.","tokens_in":18254,"tokens_out":6891,"duration_ms":63442,"significance":"MANTA is a well-executed, incremental improvement over the same authors' GTDA. Its strengths include reproducible code, ablations that isolate key design choices, and substantial efficiency gains (65.3x inference, 6.6x training, 2.8x fewer parameters). If the reported gains are robust, the model provides a more practical stochastic anticipation method. However, the claimed mechanism for long-range observed-to-future information flow is not directly measured, and the empirical gains on Assembly101 are small; the central evidence for the 'very long sequence' claim is therefore incomplete.","major_comments":[{"comment":"The paper reports no standard deviations or significance tests for Assembly101 and 50Salads. On Assembly101 the Mean MoC improvements over GTDA are small (e.g., 6.7 vs 6.4 at alpha=0.2, beta=0.1; 5.3 vs 4.5 at beta=0.2) and Top-1 MoC is actually worse at beta=0.1 (16.9 vs 18.0). Without variance estimates, it is unclear whether the claimed state-of-the-art results on these two datasets are statistically meaningful. Please add error bars or significance testing for the main comparisons.","section":"Sec. 4.3, Table 1"},{"comment":"The bidirectional layer's backward scan processes the reversed sequence, so for a future position j it only sees positions j, j+1, ..., P+F (after reversing), never the observed block at positions 1,...,P. Only the forward scan can propagate observed information into the masked future region. The bidirectionality ablation in Table 3 therefore demonstrates the value of future context, not the long-range observed-to-future flow that motivates the architecture. Please add an experiment that directly measures information flow from observed inputs to future predictions, such as perturbing observed positions and measuring the change in future outputs at varying distances, or report performance on Assembly101 broken down by anticipation horizon. This would substantiate the 'global receptive field' and 'early propagation' claims.","section":"Sec. 3.2.2, Eqs. (11)-(14) and Table 3"},{"comment":"The only duration-based analysis is on Breakfast, whose sequences max out at 10.8 minutes (5.4 minutes of anticipation at beta=0.5), and Fig. 5 aggregates over all future positions. Assembly101, with anticipation horizons up to 12.5 minutes, is not analyzed as a function of horizon or video length. Without such analysis, the claim of effectiveness on 'very long sequences' is not directly supported by the experiments.","section":"Sec. 4.3, Fig. 5"}],"minor_comments":[{"comment":"The statement 'achieves state-of-the-art results on three datasets' is an overstatement given the Assembly101 Top-1 MoC result at beta=0.1 (16.9 vs 18.0 for GTDA); please qualify the claim to reflect the exception noted in Section 4.3.","section":"Abstract and Conclusion"},{"comment":"The formula for bar-B is ambiguous as printed: 'exp(delta A)-1(exp(delta A) - I) delta B' could be misread. Please write the intended expression, e.g., (e^{delta A} - I)(delta A)^{-1} delta B, or (e^{delta A} - I) A^{-1} B, to avoid confusion.","section":"Eq. (9)"},{"comment":"In the row for 25 samples, the Mean MoC value at beta=0.1 appears to be a typo: 25.5 should likely be 27.7 (compare with Table 1 for the same configuration).","section":"Supplementary Tab. 10"},{"comment":"Consider labeling the number of videos in each duration group or adding error bars, since group sizes likely vary and the aggregated mean could be dominated by a few long videos.","section":"Fig. 5"}],"recommendation":"major_revision","confidential_remarks":"I have no concerns about research integrity. The paper is an incremental extension of GTDA but with concrete contributions in efficiency and accuracy. The main issues are the missing direct evidence for long-range information propagation and the lack of variance estimates for Assembly101 and 50Salads; both are addressable within a revision and do not require new datasets or a fundamentally different approach."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: MANTA is a genuine improvement for stochastic dense anticipation. It replaces GTDA's heavy multi-stage dilated convs with a single stack of 15 bidirectional Mamba blocks, and the payoff shows: better accuracy on three public benchmarks and a 65x inference speedup. The paper is empirical, well-ablated, and ships code. The two soft spots are missing error bars for Assembly101 and 50Salads, and an unverified story about how observed information actually reaches the masked future through the fixed-size S6 hidden state.\n\nWhat's new: the architecture, not the framing. The diffusion conditioning and zero-padded future come directly from GTDA, so the contribution is swapping the multi-stage gated convs for Mamba's S6 with a backward scan. That's incremental but sensible for this task. The paper does well by comparing against transformer baselines with global attention (Transf-15 and Transf-18) and showing MANTA beats them while being much faster. That strengthens the case that the Mamba block is doing something useful, not just that any global-receptive-field model helps.\n\nSoft spots: the stress-test note is right. The backward scan processes the reversed sequence, so at a future position it only sees later future positions, never the observed block. The bidirectional ablation (Table 3) therefore doesn't validate the claimed observed-to-future propagation; that path is carried entirely by the forward scan and the fixed-size hidden state. The paper never directly measures effective receptive field or provides a horizon-wise breakdown for Assembly101, where the anticipation interval reaches 12.5 minutes. This is a missing analysis, not a fatal flaw: the empirical gains on held-out splits, including the long-horizon Assembly101, suggest the forward scan is doing its job in practice. But the 'global receptive field' claim is theoretical unless you verify it. Also, error bars are only reported for Breakfast; Assembly101's gains over GTDA are small (e.g., 6.7 vs 6.4), so without variance we can't tell whether those are meaningful. Both are fixable with a bit more analysis.\n\nBottom line: this is a useful, carefully evaluated paper for anyone working on anticipation or on applying SSMs to video. It deserves a serious referee. I'd send it to review and expect the authors to address the information-flow question and the missing variance estimates. If the code checks out, I'd probably cite it.","headline":"MANTA is a solid, efficient architectural win for stochastic dense anticipation, but the paper's story about long-range information flow is under-verified and deserves a look before you take the 'global receptive field' claim at face value.","tokens_in":18727,"tokens_out":3349,"would_cite":true,"duration_ms":35993,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MANTA replaces the previous best diffusion generator with a bidirectional Mamba stack that reaches a dense global temporal receptive field at linear cost, and reports state-of-the-art accuracy on Breakfast, 50Salads, and Assembly101 with…","keywords":["stochastic long-term action anticipation","diffusion models","Mamba","state space models","long-term dense anticipation","temporal action modeling","efficiency","video understanding"],"falsifier":"Perturb a single observed frame in a trained MANTA and measure how far into the masked future its influence changes the output, across all 15 blocks and on the longest Assembly101 sequences; if the influence decays before the end of the anticipation horizon in early blocks, or if a model with the backward path removed matches the bidirectional model on the 25-minute videos, the dense-global-receptive-field explanation is wrong.","tokens_in":17776,"feed_emoji":"🎬","tokens_out":8046,"duration_ms":69546,"temperature":0.7,"pith_summary":"Stochastic long-term dense action anticipation asks a model to generate multiple plausible action sequences covering several minutes of future video. The paper identifies the generator's temporal receptive field as the main bottleneck: the previous state of the art, GTDA, uses multi-stage dilated gated convolutions whose receptive field is sparse and becomes global only late. MANTA replaces that generator with a stack of Mamba-based Bidirectional Selective State-Space Layers, which keep a dense global receptive field with linear complexity in sequence length. The paper reports state-of-the-art accuracy on Breakfast, 50Salads, and Assembly101, while cutting inference time per video from 71.8 seconds to 1.1 seconds (65.3×) and training time per epoch from 379 seconds to 57 seconds. If correct, this makes long-horizon stochastic forecasting both more accurate and cheap enough for real-time use.","feed_headline":"Mamba diffusion forecasts minutes-ahead actions 65x faster","feed_subtitle":"Bidirectional state-space generator beats GTDA on three benchmarks; inference runs 65x faster.","key_machinery":"The load-bearing component is the Bidirectional Selective State-Space Layer (BSSL), a Mamba-style block that processes the temporal sequence with two S6 scans, one forward and one backward, whose input-dependent matrices (B, C, and Δ) act as data-dependent gates. It carries the argument by replacing the dilated gated temporal convolutions of GTDA: it provides a dense global receptive field with linear cost, and its selectivity lets the model treat the observed and zero-masked halves of the input differently. The 15-block MANTA stack built from BSSL is the whole temporal engine; the diffusion process and DDIM sampling are inherited from GTDA unchanged.","core_discovery":"MANTA is a diffusion generator that predicts past and future one-hot action labels jointly, conditioned on observed visual features zero-padded over the future. Its core discovery is that the S6 selective state-space layer, run in both forward and backward temporal directions, gives every position a dense, locally aware global receptive field in a single stage, so the zero-padded future entries can draw on observed context from the very first block. This removes the need for GTDA's multi-stage dilated architecture: a single stack of 15 MANTA blocks outperforms GTDA on three datasets, with the largest gains on longer videos and longer anticipation horizons. The same blocks are also much cheaper: 2.8× fewer parameters, 65.3× faster inference for 25 stochastic samples, and 6.6× faster training.","pith_inferences":["A natural extension the paper does not test is to swap the BSSL generator into other masked-fill dense prediction tasks, such as future motion or audio forecasting, where the same observed-context-to-masked-region propagation problem appears.","The paper attributes the gain to dense global receptive field but does not measure effective receptive field directly; a head-to-head against a bidirectional linear-attention generator with comparable gating would isolate whether the mechanism is the receptive field or the particular SSM dynamics.","Assembly101 contains 25-minute videos, yet the duration-based analysis is only reported on Breakfast; re-running the video-length grouping on Assembly101 would provide the strongest test of the long-range modelling claim.","Because 50Salads converges with only 10 DDIM steps while Breakfast and Assembly101 use 50, the interaction between generator capacity and number of sampling steps is unexplored and could yield further speedups."],"forward_implications":["If MANTA's claim holds, long-term stochastic anticipation no longer needs multi-stage dilated generators: a single Mamba stage matches or beats GTDA's accuracy on all three datasets.","The efficiency gain (65.3× inference, 6.6× training, 2.8× fewer parameters) makes generating 25 stochastic samples per observation practical on long videos, opening the door to real-time anticipation.","The reported performance gap over GTDA widens on longer videos (5.5% versus 3.8% Mean MoC improvement), implying that long-horizon forecasting is exactly where dense global receptive fields help most.","Removing either the backward scan or the S6 selectivity degrades accuracy substantially (Top-1 MoC drops from 55.5 to 35.8 without the backward path at α=0.2, β=0.1), so both design choices are load-bearing for the reported results."],"supporting_citations":[{"why":"Supplies the GTDA baseline, the joint past/future diffusion formulation, and the training/evaluation protocol that MANTA inherits and must beat.","marker":"[57]"},{"why":"Supplies the S6 selective state-space layer that is the core temporal unit of the BSSL.","marker":"[13]"},{"why":"Supplies the bidirectional scanning design for visual sequences that BSSL adopts for its backward path.","marker":"[26]"},{"why":"Defines the stochastic long-term anticipation protocol, including observation/anticipation ratios, 25 samples, and MoC metrics.","marker":"[7]"},{"why":"Provides the non-selective S4 baseline used in the ablation showing that input-dependent selectivity is needed.","marker":"[15]"},{"why":"Provides the windowed and atrous Transformer block used as the attention-based ablation baseline.","marker":"[5]"},{"why":"Earlier diffusion-based stochastic anticipation baseline that MANTA outperforms on the same benchmarks.","marker":"[65]"},{"why":"Supplies the Assembly101 dataset, which includes the longest videos (up to 25 minutes) used in evaluation.","marker":"[44]"},{"why":"Supplies the Breakfast dataset, the primary benchmark for the main results and ablations.","marker":"[25]"},{"why":"Supplies the 50Salads dataset, one of the three evaluation benchmarks.","marker":"[51]"}],"fun_headline_variants":["Diffusion Mamba beats GTDA, 65x faster inference","Bidirectional Mamba diffusion predicts minutes ahead, 65x faster","MANTA: one-stage diffusion for long-term anticipation, 65x faster","State-space diffusion sees far future, slashes compute 65x"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The S6 selective state-space layer's fixed-size hidden state can carry the observed frames' information across the whole masked future region, and the backward scan fills in what the causal forward scan drops; the evidence for this is the final accuracy and the bidirectionality ablation, not a direct measurement of the receptive field.","fun_headline_variants_meta":{"raw":{"variants":["Diffusion Mamba beats GTDA, 65x faster inference","Bidirectional Mamba diffusion predicts minutes ahead, 65x faster","MANTA: one-stage diffusion for long-term anticipation, 65x faster","State-space diffusion sees far future, slashes compute 65x"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000497,"raw_usage":{"total_tokens":2413,"prompt_tokens":900,"completion_tokens":1513,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":516,"completion_tokens_details":{"reasoning_tokens":1436}},"tokens_in":516,"tokens_out":1513,"duration_ms":10755,"temperature":1.0,"reasoning_tokens":1436,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:17:17.320408+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Perturb a single observed frame in a trained MANTA and measure how far into the masked future its influence changes the output, across all 15 blocks and on the longest Assembly101 sequences; if the influence decays before the end of the anticipation horizon in early blocks, or if a model with the backward path removed matches the bidirectional model on the 25-minute videos, the dense-global-receptive-field explanation is wrong.","supporting_citations":[{"cited_title":"The language of actions: Recovering the syntax and semantics of goal- directed human activities","cited_arxiv_id":null,"evidence_quote":"Supplies the Breakfast dataset, the primary benchmark for the main results and ablations."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the 50Salads dataset, one of the three evaluation benchmarks."},{"cited_title":"Gated temporal diffusion for stochastic long-term dense anticipation","cited_arxiv_id":null,"evidence_quote":"Supplies the GTDA baseline, the joint past/future diffusion formulation, and the training/evaluation protocol that MANTA inherits and must beat."},{"cited_title":"Videomamba: State space model for efficient video understanding","cited_arxiv_id":null,"evidence_quote":"Supplies the bidirectional scanning design for visual sequences that BSSL adopts for its backward path."},{"cited_title":"Farha and J","cited_arxiv_id":null,"evidence_quote":"Defines the stochastic long-term anticipation protocol, including observation/anticipation ratios, 25 samples, and MoC metrics."},{"cited_title":"Efficiently mod- eling long sequences with structured state spaces","cited_arxiv_id":null,"evidence_quote":"Provides the non-selective S4 baseline used in the ablation showing that input-dependent selectivity is needed."},{"cited_title":"How much temporal long-term context is needed for action seg- mentation? In IEEE International Conference on Computer Vision (ICCV), 2023","cited_arxiv_id":null,"evidence_quote":"Provides the windowed and atrous Transformer block used as the attention-based ablation baseline."},{"cited_title":"DiffAnt: Diffusion Models for Action Anticipation","cited_arxiv_id":"2311.15991","evidence_quote":"Earlier diffusion-based stochastic anticipation baseline that MANTA outperforms on the same benchmarks."},{"cited_title":"Sener, D","cited_arxiv_id":null,"evidence_quote":"Supplies the Assembly101 dataset, which includes the longest videos (up to 25 minutes) used in evaluation."}],"review_version":1}