{"id":"70531118-c663-4165-9ae0-a333a830acde","arxiv_id":"2608.08469","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Aero Realtime aligns continuous video, audio, and text output on one 80ms grid, letting a 4B model generate lexical tokens or silence in a single stream while reusing the KV cache.","lead":"A new 4-billion-parameter model, Aero Realtime, places video, audio, and text output on a common 80-millisecond grid so a language model can listen while it speaks. The paper shows the architecture can keep within about 200 milliseconds of a live video timeline for 20 minutes, but its proactive behavior is mostly trained on question-answer data and trails stronger video LLMs.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Silence-label masking (r=0.95) is the actual controller of response timing; without it the aligned objective produces a mute model, so the central 'learns when to respond' claim is not yet supported.","rationale":"After reading the full text, the strongest load-bearing concern is the silence-label mask, matching the reader's weakest assumption. The architecture and training pipeline are coherent, the latency experiment supports the 20-minute claim, and the paper is transparent about limitations (Section 5). However, the central novelty is the claim that one aligned autoregressive objective jointly learns when and what to output. Table 4 undermines this: the objective by itself collapses to near-total silence on OVOBench; only after randomly discarding 95% of silence targets does the model speak. The random masking removes the supervision that would teach the model not to speak at most moments, including while the user is speaking, so 'native proactive' behavior is not validated. The paper also notes possible OVOBench contamination from egocentric video sources (Section 3.3) and reports a single seed, but these affect the secondary capability comparison, not the core timing mechanism. The masking concern is therefore the more fundamental issue. It is addressable by evaluating response-timing quality and by ablating the loss design, so the verdict should remain CONDITIONAL rather than move to REJECT or ACCEPT.","tokens_in":14495,"tokens_out":11584,"duration_ms":120522,"concrete_test":"On a held-out set of realtime conversations with user/assistant timestamps, measure (i) the fraction of 80-ms slots where the model emits lexical tokens while the user's audio is active (speech-overlap rate) and (ii) the onset latency of the first lexical token after a question. Then retrain Stage 1 from the same checkpoint with a class-weighted loss that keeps all [P] targets but up-weights lexical tokens by 20x to match the r=0.95 effective lexical-to-silence loss ratio, and re-run the same metrics plus OVOBench. If speech-overlap is high, or if the class-weighted variant cannot match the r=0.95 model's Realtime score while preserving low speech-overlap, then the silence mask is the actual timing controller and the aligned objective alone does not learn when to respond.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.4 / Table 4 shows that with r=0.00 (no masking) the model 'remains almost entirely silent' and scores 6.40 on OVOBench. The model only becomes useful at r=0.70 and reaches its reported behavior at r=0.95, where 95% of [P] target positions are randomly dropped from the loss. This means the aligned next-token objective in Eq. (9) is not, by itself, learning an appropriate silence/lexical balance: it is being told to ignore 95% of the evidence that the ground-truth stream is silent. Since lexical tokens are never masked, the effective objective heavily biases the model toward emitting lexical tokens at most slots, so the 'when to respond' behavior is largely set by the hand-chosen r rather than by the temporal alignment. The paper reports no direct timing-quality metric, such as whether the model emits lexical tokens while the user is speaking or how quickly it responds after a question. Consequently, the headline claim that 'one autoregressive objective learns both when to respond and what to generate' (Section 3.2) is not yet demonstrated: the what is learned, but the when is imposed by a hyperparameter. This is fixable, so it supports a conditional verdict rather than rejection.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Aero Realtime, a 4B streaming multimodal model that aligns video, audio, and text output on a shared temporal grid of roughly 80-ms slots. Each slot predicts either a lexical token or a special silence token, so input and output advance together under a single autoregressive objective. The authors provide a complete training and serving recipe: causal realtime QA construction, slot-aligned supervision, silence-aware optimization, modality-aware three-level parallelism, and cache-valid delta inference with KV-cache reuse. On OVOBench the 4B model achieves 48.31 average, comparable to several 7B methods though below the strongest baselines. On four A6000 GPUs, the system maintains 84-ms median and 173-ms P95 processing lag over the first 20 minutes of a continuously streamed video. The central claim is that one aligned autoregressive objective can learn both when to respond and what to generate, without an external response gate.","tokens_in":14763,"tokens_out":2729,"duration_ms":28224,"significance":"If the central claim is fully supported, Aero Realtime would be a valuable architectural contribution: it demonstrates a duplex, natively proactive interface that is compatible with KV-cache-reusing inference engines and that retains non-trivial video understanding in a compact 4B model. The paper is unusually complete in its engineering details, including training stages, data mixtures, parallelization strategy, and serving mechanism. The latency measurements over a 20-minute continuous stream are a useful concrete data point, and the authors are candid about limitations such as possible data contamination and the 12.5-token/s output ceiling. However, the paper's headline claim that the aligned objective 'learns when to respond' is not yet demonstrated: the silence-label masking ratio is a hand-tuned hyperparameter that is decisive for whether the model speaks at all, and no direct timing-quality metric is reported. The contribution is therefore promising but requires additional evidence before the central claim can be accepted.","major_comments":[{"comment":"The silence-label masking ratio r is load-bearing for the claim in Section 3.2 that 'one autoregressive objective learns both when to respond and what to generate.' With r=0.00 the model 'remains almost entirely silent' and scores only 6.40 on OVOBench, while r=0.95 appears necessary for useful behavior. Since lexical targets are never masked and 95% of [P] targets are dropped from the loss, the effective objective strongly biases the model toward emitting lexical tokens at most slots, so the when-to-respond behavior is substantially imposed by the masking hyperparameter rather than learned by the aligned stream itself. The paper reports no direct timing-quality metric, such as whether the model emits lexical tokens during user speech versus after a question, or how response latency compares with ground-truth response timestamps. I recommend adding an evaluation that measures timing appropriateness directly, and ablating r with respect to that metric, not only OVOBench accuracy.","section":"Section 4.4, Table 4"},{"comment":"The 'sustained realtime processing' claim is only demonstrated for the first 20 minutes of a single 30-minute Video-MME clip. The text reports median and P95 lag over the first 20 minutes and a 153-ms lag at the 20-minute boundary, but gives no information about the 20-to-30-minute interval, where the active sequence grows further. The checkpoint values in Table 3 are non-monotonic (10 min: 95 ms, 15 min: 56 ms, 20 min: 153 ms), so it is unclear whether the system is in a steady state or approaching a threshold. Please report the full 30-minute trajectory, including what happens after 20 minutes, and state whether the KV-cache growth or input backlog is bounded over longer horizons.","section":"Section 4.3, Figure 6, Table 3"},{"comment":"The claim that the model learns 'when to respond' is not directly evaluated anywhere in the experiments. OVOBench measures video understanding under online input, but its Realtime track does not appear to score whether the model's silence/lexical decisions are temporally appropriate. The qualitative examples in Appendix E are illustrative but not quantified. Without a ground-truth-aligned metric for response timing, the paper cannot distinguish 'the model learned when to speak' from 'the masking ratio and the dataset statistics make the model speak at convenient times.' A concrete proposal: use the time-aligned realtime QA data to compute precision/recall of lexical emission within a tolerance window around annotated assistant turn boundaries, and report this for r=0.70 and r=0.95.","section":"Section 3.2 and Section 4.2"}],"minor_comments":[{"comment":"The abstract states that the system remains 'within 200 ms of the source timeline' but this is only shown for the first 20 minutes of one stream; please qualify the claim as 'for the first 20 minutes of the evaluated stream' to match the evidence.","section":"Abstract"},{"comment":"The text says 'a 30-minute Video-MME clip' but reports results only through the first 20 minutes; please clarify whether the remaining 10 minutes were excluded and why.","section":"Section 4.3"},{"comment":"The paper reports a single run per configuration without variance; given that the masking-ratio ablation drives the main qualitative change, at least reporting variance across a few seeds for the central configuration would strengthen the results.","section":"Appendix D"},{"comment":"The recurrence in Eq. (6) is attributed to Voxtral Realtime, but the paper does not state what is newly introduced beyond the visual-state injection; a sentence clarifying the architectural delta would help readers position the contribution.","section":"Section 3.2, Eq. (6)"},{"comment":"The statement that 'path-level deduplication does not guarantee content-level independence from every evaluation video' is honest, but the paper does not quantify the overlap risk; a simple estimate of source-video overlap between the training mixture and OVOBench would be more informative.","section":"Section 3.3"},{"comment":"The claim that 'the larger Backward gain suggests that the focused second-stage mixture particularly improves reasoning over previously observed events' is presented as an interpretation rather than a tested hypothesis; please soften or support it with an additional control.","section":"Section 4.5"}],"recommendation":"major_revision","confidential_remarks":"The paper is well-written and the engineering contribution is substantial, but the central scientific claim ('one autoregressive objective learns when to respond') is currently undercut by the masking-ratio dependence. This is fixable with targeted experiments, so I recommend major revision rather than rejection. I would also gently note that the authors have been commendably transparent about contamination and one-run results; that transparency should not be penalized, but the corresponding evidence gaps should be filled."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. First, it delivers the most concrete recipe I have seen for a duplex multimodal streaming model: video, audio, and text output pinned to an 80-ms grid, with a silence token in the LM head, KV-cache reuse across updates, and a real training/serving stack. Second, the headline claim that one aligned objective learns both when and what to say is not fully supported by the experiments. The model only speaks because 95% of silence targets are dropped from the loss; without that masking it goes mute and scores 6.40 on OVOBench. That makes the masking ratio the real controller of response timing, not the alignment itself.\n\nWhat is genuinely good: the slot-aligned formulation is clean, the delta-inference scheme is a credible path to low-latency serving, and the two-stage training plus the parallelism infrastructure look careful. The paper openly says it trails the strongest baselines, worries about data contamination, and flags the 12.5-token/s cap. It cites Voxtral Realtime for the recurrence and does not try to hide that. That is honest engineering work, and the qualitative transcript examples suggest the proactive behavior can be real.\n\nThe soft spots are real but fixable. The masking ablation in Table 4 shows a monotone dependence on r with no timing-quality metric — no measure of whether the model speaks while the user is talking or how quickly it responds after a question. Without that, the \"when to respond\" property is asserted rather than demonstrated. Latency is reported only for the first 20 minutes of a 30-minute clip, and only for a single run; the reader should ask why the last 10 minutes are missing. The OVOBench Forward score is the lowest in the table, which matters for a paper claiming realtime understanding. None of these sink the paper; they just mean the conditional verdict is right.\n\nI would send this to a serious reviewer. The architecture is a useful data point for anyone building streaming assistants, and the recipe is reusable even if the current model trails offline systems. The required revision is clear: show timing behavior at several masking ratios, add a direct when-to-respond metric, and report the full latency curve. If those come back clean, this becomes a solid contribution rather than a promising one.","headline":"A genuinely useful duplex-streaming architecture with an honest limitations section, but the 'learns when to respond' claim is load-bearing on a hand-tuned silence masking ratio and lacks direct timing evidence.","tokens_in":15250,"tokens_out":1398,"would_cite":true,"duration_ms":17430,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single autoregressive objective can learn both when to speak and what to say, on a shared 80-millisecond clock linking video, audio, and text output.","keywords":["streaming multimodal generation","duplex interaction","silence token","slot-aligned supervision","realtime video understanding","low-latency incremental inference","KV-cache reuse","proactive response modeling"],"falsifier":"Run the same recipe at r=0.95 and r=0.90 on the same held-out stream and compare the delay between each response onset and the event it describes; if the onsets shift or vanish at r=0.90, the masking ratio, not the aligned objective, controls timing.","tokens_in":14324,"feed_emoji":"⏱️","tokens_out":9363,"duration_ms":86327,"temperature":0.7,"pith_summary":"This paper argues that turn-based prefill-then-decode interaction is not required for realtime multimodal assistants. The proposed 4B system, Aero Realtime, places video, audio, and text output on one temporal grid of roughly 80-ms slots, and at each slot predicts either a lexical token or a dedicated silence token, so a single next-token loss learns both when to respond and what to generate. Because input and output advance on the same clock, the model can perceive while speaking, and because each update appends only the newest multimodal slot and reuses the key-value cache, a 20-minute continuous stream stays within about 200 ms of the source timeline on four workstation GPUs. These results matter because they point to a serving-friendly alternative to polling loops and external response gates for continuous, proactive interaction.","feed_headline":"One 80 ms clock teaches a model when to speak and what to say","feed_subtitle":"A slot-aligned stream keeps continuous video within 200 ms for 20 minutes on four workstation GPUs.","key_machinery":"The load-bearing mechanism is the slot-aligned input-output stream: each ~80-ms audio slot is fused with the preceding output-token embedding ($e_t^a = z_t^a + E(y_{t-1})$) to form the next causal state, and the shared language-model head predicts a lexical token or the silence token $[P]$ from that state. This single mechanism carries the whole argument: it admits new observations into an active generation sequence, makes silence and speech comparable outputs of one objective, and enables delta inference in which only the newest multimodal slot is evaluated while the KV cache keeps the prefix.","core_discovery":"The central claim is that response timing is a linguistic prediction problem, not a separate control problem. Aero Realtime couples every ~80-ms audio slot with one output slot whose target is either a lexical token or the special silence token [P]; the audio representation at slot t is added to the embedding of the previous output token, so each step is conditioned on both fresh perception and the model's own most recent output, while timestamped visual tokens enter the same causal sequence. Training remains standard next-token cross-entropy, and inference appends only the newly arrived multimodal slot while carrying the sampled token into the next fused state and preserving the KV-cached prefix. On OVOBench the 4B model scores 61.49 realtime, 44.07 backward, and 39.36 forward, comparable to several prior 7B methods, and in a long-horizon service test it maintains an 84-ms median and 173-ms P95 processing lag over the first 20 minutes of a streamed video, remaining within 200 ms of the source timeline.","pith_inferences":["In our reading, the Section 4.4 ablation suggests the silence-label mask is doing more work than a training detail: at r=0.00 the model is almost mute, so a fair test of the 'timing from one objective' claim would ramp the mask down during training and check whether timing behavior survives.","A coarse-to-fine slot structure, with a fast timing scheduler over a slower lexical decoder, could lift the 12.5-token/s ceiling while keeping a shared clock; this is a natural extension the paper lists as adaptive-rate decoding.","The aligned-stream format is modular enough that it could be instantiated per speaker to handle multi-party dialogue, with each participant's stream carrying its own silence tokens and the model scheduling turn-taking on the shared grid.","A directly testable extension would report response-onset alignment: measure the lag between each initiated response and the event it comments on, rather than only head perplexity or benchmark averages, to show timing quality is learned and not just masked in."],"forward_implications":["A duplex assistant can be trained with ordinary next-token supervision: no external response gate or polling controller is needed to decide when to speak.","Long-running interactions remain servable because each 80-ms update touches only the newest slot and reuses the cached prefix, avoiding repeated full prefills.","The fixed 80-ms grid makes the architecture conversational: lexical output is capped at 12.5 tokens per second, so the same design does not directly serve long-form text generation.","Video understanding is not sacrificed by the aligned formulation: the 4B model scores comparably to several 7B online baselines on OVOBench, while being the only evaluated architecture with duplex I/O.","Because perception stays active during generation, the model can react to events that occur mid-response, which turn-based streaming models cannot do."],"supporting_citations":[{"why":"Supplies the causal audio-plus-previous-token recurrence Aero Realtime adopts and extends with timestamped visual states.","marker":"Mistral AI et al. 2026"},{"why":"Provides the Qwen Omni audio encoder and the timestamp-interleaving of visual tokens used in the aligned stream.","marker":"Xu et al. 2025a,b"},{"why":"Provides vLLM-Omni resumable requests, the serving primitive for delta-based KV-cache reuse.","marker":"Yin et al. 2026"},{"why":"Supplies LLaVA-Video QA and caption data that are converted into the aligned-stream training format.","marker":"Zhang et al. 2025b"},{"why":"Supplies LiveCC streaming speech-transcription data used in the training mixture.","marker":"Chen et al. 2025"},{"why":"Supplies QAEgo4D egocentric video QA converted to aligned-stream supervision.","marker":"Patel, Chitalia, and Yang 2025"},{"why":"Defines OVOBench, the Realtime/Backward/Forward benchmark used to measure video understanding.","marker":"Li et al. 2025"},{"why":"Sets the 80-ms audio downsampling that fixes the slot grid.","marker":"Chu et al. 2024"}],"fun_headline_variants":["80ms slots turn response timing into a language prediction","Duplex AI on an 80ms clock: talk and listen simultaneously","One autoregressive objective decides when and what to say","Fully duplex streaming: input and output share an 80ms grid","Model speaks and listens on the same 80ms clock"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that randomly dropping 95% of silence-target losses is a benign training detail: the Section 4.4 ablation shows that without this masking the model emits silence almost always and scores 6.40 on OVOBench, so the aligned objective alone is not demonstrated to learn when to respond.","fun_headline_variants_meta":{"raw":{"variants":["80ms slots turn response timing into a language prediction","Duplex AI on an 80ms clock: talk and listen simultaneously","One autoregressive objective decides when and what to say","Fully duplex streaming: input and output share an 80ms grid","Model speaks and listens on the same 80ms clock"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000808,"raw_usage":{"total_tokens":3594,"prompt_tokens":1041,"completion_tokens":2553,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":657,"completion_tokens_details":{"reasoning_tokens":2468}},"tokens_in":657,"tokens_out":2553,"duration_ms":16731,"temperature":1.0,"reasoning_tokens":2468,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:34:32.622219+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same recipe at r=0.95 and r=0.90 on the same held-out stream and compare the delay between each response onset and the event it describes; if the onsets shift or vanish at r=0.90, the masking ratio, not the aligned objective, controls timing.","supporting_citations":[{"cited_title":"Liger-Kernel: Efficient Triton Kernels for","cited_arxiv_id":null,"evidence_quote":"Supplies LiveCC streaming speech-transcription data used in the training mixture."}],"review_version":1}