{"id":"6cc750f2-9b51-43bb-a44f-c83a96b6ec20","arxiv_id":"2412.02419","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A reactive autoregressive diffusion system synthesizes two-person conversational motion from speech with trajectory control, backed by a new interaction dataset.","lead":"This paper presents a system that generates the full-body movements of two people talking with each other, driven by their speech, in real time. It combines an autoregressive diffusion model with trajectory planning and a newly captured interaction dataset.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed online/real-time capability is not supported: Eq. (2) conditions on future speech (s, s_refer) for the entire 35-frame future window, so a streaming system must wait ~1.17s of audio before generating; the reported 8ms inference time omits this mandatory lookahead.","rationale":"Reading in good faith, the paper introduces a novel task, a reasonable dual-stream autoregressive diffusion design, and a new dataset; the ablations and user study support the value of trajectory, reference motion, and SCT. My concern is narrowly about the central 'online/real-time' claim. The model is trained and evaluated with ground-truth future speech features; this is not a missing experiment but an architectural dependence on non-causal information. The 8ms figure measures only the denoising network and cannot validate real-time interaction when the system must wait 35 future audio frames before producing a clip. If the causal-variant experiment shows FDD degradation similar to the trajectory ablation, then the first-system claim is not established for online use. I therefore recommend REJECT for the current version; the work could be reconsidered after retraining with masked or predicted future speech and reporting end-to-end latency. The reader's weakest_assumption targets trajectory prediction, which I agree is unvalidated, but the future-speech dependence is more load-bearing because it is explicit in the method and cannot be fixed by evaluating G_traj alone.","tokens_in":19641,"tokens_out":10852,"duration_ms":112749,"concrete_test":"Retrain or evaluate a causal variant of the generator in which future speech conditions s and s_refer in Eq. (2) are zero-masked at inference (analogous to the existing random-mask training in Sec. 3.2.2), and compare FDD/FPD on the InterAct++ test split. If causal FDD stays close to the full model's 117.88, the lookahead is not load-bearing and a genuinely online version is achievable; if FDD rises toward the 181.95 of the 'w/o traj' ablation in Table 5, the claimed real-time interactive quality depends on non-causal future audio and the central online claim is unsupported.","verdict_should_be":"REJECT","load_bearing_attack":"Section 3.2.1 defines the generator condition as x0_hat = G_m(x_t, t; [m, p, s, m_refer, s_refer]) (Eq. 2), where s and s_refer are explicitly 'future speech features' from the main speaker and partner. Section 3.3 sets the window to N=45 frames with 35 future frames; at 30 FPS that is 1.17s of audio that must be available before a clip can be generated. Thus the system is not causal: in a live conversation, a motion response to an utterance cannot begin until 35 frames of that utterance have already been heard. The paper's real-time paragraph reports only the diffusion model's 8ms per clip and omits this buffering latency, plus SpeechTokenizer and trajectory prediction. The authors acknowledge the issue in Section 3.2.1: 'Using the future information may sound unrealistic for interactive applications, but indeed humans predict it for smooth communication... We will leave a model that predicts such information for future work.' This is an admitted use of ground-truth future audio, not a predicted one, and it directly undercuts the abstract's 'online manner' and the title's 'Real-time'/'Reactive' claims. The trajectory concern raised by the reader is valid, but the future-speech lookahead is more fundamental: even a perfect trajectory predictor would not make the pipeline causal.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces an audio-driven, autoregressive diffusion framework for generating full-body motions of two interacting characters from speech, together with a new dataset (InterAct++) that extends the existing InterAct dataset with more dynamic interactions. The method predicts root trajectories from speech and then generates two-person motion conditioned on past and future speech, past motion, and trajectories. The authors report state-of-the-art results on single-person co-speech generation (BEAT), a two-person speech-to-motion benchmark, and an interaction generation task, and they claim real-time online performance with 8 ms per motion clip. The manuscript includes ablations, a user study, and a plan to release code and data.","tokens_in":19970,"tokens_out":8046,"duration_ms":84057,"significance":"If the real-time online claim were correct, this would be a significant advance for interactive character control and conversation animation, and the InterAct++ dataset is a useful addition. The two-stage trajectory-to-full-body design and the separate token conditioning are sensible, and the experiments cover standard metrics and ablations. However, the paper's headline contribution is undermined by the use of ground-truth future speech, which makes the pipeline non-causal; the trajectory module is also not evaluated directly. These issues prevent the claims as stated from being accepted.","major_comments":[{"comment":"The claimed online/real-time capability is not supported by the model definition. Equation (2) conditions the generator on future speech features s and s_refer for the entire 35-frame future window (N=45 frames with 10 past frames, Sec. 3.3). At 30 FPS this requires roughly 1.17 seconds of audio to be buffered before a motion clip can be produced, so the system is non-causal. The acknowledgment in Sec. 3.2.1 that the future information is not predicted but used as ground truth ('We will leave a model that predicts such information for future work') confirms that this is an architectural requirement, not an implementation detail. The 8 ms per-clip inference time reported in Sec. 3.3 omits this mandatory lookahead as well as the costs of SpeechTokenizer and trajectory prediction. Consequently, the abstract's 'online manner', the title's 'Real-time', and the 'reactive' characterization are not justified. I recommend either removing these claims and repositioning the work as an offline future-aware generation system, or introducing and evaluating a causal variant that uses only past speech.","section":"Sec. 3.2.1, Eq. (2), Sec. 3.3"},{"comment":"The trajectory prediction module G_traj is a load-bearing component: the full-body diffusion model is conditioned on predicted trajectories (Sec. 3.1 and D.1), and the ablation 'w/o traj.' (Table 5) shows FPD degrading from 47.74 to 103.49 and FDD from 117.88 to 181.95 when trajectory conditioning is removed. However, the main experiments never report a direct evaluation of trajectory prediction accuracy (e.g., root position error, facing-direction error, path divergence) on held-out conversations, nor do they analyze how trajectory errors accumulate over autoregressive windows. Without such an evaluation, the claimed interactive quality and the practical usability of the fully automatic pipeline are not fully established.","section":"Sec. 3.1 / D.1 / Table 5"},{"comment":"The 'Real-time Performance' paragraph reports only the motion generation module's inference time (8 ms per clip). A real-time claim requires an end-to-end latency measurement that includes audio tokenization (SpeechTokenizer), trajectory prediction, the autoregressive context update, and the blending operations described in Sec. 3.3 and D.4. Please provide a complete latency breakdown on the evaluation hardware, and specify the total latency from audio input to rendered motion output, including the buffering requirement of the future-speech conditioning.","section":"Sec. 3.3"}],"minor_comments":[{"comment":"Audio2Photoreal is cited as [41] in Sec. 5.2 and the caption of Fig. 5, but reference [41] is 'Audio2Gestures' (Li et al.); the correct citation is [52] (Ng et al.). Please fix.","section":"Sec. 5.2 / Fig. 5"},{"comment":"The abbreviation 'SCT' is used without definition; please define 'Separated Conditional Tokens' when first used in Sec. 3.2.1 or in the table caption.","section":"Table 5"},{"comment":"The row label 'IntetAct' should be 'InterAct'.","section":"Table 3"},{"comment":"Equation (2) uses G_m while the preceding text and Eq. (1) use G; please make the notation consistent and clarify whether the subscript denotes the masked/modal variant.","section":"Sec. 3.2.1, Eq. (2)"}],"recommendation":"reject","confidential_remarks":"The paper has useful components, especially the InterAct++ dataset and the two-stage trajectory-conditioned generation, but the central advertised capability (real-time online two-person interaction generation) is contradicted by the model's explicit use of ground-truth future speech, which the authors themselves acknowledge is left for future work. This is not a local presentation issue but a fundamental mismatch between the claim and the method. A rejection at this stage seems appropriate, though a future submission that honestly repositions the work as offline future-aware generation and directly evaluates the trajectory module could be viable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nYou should know this paper is worth reading but its central claim needs careful unpacking. It builds the first autoregressive two-person co-speech generation system, with trajectory control and a new dataset (InterAct++), and the experiments back the design choices. But the \"real-time\" and \"online\" claims in the abstract and title are not supported by the actual architecture. Eq. (2) conditions generation on future speech features for a 35-frame window. At 30 fps that is about 1.17s of audio that must be available before a motion clip can be generated.\n\nThe authors are honest about this: in Sec. 3.2.1 they write that using future information \"may sound unrealistic for interactive applications\" and leave a predictive model to future work. That admission is buried in the method section, but the abstract still calls the system \"online.\" It is not. It is a buffered, non-causal system. The reported 8ms per clip is the diffusion model only, and omits speech tokenization, trajectory prediction, and the lookahead delay. So the real-time performance paragraph overstates what is actually delivered.\n\nThe trajectory concern from the reader's report is valid too. The trajectory predictor is a separate module, and its accuracy is never evaluated. The ablation shows that removing it degrades FPD from 47.74 to 103.49 and FDD from 117.88 to 181.95. That makes the trajectory module load-bearing, and it is a black box in the evaluation.\n\nThe paper does genuine things well. The task formulation is new and clearly located against ARfriend and text-driven baselines. The separated condition tokens, the random masking for reference motion, and the trajectory blending are all sensible and the ablations support them. InterAct++ is a real contribution; the authors describe the capture setup and scenarios in the supplement. The improvements over Audio2Photoreal, LDA-dual, and ReMoS are consistent, not cherry-picked. I would cite this for the dataset and task definition.\n\nThe soft spots beyond causality: metrics lack error bars, code and data are promised but not yet released, and the user study has a large preference margin that is plausible but not error-barred. These are addressable in revision. The causality issue is more fundamental: the paper should either add a predictive speech model, or reframe the contribution as offline generation with fixed lookahead, which would still be a contribution.\n\nThis is a paper for character animation and embodied-agent researchers. A serious referee should spend time on it; it deserves major revision, not desk rejection. Recommended for peer review, with the expectation that the real-time claim gets corrected or demonstrated end-to-end.","headline":"Genuinely new task and a solid system plus dataset, but the 'real-time online' claim does not survive contact with the model's own future-speech condition.","tokens_in":20513,"tokens_out":2415,"would_cite":true,"duration_ms":24322,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims the first online system that synthesizes interactive two-person full-body motion from speech in real time.","keywords":["co-speech gesture synthesis","two-person interaction generation","autoregressive diffusion model","real-time motion synthesis","speech-driven animation","trajectory control","conversational motion dataset","full-body motion generation"],"falsifier":"Measure the trajectory predictor's root-position and facing-direction error against held-out ground-truth motion for the same two-speaker audio: if the average endpoint error is large relative to the distance needed for a handshake or object handoff, then the interactive realism cannot be attributed to the planned trajectory. A second check is to inspect whether contact actions such as handshakes actually produce touching hands in generated clips, since the authors state that no explicit joint-position constraint is used.","tokens_in":19456,"feed_emoji":"👥","tokens_out":7775,"duration_ms":80901,"temperature":0.7,"pith_summary":"This paper sets out to make conversational avatars move the way people actually move in dialogue: each person's body responds to their own speech, the partner's speech, and the partner's current motion, while also walking, turning, hugging, handing objects, and performing other dynamic acts. It claims to be the first system that can do this online, in real time, rather than offline after the full conversation is known. The generator is an autoregressive diffusion model that consumes the recent motion of both characters, future speech features from both, and a predicted 2D root trajectory, producing each 45-frame window of two-person motion in roughly 8 ms (over 100 fps). To make the task learnable, the authors extend an existing two-person conversational dataset with 402 new clips covering hugging, handshaking, waving, grabbing, patting, and high-fives. If the claims hold, live virtual agents, games, and VR characters could hold natural and spatially active conversations instead of standing still and gesturing.","feed_headline":"First real-time two-person speech-to-motion model runs at 100 fps","feed_subtitle":"An autoregressive diffusion model animates two characters from both voices and a planned trajectory in real time.","key_machinery":"The engine is a conditional denoising diffusion transformer with separated condition tokens. For each sliding window it takes past self motion, the partner's past motion, future speech features (semantic tokens from a pretrained speech tokenizer, Mel-spectrogram acoustics, and rhythm), and a future 2D root trajectory (body positions plus facing directions) for both characters, and denoises the next 45 frames of two-person motion in one pass. Training uses a random mask on the partner's motion so the model can also learn from single-person co-speech data, and inference applies a classifier-free-guidance-style interpolation between conditional and unconditional predictions to tune how strongly a character follows the partner. Alternating which character is root-normalized preserves relative position between the two bodies. Long sequences are stabilized by blending the input trajectory with a self-predicted trajectory and by cross-fading overlapping clip boundaries.","core_discovery":"The central claim is that two-person interactive full-body motion can be generated autoregressively from two speech streams in real time, with each character reacting to the other's audio and motion while following a planned root trajectory. The authors report that the full model outperforms single-person co-speech baselines and two-person interaction baselines on a benchmark built from the extended dataset, and that the reactive autoregressive design is what makes the difference: removing the reference-person condition degrades interaction quality sharply. They also introduce InterAct++, an enriched dataset of dynamic conversational interactions, and show that their system can be trained on single-person data through a random masking scheme that behaves like classifier-free guidance. The stated novelty is the combination: real-time speed, autoregressive reactivity, two-character full-body output, and trajectory controllability in one system, which no prior work achieves together.","pith_inferences":["Because trajectory accuracy is never reported as a standalone number, an immediate next evaluation would be to measure endpoint and heading error against ground truth; I would expect this to be the main factor separating success from failure on unseen conversations.","The condition channels are generic, so the dual-stream design could in principle be extended to three or more characters by adding more partner tokens, but the paper does not test this.","The random-mask guidance weight already interpolates between fully reactive and fully autonomous behavior, which suggests a user-facing slider for how strongly a character follows its partner; the paper does not explore this interaction-control dimension.","The authors' own limitation about handshaking implies that contact-critical motions need an explicit geometric constraint; adding a contact-aware loss or post-contact correction is a natural testable extension."],"forward_implications":["Because each 45-frame window is generated in about 8 ms, a deployed system can animate a conversation as it happens, at over 100 fps.","Because generation is conditioned on a planned 2D trajectory, an animator or game engine can steer where characters walk and face while speech drives their gestures.","The random masking of the partner's motion lets the same model train on single-person co-speech datasets, so the approach can absorb large existing corpora beyond the collected two-person data.","The new InterAct++ clips provide test material for dynamic actions such as hugging, handshaking, waving, grabbing, patting, and high-fives that earlier two-person conversation datasets lacked.","Online applications such as VR, games, and customer-service avatars stand to gain two characters that converse and move together in real time rather than standing and gesturing in place."],"supporting_citations":[{"why":"InterAct, the only existing two-person conversational speech-and-motion dataset, which InterAct++ extends and which provides the training/test split.","marker":"[29]"},{"why":"LDA, the co-speech diffusion baseline the two-person benchmark compares against and the basis of the LDA-dual variant.","marker":"[3]"},{"why":"Audio2Photoreal, the two-person conversational baseline that lacks reactive conditioning on the partner's motion.","marker":"[52]"},{"why":"ReMoS, the interaction-reaction synthesis baseline used for the interactive motion generation comparison.","marker":"[19]"},{"why":"BEAT, the large single-person co-speech dataset used for training and for the single-person evaluation.","marker":"[45]"},{"why":"SpeechTokenizer, the pretrained tokenizer that provides the compact semantic speech tokens that replace text embeddings.","marker":"[71]"},{"why":"The 8-step diffusion formulation and trajectory extension used to reach real-time generation speed.","marker":"[11]"},{"why":"Denoising diffusion probabilistic models, the generative formulation underlying the conditional motion generator.","marker":"[24]"}],"fun_headline_variants":["Real-time two-person speech-to-motion at 100 fps","Autoregressive diffusion animates two speakers in real time","First online model for interactive two-speaker motion","Two-person co-speech motion generation goes real-time at 100 fps"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the trajectory predictor can reliably anticipate each speaker's future root path and facing direction from speech and context alone, because the full-body diffusion generator is conditioned on those predicted trajectories even though trajectory accuracy is not measured in the main evaluation.","fun_headline_variants_meta":{"raw":{"variants":["Real-time two-person speech-to-motion at 100 fps","Autoregressive diffusion animates two speakers in real time","First online model for interactive two-speaker motion","Two-person co-speech motion generation goes real-time at 100 fps"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000859,"raw_usage":{"total_tokens":3712,"prompt_tokens":914,"completion_tokens":2798,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":530,"completion_tokens_details":{"reasoning_tokens":2729}},"tokens_in":530,"tokens_out":2798,"duration_ms":21861,"temperature":1.0,"reasoning_tokens":2729,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T23:28:06.756684+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the trajectory predictor's root-position and facing-direction error against held-out ground-truth motion for the same two-speaker audio: if the average endpoint error is large relative to the distance needed for a handshake or object handoff, then the interactive realism cannot be attributed to the planned trajectory. A second check is to inspect whether contact actions such as handshakes actually produce touching hands in generated clips, since the authors state that no explicit joint-position constraint is used.","supporting_citations":[{"cited_title":"Interact: Capture and modelling of realistic, ex- pressive and interactive activities between two persons in daily scenarios","cited_arxiv_id":null,"evidence_quote":"InterAct, the only existing two-person conversational speech-and-motion dataset, which InterAct++ extends and which provides the training/test split."},{"cited_title":"Listen, denoise, action! audio-driven motion synthesis with diffusion models","cited_arxiv_id":null,"evidence_quote":"LDA, the co-speech diffusion baseline the two-person benchmark compares against and the basis of the LDA-dual variant."},{"cited_title":"From audio to photoreal embodiment: Synthesizing humans in conversations","cited_arxiv_id":null,"evidence_quote":"Audio2Photoreal, the two-person conversational baseline that lacks reactive conditioning on the partner's motion."},{"cited_title":"Remos: 3d motion- conditioned reaction synthesis for two-person interactions","cited_arxiv_id":null,"evidence_quote":"ReMoS, the interaction-reaction synthesis baseline used for the interactive motion generation comparison."},{"cited_title":"Speechtokenizer: Unified speech tokenizer for speech language models, 2023","cited_arxiv_id":null,"evidence_quote":"SpeechTokenizer, the pretrained tokenizer that provides the compact semantic speech tokens that replace text embeddings."},{"cited_title":"Taming diffusion probabilistic mod- els for character control","cited_arxiv_id":null,"evidence_quote":"The 8-step diffusion formulation and trajectory extension used to reach real-time generation speed."},{"cited_title":"Denoising dif- fusion probabilistic models","cited_arxiv_id":null,"evidence_quote":"Denoising diffusion probabilistic models, the generative formulation underlying the conditional motion generator."}],"review_version":1}