{"id":"de61fe78-f806-4198-a6ed-c25b9f3e08fa","arxiv_id":"2505.08293","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A multi-granular VQ-VAE that encodes and reconstructs motion at several time scales, paired with an audio-to-token predictor, lowers FGD and beats EMAGE in perceptual A/B tests on BEAT2 full-body gesture generation.","lead":"M3G is an AI system that creates full-body 3D gestures, including face, hands, and body movement, from audio and transcript. It works by representing movement at several time scales at once, which makes generated avatars look more natural than earlier methods in tests on the BEAT2 dataset.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The undisclosed body-hint protocol in Eq. (9) makes Table 1's headline FGD comparison ambiguous: if the first 8 ground-truth frames are used at inference, M3G is not a from-scratch generator.","rationale":"I read the paper in good faith. The multi-granular tokenization idea is plausible, and the internal ablations (Tables 2-4) are coherent and support the usefulness of the proposed components. However, the paper's central claim is an empirical superiority claim anchored on Table 1, and that claim requires a well-defined and fair inference protocol. Eq. (9) introduces ground-truth frames at training time without specifying their role at inference. This is not a matter of style or consensus; it is a correctness risk in the comparison itself. If the first 8 ground-truth frames are used at inference, the FGD gain could be explained by initialization information that baselines do not receive. If they are not used, the model faces a training/test mismatch that is never analyzed. Either way, the headline number 4.784 cannot be interpreted as stated. The reader's weakest_assumption identifies exactly this issue, and I agree. The absence of code makes the concern unresolvable from the manuscript. I do not see a reason to move the verdict beyond CONDITIONAL, because the concern could be settled by disclosure or by re-running with a zeroed hint; the architecture itself is not invalidated. The additional concerns about granularity selection on evaluation data and Beat Align being unfavorable are real but secondary. Overall, the reader's conditional verdict already captures the appropriate level of confidence.","tokens_in":13341,"tokens_out":3485,"duration_ms":37139,"concrete_test":"Ask the authors to release the inference script, then rerun the M3G row of Table 1 under three conditions: (i) first 8 frames set to ground truth as in Eq. (9), (ii) first 8 frames zeroed, and (iii) body-hint branch removed entirely. If FGD under (ii) or (iii) approaches or exceeds EMAGE's 5.643, the reported 4.784 depends on the undisclosed GT hints. If FGD under (i) matches the paper, the comparison must be restated as conditional on an 8-frame initial pose, and baselines should receive the same condition.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on Table 1: M3G's FGD 4.784 vs. EMAGE's 5.643 on BEAT2. The load-bearing condition is that both systems are evaluated under the same generation protocol. Eq. (9) in Section 3.4 defines body hints h_hints encoded from a masked gesture sequence where 'the first 8 frames are set to be the ground truth and the left frames are set to be 0 vectors.' The paper never states whether these ground-truth frames are also provided at inference. If they are, M3G receives eight real frames of the target motion, while the listed baselines generate from scratch, so the FGD gap is not a fair comparison. If they are not, Eq. (9) is trained with a conditioning signal that is absent at test time, an unquantified train/inference mismatch whose effect on FGD, Diversity, and the perceptual study is unknown. Because no source code or inference script is supplied, the reader cannot determine which protocol produced the reported numbers. This ambiguity directly undermines the strongest claim that M3G 'outperforms the state-of-the-art methods.' The Beat Align result is also not favorable to M3G, and the granularity count is selected experimentally, but those are secondary; the body-hint protocol is the single most load-bearing concern.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript introduces M3G, a two-stage framework for audio-driven full-body gesture synthesis. In the first stage, a Multi-Granular VQ-VAE (MGVQ-VAE) encodes motion sequences into token series at n temporal granularities using TCN encoders and a shared codebook, and reconstructs the motion by summing upsampled TransTCN outputs from all granularities. In the second stage, a multi-granular token predictor fuses audio and text features (with a body-hint input) to predict the token series, which are decoded by the fixed MGVQ-VAE. Experiments on BEAT2 compare M3G with S2G, Trimodal, HA2G, DisCo, CaMN, DiffStyleGesture, Habibie et al., TalkSHOW, and EMAGE; M3G reports the best FGD, MSE, LVD, and Diversity among full-body methods, while its Beat Align is lower than some baselines. A perceptual A/B study finds participants prefer M3G over CaMN and EMAGE. The paper also ablates the number of granularities and various components.","tokens_in":13624,"tokens_out":6812,"duration_ms":62180,"significance":"If the reported results hold under a fair protocol, the paper would make a solid contribution: the core idea that gestures span multiple temporal granularities and that a shared codebook plus multi-granular tokenizer can improve reconstruction and generation is clearly stated, well-motivated, and supported by a substantial set of ablations. The authors also report statistical significance across five runs, which is commendable. The main technical uncertainties—the undisclosed inference protocol for the body-hint conditioning and the uncontrolled token budget in the reconstruction comparison—must be resolved before the central claims can be accepted.","major_comments":[{"comment":"The body-hint input is not specified at inference. The text defines \\bar g as a masked gesture sequence where the first 8 frames are ground truth and the remaining frames are zero vectors, and uses h_hints as a conditioning signal. The paper never states what is used for \\bar g at inference time, where ground-truth frames are unavailable. If the same masking is used at inference, M3G is not a from-scratch generator and Table 1's comparison with EMAGE and other baselines is not under an equivalent protocol; if the masking is not used, there is a train/inference mismatch whose effect on FGD, Diversity, and the perceptual results is unquantified. The authors must report the inference protocol explicitly, and either remove the ground-truth conditioning or re-evaluate the baselines under the same conditioning; alternatively, they should ablate the body-hint at training to measure the mismatch.","section":"Section 3.4, Eq. (9)"},{"comment":"The reconstruction comparison between MGVQ-VAE and vanilla VQ-VAE is confounded by token count. MGVQ-VAE uses n token series with lengths T, T/2, ..., T/2^(n-1), so it emits roughly 2T tokens per sequence when n=4, whereas vanilla VQ-VAE emits T tokens. The observed improvements in JRMSE and FGD could reflect this increased token budget rather than a superior multi-granular representation. To support the claim that MGVQ-VAE surpasses VQ-VAE in tokenization quality, the authors should compare against a VQ-VAE with a matched total number of tokens (e.g., residual VQ with multiple levels) or report rate-distortion curves.","section":"Section 4.6, Table 4"},{"comment":"The number of granularities is selected on the test set without mention of a held-out validation split. The text says the experiment adjusts the number of granularities to find the best one, and then the best configuration (4 granularities) is used in all subsequent comparisons. This selection on the test set can overstate the benefits of the chosen hyperparameter. Please state whether the selection was made on a validation split or, if not, treat the reported numbers as partially selected.","section":"Section 4.5, Table 2"}],"minor_comments":[{"comment":"The phrase \"corase-to-fine-grained\" contains a spelling error and should be \"coarse-to-fine-grained\".","section":"Section 1"},{"comment":"The description of the \"w/o TransTCN\" variant references \"equation (4)\" and \"equation (2)\", but TransTCN appears in Eq. (5) and the encoder TCN in Eq. (2); the cross-references are incorrect.","section":"Section 4.5"},{"comment":"The p-value row does not state which statistical test was used; please specify the test (e.g., paired t-test or Wilcoxon) and the null hypothesis for each column.","section":"Section 4.4, Table 1"},{"comment":"The text contains \"Ground Turth\" in the figure caption and \"EMage\" with inconsistent capitalization; these typos should be corrected.","section":"Section 5, Figure 3"},{"comment":"The model name is inconsistent: the paper uses \"MGVQ-VAE\" in most places but \"MG-VQV AE\" in Section 4.6; please unify the notation.","section":"Section 3.3 and Section 4.6"},{"comment":"The normalization by kernel size in Eq. (6) is introduced without explanation of why kernel size affects scale; a brief justification or reference would help readers understand the motivation.","section":"Section 3.3, Eq. (6)"}],"recommendation":"major_revision","confidential_remarks":"The undisclosed body-hint protocol is the single most serious issue; it directly affects the headline FGD comparison and the perceptual study. The token-budget confound in the reconstruction comparison is also important and would weaken the second contribution if not addressed. I would request the authors to clarify the inference protocol and, if necessary, re-run the experiments, and to add a token-matched comparison for the tokenizer. If these points are resolved, the paper could be acceptable for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"M3G is a legitimate attempt at a real problem: co-speech gesture models usually tokenize motion frame-wise, which forces every gesture into a fixed temporal unit. The multi-granular idea—encoding motion patterns at several timescales into a shared codebook, then predicting tokens coarse-to-fine—is well motivated, and the paper implements it carefully. MGVQ-VAE's reconstruction results are cleanly better than vanilla VQ-VAE across every body part, and the ablations in Tables 2–4 are internally consistent and support the multi-granularity hypothesis. The component ablations (w/o TransTCN, w/o Full-Body Latent, etc.) show the expected patterns.\n\nThe soft spot is real and load-bearing. Equation (9) says the body hints are \"the first 8 frames set to be the ground truth and the left frames set to be 0 vectors,\" but the paper never states whether the same ground-truth prefix is fed at inference. If it is, M3G is not generating from scratch the way EMAGE and other baselines do, and the FGD gain in Table 1 is not a fair win. If it is not, then training used a conditioning signal that disappears at test time, and the reported gains come from an unquantified mismatch. This is not a minor omission; the headline claim depends on which protocol actually produced the numbers.\n\nA few secondary issues: the number of granularity levels is selected on the evaluation data (Table 2), the Beat Align score is actually slightly worse than EMAGE despite the overall claim, and no code is released. The token-predictor circularity (predicting the tokenizer's own tokens) is standard for this line of work and doesn't bother me because the final evaluation is on generated motion.\n\nBottom line: the architecture is worth taking seriously and deserves referee time—the reviewers can ask for the inference protocol and code. But I would not cite the quantitative comparison as a baseline until the body-hint question is answered. This is a solid paper with a fixable reporting gap, not a flawed one.","headline":"A genuine architectural contribution with clean ablations, but the undisclosed body-hint inference protocol makes the headline comparison untrustworthy until clarified.","tokens_in":658,"tokens_out":753,"would_cite":false,"duration_ms":23572,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"M3G claims that audio-driven gesture models improve when motion is tokenized at multiple temporal scales instead of one frame at a time.","keywords":["audio-driven gesture generation","full-body human motion synthesis","VQ-VAE","multi-granular tokenization","discrete latent space","co-speech gesture","BEAT2","temporal cross-attention"],"falsifier":"Run the trained M3G with zero vectors in place of the first eight ground-truth body frames at inference, then recompute FGD on BEAT2; if the score rises to or above EMAGE's 5.643, the reported advantage comes from the body-hint prefix rather than from multi-granularity.","tokens_in":13096,"feed_emoji":"💃","tokens_out":7676,"duration_ms":73088,"temperature":0.7,"pith_summary":"The paper claims that the reason audio-driven gesture models plateau is that they tokenize motion one frame at a time, even though a complete expressive gesture can last anywhere from a few frames to a longer phrase. It proposes M3G, in which a Multi-Granular VQ-VAE encodes each motion into several token streams at different temporal resolutions that share one codebook, and a token predictor reconstructs those streams from audio, text, and speaker identity. On BEAT2, M3G reports the lowest FGD among compared methods (4.784 versus EMAGE's 5.643), lower facial error, and higher diversity, and its reconstruction study finds MGVQ-VAE beats a vanilla VQ-VAE (FGD 1.497 versus 3.302). The intended significance is that granularity is a property of gesture itself, so gesture tokens should be multi-scale rather than frame-wise.","feed_headline":"Multi-granular tokens make audio-driven avatars move more naturally","feed_subtitle":"Encoding gestures at four timescales cuts FGD from 5.64 to 4.78 on BEAT2.","key_machinery":"The load-bearing object is the Multi-Granular VQ-VAE (MGVQ-VAE): a VQ-VAE whose encoder is $n$ parallel TCN streams with downsampling factors $2^i$, all mapping into one shared codebook, with the decoder summing Transpose-TCN reconstructions from every granularity. It turns 'grain size' into a first-class modeling choice instead of a fixed frame rate. The second mechanism is the multi-granular token predictor, which generates the token streams from audio and text by combining mean-pooled and TCN-derived latents per body part with temporal cross-attention and quantizing coarse-to-fine. Together these allow motion patterns with different durations to be encoded, predicted, and summed back into a full gesture.","core_discovery":"The central discovery is that a single gesture is better represented by several token streams at different temporal scales than by one token per frame. MGVQ-VAE replaces the frame-wise encoder with $n$ temporal convolutional encoders that downsample the motion by factors $2^i$, quantizes all resulting embeddings against one shared codebook, and reconstructs by summing $n$ transposed-TCN decoder outputs; the scale normalization in Eq. (6) keeps tokens from different granularities in one semantic space. The multi-granular token predictor fuses rhythmic audio features with text embeddings, separates face and body, builds two multi-granular latent views per body part (mean-pooled and TCN-derived), correlates them with temporal cross-attention, and predicts tokens coarse-to-fine. The paper reports that this pipeline outperforms compared methods on BEAT2, with FGD 4.784 versus 5.643 for EMAGE, better facial MSE and LVD, and that MGVQ-VAE reconstruction preserves motion far better than a single-granularity VQ-VAE.","pith_inferences":["A natural extension the paper leaves implicit is predicting the granularity itself per gesture or per body part, rather than fixing it to powers of two, which could adapt to fast versus slow speakers.","The reported comparison may depend on the 'body hint' protocol: the token predictor is trained with the first eight ground-truth frames, and the paper does not state whether those frames are also provided at inference; a fair head-to-head should run both settings.","If multi-granular tokens are a better motion language, they should also benefit motion editing, interpolation, and retrieval, because one codebook spans short beats and longer phrase-level movements; that is my inference, not a claim tested here."],"forward_implications":["Gesture tokenization for co-speech animation should treat temporal granularity as a variable rather than a fixed frame rate, if the reported BEAT2 gains hold.","A multi-granular VQ-VAE can serve as a standalone motion autoencoder, reconstructing BEAT2 motion with FGD 1.497 versus 3.302 for frame-wise VQ-VAE, which suggests richer discrete motion priors.","The four-granularity configuration with sizes 1, 2, 4, and 8 is the reported sweet spot; adding more granularities degrades results, so scale count and sizing are important design choices.","Facial expression, hands, upper body, lower body, and global translation can share one multi-scale token vocabulary, which is what makes holistic full-body generation tractable in this design."],"supporting_citations":[{"why":"Supplies the VQ-VAE formulation, including encoder, quantizer, codebook, and commitment loss, that MGVQ-VAE generalizes from frame-wise to multi-granular tokens.","marker":"[Van Den Oord et al., 2017]"},{"why":"Provides the EMAGE baseline and the full-body reproduction protocol for other baselines that M3G compares against on BEAT2.","marker":"[Liu et al., 2023]"},{"why":"Provides the BEAT2 dataset and the train/val/test split used for all generation and reconstruction experiments.","marker":"[Liu et al., 2022b]"},{"why":"Defines the FGD metric and the Trimodal comparison method used to judge generation quality.","marker":"[Yoon et al., 2020]"},{"why":"Supplies the TalkSHOW baseline and the A/B perceptual-study protocol used in the subjective evaluation.","marker":"[Yi et al., 2023]"},{"why":"Supplies the onset and amplitude rhythmic audio features that M3G fuses with text embeddings.","marker":"[Ao et al., 2023]"},{"why":"Supplies the pretrained word-vector model used to encode transcript text into content features.","marker":"[Bojanowski et al., 2017]"},{"why":"Defines the MSE and LVD facial-expression metrics used for evaluating face generation.","marker":"[Xing et al., 2023]"}],"fun_headline_variants":["Multi-granular tokens make audio avatars move better","Audio-driven full-body gestures at four timescales","M3G: multi-scale tokens for expressive avatar motion","One codebook, many granularities: better gestures from audio"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The token predictor is trained with the first eight ground-truth body frames as a body hint, and the paper does not state whether those frames are also fed at inference; if they are, the comparison against baselines that generate from scratch is not apples-to-apples, and if they are not, there is an unstudied training/inference mismatch.","fun_headline_variants_meta":{"raw":{"variants":["Multi-granular tokens make audio avatars move better","Audio-driven full-body gestures at four timescales","M3G: multi-scale tokens for expressive avatar motion","One codebook, many granularities: better gestures from audio"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000221,"raw_usage":{"total_tokens":1474,"prompt_tokens":996,"completion_tokens":478,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":612,"completion_tokens_details":{"reasoning_tokens":411}},"tokens_in":612,"tokens_out":478,"duration_ms":4828,"temperature":1.0,"reasoning_tokens":411,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:58:47.607486+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the trained M3G with zero vectors in place of the first eight ground-truth body frames at inference, then recompute FGD on BEAT2; if the score rises to or above EMAGE's 5.643, the reported advantage comes from the body-hint prefix rather than from multi-granularity.","supporting_citations":[{"cited_title":"Speech gesture generation from the trimodal context of text, audio, and speaker identity.ACM Transac- tions on Graphics (TOG), 39(6):1–16,","cited_arxiv_id":null,"evidence_quote":"Defines the FGD metric and the Trimodal comparison method used to judge generation quality."},{"cited_title":"Enriching word vec- tors with subword information","cited_arxiv_id":null,"evidence_quote":"Supplies the pretrained word-vector model used to encode transcript text into content features."},{"cited_title":"Codetalker: Speech-driven 3d facial animation with dis- crete motion prior","cited_arxiv_id":null,"evidence_quote":"Defines the MSE and LVD facial-expression metrics used for evaluating face generation."}],"review_version":1}