{"id":"b4895494-2535-4cb1-877e-7af7886b784a","arxiv_id":"2412.09892","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"VQTalker learns a discrete facial-motion codebook via GRFSQ and generates talking heads from speech tokens, reporting improved lip sync on non-Indo-European languages at lower bitrate.","lead":"VQTalker encodes facial motion into discrete tokens using a group-residual finite scalar quantization scheme, then generates talking-head videos from speech tokens with a BERT-based coarse-to-fine model. The paper claims better cross-lingual lip sync than prior methods, especially on non-Indo-European languages, while compressing motion to about 11 kbps.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The causal claim that discrete facial motion tokens improve cross-lingual generalization is untested: Table 4 varies audio and motion representations together and evaluates only on HDTF, so the multilingual advantage could come from CosyVoice or training data.","rationale":"The reader's weakest_assumption focuses on the tiny MNTE corpus, and that is a real problem: with 5 clips per language and no confidence intervals, the per-language numbers could easily be noise, and the self-collected, unreleased corpus makes external verification impossible. I agree with that assessment. However, the more load-bearing problem sits one level deeper: even if MNTE were replaced by a large public multilingual benchmark, the paper's central explanatory claim—that discrete facial motion tokens are what enable the cross-lingual gains—would remain unsupported. The evidence for that claim is Table 4, which is a 2x2 ablation where both audio representation (Whisper vs CosyVoice) and motion representation (continuous vs GRFSQ) change at once, and it is run only on HDTF, not on MNTE. The speech-driven baselines in Table 2 differ in many ways, so their lower scores cannot be attributed to continuous motion. A single-variable ablation on MNTE would settle whether discretizing motion actually helps multilingual generalization. Without that, the 'phoneme/viseme' motivation in the introduction is a post-hoc narrative rather than a tested mechanism. I therefore keep the CONDITIONAL verdict: the architecture is plausible and the qualitative results are suggestive, but the central claim needs a controlled test on multilingual data. I also note the bitrate comparison with continuous latent dimensions is somewhat apples-to-oranges, but I do not treat it as the most load-bearing issue.","tokens_in":19391,"tokens_out":6274,"duration_ms":64557,"concrete_test":"Run a controlled ablation on MNTE: keep CosyVoice audio, the renderer, and the training data fixed, and replace only the GRFSQ quantizer with a continuous bottleneck of the same input dimension (the 120-dimensional motion vector before quantization) while preserving all losses. Compute per-language LMD and FID with bootstrap confidence intervals. If the continuous variant is not significantly worse on a majority of the six languages, the claim that discrete facial motion tokens drive the cross-lingual improvement is falsified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"VQTalker's headline novelty is the claim that discretizing facial motion (GRFSQ) improves cross-lingual generalization relative to continuous motion representations. The paper's evidence for this is Table 4 (Discrete vs. Continuous Representation) and Table 2 (speech-driven comparison). Neither isolates the motion-quantization variable. Table 4 compares D-D (CosyVoice + GRFSQ) with C-C (Whisper + vector), varying both the audio encoder and the motion bottleneck at once; it also reports only HDTF (Indo-European) results, so it says nothing about the cross-lingual setting. Table 2 compares full systems that differ in training data, renderer, and audio features, so the win cannot be attributed to the discrete motion tokens. Thus the central mechanism asserted in the abstract and introduction is untested. The MNTE corpus's small size (30 videos, 5 per language, unreleased, and no confidence intervals) further weakens the empirical basis, but fixing that alone would not resolve the confound.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"VQTalker proposes a two-stage talking-head framework: a self-supervised facial motion tokenizer that quantizes motion differences with Group Residual Finite Scalar Quantization (GRFSQ), and a BERT-based coarse-to-fine generator that maps discrete speech tokens to four residual motion codebooks. The paper claims state-of-the-art performance in both video reconstruction and speech-driven generation, improved cross-lingual generalization from discretization, and a low bitrate of approximately 11 kbps at 512x512 resolution. Evaluation is carried out on HDTF and on a newly collected Multilingual Non-Indo-European Talking Head Evaluation Corpus (MNTE), with ablations over codebook design and discrete versus continuous representations.","tokens_in":19721,"tokens_out":4417,"duration_ms":48800,"significance":"If the central claims are validated, the paper would provide a useful architectural contribution: a codebook-free quantizer for facial motion, a clean separation between motion tokenization and rendering, and a low-bitrate discrete representation that may generalize across languages. Strengths of the manuscript include the transparent bitrate calculation (4 x 12 x log2(625) x 25), the detailed model configuration and hyperparameters, the ablation of different codebook designs, and an explicit limitations and ethics discussion. The central empirical claims, however, rest on comparisons that are partly confounded and on a very small, unreleased evaluation corpus; these need to be addressed before the stated conclusions can be accepted.","major_comments":[{"comment":"The abstract claims \"state-of-the-art performance in both video-driven and speech-driven scenarios,\" but Table 2 shows that on MNTE, EDTalker outperforms VQTalker on SSIM (0.745 vs 0.730) and LMD (0.107 vs 0.146), while on HDTF VQTalker wins. This unqualified claim is not supported by the reported numbers and should be rephrased to identify the specific metrics and datasets where VQTalker is best, with a discussion of where it is not.","section":"Table 2 and Abstract"},{"comment":"The central causal claim that discrete facial motion tokens improve cross-lingual generalization is not isolated in Table 4: the D-D condition changes both the audio encoder (CosyVoice vs Whisper vs VQ-Wav2vec) and the motion representation (vector vs GRFSQ), while the C-D and D-C rows do not hold the other component fixed. In addition, Table 4 reports only HDTF, which is Indo-European, so it says nothing directly about the multilingual setting. An experiment that varies only the motion representation while keeping the audio encoder fixed, evaluated on MNTE, is needed to support the paper's main claim.","section":"Ablation Study, Table 4"},{"comment":"No confidence intervals, per-sample breakdowns, or significance tests are reported for any of the quantitative results in Tables 1, 2, 3, 4, or 8. This is particularly problematic for the multilingual claim because MNTE contains only 30 videos total, 5 per language for 6 languages, and the authors state that the corpus is not released. Many reported differences are small (e.g., Table 1 SSIM 0.874 vs AniTalker 0.905), so without variance estimates or significance testing the conclusions are not statistically grounded. The authors should report confidence intervals or error bars, ideally with a released or expanded evaluation set.","section":"Experiment, Evaluation Metrics and MNTE"},{"comment":"The speech-driven comparison in Table 2 is a full-system comparison in which baselines are trained on different datasets (Table 5), use different decoders, and differ in audio features. Therefore the observed gains over EDTalker and EchoMimic cannot be attributed to the discrete motion representation or to GRFSQ specifically. A controlled comparison with a continuous-representation variant trained on the same data, or an ablation that isolates the proposed components under matched conditions, is necessary to support the attribution.","section":"Table 2 and Table 5"}],"minor_comments":[{"comment":"In the section headed \"Facial Motion Tokenizer,\" the text says \"Table 7 presents the results,\" but the table referenced is Table 1 in the main text; the table numbering is inconsistent with the supplementary material and should be corrected.","section":"Facial Motion Tokenizer"},{"comment":"The Limitations section refers to a \"wrapping method\" that causes blur; this appears to be a typo for \"warping method\" and should be fixed.","section":"Limitations and Future Work"},{"comment":"The statement \"we did not incorporate any pre-trained parameters\" is confusing because the framework uses a pretrained speech tokenizer from CosyVoice, Whisper, and VQ-Wav2vec in ablations, plus 3DDFA and landmark predictors; please clarify that the claim applies only to the generator and renderer, not to the entire pipeline.","section":"Training and Inference"},{"comment":"Algorithm 1 initializes \"I ← empty tensor of shape (G, R, ∗)\" but does not define how the FSQ level vector [l1, ..., ld] is converted into index tuples or what the exact shapes are; adding a concrete example or a precise index formula would improve reproducibility.","section":"Algorithm 1"},{"comment":"The language distribution percentages in Figure 5 do not obviously sum to 100%: the listed categories sum to about 97.45% before the additional 2.55% and 4.30% labels, so the figure legend or the percentages should be corrected.","section":"Figure 5"},{"comment":"In Table 3, the VQ row reports \"# Codes 8196\" with CSIM equal to MSI (0.588 and 0.588); this looks like a typo and should be verified, and the units or meaning of \"Util.\" should be stated more precisely.","section":"Table 3"}],"recommendation":"major_revision","confidential_remarks":"The paper builds directly on the authors' own AniTalker and CosyVoice, and the MNTE corpus is self-collected, small, and unreleased. The core architectural idea is plausible and the bitrate analysis is transparent, but the headline causal claim about discrete motion tokens and cross-lingual generalization needs an isolated experiment and statistical support. I see no unfixable flaw, so major revision rather than rejection seems appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nQuick take: this is a well-built system with one genuinely novel ingredient—GRFSQ, the combination of group, residual, and finite scalar quantization applied to global facial motion tokens—and a useful low-bitrate result (about 11 kbps at 512x512). But the two headline claims, \"state-of-the-art\" and \"cross-lingual generalization via discrete motion tokens,\" are not backed by the experiments as designed.\n\nWhat the paper does well: the codebook ablation in Table 3 is the strongest part. It compares VQ, GVQ, RVQ, GRVQ, and several GRFSQ configurations, and the utilization/stability numbers make a credible case that FSQ helps. The video-reconstruction comparison at 512 resolution is also informative: they beat or match continuous-vector methods at roughly one-third the bitrate. The authors give a coherent argument for why global semantic VQ beats patch-based VQ for faces, and they are honest in the limitations section about residual jitter and background blur.\n\nWhere it gets soft. The central mechanism claim is untested. Table 4 changes the audio encoder and the motion representation simultaneously (Whisper+vector vs. CosyVoice+GRFSQ) and only reports HDTF. You cannot attribute the gain to the discrete motion bottleneck; it could come from CosyVoice or the training data. Table 2 compares full systems trained on different datasets and with different renderers, so the multilingual win is not isolatable. The MNTE corpus is the other weak link: 30 videos, 5 per language, not released, no confidence intervals, and on two of the six MNTE metrics (SSIM, LMD) EDTalker actually beats VQTalker—undercutting the unqualified SOTA claim. There is also a real train/test overlap question: they train on HDTF and evaluate on HDTF without describing an identity split. And no code or data is released.\n\nIn short: the architecture is credible, the bitrate claim is concrete and reproducible from the hyperparameters, and the ablation is thoughtful. But the evidence for the multilingual advantage is circumstantial. This deserves a serious referee—the GRFSQ idea and the bitrate numbers are worth the field's attention—but it needs major revision: release code and MNTE, control the baselines, fix the ablation to isolate the motion-quantization variable, add error bars, and temper the claims.\n\nBest","headline":"Plausible, well-engineered system; the multilingual and state-of-the-art claims outrun the evidence, but the GRFSQ idea and the low-bitrate result merit a serious look.","tokens_in":20182,"tokens_out":2450,"would_cite":false,"duration_ms":27361,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"VQTalker claims that discretizing facial motion into a finite token vocabulary, rather than a continuous stream, is what lets talking-head avatars lip-sync across languages and at roughly 11 kbps.","keywords":["talking head generation","facial motion tokenization","vector quantization","GRFSQ","multilingual lip synchronization","viseme","low-bitrate facial animation","coarse-to-fine motion generation"],"falsifier":"The central claim would be falsified by a multilingual benchmark with dozens of speakers per language and phoneme-balanced test sentences that shows the discrete method's loss gap between Indo-European and non-Indo-European languages is no smaller than that of continuous baselines.","tokens_in":19149,"feed_emoji":"🗣️","tokens_out":7766,"duration_ms":74212,"temperature":0.7,"pith_summary":"VQTalker sets out to prove that a talking head should not model facial motion as a continuous stream but as a finite vocabulary of discrete motion tokens, analogous to the way speech is composed of a finite set of phonemes and corresponding visemes. The paper argues that continuous representations are the reason existing systems struggle with non-Indo-European languages: they allow infinite variation, overfit the dominant training languages, and leak identity. The proposed framework quantizes the difference between source and target face motion with a Group Residual Finite Scalar Quantization (GRFSQ) bottleneck, then generates motion tokens from speech tokens through a coarse-to-fine transformer. The authors report state-of-the-art results on both video reconstruction and speech-driven generation, with a sharper improvement on their own six-language non-Indo-European evaluation set, at roughly 11 kbps for 512x512 facial motion. A sympathetic reader would take the core proposal as: make the representation discrete and the model will transfer across languages with less data and less bandwidth.","feed_headline":"Facial motion tokens make talking avatars multilingual at 11 kbps","feed_subtitle":"A finite code of mouth and face shapes narrows the gap between Indo-European and other languages.","key_machinery":"The central object is Group Residual Finite Scalar Quantization (GRFSQ), a codebook-free quantizer that splits the 120-dimensional motion difference into 12 groups, applies 4 rounds of residual quantization, and at each step projects a low-dimensional residual onto a finite grid of levels (e.g., 5x5x5x5) via finite scalar quantization; the product grid forms an implicit codebook of 625 entries per group. Because there is no explicit learned codebook, it avoids codebook collapse and needs no auxiliary losses. On top of it, the paper builds a coarse-to-fine generation loop: for each of the 4 residual layers, a shared transformer with a layer indicator predicts all frame tokens non-autoregressively within the layer, while layers are generated autoregressively from coarse to fine, so early layers capture head pose and large motion and later layers add lip detail and suppress jitter.","core_discovery":"On its own terms, the paper's claim is that facial motion can be lossily but usefully reduced to a discrete codebook sequence, and that doing so is not merely a compression trick but the mechanism that enables multilingual generalization. The tokenizer encodes motion as the difference between source and target facial features, splits it into groups, and applies repeated residual finite scalar quantization, producing code indices that a shared transformer predicts from speech tokens plus controls for head pose, gaze, and blinks. In experiments the discrete-discrete pipeline (speech tokens in, motion tokens out) beats continuous alternatives on identity preservation, sharpness, and video distribution distance, and, crucially, the gap between Indo-European and non-Indo-European test scores is smaller for VQTalker than for comparison systems. The authors further claim a bitrate of about 11 kbps, roughly 70% of the lowest continuous-representation bitrate cited, at 512x512 resolution.","pith_inferences":["The paper does not test zero-shot transfer to a language absent from its evaluation set, but if the discrete-token argument is right, a model trained only on Indo-European speech should animate a speaker of a completely unseen language about as well as one of the six tested languages; that is a direct, checkable extension.","Because the method separates motion tokens from texture rendering, its 11 kbps bitrate is a motion-stream cost, not a full video cost; comparing end-to-end systems would require adding the renderer's output bitrate, so the headline number should be read as a motion-representation result, our reading rather than the paper's.","The phoneme-viseme framing predicts that languages with similar articulatory inventories should transfer best to each other; a per-viseme or per-phoneme error analysis on a larger corpus would reveal whether transfer is truly organized by shared mouth shapes or by something else in the discrete code."],"forward_implications":["If the central claim holds, multilingual talking-head systems no longer need large per-language training corpora; a finite motion-token vocabulary learned once should transfer to new languages through discrete speech tokens.","Facial motion can be stored or transmitted at about 11 kbps for 512x512 avatars, well below the 16 kbps floor cited for continuous representations, making real-time streaming avatars cheaper.","The discrete bottleneck doubles as an identity-leakage guard, so cross-identity reenactment can use a driving video without transferring skin tone or other appearance attributes.","The coarse-to-fine residual structure gives explicit control: separate layers carry pose, gaze, blink, and fine lip detail, so editing or retargeting one aspect without regenerating the whole sequence becomes possible."],"supporting_citations":[{"why":"Supplies the discrete multilingual speech tokens used as audio input to the motion generator; the discrete-discrete ablation shows this tokenizer outperforms both continuous and other discrete audio features.","marker":"Du et al. 2024"},{"why":"Provides the latent-image-animator backbone that the tokenizer and renderer are built on; the paper modifies it by removing linear motion decomposition and inserting quantization.","marker":"Wang et al. 2022"},{"why":"Defines Finite Scalar Quantization, the codebook-free quantizer that GRFSQ uses at every residual step, providing the implicit-codebook mechanism and straight-through estimator.","marker":"Fabian et al. 2024"},{"why":"Provides the Whisper Large v2 continuous multilingual audio features used as the continuous baseline in the discrete-versus-continuous ablation.","marker":"Radford et al. 2023"},{"why":"AniTalker is a principal prior-art baseline; its continuous 20-dimensional motion encoding is the comparison point for both quality metrics and the 16 kbps bitrate.","marker":"Liu et al. 2024"},{"why":"EchoMimic is a 512x512 baseline with strong visual quality; the user study compares against it and it defines the high-resolution comparison point.","marker":"Chen et al. 2024"},{"why":"VoxCeleb is one of the three training corpora and the main source of the 97% Indo-European language bias; its re-downloaded subset is used to train VQTalker.","marker":"Nagrani, Chung, and Zisserman 2017"}],"fun_headline_variants":["Tokenized facial motion yields 11 kbps multilingual talking avatars","Discrete facial codes improve multilingual avatar lip sync and motion","Facial motion tokenizer shrinks bitrate, widens language coverage","Multilingual talking avatars from a finite codebook of face motions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim that discrete tokens generalize across languages is tested on a corpus of only 30 short clips, five per language for six languages, collected by the authors; if those clips do not represent each language's range of sounds and mouth shapes, the cross-lingual result could disappear.","fun_headline_variants_meta":{"raw":{"variants":["Tokenized facial motion yields 11 kbps multilingual talking avatars","Discrete facial codes improve multilingual avatar lip sync and motion","Facial motion tokenizer shrinks bitrate, widens language coverage","Multilingual talking avatars from a finite codebook of face motions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000296,"raw_usage":{"total_tokens":1717,"prompt_tokens":944,"completion_tokens":773,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":560,"completion_tokens_details":{"reasoning_tokens":699}},"tokens_in":560,"tokens_out":773,"duration_ms":8443,"temperature":1.0,"reasoning_tokens":699,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T16:35:52.667059+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"The central claim would be falsified by a multilingual benchmark with dozens of speakers per language and phoneme-balanced test sentences that shows the discrete method's loss gap between Indo-European and non-Indo-European languages is no smaller than that of continuous baselines.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the latent-image-animator backbone that the tokenizer and renderer are built on; the paper modifies it by removing linear motion decomposition and inserting quantization."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines Finite Scalar Quantization, the codebook-free quantizer that GRFSQ uses at every residual step, providing the implicit-codebook mechanism and straight-through estimator."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"AniTalker is a principal prior-art baseline; its continuous 20-dimensional motion encoding is the comparison point for both quality metrics and the 16 kbps bitrate."}],"review_version":1}