{"id":"a20b94c4-4c32-4154-88a1-0f3138c5b00a","arxiv_id":"2501.04038","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"An LLM-based error corrector that re-reads both audio and lip video improves AVSR WER on clean LRS3 by 24%, but underperforms a prior method at low signal-to-noise ratios.","lead":"AVGER is a new error-correction layer for audio-visual speech recognition that feeds a large language model both the audio and lip-video again, along with the recognizer's candidate transcriptions, and asks it to pick the best one. It reports cutting the word error rate by about 24% on clean speech and some noisy conditions, but the gain is inconsistent and the technical setup has unresolved issues.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (11) defines L_WER as a Levenshtein distance with no described gradient estimator; without one, the training loss in Eq. (13) is not optimizable, so the ablation gains in Table 2 are unsupported.","rationale":"The paper's central claim—that AVGER achieves 24% WER reduction and outperforms mainstream AVSR—rests on the reported experiments. The most load-bearing assumption is that the proposed multi-level loss in Eq. (13) can actually be optimized. L_CE and L_CMD are differentiable, but L_WER as written is an edit distance over discrete sequences. Without a gradient estimator, this term cannot influence training, which contradicts the ablation study attributing a large relative WERR to it (rows #1 vs #2: 1.10 vs 1.25). This is not a stylistic omission: it determines whether the reported results are even derivable from the described procedure. The empirical overclaim in the abstract (24% at all conditions vs Table 1 at -10dB, where WERR is 4.0% and UADF is better) is a secondary reporting issue; even if fixed, the training gap remains. Since the authors release code, the concern is directly checkable, so the reader's REJECT verdict remains appropriate.","tokens_in":14365,"tokens_out":7431,"duration_ms":72001,"concrete_test":"Inspect the released training code (https://github.com/CircleRedRain/AVGER) for the implementation of L_WER. Specifically, check whether edit distance is computed on detached argmax outputs (zero gradient) or whether a REINFORCE/straight-through/soft edit-distance estimator is used. Then re-run the clean ablation with L_WER disabled and with a differentiable surrogate; if the loss cannot influence gradients, the claimed gain from L_WER is invalid.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 'Multi-Level Consistency Constraint Training' defines L_WER in Eq. (11) as D(Yhat || Y*)/len(Y*), the Levenshtein edit distance between the generated transcription and ground truth, and adds it to L_MLC in Eq. (13). The edit distance operates on discrete token sequences: Yhat is the model's decoded output, so the mapping from model parameters to L_WER is piecewise constant and its gradient is zero almost everywhere. The paper never describes a gradient estimator such as REINFORCE, straight-through, or a differentiable approximation like soft-DTW, nor does it state whether Yhat is obtained by sampling or argmax decoding. This is load-bearing because the ablation in Table 2 rows #1 vs #2 (L_WER removed) and #1 vs #3 (L_CMD removed) attributes a substantial WER improvement to these losses (1.25/1.26 vs 1.10). If L_WER cannot propagate gradients, it cannot change training, so the reported improvement is unexplained and the training procedure as written is not executable.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AVGER, a generative error correction (GER) framework for audio-visual speech recognition (AVSR). A first-stage AVSR system (AV-HuBERT) produces N-best hypotheses; a Q-Former-based multimodal synchronous encoder then re-reads the original audio and lip video, segments the frame-level features into temporally aligned clips, and compresses them into LLM-readable representations. These representations and the N-best list are combined into a cross-modal prompt for a LLaMA decoder fine-tuned with LoRA. Training uses a multi-level loss consisting of cross-entropy, a Levenshtein-distance-based WER term, and a central moment discrepancy (CMD) term. Experiments on LRS3 under clean and babble-noise conditions report WER reductions, with the abstract claiming a 24% WER reduction and superior performance to mainstream AVSR systems.","tokens_in":14598,"tokens_out":5720,"duration_ms":57148,"significance":"The underlying idea is timely and potentially useful: using the original audio and visual signals in a second-pass LLM-based correction stage is a natural extension of recent ASR GER work, and the temporal-segmentation design for modality synchronization is a sensible engineering choice. The paper also provides ablations, hyperparameter studies, and a public code/model link, which are helpful for reproducibility. However, the evidence presented does not support the claims as stated: the headline 24% figure is cherry-picked from the clean/high-SNR rows, low-SNR results are actually worse than the UADF baseline, the training loss as written is not differentiable, and the data-partition description is ambiguous. If these issues are resolved, the contribution could be a useful advance; in its current form the central claims are not established.","major_comments":[{"comment":"The abstract's claim of a 24% WER reduction and \"outperforming current mainstream AVSR systems\" is not supported by the full table. The 24% figure corresponds to the WERR relative to AV-HuBERT only at 0 dB, 5 dB, and clean conditions; at -10 dB the WERR is 4.0% and at -5 dB it is 8.9%, while UADF achieves a lower WER (21.8 vs. 29.1 at -10 dB; 10.7 vs. 12.3 at -5 dB). The authors should either restrict their claims to the conditions where AVGER is actually superior or provide a substantive analysis of why the proposed method underperforms at low SNR.","section":"Results and Analysis, Table 1"},{"comment":"L_WER is defined as a Levenshtein edit distance between the decoded transcription and the ground truth, but no gradient estimator is described. Because edit distance on discrete token sequences is piecewise constant and has zero gradient almost everywhere, the total loss L_MLC is not differentiable as written, so the training procedure is not executable without an additional mechanism such as straight-through estimation, REINFORCE, or a differentiable surrogate. This is load-bearing because the ablation in Table 2 (rows #1 vs. #2 and #1 vs. #3) attributes part of the improvement to L_WER and L_CMD. The authors must specify how gradients are obtained or remove the term from the training objective.","section":"Multi-Level Consistency Constraint Training, Eq. (11) and Eq. (13)"},{"comment":"The data-partition description is internally inconsistent. The text says the original training and test sets were combined, re-divided into a 270h training set and a 1h validation set, and that the original 0.9h test set was maintained; these statements cannot all be true simultaneously. If any test clips were used during training, the reported WERs are not valid. The authors must clarify the exact partition and explicitly confirm that the test utterances are disjoint from the training and validation sets.","section":"Dataset"},{"comment":"No error bars, confidence intervals, or multiple-seed results are reported. On a 0.9h test set, differences such as 1.10 vs. 1.19 WER (Table 2, rows #1 and #5) are small and may not be statistically reliable. At least multiple runs or a significance test are needed to support the comparative and ablation claims, especially given that some rows differ by less than 0.1 WER.","section":"Tables 1-3"}],"minor_comments":[{"comment":"The CMD loss sums from k=5, whereas the standard CMD definition uses k=2..K. If this is intentional, please explain; otherwise it appears to be a typo that omits variance, skewness, and kurtosis terms.","section":"Eq. (10)"},{"comment":"The text defines WERR as improvement relative to GER, but the values in Table 1 are computed relative to AV-HuBERT (e.g., clean WER 1.45 to 1.10 gives 24.1%). Please make the reference baseline explicit and consistent throughout.","section":"Metrics"},{"comment":"The spelling of AVSR is inconsistent ('AVSR', 'A VSR', 'A VSR'), and there are several typos (e.g., 'rencent', 'Corrposending'). Please standardize the terminology and proofread the manuscript.","section":"Throughout"},{"comment":"The insertion of RepS and RepL into the LLM embedding sequence is described only verbally; please specify whether they are concatenated with text embeddings, how length alignment is handled, and how the model distinguishes audio, video, and text segments.","section":"Experimental Setup"},{"comment":"The 'Part 1' through 'Part 5' labels in the figure are not explained in the caption, which makes the architecture diagram hard to follow. Please annotate the parts or add a matching description in the caption.","section":"Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The paper has a promising direction, but the main claims as written are contradicted by its own Table 1, and the L_WER gradient issue makes the training section non-reproducible as written. These are fixable with a revised objective, clarified data split, corrected claims, and error bars, but the authors should be asked to address them explicitly before the paper can be considered for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth reading, but with the abstract's 24% claim bracketed. The new thing here is real: AVGER is the first feature-level audio+visual GER for AVSR I know of, and the temporal-clipping scheme—Q-Former over fixed one-second segments with shared weights, separate queries per modality, segment position embeddings—is a clean way to keep audio and video aligned for an LLM second pass. The ablation is informative and shows each piece (speech, video, CMD, WER loss) contributes. The case study is nice. On clean and 0/5 dB babble, the method beats GER and UADF, and the 1.10 WER on LRS3 is a solid improvement over the 1.45 AV-HuBERT baseline.\n\nNow the soft spots. The biggest is Eq. (11). L_WER is a Levenshtein distance on discrete decoded tokens. As written, its gradient is zero almost everywhere, and the paper never says how they backprop through it—no straight-through, REINFORCE, soft-DTW, or anything. The ablation gives L_WER about 12% relative WER improvement (row #1 vs #2), so this isn't cosmetic. The authors need to state exactly what they optimize, or the training procedure is not executable as described. That's a load-bearing technical gap.\n\nSecond, the data-split sentence is dangerously ambiguous: 'combined the original training and test sets ... maintained the original 0.9h test set.' If the official LRS3 test clips went into the re-split training data, the evaluation is compromised. I suspect that's a wording slip rather than actual leakage, but reviewers need to make them pin it down.\n\nThird, the abstract says 'reduce WER by 24%' without qualification. Table 1 shows that only at clean, 0dB, and 5dB; at -5dB and -10dB, UADF beats AVGER, and at -10dB AVGER barely improves on AV-HuBERT. The paper acknowledges this in one sentence but still headlines the best case.\n\nNo error bars, and the hyperparameter search is over a single seed, so the clean-condition differences (1.10 vs 1.31) might be within run-to-run noise. That's a minor-to-moderate concern.\n\nFor a reader: someone working on GER or AVSR will get useful ideas here, especially the temporal alignment trick and the multimodal loss framing. It deserves peer review, and I'd want the reviewers to demand (1) a description of how L_WER is optimized, (2) clarification on the split, (3) a more honest abstract, and (4) at least 2-3 seeds with variance. This is a revise rather than a reject, assuming the L_WER story has a real answer.","headline":"A genuinely new audio+visual GER combination with good high-SNR results, but the 24% headline overstates it and the WER-loss gradient gap needs a real answer.","tokens_in":15133,"tokens_out":4685,"would_cite":false,"duration_ms":45858,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a second-pass language model which re-reads the original audio and lip video, together with the AVSR system's N-best hypotheses, lowers word error rate by 24% on LRS3.","keywords":["audio-visual speech recognition","generative error correction","large language models","Q-Former","multimodal synchronous encoder","cross-modal prompting","LoRA fine-tuning","LRS3 benchmark"],"falsifier":"Inspect the released training code for the implementation of the word-error-rate loss: if it has no straight-through estimator, no REINFORCE-style sampling, and no differentiable surrogate, then the objective as written cannot be minimized by gradient descent, and the ablation gains attributed to that loss term are unsupported.","tokens_in":14147,"feed_emoji":"🎤","tokens_out":10701,"duration_ms":91205,"temperature":0.7,"pith_summary":"The paper is trying to establish that generative error correction for audio-visual speech recognition should not stop at the recognizer's N-best list. AVGER, the proposed system, sends the original audio and lip video through a Q-Former-based encoder a second time, compresses them into tokens an LLM can read, and asks the LLM to pick the best transcription from the N-best hypotheses. On the LRS3 benchmark this reduces word error rate by about 24% relative to the base AVSR system on clean audio, with smaller but consistent gains at every tested noise level. The reason this matters is that it suggests the first-pass recognizer leaves recoverable information in the raw multimodal signal, and that an LLM can exploit it when the audio and visual streams are presented together with the candidates.","feed_headline":"Listening and seeing again cuts AVSR word errors by 24%","feed_subtitle":"An LLM second pass that rereads the original audio and lip video beats text-only error correction.","key_machinery":"The load-bearing mechanism is the Multimodal Synchronous Encoder built on a shared Q-Former. The Q-Former is a querying transformer that compresses long frame-level features into a fixed number of learnable query vectors; here, the speech and lip-video features are first split into equal-length temporal segments, each segment is compressed separately with modality-specific queries, and the outputs are stacked and linearly bridged into the LLM embedding space. Temporal clipping with non-overlapping segments preserves the synchronization between audio and video, which the authors identify as the key to keeping multimodal alignment. The Cross-modal Prompt then places the audio representation, the video representation, and the N-best candidate list into separate labeled sections of the prompt, and the Multi-Level Consistency Constraint loss ties the audio and video representations to the ground-truth text representation while also optimizing the final output against the reference transcription.","core_discovery":"The central claim is that an LLM-based second pass can correct AVSR errors if it re-reads the original audio and visual signals rather than working from transcripts alone. AVGER first decodes N-best transcriptions with an AVSR system, then a multimodal synchronous encoder slices the temporally aligned speech and lip-video features into fixed-length segments and compresses each segment with a shared Q-Former using modality-specific query vectors. The compressed audio representation, compressed video representation, and the N-best list are assembled into a cross-modal prompt, and a decoder LLM fine-tuned with LoRA produces the corrected transcription. Training uses a multi-level consistency constraint that combines cross-entropy, a word-error-rate term, and a central-moment discrepancy that pulls the audio and video representations toward the text representation. On LRS3 the method lifts the base AVSR word error rate from 1.45% to 1.10% on clean audio (24.1% relative reduction), outperforms text-only GER and the uncertainty-fusion UADF baseline at clean and high-SNR conditions, and improves over the base system at all tested SNRs.","pith_inferences":["At -10 dB SNR, the text-only UADF baseline beats AVGER, so a natural extension is a noise-adaptive router that down-weights the audio representation and leans on lip video when the audio is heavily corrupted.","Because the non-differentiable word-error-rate loss is added to the total loss without a stated gradient mechanism, some of the reported gain may come from the cross-entropy and CMD terms alone; a controlled run that drops only the WER term would isolate its true contribution.","The correction encoder is deliberately independent of the first-pass recognizer, which implies the same second-pass stage could be attached to a different AVSR or even ASR backend without retraining the backend, a transfer the paper does not test."],"forward_implications":["If AVGER's results hold, a second-pass LLM that re-reads the original audio and video can improve AVSR output substantially beyond what the first-pass decoder alone achieves.","The clean-audio gain shows visual information is useful even without acoustic noise, not merely as a noise-robustness mechanism.","The ablation places the temporal slicing of frame-level features as the largest single contributor, so preserving segment-level synchronization should be a design principle for multimodal speech LLMs.","The monotone improvements with more N-best hypotheses and more Q-tokens indicate that candidate diversity and representation capacity directly trade for accuracy in this pipeline."],"supporting_citations":[{"why":"Supplies the HyPoradise GER baseline and H2T-LoRA pipeline whose N-best-to-transcription mapping AVGER extends by adding raw audio and video.","marker":"Chen et al. 2023a"},{"why":"Provides the UADF uncertainty-aware fusion baseline that AVGER is compared against at multiple SNR levels.","marker":"Chen et al. 2024"},{"why":"Provides the Q-Former architecture and its learnable-query mechanism, reused as the core of the Multimodal Synchronous Encoder.","marker":"Li et al. 2023"},{"why":"Provides LLaMA-7B, the decoder that is fine-tuned with LoRA to generate the final transcription.","marker":"Touvron et al. 2023a"},{"why":"Provides LoRA, the low-rank adaptation method used to fine-tune the LLM efficiently.","marker":"Hu et al. 2022"},{"why":"Defines the Central Moment Discrepancy used in the representation-level consistency loss.","marker":"Zellinger et al. 2017"},{"why":"Defines the word error rate and Levenshtein distance used as the utterance-level loss and evaluation metric.","marker":"Davis and Mermelstein 1980"},{"why":"Supplies the AV-HuBERT model used as the first-pass AVSR system that generates N-best hypotheses and serves as the baseline in Table 1.","marker":"Shi et al. 2022"},{"why":"Supplies the VideoMAE model whose first layer is adapted into the lip-video encoder branch.","marker":"Tong et al. 2022"},{"why":"Supplies the LRS3 dataset on which the system is trained and evaluated.","marker":"Afouras, Chung, and Zisserman 2018"}],"fun_headline_variants":["LLM re-reads audio and lip video to correct AVSR, cutting WER 24%","Second-pass LLM re-reads audio and video, cuts AVSR WER by 24%","AVGER: listening and seeing again — LLM trims AVSR word errors by 24%","Hear it, see it, correct it: LLM re-reads both signals to cut AVSR WER 24%","Re-listening + re-watching: LLM second pass reduces AVSR WER 24%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The training pipeline assumes that the word-error-rate term in the loss, defined as a Levenshtein edit distance and therefore non-differentiable, can still be optimized by backpropagation, but the paper never specifies how its gradients are obtained.","fun_headline_variants_meta":{"raw":{"variants":["LLM re-reads audio and lip video to correct AVSR, cutting WER 24%","Second-pass LLM re-reads audio and video, cuts AVSR WER by 24%","AVGER: listening and seeing again — LLM trims AVSR word errors by 24%","Hear it, see it, correct it: LLM re-reads both signals to cut AVSR WER 24%","Re-listening + re-watching: LLM second pass reduces AVSR WER 24%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00181,"raw_usage":{"total_tokens":7190,"prompt_tokens":1074,"completion_tokens":6116,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":690,"completion_tokens_details":{"reasoning_tokens":5984}},"tokens_in":690,"tokens_out":6116,"duration_ms":39516,"temperature":1.0,"reasoning_tokens":5984,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:21:13.512009+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Inspect the released training code for the implementation of the word-error-rate loss: if it has no straight-through estimator, no REINFORCE-style sampling, and no differentiable surrogate, then the objective as written cannot be minimized by gradient descent, and the ablation gains attributed to that loss term are unsupported.","supporting_citations":[{"cited_title":"M.; Chen, P.; Chng, E.; and Yang, C","cited_arxiv_id":null,"evidence_quote":"Provides the UADF uncertainty-aware fusion baseline that AVGER is compared against at multiple SNR levels."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the Central Moment Discrepancy used in the representation-level consistency loss."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the word error rate and Levenshtein distance used as the utterance-level loss and evaluation metric."}],"review_version":1}