{"id":"9267f62a-9b80-4a7a-9d25-f14a6435592c","arxiv_id":"2411.17998","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Codecformer-EL replaces the waveform loss in codec-based speech separation with an MSE loss on frozen codec embeddings, cutting training cost by about half and improving perceptual metrics on WSJ0-2mix.","lead":"This paper trains codec-based speech separation models by comparing compressed audio embeddings instead of waveforms, so the codec decoder is not needed during training. The authors report roughly twice the training speed and better perceptual quality scores on WSJ0-2mix, though standard signal-quality metrics drop sharply.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The embedding-loss target is the codec encoder's continuous output, but the decoder is trained on quantized embeddings; if separator outputs are not quantized before decoding, the decoder sees out-of-distribution inputs and the reported perceptual gains may be resynthesis artifacts, not better…","rationale":"The reader correctly identifies the embedding-space suitability assumption as fragile. I sharpen it into a specific, testable mechanism: a train/inference input distribution mismatch. The separator is trained to reproduce continuous encoder embeddings, while the codec decoder was trained on quantized embeddings. The paper does not state that separator outputs are quantized before decoding, and Figure 1 implies direct decoding. This is load-bearing because the entire motivation for embedding loss is that lower embedding MSE improves decoded audio; if the decoder cannot invert continuous embeddings, the loss optimizes a proxy misaligned with the actual generation path. The very negative SI-SDR values alongside high DNSMOS/STOI in Table II are exactly what one would expect from a decoder operating out-of-distribution, so the headline perceptual improvements may not reflect better separation. I do not treat this as a proven flaw, because the embedding space may be smooth enough for the decoder to generalize; hence a concrete oracle test is needed. I also considered the batch-size confound in the speed comparison and the absence of error bars, but the decoder-input mismatch is more fundamental: if it lands, the central claim fails regardless of speedups or statistical significance. The reader's verdict of CONDITIONAL is therefore appropriate, and our concern does not change it.","tokens_in":9103,"tokens_out":8179,"duration_ms":76600,"concrete_test":"Run an oracle decoding test on a held-out set of WSJ0-2mix utterances: feed (a) the ground-truth continuous encoder embeddings e = Encoder(s_gt), exactly the targets used in Eq. 1, and (b) the corresponding quantized embeddings through the frozen codec decoder, then measure SI-SDR, STOI, and DNSMOS against s_gt. If (a) is substantially worse than (b), the continuous embedding space is not the decoder's input distribution, so the embedding-loss target is mismatched and the perceptual gains in Table II must be reinterpreted. As a complementary check, compute the nearest-codebook distance of Codecformer-EL's separator outputs; a low hit rate would confirm the decoder is being fed out-of-distribution inputs.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's core mechanism is Eq. 1: MSE between the separator output and the frozen codec encoder's embedding of the clean target. The same embedding space is then fed to the codec decoder at inference (Fig. 1). However, SoundStream, EnCodec, and DAC are autoencoders whose decoders are trained to invert quantized embeddings (RVQ codebook sums), not the unquantized continuous encoder output. The text never states that Codecformer-EL's output is quantized or matched to the codebook before decoding. If it is not, the decoder is being evaluated on out-of-distribution continuous vectors. Minimizing embedding MSE is then a surrogate for the true objective, decoded audio quality. Table II's pattern — e.g., EnCodec SI-SDR of -29.1 dB with DNSMOS OVRL 1.89 and STOI 0.80 — is consistent with the decoder generating perceptually plausible but signal-misaligned audio, rather than with better separation. Section II-A's 'spectrogram-like' analogy is not evidence that MSE in this space improves decoder output. Section IV-E concedes reliance on codec quality, but the specific encoder/decoder input mismatch is not examined. If the decoder cannot faithfully invert the continuous embeddings used as training targets, the central claim that embedding loss improves separation quality collapses; the observed DNSMOS/STOI gains would be codec resynthesis artifacts, not source-separation improvements.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Codecformer-EL, a modification of the Codecformer speech separation model that trains using an embedding-level MSE loss on the frozen encoder representations of a neural audio codec, removing the need to decode during training. The authors claim a 2x improvement in training speed and computational cost, and better DNSMOS and STOI performance on WSJ0-2mix across three pre-trained codecs (DAC, EnCodec, SoundStream), despite lower objective metrics. They also compare separation performance across codec pre-training datasets and extend Codecformer to support EnCodec and SoundStream.","tokens_in":9470,"tokens_out":3734,"duration_ms":34139,"significance":"If the central claim holds, the efficiency gain of avoiding codec decoding during training is practically valuable, and the positive perceptual results would suggest that embedding-space training is a viable alternative for codec-based separation. The paper is also useful for expanding codec-based separation to multiple codec architectures and for documenting the objective-perceptual metric gap in this setting. However, the support for the headline claim is weakened by the absence of statistical significance testing, a small contradictory DNSMOS result for one codec, and an unaddressed quantization mismatch between the training target and decoder input. The manuscript honestly acknowledges some limitations, but the load-bearing claim needs better empirical grounding.","major_comments":[{"comment":"The abstract and Table II's discussion claim 'better DNSMOS' for embedding loss relative to waveform loss, but Table II reports DAC with embedding loss DNSMOS OVRL 1.80 versus waveform loss 1.81, which is worse. The claim is only consistently true for STOI across all three codecs, and for DNSMOS on EnCodec and SoundStream. Please either revise the claim to acknowledge the DAC result or provide error bars and significance tests demonstrating that the differences (including the 0.01 OVRL gap) are robust.","section":"Abstract and Section IV-B, Table II"},{"comment":"The neural audio codec decoders (SoundStream, EnCodec, DAC) are trained to invert quantized embeddings (RVQ codebook sums), but Codecformer-EL outputs continuous embeddings at inference and the manuscript never states that these are quantized before decoding. If the decoder receives unquantized continuous vectors, it operates out-of-distribution, and the very low SI-SDR values (e.g., EnCodec embedding loss -29.1 dB) combined with high DNSMOS/STOI may be resynthesis artifacts of the generative decoder rather than genuine improvements in separation. The authors should clarify whether quantization is applied at inference, and if not, should either apply it or provide an explicit analysis of the encoder/decoder distribution mismatch, including a comparison of decoded outputs with and without quantization.","section":"Section II-A and Figure 1"},{"comment":"The training speed comparison is confounded by different batch sizes: the embedding-loss model is trained with batch size 20 while the waveform-loss Codecformer uses batch size 3. Since MACs and training time are reported for the full training procedure, the '2x' advantage could partly reflect the larger batch size of the proposed method rather than the removal of the decoder alone. Please provide a comparison at matched batch size, or at least per-sample measurements, and a breakdown of where the MAC/time savings occur (e.g., decoder forward/backward vs. loss computation).","section":"Section III-D and Table IV"},{"comment":"No error bars, confidence intervals, or significance tests are reported for any of the objective or perceptual metrics. Given that some differences are very small (e.g., DAC DNSMOS OVRL 1.80 vs. 1.81), the reader cannot assess whether the claimed perceptual improvements are reliable. Please report variance across multiple training runs or a significance test, and state the number of runs used.","section":"Tables II and III"}],"minor_comments":[{"comment":"The first contribution states '2x improvement in training speech' but the intended word is 'speed'; please fix the typo.","section":"Section I, Contributions"},{"comment":"The permutation for perceptual metrics is chosen by maximizing SI-SDR, which is an objective metric. This is consistent, but it may not be the permutation that maximizes the perceptual metric itself; consider also reporting results under per-metric optimal permutations, or justify the SI-SDR-based choice.","section":"Section III-C"},{"comment":"The justification that embeddings are 'spectrogram-like' and therefore MSE is appropriate is an analogy rather than a derivation. A small ablation comparing MSE against an alternative loss (e.g., cosine similarity or L1) would strengthen the argument, but is not essential if the empirical results are robust.","section":"Section II-A"},{"comment":"The EnCodec embedding-loss SI-SDR of -29.1 dB is drastically lower than the waveform baseline (-9.0 dB). While the paper attributes this to the objective-perceptual gap, the magnitude deserves a brief discussion to reassure readers that the output is not semantically disconnected from the target speech.","section":"Table II"},{"comment":"The comparison across pre-training datasets for DAC is confounded by embedding size (Original DAC has 1024 vs. 512 for ESPnet versions). The paper mentions this, but it would be informative to state whether the performance drop for the Original DAC embedding loss is more plausibly due to dataset or embedding-size mismatch, given the scope of the paper.","section":"Section IV-C"}],"recommendation":"major_revision","confidential_remarks":"The paper's baseline is the authors' own prior work, Codecformer. This is not a problem by itself, but the lack of an independent baseline (e.g., a standard waveform-domain separator evaluated with the same perceptual metrics) limits the novelty assessment. I would encourage the editor to request a matched-batch-size comparison and a resolution of the quantization-mismatch concern, as those affect the central claim. The paper is otherwise within the scope of the journal and presents a useful efficiency-motivated idea."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper's core idea is to train a codec-based separator with MSE on frozen codec encoder outputs instead of waveform loss. That's a clean, simple contribution, and the speedup story (no decoding during training) is credible. The paper also extends Codecformer to EnCodec and SoundStream and adds STOI/DNSMOS. It reports lower objective scores but better perceptual scores for embedding loss on EnCodec and SoundStream; for DAC, the perceptual scores are a tie, which the abstract glosses over.\n\nWhat's actually new: training on embeddings rather than waveforms in a codec-based separator, with a systematic comparison across three codecs and two pre-training sets. That's useful for anyone working on codec-based speech processing.\n\nWhere it's soft: first, the abstract says \"better DNSMOS and STOI\" but Table II shows DAC embedding loss gets DNSMOS 1.80 vs 1.81 for waveform — essentially even. So the claim is overstated. Second, the speedup comparison is confounded by batch size (20 for embedding, 3 for waveform). The MACs figure (0.8 vs 1.5 GMACs) is batch-invariant and stands, but the wall-clock time benefit is not purely from the loss. Third, and most important: the codec decoders are trained on quantized embeddings (RVQ codebook sums), but the separator outputs are continuous embeddings. The paper never says the output is quantized before decoding. If it isn't, the decoder is receiving out-of-distribution vectors, and the perceptual quality you measure may be a resynthesis artifact rather than a separation improvement. The authors' own limitation section acknowledges reliance on codec quality but never examines this encoder/decoder input mismatch. That's a load-bearing gap. Fourth, there are no error bars or significance tests, and DNSMOS is a non-intrusive metric not designed for separation; small differences may not mean anything.\n\nDoes the central argument hold up? I'd say partially. The method clearly trains — the model learns to produce something the decoder turns into intelligible speech. But the specific claim of better perceptual quality is not established, and the quantization issue could undermine the mechanism entirely. It's not fatal, but it's the first thing I'd ask an author.\n\nWho this is for: people working on codec-based speech processing or efficient separation. A serious referee should engage with it, primarily to force clarification of the quantization step and a fairer speed comparison. I'd send it to review, but with the expectation of heavy revision.","headline":"A simple, promising trick—train codec-based separators with embedding-level MSE—but the perceptual gains and speedup are less clean than the abstract claims.","tokens_in":9965,"tokens_out":3457,"would_cite":false,"duration_ms":32182,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Codec-based speech separators can be trained in the compressed embedding domain without a decoder, cutting training time and compute roughly in half while improving perceptual quality on WSJ0-2mix.","keywords":["speech separation","neural audio codecs","embedding loss","compressed-domain processing","perceptual speech quality","DNSMOS","STOI","permutation invariant training"],"falsifier":"Train Codecformer-EL with the frozen codec encoder replaced by a randomly initialized, untrained encoder (same architecture, weights fixed). If the embedding-loss models still show the reported DNSMOS and STOI gains over waveform loss, the improvements do not depend on the codec's learned embedding geometry, which would contradict the paper's premise.","tokens_in":8941,"feed_emoji":"🎧","tokens_out":6669,"duration_ms":56461,"temperature":0.7,"pith_summary":"This paper argues that a speech separation model operating inside a neural audio codec can be trained with an embedding-level mean-squared-error loss instead of a waveform-comparison loss, removing the codec decoder from the training loop entirely. The authors show, on the WSJ0-2mix benchmark, that this yields separators that train about twice as fast and at roughly half the compute while matching or improving perceptual metrics such as DNSMOS and STOI, even though conventional objective scores like SI-SDR are lower. The result matters because it suggests that compressed-domain separation can be scaled to larger datasets and deployed where waveform-domain training is too costly, and that perceptual quality rather than exact waveform reconstruction is the right target for codec-based audio.","feed_headline":"Embedding loss cuts speech separation training cost 2x","feed_subtitle":"Training on compressed codec embeddings, not waveforms, beats waveform loss on DNSMOS and STOI.","key_machinery":"The central object is the embedding loss: mean squared error between the separated embeddings produced by the separator and the ground-truth embeddings produced by the frozen codec encoder, wrapped in permutation invariant training so the loss is taken over the best speaker ordering. It replaces the waveform-domain SI-SDR/PIT loss used in the baseline and lets training skip the codec decoder and the expensive waveform comparison. The separator itself is a transformer stack with channel-adapter layers and a mask-gating block that matches the codec's activation function (ELU for EnCodec and SoundStream, snake for DAC), producing a mask on the mixture embeddings. Because the frozen encoder is fixed, ground-truth embeddings can be precomputed before training, which is what makes the loss cheap.","core_discovery":"The paper's central discovery is that the frozen encoder of a pre-trained neural audio codec provides a sufficient training target for speech separation: the separator minimizes PIT-wrapped MSE between its output embeddings and the ground-truth speaker embeddings, and the codec decoder is needed only at inference time. Across DAC, EnCodec, and SoundStream pre-trained on AMUSE, models trained this way reach comparable or better DNSMOS and STOI than the same separator trained with waveform loss, while training 2.5x faster and using 1.9x fewer MACs than the waveform-loss Codecformer baseline (and 6.8x faster than Sepformer with 97x fewer MACs). The authors also find that the embedding-loss advantage persists across different codec pre-training datasets, except for the original DAC model with 1024-dimensional embeddings, where the larger embedding size appears to hurt the method.","pith_inferences":["Going beyond the paper's experiments: if MSE on frozen codec embeddings is a reliable perceptual proxy, then other embedding-space objectives—masked prediction, contrastive losses, or fine-tuning the encoder—are natural next steps that could further close the objective-metric gap.","The same decoder-free training recipe should transfer to other audio-to-audio tasks such as enhancement, de-reverberation, and music source separation, because the loss never touches the waveform domain.","A testable consequence is that the method's advantage should grow with dataset size: since embeddings can be precomputed, the memory and compute savings compound as training corpora scale, unlike waveform-domain losses.","The objective-versus-perceptual divergence reported here suggests that SI-SDR leaderboards may systematically undervalue codec-based separators; evaluating those systems with DNSMOS and STOI or a differentiable perceptual proxy would make comparisons fairer."],"forward_implications":["Training a codec-based separator with embedding loss removes the decoder from the training graph, cutting per-epoch time from 1.0 to 0.4 hours and GMACs from 1.5 to 0.8 on 2-second 8 kHz audio in the paper's setup.","The same embedding-loss recipe works with three different codec architectures (DAC, EnCodec, SoundStream), so the approach is not tied to a single codec.","Optimizing for perceptual metrics can be a better guide than SI-SDR when the output is resynthesized by a GAN-trained codec decoder; the paper's embedding-loss models often win on DNSMOS and STOI despite lower SI-SDR.","Ground-truth embeddings can be precomputed before training, meaning the training data can be stored in compressed form and the per-step memory cost stays low.","Embedding size interacts with separator width: the 1024-dimensional original DAC embedding performs worse with embedding loss, suggesting dimension alignment is a design parameter for future work."],"supporting_citations":[{"why":"Defines permutation invariant training, the mechanism that handles speaker-order ambiguity inside the embedding loss.","marker":"[9]"},{"why":"SoundStream codec, one of the three pre-trained codecs used to validate the method.","marker":"[11]"},{"why":"EnCodec codec, one of the three pre-trained codecs used to validate the method.","marker":"[12]"},{"why":"DAC codec, the third architecture and the one with the original larger-embedding variant.","marker":"[13]"},{"why":"The Codecformer separator architecture and waveform-loss baseline that the paper modifies.","marker":"[21]"},{"why":"STOI intelligibility metric used to report the perceptual gains.","marker":"[23]"},{"why":"DNSMOS non-intrusive quality metric used to report the perceptual gains.","marker":"[24]"},{"why":"Supplies the pre-trained codec weights and the AMUSE/LibriTTS training-data comparison used in the experiments.","marker":"[26]"}],"fun_headline_variants":["Embedding loss trains codec separation 2.5x faster","Skip waveform loss: train on codec embeddings for separation","No decoding needed: codec embedding loss matches waveform loss","Speech separation without waveform loss: 2.5x training speedup","Codec embedding loss beats waveform loss on DNSMOS and STOI"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a frozen codec encoder's embedding space behaves enough like a spectrogram that pulling a mixture's embeddings toward a clean speaker's embeddings with mean squared error produces embeddings the codec decoder turns into perceptually good speech.","fun_headline_variants_meta":{"raw":{"variants":["Embedding loss trains codec separation 2.5x faster","Skip waveform loss: train on codec embeddings for separation","No decoding needed: codec embedding loss matches waveform loss","Speech separation without waveform loss: 2.5x training speedup","Codec embedding loss beats waveform loss on DNSMOS and STOI"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000573,"raw_usage":{"total_tokens":2667,"prompt_tokens":867,"completion_tokens":1800,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":483,"completion_tokens_details":{"reasoning_tokens":1712}},"tokens_in":483,"tokens_out":1800,"duration_ms":14156,"temperature":1.0,"reasoning_tokens":1712,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:37:08.987715+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train Codecformer-EL with the frozen codec encoder replaced by a randomly initialized, untrained encoder (same architecture, weights fixed). If the embedding-loss models still show the reported DNSMOS and STOI gains over waveform loss, the improvements do not depend on the codec's learned embedding geometry, which would contradict the paper's premise.","supporting_citations":[{"cited_title":"Permutation invariant training of deep models for speaker-independent multi-talker speech separation,","cited_arxiv_id":null,"evidence_quote":"Defines permutation invariant training, the mechanism that handles speaker-order ambiguity inside the embedding loss."},{"cited_title":"Towards audio codec-based speech separation,","cited_arxiv_id":null,"evidence_quote":"The Codecformer separator architecture and waveform-loss baseline that the paper modifies."},{"cited_title":"An algorithm for intelligibility prediction of time–frequency weighted noisy speech,","cited_arxiv_id":null,"evidence_quote":"STOI intelligibility metric used to report the perceptual gains."},{"cited_title":"DNSMOS: A non-intrusive perceptual objective speech quality metric to evaluate noise suppressors,","cited_arxiv_id":null,"evidence_quote":"DNSMOS non-intrusive quality metric used to report the perceptual gains."},{"cited_title":"Espnet-codec: Comprehensive training and evaluation of neural codecs for audio, music, and speech,","cited_arxiv_id":null,"evidence_quote":"Supplies the pre-trained codec weights and the AMUSE/LibriTTS training-data comparison used in the experiments."}],"review_version":1}