{"id":"96f02692-42da-4217-830d-8537ca70c3f1","arxiv_id":"2506.03515","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Quantization-aware training with ternary weights plus base-3 weight indexing reduces a JETS/HiFi-GAN TTS model from 25.66 MB to 4.39 MB while keeping naturalness MOS around 3.1 to 3.3.","lead":"BitTTS shrinks a text-to-speech model by up to 83 percent by quantizing weights to ternary values and packing five weights into each 8-bit integer. The paper reports that this compact model sounds better than a smaller non-quantized comparison model on a listening test.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 83% size reduction is arithmetically sound, but the abstract's 'similar model size' quality comparison is untested: the only unquantized small baseline is 12.78 MB versus the 4.39 MB proposed model, so a same-size 32-bit control is missing.","rationale":"The reader's weakest assumption is exactly the most load-bearing concern. I considered other candidates -- the vocoder output layer exclusion, the small listening test, and the lack of released code -- but none is as directly tied to the paper's headline claim. The excluded output layer is explicitly disclosed and the compression arithmetic remains valid for the stated storage format. The small listening test affects confidence intervals but the relevant MOS gaps are large; the proposed 4.39 MB model scores ~3.09 versus 1.20 for the 12.78 MB small model, so the qualitative direction is unlikely to flip on sample size alone. The missing same-size 32-bit baseline, however, is not a statistical or presentational detail: the abstract makes a claim about a comparison that Table 1 does not contain. A 4.39 MB unquantized model is plausible to build by reducing channels further, and until that control is measured, the sentence 'outperforming the baseline of similar model size without quantization' is an extrapolation. Since this is an evidence gap rather than a demonstrated technical error, and since the 4-bit comparisons still provide partial support for the quality-per-bit story, the existing CONDITIONAL verdict remains appropriate. No verdict change is needed; the paper should be asked to supply the missing baseline before the claim is accepted as stated.","tokens_in":8658,"tokens_out":11497,"duration_ms":111991,"concrete_test":"Train the same JETS-based acoustic model and HiFi-GAN vocoder from Section 3.1 as a 32-bit float32 model with encoder/decoder channels and HiFi-GAN initial channel set to approximately 19, tuned so the saved checkpoint is 4.39 MB within 10%; then run the identical 30-utterance, 15-rater MOS protocol from Section 3.3.1. If this same-size baseline's MOS is not significantly below BitTTS's ~3.09, the abstract's 'similar model size' outperformance claim fails; if it is below, the concern is resolved. Report the exact channel widths and checkpoint size alongside the MOS.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quality claim in the abstract -- 'outperforming the baseline of similar model size without quantization' -- is not supported by the experiment as designed. Section 3.1 defines the unquantized small baseline by setting encoder/decoder channels to 32 and the HiFi-GAN initial channel to 32. Table 1 reports this model at 12.78 MB, while the proposed 1.58-bit model with weight indexing on both acoustic model and vocoder is 4.39 MB. These differ by a factor of 2.9, so the table never compares BitTTS to a non-quantized model of its own size. A non-quantized model at 4.39 MB would require roughly sqrt(12.78/4.39) ~ 1.7x smaller linear channel widths (about 19 instead of 32), and its MOS is unknown. The large gap between the 32-bit small model MOS (1.20) and BitTTS MOS (~3.09) is therefore not evidence for the specific 'similar model size' claim; it only shows that quantization beats a much larger, aggressively narrowed architecture. The size-reduction arithmetic itself is sound, and the comparison to 4-bit quantized models is informative, but the headline advantage over a same-size 32-bit model remains unmeasured. The disclosed exclusion of the vocoder output layer from quantization is a separate, secondary issue and does not rescue the missing baseline.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"BitTTS proposes a highly compact text-to-speech model by applying quantization-aware training (QAT) to as low as 1.58-bit ternary weights and storing five ternary weights as a single int8 index (\"weight indexing\"). The method is applied to a JETS-based acoustic model and HiFi-GAN vocoder, with the vocoder's final convolutional layer excluded from quantization. The authors report a model size reduction from 25.66 MB to 4.39 MB (83%) and a MOS of about 3.09 for the fully compressed model, claiming that this outperforms a non-quantized baseline of similar model size. The paper also compares 4-bit and 1.58-bit quantized variants, measures RTF on an Apple M1 Pro CPU, and analyzes the distribution of weight indices.","tokens_in":8885,"tokens_out":5019,"duration_ms":52844,"significance":"The size arithmetic is exact and the weight-indexing idea is an elegant, lossless mapping that is practical on hardware with byte-aligned storage. The paper provides a clear algorithmic description of QAT and weight indexing, and the comparison of 4-bit versus 1.58-bit quantization is informative for the TTS community. However, the headline claim of outperforming a same-size non-quantized baseline is not supported by the current experiment, as the only non-quantized small baseline is 12.78 MB, nearly three times larger than the proposed 4.39 MB model. This is a load-bearing gap that affects the abstract and the introduction. The technical contribution is otherwise sound and potentially useful for on-device TTS, but the experimental evidence needs to be aligned with the claims.","major_comments":[{"comment":"The \"baseline of similar model size without quantization\" is not similar in size: the 32-bit small model occupies 12.78 MB, while the proposed 1.58-bit model with weight indexing on both acoustic model and vocoder is 4.39 MB, a factor of 2.9 difference. Consequently, the abstract's claim that BitTTS outperforms a baseline of similar model size without quantization is not demonstrated by Table 1. A non-quantized 32-bit model at approximately 4.39 MB (e.g., with channel widths near 19 rather than 32) is missing; the gap between MOS 1.20 and 3.09 only shows that quantization can beat a much larger, aggressively narrowed architecture. Please either add this missing control or revise the claim to specify the exact sizes of the models being compared.","section":"Section 3.1, Table 1, Abstract"},{"comment":"The vocoder's convolutional layer closest to the waveform output is excluded from quantization, and the paper says this is because preliminary experiments showed significant quality degradation. The claims \"quantizing both the acoustic model and the vocoder\" and \"83% reduction\" should be qualified by reporting the size and parameter fraction of the excluded layer, and ideally by also reporting the MOS of a fully quantized vocoder (even if poor) to quantify the severity of the degradation. The exclusion is disclosed, but the current reporting makes the scope of the headline result ambiguous.","section":"Section 3.2"},{"comment":"The MOS comparisons use 15 raters on 30 utterances, and the reported 95% confidence intervals for the 4-bit and 1.58-bit configurations overlap (e.g., 2.96±0.11 vs 3.09±0.11 for the both-quantized models). The statement that 1.58-bit models \"achieve comparable or even higher MOS\" is accordingly not statistically established. Please add a significance test (e.g., paired bootstrap or Wilcoxon) or explicitly state that no significant difference was found; increasing the evaluation size would also strengthen the result.","section":"Section 3.3.1"}],"minor_comments":[{"comment":"The handling of the final block when L is not a multiple of L* is unspecified. The loop over the set in line 6 appears to iterate in descending order, and the reconstruction rules are not given for padded or unpadded blocks. Please clarify how incomplete blocks are encoded and decoded.","section":"Algorithm 1"},{"comment":"Equation (1) defines q=2^(b-1) and clipping to [-(q-1), q], but Section 3.2 for 4-bit sets the clipping range to [-8, 7]; while this is consistent with q=8, the general notation could be made clearer to avoid confusion with the 1.58-bit ternary case.","section":"Section 2.2, Eq. (1)"},{"comment":"The Huffman-coding result (4.35 MB) is presented only in a footnote; if this is part of the proposed compression pipeline, it should be integrated into the main evaluation, or the footnote should be removed to avoid distracting from the main results.","section":"Footnote 8"},{"comment":"The index frequency plot (Figure 2) is informative, but the skew toward indices 0, 121, and 242 is unsurprising given the ternary weight distribution. A quantitative measure of entropy or compression gain would make the analysis of weight pattern bias more rigorous.","section":"Section 3.3.3"}],"recommendation":"major_revision","confidential_remarks":"The main technical idea—weight indexing for ternary QAT weights—is simple and likely correct, and the size arithmetic is exact. The primary issue is that the abstract overclaims based on a missing control condition; this should be fixable by adding a same-size 32-bit baseline or by modifying the claim. The vocoder output-layer exclusion should also be clarified with parameter counts. The paper is otherwise within scope for a TTS workshop or conference, but the claims need to be brought in line with the experimental design."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the size reduction is real and the vocoder-bottleneck result is worth reporting, but the abstract goes beyond what Table 1 shows. The only unquantized small baseline is 12.78 MB, and the proposed 1.58-bit plus weight indexing model is 4.39 MB. So \"outperforming the baseline of similar model size without quantization\" is not demonstrated; the comparison is against a model that is roughly three times larger. That is the main thing to know.\n\nWhat is genuinely new: applying 1.58-bit QAT (the BitNet b1.58 recipe) to a JETS + HiFi-GAN stack, and the weight-indexing scheme, which packs five ternary weights into one int8 index. The packing is lossless and the arithmetic checks out: 3^5=243 patterns fit in 256 int8 values, so 5*1.58=7.9 bits become 8 bits. The 83% reduction from 25.66 to 4.39 MB is exact. The finding that quantizing the vocoder hurts MOS far more than quantizing the acoustic model is a legitimate empirical observation, and the demo page lets you hear the results. The 4-bit comparisons are informative.\n\nSoft spots, in order. First, the missing same-size baseline is load-bearing for the abstract's main claim. A 32-bit model at 4.39 MB would need roughly sqrt(12.78/4.39) ~ 1.7x narrower channels (about 19 instead of 32), and its MOS is unknown. The large gap between the 12.78 MB small model (MOS 1.20) and BitTTS (MOS 3.09) only shows that quantization beats a much larger, aggressively narrowed architecture. Second, the RTF story is not a win: quantized models are slower than the small 32-bit model, and weight indexing forces weight reconstruction at load time, so the on-device speed benefit is unproven. Third, the listening test is small (15 raters, 30 utterances) and the vocoder's output layer was excluded after preliminary experiments; this is disclosed, but it is still a post hoc choice that needs a robustness check. No code or data are released, which limits reproducibility.\n\nNone of this kills the paper. The size arithmetic is correct, the method is clearly described, and the vocoder-bottleneck insight is useful. It just needs a same-size unquantized baseline, ideally with a few width/pruning variants, plus a tempered abstract.\n\nMy recommendation: send it to a serious referee, but make clear that the missing baseline is the main thing to fix. It is a solid within-subfield compression result, not a breakthrough.","headline":"BitTTS has a real, arithmetically sound compression result and a useful vocoder-bottleneck finding, but the abstract's 'similar model size' quality claim is unsupported because the only unquantized small baseline is 2.9x larger.","tokens_in":9511,"tokens_out":2717,"would_cite":true,"duration_ms":27223,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Quantizing a TTS model's acoustic model and vocoder to ternary values and storing five weights per byte shrinks it from 25.66 MB to 4.39 MB while naturalness stays comparable to 4-bit quantization.","keywords":["text-to-speech","quantization-aware training","1.58-bit quantization","ternary weights","weight indexing","model compression","on-device TTS","vocoder quantization"],"falsifier":"A replication with a listening test that pits the 4.39 MB 1.58-bit model against a full-precision 32-bit model of genuinely the same 4.39 MB size would settle whether quantization helps beyond channel narrowing; if the full-precision model ties or wins, the compression advantage would not survive an equal-size comparison. A second check is whether the 1.58-bit vocoder consistently beats the 4-bit vocoder in MOS, since the paper's reported gap depends on a single training run.","tokens_in":8357,"feed_emoji":"🗣️","tokens_out":8215,"duration_ms":84086,"temperature":0.7,"pith_summary":"BitTTS tries to establish that extreme low-precision quantization is a workable way to compress full text-to-speech systems for on-device use. The paper trains a lightweight acoustic model and a HiFi-GAN vocoder with weights restricted to the ternary set {-1, 0, 1}, then stores five consecutive ternary weights as a single 8-bit index. It reports that this reduces the model from 25.66 MB to 4.39 MB, an 83 percent cut, while synthesized speech remains comparable in naturalness to 4-bit quantized models and clearly better than a 32-bit model made small by reducing channel widths. A sympathetic reading of the experiments is that quantization, rather than architecture shrinking, is the more promising route to very small TTS models.","feed_headline":"1.58-bit quantization cuts TTS model size by 83 percent","feed_subtitle":"Packing five ternary weights into one byte keeps speech naturalness near 4-bit models.","key_machinery":"The load-bearing mechanism is weight indexing, a storage scheme for ternary weights. After quantization-aware training rounds scaled weights to {-1,0,1}, the flattened weight vector is split into blocks of length 5; because there are $3^{5}$ = 243 possible five-value patterns, each block maps to an integer between 0 and 242 that fits in one byte. Retrieval rebuilds the weight tensor by looking up the pattern for each index. This recovers near-ideal 1.58-bit-per-weight storage on hardware that natively handles 8-bit values, and works alongside the absmax scaling and straight-through gradient estimator used during training.","core_discovery":"The central claim is that 1.58-bit quantization-aware training transfers from language models to TTS: a model with almost all convolutional weights in {-1,0,1} can still generate natural speech. The paper reports a naturalness MOS of 3.09 for the fully quantized 4.39 MB model with both the acoustic model and vocoder quantized, 3.30 for the 4-bit model with only the acoustic model quantized, and 1.20 for the naive 32-bit small model, with ground truth at 4.24. It also finds that quantizing the acoustic model alone keeps a 70 percent size reduction with better quality, and that the final convolutional layer before the waveform output should be left unquantized. The authors present this as evidence that quantization is effective for building small TTS models without significant quality loss.","pith_inferences":["Beyond the paper, the non-uniform index distribution (peaks at indices 0, 121, and 242) suggests that variable-length or learned coding over five-weight blocks could shave more bytes; the paper's own Huffman coding only reduces 4.39 MB to 4.35 MB.","A cleaner test of the 'quantization beats shrinking' claim would compare the 4.39 MB ternary model with a 32-bit model compressed to the same 4.39 MB by channel reduction; the paper's 12.78 MB control does not settle that comparison.","Because only 1D convolutional layers are quantized, extending QAT to embedding and attention-like layers in the TTS stack is the natural next experiment; if those also tolerate ternary weights, even smaller footprints are plausible.","The RTF results show compression and speed do not align: quantized models are slightly slower than the small 32-bit model, so latency-critical deployments would need orthogonal acceleration rather than relying on this compression alone."],"forward_implications":["A full TTS stack can fit in less than 5 MB, making on-device deployment in car navigation and conversational agents plausible for storage-constrained settings.","If quality is the priority, quantizing only the acoustic model is the recommended operating point: roughly 70 percent size reduction with a MOS of 3.18 to 3.30.","The vocoder is the quality-critical component: quantizing it costs more naturalness than quantizing the acoustic model, and its final convolutional layer should remain full precision.","Weight indexing gets storage close to the 1.58-bit ideal (64.0 kB versus 63.4 kB in the paper's example layer), so further compression must come from coding the index distribution or shrinking non-quantized parts."],"supporting_citations":[{"why":"Supplies the 1.58-bit quantization equations (absmax scaling, rounding, straight-through gradient) that the paper adapts.","marker":"[20]"},{"why":"Provides the fake quantization-aware training loop used to adapt the model to low-precision weights.","marker":"[16]"},{"why":"Defines the base TTS architecture (JETS) whose transformer blocks are replaced by four convolutional layers.","marker":"[31]"},{"why":"Provides the HiFi-GAN vocoder that is quantized and evaluated in the experiments.","marker":"[33]"},{"why":"Supplies the LibriTTS-R corpus used for training, validation, and test.","marker":"[29]"},{"why":"Yields phone-level durations used to train the duration model.","marker":"[30]"},{"why":"Adds a mixture density network layer that the paper uses to improve naturalness.","marker":"[34]"}],"fun_headline_variants":["BitTTS packs TTS into 1.58-bit weights, slashing size 83%","1.58-bit quantization shrinks TTS models 83% with natural speech","Weight indexing + 1.58-bit quantization: TTS size cut by 83%","1.58-bit TTS: 83% smaller model, quality beats baseline"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The quality comparison rests on treating the 32-bit channel-reduced model as the non-quantized baseline of similar size, but at 12.78 MB it is about three times larger than the 4.39 MB proposed model, leaving the 'similar model size' claim unverified.","fun_headline_variants_meta":{"raw":{"variants":["BitTTS packs TTS into 1.58-bit weights, slashing size 83%","1.58-bit quantization shrinks TTS models 83% with natural speech","Weight indexing + 1.58-bit quantization: TTS size cut by 83%","1.58-bit TTS: 83% smaller model, quality beats baseline"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000621,"raw_usage":{"total_tokens":2839,"prompt_tokens":869,"completion_tokens":1970,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":485,"completion_tokens_details":{"reasoning_tokens":1877}},"tokens_in":485,"tokens_out":1970,"duration_ms":16038,"temperature":1.0,"reasoning_tokens":1877,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:01:35.763299+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A replication with a listening test that pits the 4.39 MB 1.58-bit model against a full-precision 32-bit model of genuinely the same 4.39 MB size would settle whether quantization helps beyond channel narrowing; if the full-precision model ties or wins, the compression advantage would not survive an equal-size comparison. A second check is whether the 1.58-bit vocoder consistently beats the 4-bit vocoder in MOS, since the paper's reported gap depends on a single training run.","supporting_citations":[{"cited_title":"Learning trans- ferable architectures for scalable image recognition,","cited_arxiv_id":null,"evidence_quote":"Provides the fake quantization-aware training loop used to adapt the model to low-precision weights."},{"cited_title":"LibriTTS-R: A re- stored multi-speaker text-to-speech corpus,","cited_arxiv_id":null,"evidence_quote":"Provides the HiFi-GAN vocoder that is quantized and evaluated in the experiments."},{"cited_title":"2-bit conformer quantization for automatic speech recog- nition,","cited_arxiv_id":null,"evidence_quote":"Yields phone-level durations used to train the duration model."}],"review_version":1}