{"id":"cc873448-8db0-4142-b4b2-99bc716eea4e","arxiv_id":"2505.10202","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"VQ-Logits replaces a language model's large output matrix with a small codebook and a fixed word-to-code mapping, reporting up to 99% output-layer parameter reduction and 6x logit speedup at a 4% perplexity cost.","lead":"This paper proposes a way to shrink the final word-scoring layer of a language model by grouping similar vocabulary words and letting them share one score. The authors claim large savings in memory and speed with only a small perplexity increase, but the evidence is thin and no code is released.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim is plausible, but the comparison against low-rank factorization is arguably unfair and the O(B·S·V) scatter cost is excluded from the speedup claim; the paper is conditionally acceptable pending a fairer baseline and end-to-end profiling.","rationale":"The reader's verdict of CONDITIONAL is appropriate: the method is simple, the core mechanism is coherent, and the reported perplexity deltas are not implausible given softmax-over-clusters. But the reader's weakest_assumption focused on cluster coherence, which is a real modeling cost and is partially self-acknowledged (Section 3.3, examples in Section 5.5, Limitations). My stress-test pass identifies a different, arguably more load-bearing gap: the empirical claims of speedup and superiority over low-rank factorization rest on measurements and baselines that are not fairly stated. The low-rank baseline is parameter-inflated, and the speedup figure excludes the scatter operation whose cost the paper itself characterizes as O(B·S·V) in a naive implementation. Neither issue is fatal — the paper explicitly discusses mapping cost and acknowledges the naive scatter cost — but both are checkable and central to the headline claims. I therefore keep CONDITIONAL rather than moving to ACCEPT or REJECT. I partially agree with the reader because the reader's weakest_assumption was the cluster coherence; I would add that the comparison fairness and measurement boundary are equally weak, so the corrective tests should include a parameter-matched low-rank baseline and end-to-end profiling.","tokens_in":11617,"tokens_out":2074,"duration_ms":17489,"concrete_test":"Run an end-to-end output-layer benchmark on WikiText-103 (d=768, V≈267k) that (1) measures wall-clock time for the full path h → codebook logits → scatter → softmax, including the gather, and compares to full softmax; (2) trains a low-rank baseline with total output parameters matched to VQ-Logits K=1024 (i.e., rank r satisfying r·(d+V) ≈ d·K, so r≈2 or 3), and reports its PPL; and (3) reports the same PPL numbers with variance across at least 3 seeds. If the end-to-end speedup drops below 2x or the parameter-matched low-rank baseline matches or exceeds VQ-Logits in PPL at similar speed, the central claims of 6x speedup and superiority over low-rank factorization would need to be substantially revised.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central claim is that replacing the V×d output projection with a K×d codebook plus a fixed scatter mapping yields up to 99% parameter reduction and 6x faster logit computation with only ~4% perplexity degradation. The argument's load-bearing condition is that the speedup and parameter reduction are not bought by an unfair comparison or hidden costs. Two concerns stand out. First, the low-rank baseline is configured with drank=512 to make dmodel×drank 'somewhat comparable' to dmodel×K, but this ignores the drank×V term: at V=267k, drank×V = 136.7M parameters, which is why the baseline has 137.1M output params. A fair low-rank comparison with comparable total output parameters (e.g., rank r with r(d_model+V) ≈ d_model·K) would yield a dramatically smaller rank (r ≈ 2-3), and it is unknown whether such a low-rank baseline performs better or worse than VQ-Logits. Second, the 6x logit speedup counts only hC^T (O(B·S·d·K) FLOPs) and excludes the scatter/gather operation, which the paper itself notes involves O(B·S·V) memory accesses in a naive implementation. If the scatter is not nearly free on real hardware, the end-to-end output-layer speedup will be far below 6x, and the headline claim weakens. The paper acknowledges this in Appendix D but does not quantify it. A direct end-to-end measurement of the output layer (including scatter) against a properly parameter-matched low-rank baseline is the decisive test.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"VQ-Logits proposes to replace the standard V×d_model output projection of an LLM with a K×d_model codebook C and a fixed vocabulary-to-codebook mapping M. The model computes logits over the K codebook vectors and scatters them to the full vocabulary before softmax. With K ≪ V, the output-layer parameters drop from d_model·V to d_model·K plus V integers for M, and the matrix multiplication cost drops from O(B·S·d_model·V) to O(B·S·d_model·K). On WikiText-103, the authors report perplexity 19.2 for K=1024 versus 18.0 for a full softmax (≈4% increase), a 99.2% reduction in output-layer parameters, and a 6.5× speedup in logit computation; similar trends are reported on C4 and PTB. They also study codebook initialization via k-means on pretrained embeddings, fixed versus learnable codebooks, mapping strategies, and fine-tuning scope.","tokens_in":11893,"tokens_out":4054,"duration_ms":40528,"significance":"If the central claim survives scrutiny, VQ-Logits is a simple, architecture-agnostic output-layer compression scheme that is complementary to weight quantization and other model-compression methods. The use of k-means on pretrained output embeddings to define a tie-breaking-free quantization is natural, and the paper provides a clean complexity analysis (Appendix D) plus ablations isolating the contributions of mapping initialization, codebook learning, and tying. The main weakness is empirical verification: no code or data are released, no error bars are given, one figure is explicitly labeled illustrative, and the low-rank baseline is not parameter-matched. These issues are fixable and do not undermine the mathematical correctness of the method itself, but they must be addressed before the headline claims (99% compression, 6× speedup, better-than-low-rank) can be accepted.","major_comments":[{"comment":"The low-rank factorization baseline is not parameter-matched, and the stated rationale is incorrect. The paper says drank=512 is chosen to make d_model·drank 'somewhat comparable' to d_model·K, but the actual parameter count of a factorized output W_out = W1·W2 is d_model·drank + drank·V, not d_model·drank. At V≈267k, the drank·V term contributes 136.7M parameters, giving 137.1M output parameters, which is ~175× larger than VQ-Logits's 0.78M. A fair comparison against a low-rank baseline with comparable total output parameters would require rank r ≈ d_model·K/(d_model+V), i.e., r≈2–3 for K=1024. The paper should either run that matched-rank baseline or explicitly justify why the current comparison remains informative.","section":"§4.3, Table 1"},{"comment":"The 6× speedup claim must be pinned to a precise definition of 'logit computation.' Section 5.4 states that the timings 'include hC^T and the scatter operation,' but Table 1's 'Speed (Logits)' column does not say this, and Appendix D notes that a naive gather implementation of the scatter involves O(B·S·V) memory accesses. If the Table 1 speedup excludes scatter, the headline number overstates the output-layer speedup; if it includes scatter, state that explicitly in the caption and main text. Either way, report the scatter cost separately and, if possible, provide a wall-clock breakdown of the full output layer (hC^T, scatter, softmax) on the actual hardware.","section":"Tables 1 and 3, §5.4, Appendix D"},{"comment":"Figures 2a and 2b are labeled 'illustrative based on trends from Table 1,' which is inappropriate for a results section in a research paper. Replace them with actual measured values from the runs, including error bars if multiple seeds are available. The absence of any repeated-seed runs or variance information in Tables 1, 2, 9, and 10 makes it unclear whether the 0.3–0.9 perplexity differences between variants are meaningful. Please add at least a few repeated runs for the main configurations or an explicit statement that all results are single-run.","section":"Fig. 2, §5.2"},{"comment":"The uniform-within-cluster penalty is never analyzed. Because Eq. (3) assigns exactly the same logit to every vocabulary token in a code, the softmax over scattered logits gives all words in a cluster identical probability. The quality of the method therefore depends on the purity of the k-means clusters, and the paper's own examples (Monday/Tuesday/Wednesday, January/February, etc.) show that merging destroys distinctions that matter for prediction. A token-weighted cluster-entropy bound, or at least a quantitative measure of how often the target token's code contains other tokens with high frequency, would make the trade-off principled. Without this, the method's performance is empirically rather than theoretically justified.","section":"§3.2, Eq. (3), §3.3"}],"minor_comments":[{"comment":"The description of Option B for random initialization is inconsistent: §3.3 says random initialization of M uses 'dividing vocabulary into K contiguous blocks' when no prior embeddings are available, while §4.4 says M is initialized by 'k-means on input embeddings.' The relationship between these two cases should be clarified.","section":"§3.3, §4.4"},{"comment":"There is a placeholder '(??)' in the sentence about quantization of weights and activations; please insert the intended citations.","section":"§2.3"},{"comment":"The C4 paragraph says a SentencePiece unigram model was trained with a target vocabulary of 32,000 BPE tokens; unigram and BPE are different subword algorithms, so the wording should be corrected.","section":"Appendix A"},{"comment":"The row 'Random init, Learned C, FixedM' does not specify how M is initialized in that case; the text in §5.3 says 'k-means mapping from input embeddings,' but the table caption should state this to avoid ambiguity.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid methods contribution with a plausible central idea, but the empirical support is currently too thin for the strength of the claims. The low-rank comparison issue is the most consequential; fixing it could change the paper's main conclusion about superiority over factorization. The authors should also release code and checkpoints or at least provide a detailed reproducibility appendix. The paper is within scope for the journal, but I would not accept it in its current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"VQ-Logits is a clean, well-tested restatement of clustered softmax: cluster the output embeddings, predict logits over K centroids, scatter to V. The engineering value is real, but the novelty is not. The authors don't cite the closest prior art (clustered softmax, shared embedding layers, product quantization on outputs), so the claim of a 'novel approach' overstates the case. The numbers are plausible: ~4% PPL degradation at K=2048 on WikiText-103 with 99% output-layer parameter reduction. That's a useful datapoint.\n\nThe method itself is simple and correct. The ablations over codebook size, initialization, and mapping strategy are careful and informative. K-means initialization from pretrained embeddings clearly helps. The qualitative examples of day/month/unit clusters show where the method succeeds and where it breaks. The paper's stated limitations are honest; they don't hide the information loss.\n\nSoft spots: the low-rank baseline is not parameter-matched — fixed rank 512 leaves a big V×rank term unaccounted for. That's an unfair comparison and needs fixing. The speedup story is muddier than the headline suggests. Table 3 claims the measured logit time includes the scatter (15.4 ms vs 100 ms), which actually supports the 6.5x figure, but the FLOP accounting in the text only counts hC^T. And no error bars, no released code or data, and the figures are explicitly illustrative. There's also a '??' placeholder in the references and an unfinished citation for weight quantization. None of this is fatal, but it's the kind of sloppiness that needs referee attention.\n\nWho this is for: people working on output-layer compression or serving cost-sensitive LMs. The paper is a solid engineering study, not a conceptual breakthrough. I'd be happy to referee it; I wouldn't cite it as a primary source. It needs fairer baselines and end-to-end profiling before acceptance.","headline":"A solid engineering study that restates clustered softmax with better ablations, but needs fair baselines and honest speedup accounting before it's publishable.","tokens_in":12460,"tokens_out":2536,"would_cite":false,"duration_ms":24299,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The final vocabulary-sized projection of an LLM can be replaced by a K-vector codebook and a fixed word-to-code mapping, cutting output-layer parameters by up to 99% and logit computation by up to 6x with only ~4% perplexity loss.","keywords":["vector quantization","output layer compression","language modeling","softmax approximation","codebook","perplexity","LLM inference"],"falsifier":"Evaluate VQ-Logits on a held-out set of rare but information-dense words, such as proper nouns or technical terms that k-means forces to share codes with very frequent words, and compare their per-token perplexity against the full softmax baseline; if those words degrade by far more than the average 4% gap, the cluster-coherence assumption behind the method fails.","tokens_in":1769,"feed_emoji":"⚡","tokens_out":1852,"duration_ms":64284,"temperature":0.7,"pith_summary":"This paper claims that the large output projection of a language model, which maps hidden states to a full vocabulary of logits, can be replaced by a small codebook of K prototype vectors plus a fixed assignment from each word to one prototype. The model computes logits only over K codes and then scatters them to the full vocabulary before softmax, shrinking output-layer parameters by up to 99% and speeding logit computation by up to 6x on WikiText-103 while raising perplexity by only about 4%. A sympathetic reader would care because this offers a simple architectural swap that eases the dominant memory and compute bottleneck of large-vocabulary inference without requiring hierarchical softmax trees or frequency cutoffs.","feed_headline":"Vector-quantized logits shrink LLM output layers by 99%","feed_subtitle":"A K-code shared vocabulary cuts logit computation 6x while raising WikiText-103 perplexity by about 4 percent.","key_machinery":"The central object is the pair of the codebook $C$ and the fixed vocabulary-to-codebook mapping $M$. The codebook is a small learnable matrix of $K$ representative embedding vectors, and $M$ is typically obtained by k-means clustering of a pre-trained model's output embeddings, then held fixed during fine-tuning. The scatter operation copies each code logit to every vocabulary position assigned to that code, so the expensive $d_{\\text{model}} \\times V$ multiplication is replaced by a $d_{\\text{model}} \\times K$ multiplication followed by memory-bound indexing; the mapping's cluster coherence is what determines how much information is lost by giving distinct words the same probability.","core_discovery":"VQ-Logits establishes that the output embedding matrix $W_{\\text{out}} \\in \\mathbb{R}^{d_{\\text{model}} \\times V}$ can be replaced by a codebook $C \\in \\mathbb{R}^{K \\times d_{\\text{model}}}$ and a map $M: \\{0,\\dots,V-1\\} \\to \\{0,\\dots,K-1\\}$, so that each vocabulary token's logit is exactly the logit of its assigned code vector: $L_v[b,s,i] = L_c[b,s,M(i)] = h[b,s] \\cdot c_{M(i)}$. Because softmax is applied after scattering, all words assigned to the same code receive identical logits and therefore identical probabilities. The paper's experiments show that this forced sharing is not too costly: with $K=2048$ on WikiText-103, perplexity rises from 18.0 to 18.7 while output-layer parameters drop from 205.2M to 1.57M and logit computation becomes 5x faster, with similar trends on C4 and Penn Treebank.","pith_inferences":["The paper leaves implicit that the perplexity loss should be predictable from the token-weighted entropy of the code clusters: if words in a cluster are not approximately exchangeable in context, the fixed map will underperform; this suggests a quantitative bound could guide the choice of $K$ before training.","A natural extension is a hierarchical codebook that keeps frequent words on their own codes while letting rare related words share prototypes, which would preserve the speedup for the bulk of the vocabulary without forcing all clusters to be equally coarse.","Because the scatter operation gives every word in a code the same probability, the method is effectively a hard version of word-class modeling; one could test whether a soft, probabilistic assignment between words and codes recovers the lost rare-word accuracy at modest extra cost."],"forward_implications":["Output-layer storage and FLOPs scale with $K$ instead of $V$, so larger vocabularies become feasible on a fixed GPU budget.","The perplexity-versus-compression trade-off is controlled by a single integer $K$, with $K \\in [1024, 4096]$ described as a useful operating range for a 267k-word vocabulary.","VQ-Logits is compatible with other compression methods because it targets only the output projection, leaving the rest of the network unchanged.","Tying input embeddings to the codebook cuts parameters further but raises perplexity by about 1.2 points, so retaining separate full-vocabulary input embeddings is preferred.","The fixed mapping can be initialized by k-means on pre-trained output embeddings, random initialization, or frequency binning, with k-means giving the best perplexity."],"supporting_citations":[{"why":"Supplies the vector-quantization codebook concept and the commitment-loss idea that the paper adapts to output logits.","marker":"(van den Oord et al., 2017)"},{"why":"Provides the adaptive softmax baseline that VQ-Logits compares against and aims to outperform in compression.","marker":"(Grave et al., 2017)"},{"why":"Defines hierarchical softmax, an earlier output-layer compression approach that VQ-Logits positions itself against.","marker":"(Morin and Bengio, 2005)"},{"why":"Establishes tied input-output embeddings, which the paper tests as an alternative parameter-sharing scheme.","marker":"(Press and Wolf, 2016)"},{"why":"Provides the WikiText-103 dataset and standard word-level split used for the main experiments.","marker":"(Merity et al., 2017)"},{"why":"Defines the GPT-2-style Transformer decoder architecture that the experimental models are based on.","marker":"(Radford et al., 2019)"},{"why":"Supplies the C4 corpus from which the 10B-token subset for additional language-modeling experiments is drawn.","marker":"(Raffel et al., 2020)"},{"why":"Offers the efficient k-means implementation used to initialize the codebook and mapping from pre-trained embeddings.","marker":"(Johnson et al., 2021)"},{"why":"Provides the SentencePiece unigram tokenizer used to build the 32k vocabulary for the C4 experiments.","marker":"(Kudo and Richardson, 2018)"}],"fun_headline_variants":["VQ-Logits: 99% smaller LLM output layer, 6x faster","Vector-quantized logits cut LLM vocab layer 99% and 6x compute","VQ-Logits: vector-quantized output layer, 99% fewer params","Shrink LLM vocab layer 99% with vector-quantized logits","VQ-Logits: 99% param cut in LLM output layer, 6x speedup"],"cache_read_input_tokens":14464,"weakest_assumption_plain":"The load-bearing premise is that a fixed k-means mapping from the full vocabulary to $K$ code vectors creates clusters coherent enough that giving every word in a cluster the same logit, and hence the same probability, costs only a few percent in perplexity; if the clustering is poor, the reported trade-off collapses.","fun_headline_variants_meta":{"raw":{"variants":["VQ-Logits: 99% smaller LLM output layer, 6x faster","Vector-quantized logits cut LLM vocab layer 99% and 6x compute","VQ-Logits: vector-quantized output layer, 99% fewer params","Shrink LLM vocab layer 99% with vector-quantized logits","VQ-Logits: 99% param cut in LLM output layer, 6x speedup"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000718,"raw_usage":{"total_tokens":3278,"prompt_tokens":1051,"completion_tokens":2227,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":667,"completion_tokens_details":{"reasoning_tokens":2107}},"tokens_in":667,"tokens_out":2227,"duration_ms":14753,"temperature":1.0,"reasoning_tokens":2107,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:15:27.698449+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Evaluate VQ-Logits on a held-out set of rare but information-dense words, such as proper nouns or technical terms that k-means forces to share codes with very frequent words, and compare their per-token perplexity against the full softmax baseline; if those words degrade by far more than the average 4% gap, the cluster-coherence assumption behind the method fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines hierarchical softmax, an earlier output-layer compression approach that VQ-Logits positions itself against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the WikiText-103 dataset and standard word-level split used for the main experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the SentencePiece unigram tokenizer used to build the 32k vocabulary for the C4 experiments."}],"review_version":1}