LFQ: Logit-aware Final-block Quantization for Boosting the Generation Quality of Low-Bit Quantized LLMs

Eunho Yang; June Yong Yang; Jung Hyun Lee; Jungwook Choi

arxiv: 2605.29756 · v1 · pith:XNLZRIZTnew · submitted 2026-05-28 · 💻 cs.AI

LFQ: Logit-aware Final-block Quantization for Boosting the Generation Quality of Low-Bit Quantized LLMs

Jung Hyun Lee , June Yong Yang , Jungwook Choi , Eunho Yang This is my paper

Pith reviewed 2026-06-29 07:36 UTC · model grok-4.3

classification 💻 cs.AI

keywords post-training quantizationlarge language modelslogit alignmentcross-entropy lossfinal block quantizationlow-bit LLMsgeneration qualitytoken probability

0 comments

The pith

Aligning token probabilities at the logit level in the final block boosts generation quality of low-bit quantized LLMs

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper claims that block-wise PTQ for LLMs degrades on generative tasks due to ignoring the unembedding layer and using MSE loss, which misaligns token probabilities. LFQ fixes this by quantizing only the final block to match FP logits via cross-entropy minimization. This approach boosts accuracy on complex generation tasks while keeping performance on par with full-precision models for basic language tasks. If true, it offers a simple way to enhance quantized LLM deployment for tasks requiring long or complex outputs.

Core claim

LFQ quantizes the final Transformer block by minimizing the cross-entropy between the logits of the full-precision model and its quantized counterpart. This aligns the token probability distributions, addressing the shortfall in standard block-wise PTQ caused by omitting the LM head and relying on MSE. As a result, the method improves accuracy on generation benchmarks across model families without sacrificing performance on language modeling and understanding tasks.

What carries the argument

Logit-aware Final-block Quantization (LFQ), a method that applies cross-entropy loss to the final block's output logits instead of MSE during quantization.

If this is right

Consistently improves accuracy of complex generation tasks over state-of-the-art block-wise PTQ.
Maintains parity with FP baselines on language modeling and understanding.
Works across diverse model families.
Addresses misalignment in token probability distribution for better long-response generation.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Extending LFQ to multiple final blocks could further close the gap to FP models on very long generations.
Similar logit-alignment techniques might benefit other compression methods like pruning or distillation.
The approach suggests that final-layer calibration is critical for preserving output distribution in quantized networks.

Load-bearing premise

The quality degradation in block-wise PTQ stems mainly from omitting the unembedding layer and using MSE, and that correcting only the final block is enough to fix misalignment throughout the model.

What would settle it

An experiment showing no improvement in generation task accuracy when applying LFQ compared to standard PTQ on the same models and bit-widths would falsify the claim.

Figures

Figures reproduced from arXiv: 2605.29756 by Eunho Yang, June Yong Yang, Jung Hyun Lee, Jungwook Choi.

**Figure 1.** Figure 1: Reasoning trajectory of L1-Qwen-7B-Max under greedy decoding on AIME 2024 Problem 28. We compare token-level probability distributions for the FP baseline, block-wise PTQ (“blockPTQ” in this figure), and LFQ (ours) at two instants: (a) the first step where block-wise PTQ’s top-1 token diverges from the FP baseline, and (b) the first “aha” moment guiding the reasoning onto the correct path. In (a), block-wi… view at source ↗

**Figure 2.** Figure 2: Performance of Llama 3.1 8B Instruct as the number of topmost Transformer blocks optimized with LFQ increases from 1 (ours) to 3, with the remaining blocks quantized via standard MSE reconstruction; shown for FlexRound (left), OmniQuant (center), and Block-AP (right). In each subfigure, the left y-axis shows MMLU accuracy, while the right y-axis reports the IFEval+GSM8K average. All results use 4-bit per-c… view at source ↗

read the original abstract

As large language models continue to scale, low-bit weight-only post-training quantization (PTQ) offers a practical solution to their memory-efficient deployment. Although block-wise PTQ is capable of matching the full-precision (FP) baseline on basic language modeling and understanding, its quality is degraded for generative tasks -- especially at longer responses and extended chains of thought, which is critical in boosting task accuracy. We attribute this shortfall to two factors: (i) the omission of the unembedding layer (the LM head) in block-wise optimization and (ii) the reliance on the mean squared error (MSE) objective. Both factors cause the token probability distribution of the quantized model to misalign with that of the FP model, yielding notable accuracy drops on text generation benchmarks. To rectify the discrepancy, we introduce Logit-aware Final-block Quantization (LFQ), a simple yet effective enhancement to block-wise PTQ that quantizes the final Transformer block by minimizing the cross-entropy between the logits of the FP model and those of its quantized counterpart. By aligning token probabilities at the logit level in the final block, LFQ consistently improves the accuracy of complex generation tasks over state-of-the-art block-wise PTQ across diverse model families, while maintaining parity with FP baselines on language modeling and understanding.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

LFQ is a narrow tweak that applies cross-entropy on the final block's logits to fix generation drops in block-wise PTQ, but the key assumption about upstream error correction looks shaky and the abstract gives no data to check it.

read the letter

LFQ tries to improve low-bit weight-only PTQ for LLMs by quantizing only the last Transformer block with a cross-entropy loss that aligns logits to the full-precision model instead of using MSE. The rest of the model stays under standard block-wise PTQ. The stated motivation is that skipping the unembedding layer and relying on MSE causes token probability misalignment that hurts long generations and chain-of-thought tasks more than basic language modeling.

The new piece is the deliberate choice to do the logit alignment exclusively on the final block. That is a direct response to the identified shortcomings of existing block-wise methods.

The paper does a service by calling out the practical gap between PTQ performance on perplexity or understanding benchmarks versus actual generative workloads. That distinction matters for deployment.

The soft spot is the assumption that misalignment is mainly from the omitted unembedding and MSE, and that correcting only the final block is enough to fix it. Earlier blocks are still quantized under MSE, so the hidden-state inputs to the last block are already altered. It is not obvious that logit matching at the output can compensate for those shifts, especially over long sequences. The abstract contains no numbers, no tables, no ablation on whether last-block-only is sufficient, and no comparison details, so the claim of consistent gains across model families cannot be evaluated.

This is for people doing practical LLM compression and inference optimization. A reader in that area might pick up the idea if the full experiments are clean, but the current write-up leaves the central claim untested.

I would send it to peer review. The method is cheap to implement and the problem is real, so referees can check the results quickly.

Referee Report

1 major / 0 minor

Summary. The manuscript claims that standard block-wise post-training quantization (PTQ) of LLMs degrades generation quality on complex tasks because it omits the unembedding layer and optimizes blocks with MSE, causing token-probability misalignment with the FP model. It proposes LFQ as a simple fix: quantize all blocks with standard block-wise PTQ except the final Transformer block, which is instead optimized by minimizing cross-entropy between the FP model's logits and the quantized model's logits. This logit-level alignment is asserted to restore generation accuracy across model families while preserving parity with FP baselines on language modeling and understanding benchmarks.

Significance. If the experimental claims hold, LFQ provides a low-overhead, targeted intervention that improves the practical utility of low-bit weight-only quantized LLMs specifically for generative workloads. The approach is notable for requiring no new parameters and for being a direct change to the loss on one block rather than a global retraining or architectural modification.

major comments (1)

[Abstract and method description] Abstract and method description: The central claim that optimizing only the final block with cross-entropy is sufficient to correct token-probability misalignment rests on the untested assumption that upstream quantization errors (under MSE) do not produce input hidden-state distributions to the final block that cannot be adequately compensated by last-block logit alignment. No analysis of distribution shift, ablation on upstream vs. final-block effects, or comparison of hidden-state statistics is referenced, leaving the sufficiency of the final-block correction as a load-bearing but unsupported point.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for highlighting the need for stronger empirical support of the final-block correction's sufficiency. We address the comment below and will revise the manuscript accordingly.

read point-by-point responses

Referee: [Abstract and method description] Abstract and method description: The central claim that optimizing only the final block with cross-entropy is sufficient to correct token-probability misalignment rests on the untested assumption that upstream quantization errors (under MSE) do not produce input hidden-state distributions to the final block that cannot be adequately compensated by last-block logit alignment. No analysis of distribution shift, ablation on upstream vs. final-block effects, or comparison of hidden-state statistics is referenced, leaving the sufficiency of the final-block correction as a load-bearing but unsupported point.

Authors: We agree that the current manuscript lacks explicit analysis of distribution shift or ablations isolating upstream MSE quantization effects from the final-block correction. While the empirical gains on generative tasks across model families provide indirect support that the final block can compensate, this does not directly test the assumption. In revision we will add: (i) an ablation comparing standard block-wise PTQ (all blocks MSE) against variants where only upstream blocks use MSE and the final block uses cross-entropy, (ii) statistics (mean, variance, cosine similarity) of hidden-state distributions at the input to the final block for FP, standard PTQ, and LFQ models, and (iii) a short discussion of how logit-level optimization enables the final block to adapt to shifted inputs. These additions will directly address the load-bearing assumption. revision: yes

Circularity Check

0 steps flagged

No circularity; method is an empirical loss change with no self-referential reduction

full rationale

The paper defines LFQ directly as quantizing only the final Transformer block via cross-entropy minimization between FP and quantized logits, then reports empirical gains on generation tasks. No equations, derivations, or self-citations are presented that reduce the claimed improvement to a quantity fitted or defined by the method itself. The attribution of prior PTQ shortfalls to omitted unembedding and MSE is an explanatory hypothesis, not a load-bearing derivation that collapses into the new procedure. The approach remains self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The contribution rests on the domain assumption that the identified factors (omitted unembedding layer and MSE) are the dominant causes of misalignment and that final-block logit alignment suffices to correct them; no free parameters or new entities are introduced.

axioms (1)

domain assumption The shortfall in generative quality for block-wise PTQ stems from omission of the unembedding layer and reliance on MSE, causing token probability misalignment.
Explicitly stated in the abstract as the attribution for the observed accuracy drops.

pith-pipeline@v0.9.1-grok · 5774 in / 1202 out tokens · 35637 ms · 2026-06-29T07:36:37.053780+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

4 extracted references · 4 canonical work pages · 1 internal anchor

[1]

URL https: //aclanthology.org/2025.acl-long.498/

doi: 10.18653/v1/2025.acl-long.498. URL https: //aclanthology.org/2025.acl-long.498/. Cheng, W., Zhang, W., Shen, H., Cai, Y ., He, X., Kaokao, L., and Liu, Y . Optimize weight rounding via signed gradient descent for the quantization of LLMs. In Al-Onaizan, Y ., Bansal, M., and Chen, Y .-N. (eds.), Findings of the Association for Computational Linguis- t...

work page doi:10.18653/v1/2025.acl-long.498 2025
[2]

Training Verifiers to Solve Math Word Problems

URL https://aclanthology.org/2024. findings-emnlp.662/. Cobbe, K., Kosaraju, V ., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems, 2021. URL https://arxiv. org/abs/2110.14168. DeepSeek-AI, Guo, D., Yang, D., Zhang, H., Song, J., Zhang,...

work page internal anchor Pith review Pith/arXiv arXiv 2024
[3]

URL https://proceedings.mlr.press/ v202/lee23h.html. Lee, J. H., Kim, J., Yang, J. Y ., Kwon, S. J., Yang, E., Yoo, K. M., and Lee, D. LRQ: Optimizing post- training quantization for large language models by learn- ing low-rank weight-scaling matrices. In Chiruzzo, L., Ritter, A., and Wang, L. (eds.),Proceedings of the 2025 Conference of the Nations of th...

work page doi:10.18653/v1/2025.naacl-long 2025
[4]

W4” and “W3g128

URL https://aclanthology.org/2025. naacl-long.393/. Li, Z., Su, Y ., Yang, R., Xie, C., Wang, Z., Xie, Z., Wong, N., and Yang, H. Quantization meets reasoning: Exploring llm low-bit quantization degradation for mathematical reasoning, 2025. URL https://arxiv.org/abs/ 2501.03035. Lightman, H., Kosaraju, V ., Burda, Y ., Edwards, H., Baker, B., Lee, T., Lei...

work page doi:10.18653/v1/2025.acl-long.1382 2025

[1] [1]

URL https: //aclanthology.org/2025.acl-long.498/

doi: 10.18653/v1/2025.acl-long.498. URL https: //aclanthology.org/2025.acl-long.498/. Cheng, W., Zhang, W., Shen, H., Cai, Y ., He, X., Kaokao, L., and Liu, Y . Optimize weight rounding via signed gradient descent for the quantization of LLMs. In Al-Onaizan, Y ., Bansal, M., and Chen, Y .-N. (eds.), Findings of the Association for Computational Linguis- t...

work page doi:10.18653/v1/2025.acl-long.498 2025

[2] [2]

Training Verifiers to Solve Math Word Problems

URL https://aclanthology.org/2024. findings-emnlp.662/. Cobbe, K., Kosaraju, V ., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems, 2021. URL https://arxiv. org/abs/2110.14168. DeepSeek-AI, Guo, D., Yang, D., Zhang, H., Song, J., Zhang,...

work page internal anchor Pith review Pith/arXiv arXiv 2024

[3] [3]

URL https://proceedings.mlr.press/ v202/lee23h.html. Lee, J. H., Kim, J., Yang, J. Y ., Kwon, S. J., Yang, E., Yoo, K. M., and Lee, D. LRQ: Optimizing post- training quantization for large language models by learn- ing low-rank weight-scaling matrices. In Chiruzzo, L., Ritter, A., and Wang, L. (eds.),Proceedings of the 2025 Conference of the Nations of th...

work page doi:10.18653/v1/2025.naacl-long 2025

[4] [4]

W4” and “W3g128

URL https://aclanthology.org/2025. naacl-long.393/. Li, Z., Su, Y ., Yang, R., Xie, C., Wang, Z., Xie, Z., Wong, N., and Yang, H. Quantization meets reasoning: Exploring llm low-bit quantization degradation for mathematical reasoning, 2025. URL https://arxiv.org/abs/ 2501.03035. Lightman, H., Kosaraju, V ., Burda, Y ., Edwards, H., Baker, B., Lee, T., Lei...

work page doi:10.18653/v1/2025.acl-long.1382 2025