{"id":"ac755642-380c-40b8-9b15-3ea1c79d800f","arxiv_id":"2504.13989","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Gradual Binary Search over per-projection clipping ratios, combined with Hadamard rotations and dimension expansion, enables 3-bit WAKV quantization with better benchmark accuracy than QuaRot.","lead":"This paper presents a post-training quantization method that uses a gradual binary search to tune clipping ratios per projection, allowing 3-bit quantization of weights, activations, and KV caches in large language models. The approach also pads model dimensions so Hadamard rotation works on non-power-of-2 architectures like Qwen, improving accuracy over existing rotation-based methods.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unimodality of the per-projection perplexity landscape is asserted, not tested; if it fails, GBS's clipping ratios and the reported 3-bit gains are not guaranteed to generalize.","rationale":"The reader's weakest assumption is the same one I find most load-bearing. The entire quantitative story—3-bit WAKV with large accuracy gains—depends on GBS selecting good clipping ratios; if the underlying objective is multimodal, the binary search has no guarantee and the reported results may owe more to the particular projection order and starting precision than to the method. This is not a rejection: the paper reports consistent gains across six models and against SpinQuant and DFRot baselines, which is real empirical evidence that the method works in the tested configurations. But the paper asserts the unimodality assumption rather than testing it, and its own Appendix C documents instability when the starting precision changes, which is at least consistent with multimodality. A grid-search check on a few projections would settle this directly. I also note the '40%' headline is cherry-picked (it is roughly 39 percentage points on one Mistral model; relative improvements elsewhere are much larger or smaller), and the dimension-expansion cost statements conflict, but those are presentation issues that do not by themselves undermine the existence of a real effect. The reader's CONDITIONAL verdict remains appropriate, with the additional condition that the unimodality assumption be validated or the algorithm modified to handle multimodality.","tokens_in":16579,"tokens_out":17342,"duration_ms":164119,"concrete_test":"On a fixed model (e.g., Mistral-7B-Instruct-v0.3), choose 5 projections spanning q/k/v/o/gate/up/down. For each, keep all other projections in FP16, sweep the clipping ratio over 51 evenly spaced values in [0,1], and record WikiText2 PPL at each point. Count the local minima in each curve. Then run Algorithm 1 from three different initial intervals ([0,1], [0,0.5], [0.5,1]) and compare its returned ratio with the sweep's global minimum. If any curve has more than one local minimum, or if GBS returns a ratio whose PPL is more than 5% above the sweep minimum, the unimodality assumption fails for that projection and the central claim is not supported for it.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is that GBS finds near-optimal per-projection clipping ratios, and that this is what rescues 3-bit WAKV quantization. The method rests on an explicit unimodality assumption stated in Section 4.1: 'assuming a single minimum and a convex landscape.' The paper provides no direct evidence for this assumption; it only asserts after the fact that it 'appears to hold true.' Algorithm 1 is a bracketing search that can provably find the global minimum only when the one-dimensional perplexity curve is unimodal. If that curve has multiple local minima—which is plausible because quantization noise can create non-smooth, non-convex dependencies, and the paper's own Appendix C shows that the final clipping configuration is unstable when starting from a 4-bit model—then the binary search can stop at a local optimum. The reported 3-bit gains would then be initialization- and order-dependent rather than a robust property of the method, and the 'general method' claim would not transfer to other models or calibration sets.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes GBS (Gradual Binary Search), a post-training quantization method that optimizes per-projection clipping ratios using perplexity as the objective, and combines it with Hadamard rotations and a dimension-expansion scheme (based on Paley Hadamard matrices) to handle non-power-of-2 embedding dimensions. The method targets 3-bit and 4-bit quantization of weights, activations, and KV cache in LLMs. The paper reports large improvements in perplexity and six zero-shot benchmarks relative to QuaRot, SpinQuant, and DFRot across Mistral, LLaMA, and Qwen model families, and also presents theoretical results claiming that Hadamard matrices reduce outlier amplitude more effectively than random orthogonal rotations.","tokens_in":16727,"tokens_out":7733,"duration_ms":65290,"significance":"If the empirical results hold, this would be a substantial advance: 3-bit WAKV quantization with only a modest perplexity increase (e.g., Mistral-7B Instruct from 5.49 to 7.04) and a large retained benchmark average would have clear practical value. The paper's strengths include the breadth of experiments (six models, three baselines), the consistency and size of the reported gains, and a clean theoretical explanation for the advantage of Hadamard rotations over random rotations. However, the central search rests on an unverified unimodality assumption, the paper contains a direct contradiction about inference cost, and the experimental section lacks code/data release and error bars; these issues mean the generality of the method is not yet fully established. With revision, the paper could become a solid contribution to the quantization literature.","major_comments":[{"comment":"The algorithm's convergence to a near-optimal clipping ratio relies on the assertion in Section 4.1 that the perplexity landscape has a single minimum and is convex, and Section 5.2.1 merely states that this 'appears to hold true.' The evidence in Appendix C contradicts this: Figure 5 shows that running GBS from a model initialized in 4-bit precision converges to a configuration with many clipping ratios at 1 and high variance, while initialization from FP16 produces a different configuration, with test PPL 7.69 vs 7.40. That is direct evidence of initialization dependence, which is incompatible with a single-global-minimum landscape. Please provide a direct test of unimodality (e.g., grid evaluation of PPL versus clipping ratio for several representative projections) and a sensitivity analysis with respect to projection order and initialization; without this, the reported 3-bit gains cannot be robustly attributed to a global optimum search.","section":"Section 4.1 / Algorithm 1 / Appendix C"},{"comment":"There is a contradiction about inference cost. The paragraph after Eq. (2) states that dimension expansion 'increases the model size and computational cost' and then, two sentences later, claims 'there is no additional computational cost at inference.' Lemma 4.1 itself bounds d so that BitOps(A'B') <= BitOps(AB), which only matters if the expanded product is computationally more expensive unless the bound is enforced. Please state precisely, for the fused architecture used in the experiments, whether the number of FLOPs per token changes with d. If it does increase, report the accuracy-versus-FLOPs trade-off (Figure 2 already mentions a 'very high computational cost') and compare against the alternative of using one more bit instead of expansion.","section":"Section 4.2 / Eq. (2) / Lemma 4.1"},{"comment":"The paper reports single-run numbers without error bars, confidence intervals, or a code/data release. The central quantitative claim, e.g., the Mistral-7B Instruct improvement from 22.06 to 61.32 average accuracy in Table 2, is dramatic and could be sensitive to the calibration subset (10% of WikiText2 train) and to hyperparameters such as epsilon, expansion size d, and projection order. Please provide a stability analysis over at least 3-5 calibration draws and report the spread of final PPL/AVG, and report sensitivity to epsilon and to the calibration fraction (the Limitations section itself suggests 10% may be excessive). This is important for supporting the 'general method' claim.","section":"Section 5 / Tables 1-4"}],"minor_comments":[{"comment":"The phrase '40% increase in accuracy' is a 40-percentage-point improvement on a 0-100 scale, not a 40% relative increase; for the cited Mistral-7B Instruct case the relative increase is approximately 178%. Please rephrase to avoid misleading readers.","section":"Abstract / Conclusion"},{"comment":"The text says 'We first chose to add 8 dimensions, reaching 1542' but 1536+8=1544. The value 1542 would not satisfy the Paley condition (1542-1=1541, which is composite), whereas 1544-1=1543 is prime and 1543 ≡ 3 (mod 4). Please correct the number or the arithmetic.","section":"Section 4.2"},{"comment":"In the QuaRot+ row for Qwen2.5 7B Inst, PIQA, HS, and ARC-E are all reported as exactly 1.14; this appears to be a copy-paste error and should be checked.","section":"Table 2, Qwen2.5 7B Inst row"},{"comment":"The pseudocode does not make explicit how the model state evolves across the for loop: are projections 1..i-1 kept quantized with their selected ratios when projection i is being evaluated, and is the quantization of earlier projections re-applied or frozen when the clipping ratios are updated? Please clarify.","section":"Algorithm 1"},{"comment":"There are several typographical issues: 'demonsatrate' in Section 3.2, 'Figue' in Appendix A.1.2, and 'W AKV' in the table captions should be 'WAKV'.","section":"Throughout"},{"comment":"The text says the computational limit for LLaMA3-8B in 3 bits is d=1366, but Figure 2's x-axis extends to 3500 and shows a 'Computational limit' line. Please clarify how the limit is computed for each model and which bit budget is used for the comparison, since the curves appear to extend beyond the stated limit.","section":"Figure 2 / Lemma 4.1"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing you should know about this paper is that the core method is genuinely new: a per-projection binary search over clipping ratios, driven by perplexity, applied gradually one projection at a time. The reported 3-bit WAKV results are striking—QuaRot's average accuracy jumps from 22.06 to 61.32 on Mistral-7B-Instruct, for example—and the improvements are consistent across Mistral, LLaMA, and Qwen families, and across three different rotation baselines (QuaRot, SpinQuant, DFRot). That breadth is the strongest evidence that the central claim is real, and the evaluation is not circular: the ratios are fit on 10% of WikiText2 train, while perplexity is reported on test and benchmark accuracies are on separate tasks.\n\nWhat is actually new: the gradual binary search itself, and the Paley-based dimension expansion for non-power-of-2 hidden sizes. The Hadamard optimality result is elementary (the max entry of an orthogonal matrix is at least 1/√n, and Hadamard achieves it), but it is correct, and it gives a clean explanation for why Hadamard rotations reduce outliers more than random rotations. The paper also honestly acknowledges that dimension expansion was independently developed in [10].\n\nThe soft spots are real but mostly fixable. First, no code or data release, and no error bars; for a method that relies on a search procedure, that is a nontrivial reproducibility gap. Second, there is a direct contradiction about inference cost: Section 4.2 says dimension expansion adds \"no additional computational cost at inference,\" but Lemma 4.1 and the surrounding text explicitly analyze the increased BitOps. The \"no cost\" statement is simply wrong—expanded activations still need to be rotated. Third, the arithmetic in the example is off: for n=4096, b=4, b'=3, Lemma 4.1 gives d ≤ 1365.33, so the claim that d=1366 is within the limit is a small error. Fourth, the \"40% increase in accuracy\" headline is cherry-picked from the best Mistral case; the mean improvement is smaller, though still substantial.\n\nThe stress-test concern about the unimodality assumption is fair but not fatal. The paper states \"assuming a single minimum and a convex landscape\" and then later says it \"appears to hold true,\" but provides no direct evidence. The one-dimensional perplexity curve could have multiple minima, and the binary search would then stop at a local optimum. However, the empirical results are consistent across many models and three baselines, which suggests the search is working in practice even if the theoretical justification is loose. I would ask the authors to add a sensitivity analysis or at least plot a few representative perplexity curves.\n\nOverall: this is a solid, useful engineering contribution that deserves peer review. I would send it to a serious referee and ask for code release, correction of the cost contradiction, and a more careful discussion of the unimodality assumption. The central claim is likely to survive.\n\nSerious thinker: yes. The work is coherent, cites the right literature, and acknowledges its limitations. I would bring it to a reading group focused on LLM quantization.","headline":"A novel and empirically promising per-projection clipping search (GBS) for 3-bit WAKV quantization, with large reported gains that are probably real, but the paper needs reproducibility fixes and some internal corrections before the numbers can be taken at face value.","tokens_in":17323,"tokens_out":2845,"would_cite":true,"duration_ms":27190,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A perplexity-guided gradual binary search over per-projection clipping ratios makes 3-bit weights, activations, and KV-cache quantization practical, lifting average benchmark accuracy by about 40% over rotation-based baselines.","keywords":["post-training quantization","large language models","Hadamard matrices","activation outliers","clipping ratio","perplexity optimization","binary search","KV cache"],"falsifier":"On a fixed model and projection, compute perplexity over a fine grid of clipping ratios in [0,1] at 3-bit WAKV; if the resulting curve shows two separated minima, or a non-convex valley, the unimodality assumption fails and GBS can return a suboptimal ratio — observing such a curve for any model in the paper would directly challenge the method's claimed near-optimality.","tokens_in":16307,"feed_emoji":"🧮","tokens_out":10928,"duration_ms":89743,"temperature":0.7,"pith_summary":"The paper claims that 3-bit quantization of large language model weights, activations, and key-value caches is practical when each quantized projection is given its own clipping ratio — the fraction of the activation range kept before rounding — chosen automatically by a gradual binary search that minimizes the model's perplexity. On the QuaRot rotation-based pipeline, this search lifts the average six-benchmark accuracy of 3-bit Mistral-7B-Instruct from 22.06% to 61.32% (FP16: 71.62%) and cuts perplexity from 38.28 to 7.04; similarly large gains appear on LLaMA and Qwen models. The paper also proves that Hadamard matrices reduce activation outliers more than random orthogonal rotations (by $1/\\sqrt{n}$ versus $\\sqrt{2\\log n}/\\sqrt{n}$ at the level of the largest entry), and it extends rotations to non-power-of-2 embedding dimensions by zero-padding to a Paley-constructible size. A sympathetic reader would take the central insight to be that low-bit LLM quantization fails mainly because of badly chosen clipping ratios, not because of the rotation or the quantizer itself.","feed_headline":"3-bit LLM quantization jumps 40% with perplexity-guided search","feed_subtitle":"Tuning one clipping ratio per projection turns 3-bit QuaRot's unusable outputs into near-FP16 accuracy.","key_machinery":"The central mechanism is the Gradual Binary Search (GBS) of Algorithm 1: the model's linear projections are processed one by one; for each, the projection is quantized while the rest of the model remains in FP16 (or uses already chosen ratios), and the search probes clipping ratios in $[0,1]$, evaluating perplexity on 10% of WikiText2 at each probe and halving the interval toward the lower-perplexity side until the interval width is below $\\epsilon$. The search assumes the perplexity-versus-ratio curve is convex with one minimum. Around it, the supporting machinery is the Hadamard rotation (an orthogonal matrix with entries $\\pm 1$, applied with the fast Hadamard transform), the Paley construction that builds a Hadamard matrix of order $p+1$ when $p$ is prime and $p \\equiv 3 \\pmod 4$, and dimension expansion: zero-padding weights from dimension $n$ to $n+d$ so a Hadamard rotation exists, with Lemma 4.1 bounding $d \\le n(b-b')/b'$ so the padded multiplication uses no more bit-operations than the unpadded one at a higher bitwidth.","core_discovery":"On its own terms, the paper's central discovery is that 3-bit quantization of LLM weights, activations, and KV caches is attainable when each quantized projection receives its own clipping ratio, chosen to minimize model perplexity rather than quantization error. The strongest evidence is the 3-bit WAKV result for Mistral-7B-Instruct: QuaRot alone reaches 22.06% average accuracy over six benchmarks with perplexity 38.28, and adding the Gradual Binary Search raises this to 61.32% with perplexity 7.04, compared with FP16's 71.62% and 5.49. The paper also claims a theoretical proof that a Hadamard matrix reduces a single outlier by a factor $1/\\sqrt{n}$ while a random orthogonal rotation reduces it by $\\sqrt{2\\log n}/\\sqrt{n}$, so Hadamard matrices attain the optimal bound among all orthogonal matrices. Finally, it claims that zero-padding weights to a dimension that admits a Paley-constructed Hadamard matrix generalizes rotations to non-power-of-2 architectures such as Qwen.","pith_inferences":["Editorial inference: since the paper's final clipping ratios vary widely by projection (some near 1, some below 0.3), a cheap proxy — a small per-layer-type lookup table of ratios — could test how much of GBS's gain comes from per-projection tuning versus simply clipping aggressively.","Editorial inference: the unimodality assumption is the fragile point; if perplexity landscapes turn out to have multiple minima on other architectures or calibration sets, a multi-start or coarse-grid pre-pass would preserve the method while removing its main failure mode.","Editorial inference: the Hadamard-optimality proof for a single-outlier vector suggests a broader message: outlier reduction in LLM quantization is an incoherence problem, and any orthogonal matrix whose entries are all near $1/\\sqrt{n}$ in absolute value should behave similarly to Hadamard rotations.","Editorial inference: the paper's benchmarks are six next-token-prediction tasks; testing GBS-quantized models on instruction-following or generation-quality metrics would show whether the perplexity gains translate beyond perplexity-like tasks."],"forward_implications":["3-bit WAKV quantization becomes usable on several 7-8B model families: the best 3-bit result (Mistral-7B-Instruct at 61.32 average accuracy) comes within about 10 points of FP16 while spending roughly a third of the bits on weights, activations, and KV cache.","GBS is rotation-agnostic: applying it to SpinQuant and DFRot improves their 4-bit and 3-bit results, so the ratio-search benefit is a general add-on to rotation-based quantizers, not a QuaRot-specific fix.","Dimension expansion plus Paley Hadamard construction removes the power-of-2 restriction, making rotation-based quantization available to architectures like Qwen with 1536- or 3584-dimensional embeddings.","Perplexity, not mean-squared quantization error, is the right guiding objective for low-bit post-training quantization; the paper's plots show lower perplexity tracking higher benchmark accuracy across all runs.","Expanding dimensions improves accuracy monotonically up to the Lemma 4.1 bit-operations budget, after which the padded computation costs as much as quantizing with one extra bit."],"supporting_citations":[{"why":"QuaRot: the rotation-based quantization pipeline (per-token activation quant, GPTQ weights, group-local rotation) that GBS is built on and compared against.","marker":"[2]"},{"why":"SpinQuant: a learned-rotation baseline; Appendix E shows GBS can be bolted onto it, supporting the generality claim.","marker":"[19]"},{"why":"DFRot: a refined-rotation baseline; Appendix E shows GBS improves its 3-bit results, supporting the generality claim.","marker":"[28]"},{"why":"PACT: established clipping ratio as an activation-quantization parameter; GBS replaces training with perplexity-guided search.","marker":"[5]"},{"why":"PrefixQuant: prior work using a grid search over quantization configurations; the paper cites it as the grid-search antecedent of GBS.","marker":"[4]"},{"why":"Vershynin: supplies the Gaussian approximation of random orthogonal matrix entries used to bound random-rotation outlier reduction.","marker":"[25]"},{"why":"De Haan and Ferreira: supplies the expected maximum of normal variates used to derive the random-rotation reduction factor.","marker":"[6]"},{"why":"Franco et al.: independently proposed post-training model expansion; the paper uses it as the precedent for dimension expansion.","marker":"[10]"},{"why":"GPTQ: the weight-quantization method used inside the QuaRot-based evaluation pipeline.","marker":"[11]"}],"fun_headline_variants":["Per-projection clipping unlocks 3-bit LLM quantization","Hadamard + perplexity search: 3-bit LLM with 40% boost","3-bit LLMs: Hadamard and per-axis search beat SoTA","Gradual binary search: 3-bit WAKV, 40% better accuracy","Dim expansion + Hadamard: 3-bit quantization for LLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that perplexity as a function of a single projection's clipping ratio is convex with a single minimum, so the gradual binary search in Algorithm 1 cannot get stuck in a local minimum.","fun_headline_variants_meta":{"raw":{"variants":["Per-projection clipping unlocks 3-bit LLM quantization","Hadamard + perplexity search: 3-bit LLM with 40% boost","3-bit LLMs: Hadamard and per-axis search beat SoTA","Gradual binary search: 3-bit WAKV, 40% better accuracy","Dim expansion + Hadamard: 3-bit quantization for LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000318,"raw_usage":{"total_tokens":1840,"prompt_tokens":1030,"completion_tokens":810,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":646,"completion_tokens_details":{"reasoning_tokens":709}},"tokens_in":646,"tokens_out":810,"duration_ms":6208,"temperature":1.0,"reasoning_tokens":709,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:02:58.249863+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a fixed model and projection, compute perplexity over a fine grid of clipping ratios in [0,1] at 3-bit WAKV; if the resulting curve shows two separated minima, or a non-convex valley, the unimodality assumption fails and GBS can return a suboptimal ratio — observing such a curve for any model in the paper would directly challenge the method's claimed near-optimality.","supporting_citations":[{"cited_title":"High-Dimensional Probability: An Introduction with Applica- tions in Data Science","cited_arxiv_id":null,"evidence_quote":"Vershynin: supplies the Gaussian approximation of random orthogonal matrix entries used to bound random-rotation outlier reduction."}],"review_version":1}