{"id":"cc1ac159-aabe-461a-8fe2-401c5a841bb4","arxiv_id":"2411.11745","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"BitMoD quantizes LLM weights to 3-4 bits using per-group custom floating-point formats with a repurposed zero value, and accelerates them with a bit-serial PE and dequantization unit.","lead":"BitMoD assigns each group of 128 weights in a large language model its own custom 3-bit or 4-bit numeric format, and pairs that scheme with a bit-serial chip design to run inference faster. The paper reports lower perplexity at 3-bit precision than prior quantization methods, plus 1.5-1.7x speedups over state-of-the-art LLM accelerators, which could make on-device AI more practical.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The FP3/FP4 special values are chosen by fitting the six evaluation LLMs (Fig. 3, §III-A) and are never validated on held-out models, so the 3-bit perplexity and lossy-hardware speedup claims may be benchmark-specific.","rationale":"The paper is a solid algorithm-hardware co-design with a plausible mechanism: replacing the redundant signed zero with one of four tunable special values, then per-group selection, and a bit-serial PE that makes these formats cheap. The strongest support is that the method also improves AWQ/OmniQuant combinations (Table XI), which suggests the data type is useful beyond the bare comparison of Table VI. However, the headline 3-bit result is the main differentiator, and the fixed special values are a small set of hyperparameters selected on the evaluation set. The authors disclose this in §III-A. With only six models and two text datasets, a few tunable values can be selected to reduce average quantization error without implying transfer. A leave-one-out check is the minimal experiment that would establish generalization. The hardware claims are simulation-based, but RTL synthesis plus cycle-level simulation is standard for architecture papers; the more decision-relevant weakness is the special-value transfer, since the lossy speedup depends on 3-bit usability. I therefore agree with the reader's weakest assumption. The existing CONDITIONAL verdict remains appropriate; the condition should explicitly require a holdout/leave-one-out validation of the special-value choices and, secondarily, release of the RTL with a commit hash, as the reader suggests.","tokens_in":25563,"tokens_out":13277,"duration_ms":124059,"concrete_test":"Use the released bitmod-quant code to perform leave-one-out special-value selection on the six evaluated LLMs: for each held-out model, select the FP3/FP4 special-value sets using Fig. 3's quantization-error criterion on the other five models, then compute 3-bit and 4-bit Wikitext-2/C4 perplexity on the held-out model. Compare the held-out mean ΔPPL with the reported 2.94 (Table VI) and with INT3-Asym; also count how often the selected outside-range value (currently ±6) changes across folds. If the held-out 3-bit advantage over INT3-Asym roughly halves, or the selected value changes in a majority of folds, the fixed special values are overfit and the headline should be re-qualified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central algorithmic novelty is the set of four special values that replace the redundant zero in FP3/FP4. These values are not derived; they are selected as the candidates that minimize per-group quantization error on the exact six LLMs used in the evaluation (§III-A, Fig. 3; FP4 analogues ±5, ±8 are chosen the same way). The paper even acknowledges this: \"we have fixed the four special values given that they can minimize the quantization error for the diverse set of LLMs that we evaluate.\" Table IX is the only special-value ablation, but it is performed on the same models used for selection (OPT-1.3B, Phi-2B, Llama-2-7B, Llama-3-8B), so it cannot detect overfitting. Because the per-group algorithm then picks the best of these four values for each weight group, the reported large 3-bit gain (mean ΔPPL 2.94 vs 24.34 for INT3-Asym in Table VI) could partly reflect tuning to the benchmark family. The hardware speedup at 3-bit in Fig. 7 is contingent on 3-bit remaining acceptable; if the special values do not transfer to a new model family, the 'lossy' configuration would need 4-bit, eroding the memory-bound generative speedup. A leave-one-out or held-out-model check is therefore the load-bearing test for both the algorithmic and hardware headline.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes BitMoD, a post-training weight-quantization scheme and co-designed accelerator for LLMs. On the algorithm side, BitMoD extends low-precision floating-point formats (FP3/FP4) by replacing the redundant zero with one of four special values (e.g., ±3 and ±6 for FP3) and selecting the best special value per group of 128 weights via 2-bit metadata. This 'fine-grained data type adaptation' is claimed to improve perplexity at 3-bit/4-bit over prior methods and to integrate with AWQ, OmniQuant, and SmoothQuant. On the hardware side, the paper presents a bit-serial PE with a unified representation for INT8/INT6/FP4/FP3 and an on-the-fly bit-serial dequantization unit. Evaluation on six LLMs reports <0.5% average accuracy loss at 4-bit on discriminative tasks, and average speedups of 1.69× and 1.48× over ANT and OliVe under an iso-compute area constraint. The authors provide an open-source artifact and RTL synthesis in TSMC 28nm with cycle-level simulation.","tokens_in":25900,"tokens_out":9315,"duration_ms":85561,"significance":"If the reported gains generalize beyond the evaluation suite, the work is significant: it shows that reusing the zero codepoint in low-precision float formats can push the accuracy-efficiency frontier of LLM weight quantization, and the hardware design is thorough and reproducible (public artifact, RTL synthesis, cycle-accurate simulator, and a clear iso-compute-area methodology). The 3-bit perplexity results are substantially better than the per-group integer baselines on the evaluated models. However, the central algorithmic innovation—the specific choice of special values—is selected on the same six models used for the final evaluation, so the generality of the headline claims is currently unproven. The 3-bit hardware speedup in Fig. 7 is contingent on that generality.","major_comments":[{"comment":"The special values ±3 and ±6 for FP3, and ±5 and ±8 for FP4, are chosen by minimizing per-group quantization error on the same six LLMs that are later used to report all perplexity and accuracy results. The only special-value ablation (Table IX) is also performed on a subset of those same models (OPT-1.3B, Phi-2B, Llama-2-7B, Llama-3-8B). The paper explicitly states, 'we have fixed the four special values given that they can minimize the quantization error for the diverse set of LLMs that we evaluate.' Consequently, the reported 3-bit perplexity advantage and the associated 3-bit hardware speedup in Fig. 7 are in-sample. This is load-bearing: if the selected values do not transfer to other LLM families, the algorithmic advantage may vanish and the lossy hardware configuration would need to fall back to 4-bit, eroding the speedup. I request a held-out validation, such as leave-one-model-out selection or evaluation on additional LLMs (e.g., Mistral, Gemma) that were not used in Fig. 3, to demonstrate that the chosen special values remain near-optimal.","section":"Section III-A, Fig. 3, Table IX"},{"comment":"The procedure for selecting the special values is not described in sufficient detail to assess overfitting risk. The text says 'we determine the two remaining special values that can minimize the quantization error,' but the candidate set, the search strategy, and the stopping criterion are unspecified. Fig. 3 plots only a few candidates, and it is unclear whether the search space included non-integer values or values beyond ±8. Please provide the complete candidate set, the selection criterion applied, and the resulting quantization-error values for all candidates, or prefer a first-principles derivation if one exists. Without this, the claim that the chosen values 'minimize' the error is not reproducible.","section":"Section III-A, Fig. 3"}],"minor_comments":[{"comment":"The phrase 'better perplexity than prior LLM quantization scheme' is vague; please specify which prior scheme(s) are meant, or state the comparison more generally as 'prior low-bit LLM quantization methods.'","section":"Abstract"},{"comment":"The sentence 'it necessities a floating-point pipeline' should read 'it necessitates a floating-point pipeline.'","section":"Section II-C, text near Table III"},{"comment":"The argument that bit-serial dequantization 'never stalls' the pipeline assumes that dequantization of one group can overlap with the dot-product of the next group. The paper does not describe double-buffering or a pipeline schedule for the shared column accumulator; a short clarification of the scheduling would help.","section":"Section IV-B, paragraph on pipeline timing"},{"comment":"The 'lossy' configuration uses 3-bit weights for generative tasks, but Table VI shows that some models (e.g., OPT-1.3B, Phi-2B) incur perplexity increases of 8 and 3 points, respectively. It would be useful to state the quality threshold used to declare this configuration 'good', and to report the speedup if a fixed quality target (e.g., within 1 perplexity point of FP16) were enforced.","section":"Section V-C, Fig. 7"},{"comment":"The comparison with ANT and OliVe uses per-group extensions of their data types, which is transparently stated, but the paper should also report the original per-channel results for ANT/OliVe on the same models to let the reader see the effect of the extension.","section":"Table III and Section V-A"}],"recommendation":"major_revision","confidential_remarks":"The paper is well-written and the hardware evaluation is solid and reproducible, with a public artifact and RTL synthesis. The main concern is the in-sample selection of the special values, which is acknowledged in §III-A and affects both the algorithmic and hardware headline claims. I do not see this as fatal, because the issue can be addressed with a held-out or leave-one-out validation, but it is a load-bearing gap that should be resolved before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: BitMoD is a well-executed co-design paper. The core idea—repurposing the redundant zero in low-precision floating-point to add either extra resolution (ER) or extra asymmetry (EA) per group—is genuinely new. The hardware side, a unified bit-serial representation plus bit-serial dequantization, is clean and the reported efficiency numbers are plausible. I think the paper deserves a serious referee.\n\nWhat's actually new: the FP3/FP4 extensions and the per-group adaptation that picks one of four special values for each 128-weight group. The 2-bit encoding overhead is tiny, and the idea of using the redundant zero is elegant—at 3 bits that zero is 12.5% of the quantization levels, so replacing it with something useful is a real lever. The unified bit-serial PE that handles INT8, INT6, FP4, and FP3 with one datapath is also a solid contribution, and the bit-serial dequantization with INT8 scaling factors sensibly avoids a floating-point pipeline.\n\nThe paper does most things right. Comparisons against ANT, OliVe, and MX are extended to per-group quantization for fairness, which I appreciate. The ablations (data types, special values) are there. The artifact appendix is real—code, RTL, and simulator are archived—and the RTL synthesis plus cycle simulation is standard practice in this community. Combining BitMoD with AWQ/OmniQuant/SmoothQuant shows the idea is orthogonal to existing optimizations, and those results strengthen the paper.\n\nThe soft spot is the one you flagged: the four special values are selected by minimizing per-group quantization error on the exact six LLMs used in the evaluation (Fig. 3, Table IX). The paper is transparent about this, but a leave-one-out or held-out-model check is missing. This is a moderate concern, not a fatal one. The selection criterion is weight MSE, not downstream accuracy, and the chosen values are the best on five of six models, so the risk is that the headline 3-bit advantage is somewhat optimistic for unseen families, not that the mechanism fails. Still, the load-bearing claim at 3-bit would be much stronger with a Mistral or Gemma datapoint. The hardware speedup claims are simulation-based, which is fine; they are also partly contingent on 3-bit remaining acceptable, but the lossless INT6 configuration gives a solid alternative.\n\nBottom line: the paper is a legitimate HPCA/ISCA-caliber submission. The requested revision is clear: validate the special values on held-out models, and maybe report a couple more model families. I'd cite the data type idea in my own work.","headline":"A strong algorithm-hardware co-design paper with a genuinely new per-group data-type idea, held back slightly by special values that are chosen on the same six LLMs used in evaluation.","tokens_in":26453,"tokens_out":2682,"would_cite":true,"duration_ms":28685,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"BitMoD shows that LLM weights can be quantized to 3-bit and 4-bit with better perplexity and accuracy than prior quantization schemes, by giving each small group of weights its own asymmetric floating-point data type and building a…","keywords":["LLM quantization","weight-only quantization","per-group quantization","asymmetric floating-point formats","bit-serial accelerator","post-training quantization","algorithm-hardware co-design","FP3/FP4 data types"],"falsifier":"Quantize a held-out set of LLMs (e.g., Mistral, Falcon, or a 30B-parameter model) at 3-bit with BitMoD's fixed special values {±3, ±6} and compare perplexity against per-group asymmetric integer quantization; if the perplexity advantage over INT3-Asym shrinks below the reported margin or reverses, the special-value selection is overfit to the benchmark models. A more direct test is to measure normalized per-group quantization error across a broad model zoo and check whether ±3 and ±6 still minimize the average error.","tokens_in":25418,"feed_emoji":"⚡","tokens_out":5427,"duration_ms":42200,"temperature":0.7,"pith_summary":"This paper argues that the limiting factor in low-precision LLM weight quantization is not the bit-width itself but the data type: standard integer and floating-point formats waste quantization levels on a redundant zero and cannot adapt to the asymmetric, Gaussian-like distribution of weights within a group. BitMoD introduces extended FP3 and FP4 formats that replace the redundant zero with one of four special values, and lets each group of 128 weights pick whichever special value minimizes that group's quantization error. On six LLMs, this fine-grained data type adaptation keeps 4-bit accuracy loss under 0.5% on average and makes 3-bit weights outperform prior quantization schemes in perplexity. The hardware half of the paper shows that a bit-serial processing element can execute these mixed data types with a unified representation, achieving average speedups of 1.69x and 1.48x over the ANT and OliVe accelerators respectively.","feed_headline":"3-bit LLM weights beat prior quantization schemes","feed_subtitle":"A bit-serial accelerator converts the new weight formats into 1.69x and 1.48x speedups over ANT and OliVe.","key_machinery":"The load-bearing mechanism is the extended asymmetric floating-point data type, derived by repurposing the redundant ±0 of a sign-magnitude format: FP3-ER/EA adds special values ±3 and ±6 to the basic FP3 set {0, ±1, ±2, ±4}, and FP4-ER/EA adds ±5 and ±8 to the basic FP4 set. A fine-grained data type adaptation algorithm (Algorithm 1) quantizes each group of 128 weights against the basic values plus each candidate special value, and selects the special value with the lowest mean-square error, storing the choice in a 2-bit per-group metadata field. On the hardware side, the accelerator decomposes every supported weight format (INT8, INT6, FP4, FP3) into a unified bit-serial representation with sign, exponent, mantissa, and bit-significance fields, and uses a bit-serial dequantization unit that multiplies the accumulated group partial sum by one bit of the 8-bit integer scaling factor per cycle, so per-group dequantization never stalls the dot-product pipeline.","core_discovery":"The paper's central claim is that replacing the redundant zero in low-precision floating-point formats with carefully chosen special values turns 3-bit and 4-bit weights into practical formats for LLMs. For FP3 the special values are ±3 (extra resolution inside the FP3 range) and ±6 (extra asymmetry extending the range); for FP4 they are ±5 and ±8. Each weight group is quantized with the basic values plus whichever single special value gives the lowest mean-square error, so different groups in the same tensor can use different data types. The authors report that this per-group adaptation achieves better perplexity than prior data types at 4-bit, and that at 3-bit it keeps perplexity much closer to the FP16 baseline than ANT, OliVe, or Microscaling formats. They further claim that the accompanying bit-serial accelerator, which processes weight values as sign, exponent, mantissa, and bit-significance terms and dequantizes per-group partial sums in a bit-serial manner, delivers 1.69x and 1.48x average speedups over ANT and OliVe under an iso-compute-area constraint.","pith_inferences":["The special-value search procedure could be made per-tensor or per-layer rather than fixed globally, and the paper's hardware already supports reprogrammable special-value registers, so a data-dependent calibration pass is a natural extension.","The same zero-repurposing trick could plausibly extend to 2-bit formats, where the redundant zero occupies an even larger fraction of quantization levels, although accuracy would need to be re-validated.","Because the accelerator's bit-serial decoder is not limited to the four chosen special values, the approach may generalize to future data types without redesign, making the hardware a flexible substrate for quantization research.","The per-group dequantization unit's cost is amortized because the group dot-product takes at least 64 cycles while dequantization takes 8, so the design should scale to smaller group sizes before pipeline stalls appear."],"forward_implications":["At 4-bit weight precision, BitMoD claims average accuracy loss below 0.5% on discriminative tasks, which would make 4-bit weights a drop-in option for edge LLM deployment.","At 3-bit, BitMoD reports better perplexity than prior quantization schemes, implying that sub-4-bit weights can be used for generative tasks without the quality collapse seen with integer formats.","Because the data types are orthogonal to AWQ- and OmniQuant-style optimizations, combining BitMoD with those methods reduces average perplexity loss to below 1.0 at both 3-bit and 4-bit, extending the reach of existing software pipelines.","The bit-serial PE computes four multiply-accumulates in two cycles for FP4/FP3 and three cycles for INT6, giving 1.33x and 2x throughput over an FP16 MAC unit, so the accelerator gains scale with how aggressively weights are quantized.","With INT8 per-group scaling factors and 2-bit special-value metadata, the memory overhead of per-group quantization stays at 10 bits per 128-weight group, which is lower than the 24-bit overhead of standard asymmetric integer formats."],"supporting_citations":[{"why":"Supplies the Flint data type and the adaptive per-channel baseline that BitMoD compares against and extends with per-group quantization.","marker":"[26]"},{"why":"Supplies the outlier-victim pair quantization baseline and the OliVe accelerator that BitMoD compares against for speedup and energy.","marker":"[25]"},{"why":"Supplies the Microscaling (MX) shared-exponent format baseline and the group-size-32 convention that BitMoD benchmarks against.","marker":"[40]"},{"why":"Introduces block-wise quantization, the source of the per-group granularity and the concept of group size that BitMoD builds on.","marker":"[16]"},{"why":"Supplies activation-aware weight quantization and the per-group scaling-factor baseline that BitMoD integrates with in the orthogonality experiments.","marker":"[30]"},{"why":"Supplies the clipping-threshold optimization that BitMoD's data types are combined with to reduce perplexity loss.","marker":"[42]"},{"why":"Supplies the activation quantization scheme used to demonstrate that BitMoD's weight formats remain beneficial with INT8 activations.","marker":"[52]"},{"why":"Supplies the second-level quantization of scaling factors that BitMoD uses as the basis for its INT8 per-group scaling factors.","marker":"[14]"}],"fun_headline_variants":["BitMoD: 3-bit LLM weights keep accuracy","Per-group types make 3-bit LLMs practical","Bit-serial dequantization speeds LLMs to 1.69x","Adaptive data types enable 3-bit LLM accuracy","BitMoD: 3-bit LLMs with 1.69x speedup"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The four special values were chosen by minimizing per-group quantization error on the six evaluated LLMs, so the claim that they stay near-optimal for other LLMs and datasets is the load-bearing premise.","fun_headline_variants_meta":{"raw":{"variants":["BitMoD: 3-bit LLM weights keep accuracy","Per-group types make 3-bit LLMs practical","Bit-serial dequantization speeds LLMs to 1.69x","Adaptive data types enable 3-bit LLM accuracy","BitMoD: 3-bit LLMs with 1.69x speedup"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001173,"raw_usage":{"total_tokens":4935,"prompt_tokens":1114,"completion_tokens":3821,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":730,"completion_tokens_details":{"reasoning_tokens":3730}},"tokens_in":730,"tokens_out":3821,"duration_ms":29220,"temperature":1.0,"reasoning_tokens":3730,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T18:11:34.113171+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Quantize a held-out set of LLMs (e.g., Mistral, Falcon, or a 30B-parameter model) at 3-bit with BitMoD's fixed special values {±3, ±6} and compare perplexity against per-group asymmetric integer quantization; if the perplexity advantage over INT3-Asym shrinks below the reported margin or reverses, the special-value selection is overfit to the benchmark models. A more direct test is to measure normalized per-group quantization error across a broad model zoo and check whether ±3 and ±6 still minimize the average error.","supporting_citations":[{"cited_title":"ANT: Exploiting adaptive numerical data type for low-bit deep neural network quantization,","cited_arxiv_id":null,"evidence_quote":"Supplies the Flint data type and the adaptive per-channel baseline that BitMoD compares against and extends with per-group quantization."},{"cited_title":"OliVe: Accelerating large language models via hardware-friendly outlier-victim pair quantization,","cited_arxiv_id":null,"evidence_quote":"Supplies the outlier-victim pair quantization baseline and the OliVe accelerator that BitMoD compares against for speedup and energy."},{"cited_title":"With shared microexponents, a little shifting goes a long way,","cited_arxiv_id":null,"evidence_quote":"Supplies the Microscaling (MX) shared-exponent format baseline and the group-size-32 convention that BitMoD benchmarks against."},{"cited_title":"AWQ: Activation-aware weight quanti- zation for llm compression and acceleration,","cited_arxiv_id":null,"evidence_quote":"Supplies activation-aware weight quantization and the per-group scaling-factor baseline that BitMoD integrates with in the orthogonality experiments."},{"cited_title":"VS-Quant: Per-vector scaled quantization for accurate low-precision neural network inference,","cited_arxiv_id":null,"evidence_quote":"Supplies the second-level quantization of scaling factors that BitMoD uses as the basis for its INT8 per-group scaling factors."}],"review_version":1}