{"id":"277328a2-d8b5-4c5e-87a6-d9168b021c61","arxiv_id":"2502.01703","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"QLESS quantizes the gradient features used in LESS data selection down to 1 bit without losing much fine-tuning quality, cutting gradient storage up to 16x across several LLM families.","lead":"Researchers added a quantization step to the LESS data-selection method, reducing gradient storage by up to 16x while keeping fine-tuning performance similar. The method, called QLESS, may make it practical to select training data for large language models on machines with limited memory.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section 3.1's Eq. (5) gives alpha=0 for b=1, so the stated 1-bit QLESS stores only zero vectors, making the headline '1-bit preserves quality' claim unreproducible as written.","rationale":"I read the paper as an engineering extension of LESS: quantize the projected gradient datastore, compute influence via cosine similarity on the quantized vectors, and show that data selection quality holds down to 1 bit. For that central claim to stand, Eq. (5) must produce a valid nonzero quantized vector at b=1, and Eq. (7) must define a meaningful comparison. The paper fails this condition on its face: alpha=2^{b-1}-1 is zero for b=1, so every 1-bit gradient collapses to the zero vector. The reported experiments must therefore have used a different quantization rule than the one defined, most likely sign quantization or alpha=1. This is an internal inconsistency rather than a mere disagreement with consensus, and it is concrete enough to resolve by reading the code or re-running one experiment. I do not reject the paper outright: the 8-bit, 4-bit, and 2-bit results are fully defined, and the 1-bit results might survive under a documented sign-based rule. The reader's conditional verdict is appropriate; my concern is narrower than the reader's chosen weakest assumption, which concerned normalizing quantized integers rather than dequantized floats. I agree that this normalization choice also lacks theoretical support, but the degenerate alpha is the more decisive issue because it affects the headline bit width directly. The proposed test settles the question: check the repository implementation, and if a special case exists, re-run with it documented and confirm the published numbers.","tokens_in":955,"tokens_out":877,"duration_ms":34110,"concrete_test":"Inspect the released repository (https://github.com/mosesananta/QLESS) for the quantization function at bits=1. If it literally follows Eq. (5), every stored gradient vector is zero and the 1-bit rows of Tables 1, 2, and 4 cannot be generated by the described method. If it instead uses sign quantization or alpha=1 as a special case, rerun the reported 1-bit selection experiments with that special case explicitly documented and verify that the resulting averages remain within the stated standard deviations. A minimal analytical check is to feed any nonzero projected gradient into the stated formula with b=1 and confirm that the output vector and its norm are both zero.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that the quantized vectors in Eq. (5) are valid nonzero representatives for every bit width. Section 3.1 defines q_{z,i,m} = round(alpha * g_{z,i,m} / S_{z,i}) with alpha = 2^{b-1} - 1 for b in {1,2,4,8}. For b=1, alpha = 0, so every stored vector is identically zero. Eq. (6) then divides by ||q|| = 0, and Eq. (7) produces no signal. Yet Tables 1, 2, and 4 report 1-bit averages of 58.59-71.21, far from random-selection behavior. The plausible fix is alpha = 2^b - 1, which for b=1 reduces to sign quantization (as in the separate 'Sign 1-bit' row of Table 3), but the paper neither specifies this nor analyzes it. This is more load-bearing than the reader's normalization concern: even if normalizing quantized integer vectors were theoretically justified, there is no defined nonzero 1-bit representation to normalize. The 1-bit results therefore cannot be reproduced from the stated algorithm, and the paper's most striking claim rests on an undocumented implementation variant.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes QLESS, a memory-efficient extension of the LESS data-valuation framework. QLESS first maps LoRA-based gradients through a random projection and then quantizes the projected vectors with an absmax uniform quantizer at bit widths {1,2,4,8}; influence scores are computed by cosine similarity between the normalized quantized vectors. Experiments on five LLMs (LLaMA-2, LLaMA-3.1/3.2, Mistral, Qwen) and three benchmarks (TyDiQA, MMLU, BBH) report that QLESS achieves performance comparable to LESS while reducing gradient-datastore memory by up to 16x, and that 1-bit quantization preserves data-valuation quality. The paper also includes QLoRA ablations, a comparison of absmax vs. absmean quantization, and a qualitative analysis of selected examples.","tokens_in":17886,"tokens_out":2183,"duration_ms":20973,"significance":"If the central claim holds, QLESS is a practical, low-cost ingredient for making gradient-based data selection feasible at large scale, and its 1-bit result would be a notable empirical finding about the robustness of influence computation under extreme compression. The paper ships code, follows the LESS pipeline closely, and reports results across multiple model families and benchmarks, which is a genuine strength. However, the most striking claim (1-bit viability) is currently unreproducible as written because the stated quantization formula degenerates at b=1, and the aggregate 'comparable' claim is not backed by significance testing. The contribution is incremental but useful; the experimental breadth is adequate for a systems-oriented venue if the algorithmic defect and statistical gaps are fixed.","major_comments":[{"comment":"The quantization formula defines alpha = 2^{b-1} - 1, which gives alpha = 0 for b=1. Under this definition every stored q_{z,i,m} is zero, so Eq. (6)'s division by ||q_{z,i}|| is undefined and Eq. (7) produces no signal. Yet Tables 1, 2, and 4 report nontrivial 1-bit results (e.g., Qwen 1-bit avg 70.72 and Llama 3.1 1-bit avg 65.93). The paper therefore does not actually specify the 1-bit algorithm whose results it reports. Please state the exact 1-bit mapping (e.g., alpha = 2^b - 1, which reduces to sign quantization, as in the separate 'Sign 1-bit' row of Table 3) and re-run or explicitly re-derive the 1-bit experiments under that definition.","section":"Section 3.1, Eq. (5)"},{"comment":"The aggregate claim that QLESS is 'comparable to LESS' is not supported for Mistral 7B: QLESS 8-bit (58.45), 4-bit (57.76), 2-bit (58.52), and 1-bit (58.59) all fall below the random 5% baseline (59.41), and LESS itself also falls below random 5% (59.70 vs. 59.41). Because the headline comparison averages across very different per-model outcomes, at least one per-model discussion and a paired statistical test across seeds (or a clear statement of seed counts) should be provided; otherwise the 'comparable' claim holds only as an unverified aggregate.","section":"Section 4.1 / Table 4"},{"comment":"QLESS normalizes the quantized integer vector q rather than the dequantized float vector, and then computes cosine similarity in Eq. (7). This is a specific algorithmic choice, but the paper provides no analysis of it: the cited QRP guarantees (Section 2.4) concern distance preservation of the underlying continuous vector after projection and quantization, not the cosine of the normalized integer vector. Please either provide a short proof or an ablation comparing (a) cosine on normalized dequantized vectors, (b) cosine on normalized raw quantized integers, and (c) the sign-only variant, for at least one model and benchmark; this would also help isolate the sparsity effect discussed in Section 5.","section":"Section 3.2, Eqs. (6)-(7)"},{"comment":"No significance tests are reported anywhere in the paper. The reported standard deviations (in parentheses) are large relative to the differences that support the 'comparable' claim (e.g., Table 1's average gaps of 0.1-0.6 points vs. stds of 0.1-2.6). Please add paired significance tests (e.g., paired t-test or Wilcoxon over seeds) or report per-seed results so the reader can assess whether the QLESS-vs-LESS differences and the QLESS-vs-random differences are real.","section":"Section 4.2 and Tables 1-5"}],"minor_comments":[{"comment":"The phrase 'The resulting codes qi = Q(yi)' uses a subscript inconsistency with the main text's q_{z,i,m}; please unify the notation.","section":"Section 2.4"},{"comment":"Figure 1 lists 'Random 100%' and 'Random 5%' as baseline methods but does not define their markers in the caption; please clarify which bars are which and add error bars if available.","section":"Section 1 / Figure 1"},{"comment":"The claimed 'three trials using distinct random seeds' should specify whether the standard deviations in Tables 1-5 are over the three trials, and should state the seed values or a seeding protocol for reproducibility.","section":"Section 4.1 / Appendix A"},{"comment":"The sentence 'Interestingly, 1-bit quantization does not exhibit the extreme sparsity problem because its representation inherently omits a zero bin' is not consistent with the stated formula (which contains a zero bin when alpha=0); please reconcile once the 1-bit definition is corrected.","section":"Section 5"},{"comment":"The reference list is missing the QLESS-specific comparison to the very recent LESS follow-ups; please cite the relevant 2024-2025 data-selection work for context and to avoid overclaiming novelty of the quantization-into-datastore idea.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper's core idea is simple and the experimental setup is extensive, but the b=1 alpha=0 defect is a load-bearing algorithmic error, not a mere typo, because the paper's most novel claim depends on those 1-bit numbers. The lack of significance testing is also a systemic issue for a paper whose central assertion is 'comparable performance.' Given that the authors presumably ran the experiments with some actual 1-bit procedure, the fix is localized, so major revision rather than rejection is appropriate. I would also check whether the repository's code matches the corrected formula, since that is easy to verify and would strengthen the revised version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"QLESS is a simple, sensible extension of LESS: quantize the stored projected gradients. The paper's main value is the empirical sweep—8-, 4-, 2-, 1-bit across five models and three benchmarks—and the memory numbers are real: 16.54 GB down to 1.03 GB for the same datastore. The 8-bit and 4-bit results mostly track LESS, and the code and data are promised, which makes this reproducible in principle. That is worth having.\n\nThe soft spots are real, though. The biggest is the 1-bit definition. Eq. (5) sets alpha = 2^{b-1} - 1, so for b=1 alpha=0 and every stored vector is identically zero. Eq. (6) then divides by ||q||=0. Yet Tables 1, 2, and 4 report 1-bit averages from 58.59 to 71.21. The obvious intended fix is alpha = 2^b - 1 (sign quantization), and Table 3 has a separate \"Sign 1-bit\" row, which suggests that is what was run—but the paper never says so. As written, the headline claim that 1-bit preserves quality cannot be reproduced from the stated algorithm. This is more than a typo; it sits at the center of the paper's most interesting claim.\n\nSecond, Mistral 7B is a counterexample. In Table 4, every QLESS variant (8-bit through 1-bit) averages below random 5% selection, and LESS itself is barely above random. The text says QLESS \"performs competitively\" across models, but this model family says the opposite. Maybe it's variance—the standard deviations overlap—but no significance testing is reported anywhere, so we don't know.\n\nThird, the normalization choice in Eq. (6) normalizes the quantized integer vector, not the dequantized float vector. This changes cosine similarity in a way that isn't analyzed, and there's no ablation against normalizing after dequantization. It might be fine, but it's untested.\n\nThese are fixable. The 8-bit/4-bit story is solid and worth publishing. The 1-bit claim needs either a corrected formula and a clear specification of the sign variant, or removal of the \"1-bit works\" narrative. A referee should be asked to check that.\n\nI'd send this to peer review with the expectation of major revision. A careful reader can get value from the comparison, but only after the 1-bit definition is sorted out.","headline":"Useful empirical study of quantized gradient datastores for data selection, but the 1-bit result is unreproducible as written because Eq. (5) makes all 1-bit vectors zero.","tokens_in":18398,"tokens_out":2391,"would_cite":true,"duration_ms":21913,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"QLESS claims that quantizing LoRA gradients to as few as 1 bit preserves data-selection quality in LLM fine-tuning while cutting gradient storage up to 16x.","keywords":["data valuation","data selection","gradient quantization","instruction tuning","large language models","LoRA","random projection","influence functions"],"falsifier":"One concrete test: on the paper's own gradient datastore, compute the rank correlation (for example Kendall's tau) between LESS influence scores and QLESS influence scores for each bit width, and re-run fine-tuning on the top 5% selected by a dequantize-then-normalize ablation. If that ablation consistently outperforms QLESS, or if rank correlation collapses at 1-bit while performance stays high, then the claimed preservation of valuation quality is not explained by the quantized-normalized cosine similarity itself.","tokens_in":17463,"feed_emoji":"💾","tokens_out":5658,"duration_ms":48119,"temperature":0.7,"pith_summary":"QLESS (Quantized Low-rank Gradient Similarity Search) extends the LESS data-selection framework by quantizing the stored gradient features, replacing floating-point vectors with low-bit integer codes plus one scale factor per vector. The paper's central claim is that this compression preserves enough directional information for influence-based data valuation: across several LLM families and benchmarks, models fine-tuned on QLESS-selected data match or nearly match those trained on LESS-selected data, while the gradient datastore shrinks by up to 16x. The headline result is that 1-bit quantized gradients, where each dimension is essentially a sign, still produce competitive selections, suggesting that coarse directional agreement rather than precise magnitudes drives the value of gradient similarity for instruction-tuning data selection. The paper also shows that absmean quantization outperforms absmax at low bit widths because it avoids collapsing most entries into the zero bin, and that stacking QLoRA model quantization on top of gradient quantization keeps performance close to full-precision baselines.","feed_headline":"1-bit gradients pick training data as well as 16-bit","feed_subtitle":"Quantizing LoRA gradients to as few as 1 bit keeps LLM fine-tuning data selection accurate while slashing memory.","key_machinery":"The central object is the quantized-normalized gradient vector $\\hat{q}_{z,i} = q_{z,i}/\\|q_{z,i}\\|$, where $q_{z,i}$ is the absmax-quantized random projection of a LoRA training gradient. Influence is scored by the learning-rate-weighted cosine similarity $\\sum_i \\eta_i \\langle \\hat{q}_{z',i}, \\hat{q}_{z,i}\\rangle$, mirroring LESS but operating on integer codes. The two-step compression, LoRA-based random projection down to 8192 dimensions followed by absmax quantization to $b$ bits, is what carries the memory savings, while the normalization of the quantized vector rather than the dequantized float is the mechanism that allegedly keeps cosine similarities meaningful at extreme compression.","core_discovery":"On the paper's own terms, the discovery is that a simple absmax-based quantization of randomly projected LoRA gradients, storing k b-bit integers and one scale per vector instead of k 16-bit floats, retains the information needed for LESS-style influence estimation. QLESS computes influence as the learning-rate-weighted cosine similarity between normalized integer vectors, and the experiments show that 8-bit QLESS performs on par with LESS, 4-bit and 2-bit remain competitive, and even 1-bit quantization (sign-only codes) often beats a random 5% selection baseline and stays within a few points of 16-bit performance. This holds across LLaMA-2, LLaMA-3, Mistral, and Qwen models on MMLU, BBH, and TyDiQA, with the gradient datastore reduced from 16.54 GB to 1.03 GB at 1-bit. A follow-on analysis finds that absmean quantization is more robust than absmax at low bit widths because absmax shifts most small gradient values into the zero bin, and that combining QLoRA with QLESS preserves most of the selection quality while further cutting the memory footprint.","pith_inferences":["Editorial extension: if 1-bit quantization preserves selection quality, then the effective signal in gradient-based data valuation may be mostly the sign pattern of the projected gradient, which would connect QLESS to sign-based and binary-hash similarity methods and could be tested by comparing QLESS against pure sign random projections without learned scales.","Editorial extension: the paper does not analyze why normalizing the integer vector preserves ranking; a direct rank-correlation study between dequantized-normalized and quantized-normalized influence scores across bit widths would separate the contribution of the scale factor from the contribution of the quantization grid.","Editorial extension: the absmean-vs-absmax trade-off suggests an adaptive or learned quantization scale per gradient vector could improve low-bit performance further, a testable variant the paper does not run.","Editorial extension: because the selected-data fraction experiments show performance plateauing at 0.5-5%, the practical value of QLESS may be in enabling much smaller but better-curated training sets rather than only in compressing storage."],"forward_implications":["Gradient datastores for data selection can be stored in 1-8 bits, enabling selection over much larger instruction-tuning corpora within a fixed memory budget.","8-bit QLESS can replace 16-bit LESS as a drop-in memory-saving step with no expected performance loss on the evaluated benchmarks.","Even 1-bit sign-based gradients pick training subsets that outperform random selection, so extreme compression is a viable regime for influence-based data valuation.","Combining QLoRA model quantization with QLESS gradient quantization reduces the total memory footprint to roughly 22-26 GB for 7B models with only minor performance degradation.","Low-bit selections are qualitatively consistent: 16/8/4/1-bit variants select similar top examples, whereas 2-bit can shift because of sparsity, indicating bit-width calibration matters at intermediate precision."],"supporting_citations":[{"why":"Defines the LESS framework and pipeline that QLESS extends, including LoRA gradient extraction, random projection, normalization, and the data-selection protocol.","marker":"Xia et al., 2025"},{"why":"Provides the TracIn influence-formula ancestor whose checkpoint-weighted gradient dot-product QLESS quantizes and reuses.","marker":"Garima et al., 2020"},{"why":"Supplies the random-projection guarantee that justifies reducing gradient dimension before quantization.","marker":"Johnson & Lindenstrauss, 1984"},{"why":"Gives the quantized Johnson-Lindenstrauss-style guarantee that the paper relies on for preserving geometry after quantization.","marker":"Jacques, 2015"},{"why":"Supplies the QSGD-style gradient quantization idea that motivates compressing the gradient datastore.","marker":"Alistarh et al., 2017"},{"why":"The QLoRA method used in the ablation to quantize the model itself while QLESS quantizes gradients.","marker":"Dettmers et al., 2024"}],"fun_headline_variants":["1-bit gradients: data selection without the memory bloat","QLESS shrinks gradient memory 16x, keeps data quality","Sign-only gradients pick fine-tuning data as well as full-precision","Quantize LoRA gradients to 1 bit, preserve data value","Memory-efficient data selection: 1-bit gradients match 16-bit"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that normalizing the quantized integer vector, rather than dequantizing back to floats and then normalizing, still ranks training examples in nearly the same order as LESS would, so the top 5% selection stays useful.","fun_headline_variants_meta":{"raw":{"variants":["1-bit gradients: data selection without the memory bloat","QLESS shrinks gradient memory 16x, keeps data quality","Sign-only gradients pick fine-tuning data as well as full-precision","Quantize LoRA gradients to 1 bit, preserve data value","Memory-efficient data selection: 1-bit gradients match 16-bit"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000511,"raw_usage":{"total_tokens":2481,"prompt_tokens":935,"completion_tokens":1546,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":551,"completion_tokens_details":{"reasoning_tokens":1457}},"tokens_in":551,"tokens_out":1546,"duration_ms":10624,"temperature":1.0,"reasoning_tokens":1457,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T15:56:27.167090+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"One concrete test: on the paper's own gradient datastore, compute the rank correlation (for example Kendall's tau) between LESS influence scores and QLESS influence scores for each bit width, and re-run fine-tuning on the top 5% selected by a dequantize-then-normalize ablation. If that ablation consistently outperforms QLESS, or if rank correlation collapses at 1-bit while performance stays high, then the claimed preservation of valuation quality is not explained by the quantized-normalized cosine similarity itself.","supporting_citations":[{"cited_title":"Less: selecting influential data for targeted instruction tuning","cited_arxiv_id":null,"evidence_quote":"Defines the LESS framework and pipeline that QLESS extends, including LoRA gradient extraction, random projection, normalization, and the data-selection protocol."},{"cited_title":"Estimating training data influence by tracing gradient descent","cited_arxiv_id":null,"evidence_quote":"Provides the TracIn influence-formula ancestor whose checkpoint-weighted gradient dot-product QLESS quantizes and reuses."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the random-projection guarantee that justifies reducing gradient dimension before quantization."},{"cited_title":"A quantized johnson–lindenstrauss lemma: The finding of buffon’s needle","cited_arxiv_id":null,"evidence_quote":"Gives the quantized Johnson-Lindenstrauss-style guarantee that the paper relies on for preserving geometry after quantization."},{"cited_title":"Z., Tomioka, R., and Vojnovic, M","cited_arxiv_id":null,"evidence_quote":"Supplies the QSGD-style gradient quantization idea that motivates compressing the gradient datastore."},{"cited_title":"Qlora: efficient finetuning of quantized llms","cited_arxiv_id":null,"evidence_quote":"The QLoRA method used in the ablation to quantize the model itself while QLESS quantizes gradients."}],"review_version":1}