{"id":"a79be949-73bf-4664-989a-9c3f71a18713","arxiv_id":"2504.13237","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"ImPart sparsifies delta weights in SVD space by assigning lower drop rates to high-importance singular vectors, reporting better accuracy than DARE and LowRank at high compression ratios.","lead":"This paper introduces ImPart, a method that compresses the differences between fine-tuned and base large language models by selectively dropping parts of the matrix factors from a singular value decomposition, keeping the most important ones. It reports higher compression ratios than two existing baseline methods while preserving task performance, and shows the approach can also be combined with quantization and model merging.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Algorithm 1 as printed cannot achieve the stated target sparsity, and no rectangular extension is given; tabled CRs and the 2x claim are not yet anchored.","rationale":"The reader's weakest assumption was the missing rectangular generalization. My read is broader: the compression-ratio accounting is insecure even in the square case because Algorithm 1's printed target check is internally inconsistent, and the rectangular case only compounds the problem. This is the most load-bearing issue because every quantitative comparison in the paper is expressed through CR: the abstract's '2× higher compression ratio', Table 1 and Table 5 are all anchored on the achieved sparsity α. If the actual sparsity exceeds the reported value, IMPART is being compared at a different point on the compression/performance curve, and the claimed advantage is not established by the tables. I am not asserting that the released code has this bug; the manuscript text is what I can evaluate, and it is inconsistent as written. A single reproducibility check on the released implementation would settle whether the concern lands. This does not change the reader's conditional verdict: the paper should be accepted only after confirming that the measured CR matches the reported CR for both square and rectangular layers, and after the algorithm text is corrected or clarified.","tokens_in":16384,"tokens_out":11410,"duration_ms":113936,"concrete_test":"Run the released implementation of Algorithm 1 (and Algorithm 3) on a toy square matrix, e.g. n = 512, with α = 31/32, β = 0.6, C = 1; measure the resulting sparsity of Rhat U and Rhat V and compute CR = n² / (#nonzero in Rhat U + #nonzero in Rhat V). If the measured CR is not 32, the tabled compression ratios need to be recomputed. Then repeat the same measurement on a LLaMA-shaped rectangular layer, e.g. 4096×11008, and compute CR from the actual retained factor entries r(m+n); if this differs from the square-formula CR, re-plot Figure 1 and Table 5 using the true CR.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is a claim about compression ratio (CR), so the sparsity actually produced by the allocation procedure must equal the α used in every table. As printed, Algorithm 1 (Appendix A.1) does not enforce this, even for square matrices. After line 1, α is updated to (1+α_orig)/2, the intended average sparsity of U and V. With pre-prune ratio β, the retained columns i ≤ r must have average p = (α − β)/(1 − β), so that the overall average is β + (1−β)·avg = α. Line 4 sets γ to achieve exactly that retained-block average. But the while loop on line 7 checks (1/r)Σ_{k=1}^r p_k < α, rather than < (α − β)/(1 − β). Since (α − β)/(1 − β) < α for β > 0, the loop keeps firing and sets additional p_i to 1 until the retained-block average reaches α, overshooting the target sparsity and making the actual CR larger than reported. For α_orig = 31/32 and β = 0.6, the required retained average is 0.9219, but the printed loop continues until 0.96875, pushing the achieved CR far above 32. Independently, the derivation and Algorithm 1 are explicitly square-only, while LLaMA projections are rectangular (e.g., 4096×11008). For an m×n weight with rank r = min(m,n), the factor storage is r(m+n), not 2n², and no rule is given for allocating sparsity across U and V when m ≠ n. If the square formula is applied verbatim to rectangular layers, the reported CR in Tables 1 and 5, and therefore the claimed 2× advantage, does not correspond to the actual storage.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ImPart, an SVD-based delta-sparsification method for LLMs. It allocates a per-singular-vector sparsity ratio p_k that decreases with the singular value magnitude, pre-prunes the long tail of small singular components, and rescales surviving entries by 1/(1−p_k). The authors prove that the expectation of the reconstructed hidden state is preserved and present experiments on math, code, and chat tasks showing that ImPart outperforms DARE and a low-rank baseline at reported compression ratios, and that it composes with Delta-CoMe quantization and with task-arithmetic/TIES merging. The abstract claims a 2× higher compression ratio than baselines at the same performance level.","tokens_in":16770,"tokens_out":11975,"duration_ms":115659,"significance":"If the reported compression ratios are correct, ImPart would be a practical and simple improvement over DARE-style delta sparsification, with ablations and sensitivity analysis supporting the design. The paper is clearly written and includes public code. The main contribution is empirical; the expectation-preservation proof is a direct extension of DARE's argument. However, the paper's central quantitative claim (the CR values and the 2× advantage) depends on two currently unverified elements: the correctness of Algorithm 1's sparsity enforcement and a non-square generalization for LLaMA's rectangular layers. Both are fixable, but they need to be addressed before the results can be taken as stated.","major_comments":[{"comment":"The while-loop condition checks (1/r)Σ_{k=1}^r p_k < α, but after line 1, α denotes the target average sparsity of the full U and V matrices. Since the tail components are pre-pruned to p_k = 1, the retained block must average (α−β)/(1−β), which is strictly smaller than α for β > 0. As written, the loop therefore continues past the intended sparsity and sets additional p_i to 1 until the retained-block average reaches α, overshooting the target. For α_orig = 31/32 (CR=32) and β = 0.6, the intended retained average is 0.9219, while the printed loop terminates only when it reaches 0.96875, so the actual U/V sparsity exceeds the intended value and the resulting compression ratio differs from the reported CR. The condition should be (1/r)Σ p_k < (α−β)/(1−β). This affects every CR in Tables 1, 5, 6, and 7, because Algorithm 1 maps a requested α to the per-vector sparsity ratios used throughout the paper.","section":"Appendix A.1, Algorithm 1, line 7"},{"comment":"The sparsity allocation and the compression-ratio derivation are presented only for square matrices ('For simplicity, we only present the case of square matrices' in A.1 and B.2), but all LLaMA layers used in the experiments are rectangular (e.g., the MLP projections are 4096×11008 or 5120×13824). For an m×n weight, the factor storage is r(m+n) for the r unpruned singular components, not 2n^2, and the relationship between per-matrix sparsity and the overall delta sparsity α changes with the aspect ratio m:n. The paper does not specify how sparsity ratios are allocated across the left and right singular vectors when m ≠ n, nor how CR is computed in that case. Because the abstract's '2× higher compression ratio' and the CR columns in Tables 1 and 5 depend on this quantity, the central quantitative claim is not anchored unless a correct rectangular-case accounting is provided and verified.","section":"Appendix A.1 and Section 3.2"},{"comment":"The '2× higher compression ratio than baselines at the same performance level' claim is not backed by a direct iso-performance measurement. The tables report performance at fixed CR values, not the CR needed to reach a given performance level. For example, Table 5 shows that at CR=32 IMPART scores 60.20 on GSM8K while DARE scores 56.63; the CR at which DARE would reach 60.20 is not reported, so the factor of 2 is an interpretation of the plotted curves rather than a measured quantity. This headline claim should either be supported by explicit iso-performance CR extraction (e.g., from fine-grained CR sweeps with interpolation) or softened to a statement about performance at matched CR.","section":"Abstract and Figure 1"}],"minor_comments":[{"comment":"The formula for p_k is ambiguous: it reads p_k = 1 − (σ_k/σ_1)^C · γ, which can be read as 1 − γ (σ_k/σ_1)^C, while Algorithm 1 line 5 uses p_i = (1 − (σ_i/σ_1)^C) γ. Please align the main-text formula with the algorithm.","section":"Equation (7)"},{"comment":"The derivation has a notational slip: E[\\hat W^{ft} X_j] should be E[\\sum_j \\hat W^{ft}_{ij} X_j], and the final sentence says the reconstructed embedding 'approximates the origin,' which should be 'approximates the original.'","section":"Section 3.3, Equation (9)"},{"comment":"The validation set for mathematics is written 'SV AMP'; this appears to be SVAMP (a well-known math word problem benchmark).","section":"Section 5.2"},{"comment":"The theoretical result only establishes preservation of the expectation of the hidden state, not of the full output distribution or variance. The text in Section 1 that re-scaling 'ensures the performance is maintained' is stronger than what the proof supports; please soften this to 'preserves the expectation of the output.'","section":"Section 1 and Section 3.3"},{"comment":"The method name is spelled inconsistently as IMPART-QT and IMPART-Qt; please use one convention throughout.","section":"Tables 6 and 7"}],"recommendation":"major_revision","confidential_remarks":"The core idea is simple and the experiments are extensive, but the two technical gaps (the Algorithm 1 sparsity bug and the missing rectangular-case CR accounting) directly affect the headline compression-ratio claims. I would ask the authors to provide the exact sparsity allocation code and a precise storage model for rectangular weight matrices, and to re-run or re-derive the reported CRs accordingly. If the corrected numbers change the 2× claim, the abstract and figures should be revised. No circularity or novelty concerns from my side."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, the useful part: the idea of allocating different sparsity rates to singular vectors according to singular value magnitude is a natural and sensible extension of DARE's drop-and-rescale. The ablations show the importance-aware component matters, and the expectation-preservation proof is correct as far as it goes (it only covers the mean, not variance, but that is a minor limitation). The integration with Delta-CoMe-style quantization and with TIES merging is a reasonable bonus.\n\nWhere it falls down: Algorithm 1 in Appendix A.1 does not do what the rest of the paper says. After updating α to (1+α)/2, the γ in line 4 is chosen so the average sparsity over the retained block is exactly (α−β)/(1−β), which makes the overall average α. But the while loop then checks whether the retained-block average is below α, not below (α−β)/(1−β). Since (α−β)/(1−β)<α for β>0, the loop always fires and keeps setting p_i=1 until the retained-block average reaches α. That overshoots the overall target. For the paper's own CR=32 setting (α=31/32) and β=0.6, the printed algorithm produces an overall average sparsity of 0.99375, which is CR=80, not 32. So the tables report compression ratios the algorithm as written cannot produce, and every comparison against DARE and LowRank at CR=32 is apples-to-oranges. This is not a nitpick: the paper's headline is the 2x compression-ratio advantage.\n\nSecond, the whole derivation and Algorithm 1 are square-matrix only, while LLaMA linear layers are mostly rectangular (e.g., 4096x11008). There is no stated rule for how sparsity is allocated when U and V have different shapes, nor what the compression ratio formula becomes. The reader has to guess, and the reported CRs are not reproducible.\n\nMinor: baselines are only DARE and rank-truncation; no error bars; the allocation schedule is a heuristic with hyperparameters tuned on validation. Those are normal for this subfield, though they limit the strength of the SOTA claim.\n\nNet: the core method is plausible and likely worth pursuing; the ablation evidence suggests the importance-aware allocation does real work. But the central quantitative claim needs the algorithm fixed and the tables redone (or actual CRs reported). I'd send to serious peer review because the idea is useful and the issues are fixable, but I would not cite the current version.","headline":"A sensible SVD-based delta-sparsification idea with solid ablations, but the printed allocation algorithm overshoots the target sparsity, so the headline compression ratios are not anchored.","tokens_in":17275,"tokens_out":7787,"would_cite":false,"duration_ms":68587,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ImPart claims that importance-aware SVD-space sparsification roughly doubles the compression ratio of fine-tuned LLM deltas at the same accuracy, and that this transfers to quantization and merging.","keywords":["delta compression","sparsification","singular value decomposition","model merging","delta quantization","importance-aware pruning","large language models"],"falsifier":"Run Algorithm 1 on a real rectangular LLaMA layer, such as an MLP up-projection with $m \\neq n$, at a target compression ratio of 32 and count the stored nonzeros including the deterministic-mask overhead; if the realized compression falls below the reported value or below DARE's realized ratio at the same accuracy, the central $2\\times$ claim is refuted.","tokens_in":16215,"feed_emoji":"🗜️","tokens_out":9015,"duration_ms":91795,"temperature":0.7,"pith_summary":"ImPart's claim is that the right place to sparsify a fine-tuned model's delta is the SVD domain, and the right question is which singular vector matters, not which entry is large. By assigning each singular-vector pair a drop probability that rises as its singular value falls, with a long tail pre-pruned entirely and survivors rescaled by $1/(1-p_k)$, the method keeps the reconstructed output unbiased in expectation. The reported consequence is a compression ratio about twice as high as DARE or LowRank sparsification at the same task accuracy, plus gains when the sparsified deltas are quantized or merged. If correct, ImPart would make it cheaper to store, serve, and combine many task-specific models from a shared backbone.","feed_headline":"ImPart doubles delta compression at equal model quality","feed_subtitle":"By pruning singular vectors by importance, ImPart keeps task models accurate even at 93.75% sparsity.","key_machinery":"The engine is an adaptive allocation of sparsity ratios to singular-vector pairs. After writing the delta as $\\Delta W = U\\Sigma V^{\\top} = \\sum_{k=1}^{n} \\sigma_k U_k V_k^{\\top}$, the method assigns pair $k$ a drop probability $p_k = (1-(\\sigma_k/\\sigma_1)^C)\\gamma$ for the retained singular components, sets $p_k=1$ for the pre-pruned long tail beyond rank $\\lfloor n(1-\\beta)\\rfloor$, and rescales surviving entries by $1/(1-p_k)$. Independent Bernoulli masks then randomize $U_k$ and $V_k$, and the rescale gives $\\mathbb{E}[\\hat{h}] = h$, so the sparsified delta is an unbiased approximation of the original in expectation. This formula converts singular-value magnitude into per-vector sparsity while keeping the target global sparsity $\\alpha$; the allocation algorithm is written out for square matrices in the appendix.","core_discovery":"The paper introduces ImPart and claims it is a state-of-the-art delta-sparsification method. Its central claim is that sparsifying delta parameters in SVD space with per-singular-vector sparsity ratios set by singular-value importance, rather than DARE's random dropping or LowRank's coarse truncation, preserves task-specific abilities at high sparsity. On GSM8K, MATH, HumanEval, MBPP, IFEval, and AlpacaEval across LLaMA2-13B, LLaMA2-7B, and LLaMA3-8B, ImPart outperforms DARE by 4.01 points on average and LowRank by 2.11, and at the same performance level reaches about $2\\times$ the compression ratio. The same mechanism, combined with $\\Delta$-CoMe, gives ImPart-QT the best delta-quantization scores, and used before Task Arithmetic or TIES merging it improves merged-model averages.","pith_inferences":["An implication not drawn in the paper: since the rescaling makes the reconstruction unbiased in expectation, the observed quality loss at high sparsity should be governed by the variance of the randomized reconstruction, so measuring that variance per layer could predict which layers need lower sparsity without a validation sweep.","The square-matrix presentation of Algorithm 1 means the $2\\times$ claim should be re-derived for rectangular LLaMA layers; a direct check is to count stored nonzeros after applying Algorithm 1 to a real MLP or attention delta and compare with the reported compression ratio.","The authors note that validation-set hyperparameters can be misaligned with test tasks, so a natural follow-up is to test whether a single $\\beta,C$ pair learned on math transfers to code and chat, which would remove the per-task tuning cost."],"forward_implications":["At the same accuracy level, ImPart achieves roughly twice DARE's compression ratio on GSM8K and IFEval and about three times on HumanEval, so the same model quality can be stored in about half the space.","At a compression ratio of 16 (93.75% sparsity), the sparsified model retains 95.8% of the fine-tuned model's average performance, indicating that very sparse deltas remain practically usable.","ImPart-QT sets the best combined sparsification-plus-quantization numbers in the paper: 36.98 average versus 35.17 for Delta-CoMe, 36.12 for DARE-QT, and 32.71 for BitDelta at $CR_{qt}=32$.","Pre-sparsifying with ImPart before Task Arithmetic or TIES merging improves merged-model averages, so delta compression and model merging are compatible rather than competing goals.","Because the sparsity mask can be regenerated from the singular value's random seed, no mask storage is needed, so the reported compression ratio is not inflated by side information."],"supporting_citations":[{"why":"Defines the DARE random-sparsification baseline and the drop-and-rescale strategy that ImPart adapts to singular vectors.","marker":"Yu et al., 2024"},{"why":"Supplies the LowRank/ERE truncation baseline and the observation that small singular components can be discarded.","marker":"Ryu et al., 2023"},{"why":"Provides Delta-CoMe, the SVD-space mixed-precision quantization method that ImPart-QT builds on and compares against.","marker":"Ping et al., 2024"},{"why":"Provides BitDelta, a quantization baseline that ImPart-QT must beat and that DARE-QT is constructed from.","marker":"Liu et al., 2024"},{"why":"Supplies the GPTQ column-wise quantization routine that ImPart-QT extends to sparse weight matrices.","marker":"Frantar et al., 2023"},{"why":"Defines Task Arithmetic, the first merging strategy into which ImPart is integrated.","marker":"Ilharco et al., 2023"},{"why":"Defines TIES-Merging, the second merging strategy and sign-election baseline for the merging experiments.","marker":"Yadav et al., 2023"}],"fun_headline_variants":["ImPart: Importance-aware sparsification doubles LLM delta compression","SVD-based ImPart prunes by importance, retains task skills at high sparsity","ImPart's importance-aware SVD sparsification improves delta merging","ImPart: 2x compression via per-singular-vector sparsity","Importance-aware delta sparsification: ImPart beats DARE and LowRank"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the sparsity-allocation scheme and its compression-ratio formula, which the paper writes out only for square matrices, extend to the rectangular MLP and attention layers of the LLaMA models tested without changing the reported ratios.","fun_headline_variants_meta":{"raw":{"variants":["ImPart: Importance-aware sparsification doubles LLM delta compression","SVD-based ImPart prunes by importance, retains task skills at high sparsity","ImPart's importance-aware SVD sparsification improves delta merging","ImPart: 2x compression via per-singular-vector sparsity","Importance-aware delta sparsification: ImPart beats DARE and LowRank"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000178,"raw_usage":{"total_tokens":1275,"prompt_tokens":904,"completion_tokens":371,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":520,"completion_tokens_details":{"reasoning_tokens":270}},"tokens_in":520,"tokens_out":371,"duration_ms":4237,"temperature":1.0,"reasoning_tokens":270,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:16:11.637837+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run Algorithm 1 on a real rectangular LLaMA layer, such as an MLP up-projection with $m \\neq n$, at a target compression ratio of 32 and count the stored nonzeros including the deterministic-mask overhead; if the realized compression falls below the reported value or below DARE's realized ratio at the same accuracy, the central $2\\times$ claim is refuted.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides Delta-CoMe, the SVD-space mixed-precision quantization method that ImPart-QT builds on and compares against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines TIES-Merging, the second merging strategy and sign-election baseline for the merging experiments."}],"review_version":1}