{"id":"d3b20f1d-17e6-4537-89f4-7aa5da1f27b4","arxiv_id":"2411.10543","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A per-layer learnable soft-thresholding mechanism for SVD-based low-rank compression reduces language model parameters by 50% with roughly 1% average accuracy loss.","lead":"Researchers propose SoftLM, a way to shrink large language models by cutting out unimportant parts of each layer's weight matrix using a learnable 'soft threshold'. The method reportedly keeps accuracy nearly intact while reducing parameters by half and speeding up inference up to 1.72x.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Static SVD baselines for GPT-2/TinyLlama may not be fine-tuned, so the claimed adaptive-rank advantage could be an artifact of fine-tuning.","rationale":"The reader's weakest assumption focuses on whether the soft-threshold gradients faithfully reflect the rank-accuracy trade-off, which is a plausible concern but not directly falsifiable from the paper. My stress-test identifies a more directly checkable and load-bearing issue: whether the static SVD baselines for the generative models were fine-tuned. The BERT baseline is explicitly described as fine-tuned, but the GPT-2 and TinyLlama baselines are not, and the paper's wording leaves this ambiguous. If the static baselines were not fine-tuned, the experimental evidence for the central claim 'adaptive rank beats static rank' is invalid. Even if they were fine-tuned, the omission violates reproducibility standards. The concrete test I propose—re-running the static SVD baselines with the same fine-tuning budget—would settle the question. This does not change the reader's CONDITIONAL verdict, which already depends on missing code and details, but it sharpens the condition to include baseline fairness. I therefore keep the verdict unchanged rather than escalate, because the paper's BERT results provide some independent support for the adaptive mechanism even if the generative comparisons are flawed.","tokens_in":10287,"tokens_out":15136,"duration_ms":136231,"concrete_test":"Release code and checkpoints, then fine-tune SVD-GPT2-medium and SVD-TinyLlama using the identical fine-tuning schedule, optimizer, and total steps as SoftLM at the same parameter budgets, and compare WikiText-2 perplexity. Also report the exact value of s (Eq. 6) and γ (Eq. 10) used for each experiment. If fine-tuned static SVD matches SoftLM's perplexity, the adaptive threshold mechanism adds no measurable benefit beyond fine-tuning a decomposed model.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central comparative claim is that adaptive low-rank decomposition (SoftLM) outperforms static SVD at matched parameter budgets. For BERT, the text explicitly states SVD-BERT was fine-tuned; for GPT-2 and TinyLlama, Tables 4 and 5 do not state whether the static SVD baselines received the same fine-tuning procedure. If those baselines were not fine-tuned, their poor perplexity (e.g., 32.1 at CR 0.50 for GPT-2) reflects the lack of adaptation, not the inferiority of static rank assignment. SoftLM fine-tunes U, V, and Σ, which would improve any SVD-initialized model regardless of the adaptive threshold. The claimed gap could therefore be entirely due to fine-tuning. Reproducibility is further blocked by the unspecified sharpness parameter s in Eq. (6) and the absence of per-task γ values for Eq. (10). These gaps directly undermine the paper's headline claim because the comparison basis for 'outperforms static low-rank decomposition' is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes SoftLM, a post-training compression method for transformer and state-space language models. Each linear layer is replaced by an SVD factorization UΣV^T, and a differentiable soft-threshold function with a per-layer learnable threshold α is applied to the diagonal entries of Σ; an exponential compression loss encourages α to grow so that the number of surviving singular values, and hence the rank, is chosen automatically during fine-tuning. The threshold is frozen once the target parameter budget is reached, and the compressed factors are merged at the end of fine-tuning. The method is evaluated on BERT (GLUE tasks), GPT-2 and TinyLlama (WikiText-2 perplexity), and Mamba (GLUE tasks), reporting that adaptive rank allocation outperforms static SVD at matched budgets and yields 1.33–1.72× inference speedups at 50% compression.","tokens_in":10513,"tokens_out":8631,"duration_ms":82294,"significance":"If the results hold, this is a useful contribution to low-rank model compression: layer-wise ranks are learned during fine-tuning rather than chosen by an external search, and the approach transfers across attention-based and state-space architectures. The consistent improvements over static SVD across four model families, the compression-ratio ablations, and the attempt to make thresholding differentiable are concrete strengths. The main caveats are that the comparison may conflate fine-tuning with adaptive ranking, that the SVD semantics are not preserved after unconstrained fine-tuning of U, V, and Σ, and that several hyperparameters controlling the threshold dynamics are not reported. With those points addressed, the work would be a solid systems-oriented compression result; as it stands, the central comparative claim is not fully established.","major_comments":[{"comment":"The fine-tuning loop optimizes U, Σ, and V without any orthogonality constraint. After the first AdamW step, U and V are no longer orthogonal, so the diagonal entries of Σ are not singular values of the current weight matrix; the soft-threshold operation is then a diagonal mask on a factorized matrix rather than a truncation of an SVD. Because the paper motivates the method as clipping singular values and compares against truncated SVD, this discrepancy should be resolved either by enforcing orthogonality (e.g., re-orthogonalizing or parameterizing on the Stiefel manifold) or by explicitly reframing the method as a gated low-rank factorization initialized from SVD.","section":"§4.1 and §4.5, Eq. (5)"},{"comment":"The static SVD baselines for GPT-2 and TinyLlama are not described as fine-tuned; the text states only that static SVD applies the same compression ratio across all blocks, whereas for BERT the SVD baseline is explicitly said to be fine-tuned. If SVD-GPT2 and SVD-TinyLlama are not fine-tuned, their worse perplexities (e.g., 32.1 versus 19.6 at CR 0.50 for GPT-2) reflect the lack of adaptation rather than the inferiority of a static rank assignment. Please specify the baseline protocol and, if possible, include a static SVD baseline that receives the same U, V, Σ fine-tuning budget as SoftGPT2 and SoftTinyLlama.","section":"§5.2, Tables 4 and 5"},{"comment":"The sharpness parameter s in Eq. (6) controls the gradient of the soft threshold with respect to α and hence the rank-selection dynamics, but its value or schedule is never reported. Relatedly, Table 6 lists ranges for the threshold learning rate and for γ (called \"w\") rather than the values used for each dataset and model. Since the learned thresholds are the core mechanism that determines each layer's rank, these omissions prevent reproduction and make it impossible to assess sensitivity of the reported results.","section":"Eq. (6), §4.4, Table 6"},{"comment":"The abstract and execution-cost section claim a 50% reduction in total parameters, but the compression is applied only to the linear layers in the encoder/decoder (or to the in/out projections in Mamba), not to the whole model. For BERT the total size drops from 110M to 67M (about 39%), for Mamba from 130M to 87M (about 33%), and for GPT-2 from 345M to roughly 195M (about 43%); only Table 1's \"w/o emb\" row is a true 50% reduction. The reported speedups and memory savings should be restated on the same basis as the model totals.","section":"Abstract and §5.3"},{"comment":"All experimental tables report a single run per configuration, with no seeds, standard deviations, or statistical tests. In several places the reported advantage is small (e.g., SoftBERT G-Avg 82.6 versus DistilBERT 81.2, or SoftMamba 86.3 versus Mamba 87.5), so without variance information the reader cannot determine whether the observed differences are meaningful. Please report multiple seeds or otherwise justify that the comparisons are stable.","section":"Tables 1–5"}],"minor_comments":[{"comment":"There are typos and inconsistent notation: \"1.33X\" (Abstract) versus \"1.33×\" (Figure 8), \"upto\" (Section 1), \"encoder/ decoder\" (Abstract), \"compression ration\" (Section 5.2), and the regularizer is called w in Table 6 but γ in Eqs. (7) and (10).","section":"Throughout"},{"comment":"Table 3 compares against S4, but no S4 reference appears in the bibliography; please add it.","section":"Table 3"},{"comment":"Algorithm 1 ends with \"Ensure: #param in (U and V S) < #param in W\", but the post-merge shapes of U and V S are not explained; clarify how the dimensions are derived from the thresholded Σ.","section":"Algorithm 1"},{"comment":"Figure 1(b) would benefit from a label explaining what \"static\" means (uniform rank across blocks) and whether the ranks shown are before or after fine-tuning.","section":"Figure 1"},{"comment":"The fine-tuned SVD-BERT baseline is not described in terms of training budget; please state whether it uses the same number of epochs, learning rate, and fine-tuning procedure as SoftBERT.","section":"§5.2, Table 1"},{"comment":"The appendix lists batch sizes and learning rates but not the number of training epochs or the stopping criterion for the compression phase; please add these details so the procedure is fully specified.","section":"Appendix A.2"}],"recommendation":"major_revision","confidential_remarks":"The paper has a promising idea and the experiments are internally consistent, but the central comparative claim needs strengthening. I recommend major revision rather than rejection because the key issues—clarifying the static SVD baselines, addressing the loss of orthogonality, reporting sharpness and per-task hyperparameters, and correcting the parameter-reduction claims—are fixable within the manuscript's scope. The novelty should also be repositioned: without orthogonality constraints, the method is better described as a learned gated low-rank factorization initialized from SVD, and the authors should discuss how this differs from earlier dynamic-rank methods such as AdaLoRA and RankDyna."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know about this paper: the mechanism is plausible and the BERT results are useful, but the central 'adaptive beats static' claim for GPT-2 and TinyLlama rests on a baseline detail the authors never specify. The stress-test note is right — if those SVD baselines weren't fine-tuned, the comparison is a fine-tuning artifact.\n\nWhat's genuinely new: a differentiable soft threshold on singular values with a per-layer learnable alpha, paired with an exponential compression loss that tapers off. That combination isn't in the cited work, and extending it to Mamba as well as transformer blocks is a good test. The BERT comparison is honest: they state SVD-BERT was fine-tuned, and SoftBERT at 67M parameters beats DistilBERT and SVD-BERT on GLUE average. The appendix rank distributions give a useful picture of which layers tolerate compression.\n\nThe soft spots are real. Tables 4 and 5 don't say whether SVD-GPT2 and SVD-TinyLlama were fine-tuned, and that's not a small detail. SoftLM fine-tunes U, V, and Sigma; any SVD-initialized model would improve under fine-tuning. If the baselines were frozen, the perplexity gaps at matched parameter counts are meaningless. The paper also omits the sharpness parameter s in Eq. (6), the per-task gamma values, and any error bars. Single-run results are okay for a workshop paper, but for a claim that 'outperforms static low-rank decomposition' you need the control arm fully specified.\n\nI think the paper deserves peer review, not a desk reject — the method is interesting and the BERT evidence is credible — but the referee should ask for code, exact baseline protocols, and the missing hyperparameters before taking the GPT-2/TinyLlama numbers at face value. As written, I wouldn't cite it for the adaptive-vs-static claim. I'd bring it to a reading group, though, because the fine-tuning-vs-adaptation question is a good discussion.","headline":"The adaptive-rank idea is real and the BERT results are credible, but the GPT-2/TinyLlama baseline omission makes the headline comparison unverifiable as written.","tokens_in":11000,"tokens_out":3586,"would_cite":false,"duration_ms":30343,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a differentiable soft-threshold on singular values lets each linear layer learn its own rank during fine-tuning, so that compressing a language model to half its parameters costs about a 1% average accuracy drop and…","keywords":["low-rank approximation","singular value decomposition","learnable threshold","soft thresholding","model compression","fine-tuning","language models","efficient inference"],"falsifier":"Run the same fine-tuning on a small model with the compression loss omitted and the thresholds replaced by random fixed values; if the learned thresholds still match the best brute-force rank assignment at the same parameter budget, the gradient signal is not what drives the result. A more direct version is to enumerate integer ranks around each learned value on the validation set; if a nearby hand-picked assignment beats the learned model by a large margin across seeds, the soft threshold is not finding the true trade-off.","tokens_in":10113,"feed_emoji":"⚡","tokens_out":11792,"duration_ms":99342,"temperature":0.7,"pith_summary":"The method compresses a pre-trained language model by replacing each large linear weight with its SVD factorization and letting a single learnable threshold per layer decide which singular values survive. The threshold is optimized during fine-tuning through a differentiable soft-threshold function, so the network itself chooses each layer's rank under a compression loss. At a 50 percent reduction in total parameters, the paper reports 1.33 times to 1.72 times inference speed-ups, 33 to 44 percent memory savings, and roughly 1 percent average accuracy loss on GLUE for BERT. The central comparison is against static SVD with the same rank across layers: adaptive ranks give GPT-2 perplexity 19.6 versus 32.1 and TinyLlama 15.1 versus 20.4 at 50 percent compression. The paper's conclusion is that per-layer rank allocation, not the SVD itself, drives the gain.","feed_headline":"Soft thresholding learns each layer's rank, halving model size","feed_subtitle":"Learnable singular-value cutoffs beat static SVD at the same 50% parameter budget on BERT, GPT-2, TinyLlama, and Mamba.","key_machinery":"The machinery is the soft-thresholded SVD layer: each linear weight $W$ is decomposed once into $U\\Sigma V^T$, and the singular-value matrix is passed through $\\mathrm{Th}_s(x)$, which keeps values above the learnable cutoff roughly as $x\\tanh(s(x-\\alpha))$ and sends values below it to zero. A per-layer scalar $\\alpha_i$ is trained together with the model under total loss $L_{\\text{acc}}+\\gamma\\sum_i e^{-\\alpha_i}$, so the exponential compression term gives a large push early and nearly vanishes once a threshold is large. When the parameter count reaches the target, $\\alpha_i$ is frozen; at the end, $V\\cdot \\mathrm{Th}_s(\\Sigma)$ is merged into one matrix, so inference stores two smaller matrices instead of three. This makes rank selection a continuous, learned part of fine-tuning rather than a post-hoc static choice.","core_discovery":"The paper's central claim is that the optimal low-rank approximation of a language model is not a single rank applied to every layer but a per-layer rank chosen by the model itself during fine-tuning. By substituting each linear layer with $W \\approx U\\,\\mathrm{Th}_s(\\Sigma)\\,V^T$, where $\\mathrm{Th}_s$ smoothly clips singular values below a learnable threshold $\\alpha_i$, the training loss can push each layer's threshold up until the target parameter budget is met. Because the threshold is differentiable, gradient descent replaces exhaustive search over rank combinations. The evidence is the consistent gap over static truncation: SoftBERT reaches 90.7 F1 on MRPC versus 86.4 for SVD-BERT at the same 50% encoder compression, and the generative models show perplexity reductions of 12.5 points (GPT-2) and 5.3 points (TinyLlama) at CR 0.50. The paper also reports near-parity for SoftMamba, with a 1.2% average task degradation at 50% compression of the projection layers.","pith_inferences":["Because the paper leaves the sharpness parameter $s$ unspecified and ties the compression loss only indirectly to the parameter count, one direct test would be to read off the final rank each threshold produces and compare it with a brute-force per-layer rank search at the same parameter budget.","The reported gains come from fine-tuning on each target task; a natural extension is task-agnostic or few-shot compression, where the thresholds might need a calibration set to transfer.","The SVD-at-initialization cost could be reduced with randomized or incremental SVD, and if the gains persist the approach becomes practical for billion-parameter models without the fine-tuning memory overhead the paper reports."],"forward_implications":["If the central claim holds, static SVD compression of transformers should be replaced by adaptive rank allocation whenever fine-tuning is possible, because the same parameter budget buys substantially lower perplexity.","The method extends beyond attention models: Mamba's dominant projection layers can be halved with only about 1.2% average task degradation, so state-space language models are viable targets for this compression.","The 1.33–1.72× speed-up and 33–44% memory savings at CR 0.50 mean models compressed this way fit into tighter device budgets without task-specific distillation or quantization.","The learned rank distributions show earlier layers keep higher ranks, implying layer position is a real factor in rank sensitivity, not just matrix shape.","At higher compression ratios the gap over static SVD widens, so adaptive rank allocation is especially valuable under aggressive compression."],"supporting_citations":[{"why":"It supplies the attention encoder architecture whose linear layers are the compression targets.","marker":"Vaswani et al., 2023"},{"why":"It supplies the BERT-Base pre-trained model and the GLUE fine-tuning setup used in the BERT experiments.","marker":"Devlin et al., 2018"},{"why":"It supplies the GPT-2-medium decoder model and the WikiText-2 language modeling benchmark.","marker":"Radford et al., 2019"},{"why":"It supplies the Mamba state-space architecture whose in-projection and out-projection layers are compressed.","marker":"Gu and Dao, 2024"},{"why":"It defines the FWSVD weighted low-rank baseline that static compression must beat.","marker":"Hsu et al., 2022"},{"why":"It defines SVD-LLM, the truncation-aware static SVD baseline whose compression loss is extended here.","marker":"Wang et al., 2024"},{"why":"It defines RankDyna, the earlier dynamic-rank inference method whose gradient-memory overhead motivates the threshold approach.","marker":"Hua et al., 2023"},{"why":"It defines AdaLoRA, the dynamic-rank fine-tuning baseline that allocates rank during training rather than inference.","marker":"Zhang et al., 2023"},{"why":"It provides the singular value decomposition algorithm that produces the U, Sigma, and V factors used in every compressed layer.","marker":"Golub and Reinsch, 1971"}],"fun_headline_variants":["Soft thresholding picks per-layer ranks, halves model size","Differentiable soft thresholds choose each layer's rank","Per-layer rank via soft thresholding, 50% smaller models","Soft-threshold SVD: learnable rank, 1.33-1.72X speedup","Learnable thresholding automates per-layer compression, halves params"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the smooth soft-threshold function provides gradients that faithfully represent the true accuracy-versus-compression trade-off of each layer; if those gradients do not track the effect of dropping singular values, the learned thresholds are arbitrary and the gain over static SVD would not reproduce.","fun_headline_variants_meta":{"raw":{"variants":["Soft thresholding picks per-layer ranks, halves model size","Differentiable soft thresholds choose each layer's rank","Per-layer rank via soft thresholding, 50% smaller models","Soft-threshold SVD: learnable rank, 1.33-1.72X speedup","Learnable thresholding automates per-layer compression, halves params"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001154,"raw_usage":{"total_tokens":4796,"prompt_tokens":973,"completion_tokens":3823,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":589,"completion_tokens_details":{"reasoning_tokens":3731}},"tokens_in":589,"tokens_out":3823,"duration_ms":26893,"temperature":1.0,"reasoning_tokens":3731,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T19:35:04.927643+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same fine-tuning on a small model with the compression loss omitted and the thresholds replaced by random fixed values; if the learned thresholds still match the best brute-force rank assignment at the same parameter budget, the gradient signal is not what drives the result. A more direct version is to enumerate integer ranks around each learned value on the validation set; if a nearby hand-picked assignment beats the learned model by a large margin across seeds, the soft threshold is not finding the true trade-off.","supporting_citations":[],"review_version":1}