{"id":"ab7fe612-db0b-4524-887b-ed72622b74e4","arxiv_id":"2411.09816","paper_version":6,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A shared low-rank basis plus sparse per-layer projections compresses transformer MLPs by up to 75% with small accuracy loss.","lead":"Researchers built a compression recipe that lets many layers of a neural network reuse the same small set of learned basis weights, with tiny per-layer sparse adjustments. It reports shrinking the largest layer blocks of vision and language transformers by roughly half to three quarters while keeping accuracy and language quality nearly unchanged.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 3 contradicts the 'negligible perplexity degradation' claim for Llama models: Llama-3.2-3B rises from 6.19 to 9.70 at a 50% MLP budget (+57% relative), and Llama-3.1-8B from 4.75 to 7.11 (+50%).","rationale":"The reader's CONDITIONAL verdict is appropriate and I do not move it. However, my load-bearing concern differs from the reader's weakest_assumption. The reader focused on whether local L2 reconstruction on a small calibration set is a sufficient proxy for global task performance. That is a legitimate risk, but it is an extrapolation risk: the paper's experiments provide indirect evidence that the proxy works reasonably. My concern is more direct: the paper's own reported numbers falsify the qualitative claim of 'negligible degradation' for Llama models at the advertised 50% MLP budget. This is an internal inconsistency, not a speculation about unobserved settings. It is load-bearing because the paper's central contribution is framed around practical, high-ratio MLP compression with minimal loss; if the loss is 50–57% relative perplexity on two of four LLMs, the practical claim is materially weaker. The fix is straightforward: either report the Llama results as moderate degradation and adjust the abstract/conclusion, or provide additional experiments (error bars, larger calibration data, multi-seed runs) demonstrating that the reported gap is a conservative artifact. The abstract's unsupported QAT and downstream-benchmark claims reinforce that the paper overstates its support but are secondary to the numerical contradiction in Table 3. Verdict remains CONDITIONAL: the method's technical core (shared basis + sparse projections + block-wise optimization) is plausible and the SVD comparisons are strong, but the paper must correct or contextualize the 'negligible' characterization before acceptance.","tokens_in":19837,"tokens_out":11152,"duration_ms":109549,"concrete_test":"Define 'negligible' a priori as, say, <10% relative perplexity increase, then apply it to Table 3: at the 50% budget only Gemma-2-9B passes; 3 of 4 LLM entries fail. Independently reproduce Llama-3.2-3B at 50% budget using multiple seeds and a held-out calibration set (e.g., a different 163,840-token sample from SlimPajama), and report mean and standard error of WikiText-2 perplexity. If the +3.5 ppl gap is stable, the 'negligible degradation' claim is empirically false and must be removed or substantially softened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that FiPS compresses LLM MLPs by 40–50% 'while maintaining perplexity with negligible degradation' is contradicted by the paper's own Table 3. At a 50% parameter budget, relative WikiText-2 perplexity increases are: Gemma-2-2B +12%, Gemma-2-9B +9%, Llama-3.2-3B +57%, Llama-3.1-8B +50%. Only Gemma-2-9B is arguably within a 'negligible' threshold; the Llama models show large, practically significant degradation. The abstract further claims downstream benchmark improvements and QAT results (3-bit FiPS on Gemma-2-2B) that appear nowhere in the full text. The method may still outperform SVD baselines—at 50% budget SVD gives 41.03 ppl on Gemma-2-2B versus 7.47 for FiPS—so the core decomposition is not invalidated. But the qualitative claim of negligible degradation, which is central to the paper's framing and conclusion, is not supported by the reported data for 3 of 4 models at the advertised budget. Either the characterization must be revised, or additional evidence (e.g., error bars, larger calibration sets) is needed to show the Llama results are conservative artifacts rather than genuine limitations.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FiPS, a post-training compression method for transformer MLPs that combines cross-layer parameter sharing, low-rank tensor decomposition, and sparsity. A group of MLP weight matrices is concatenated and decomposed as W_i = U V_i, where U is a shared dense basis and V_i are sparse layer-specific projections; the factors are initialized by truncated SVD and then optimized by block-wise L2 activation reconstruction (Eq. 1), with optional end-to-end fine-tuning. Experiments on DeiT-B and Swin-L report ImageNet-1k top-1 accuracy within 1 point of the original at 25-50% parameter budgets, and experiments on Gemma-2 and Llama-3 models report WikiText-2 perplexity at 50-60% MLP parameter budgets, with large gains over plain SVD baselines where those are reported.","tokens_in":20171,"tokens_out":3709,"duration_ms":37379,"significance":"If the central claims hold, fine-grained cross-layer parameter sharing is a practical and previously underexplored compression axis that can be stacked with pruning and quantization. The paper has several concrete strengths: it ships code, gives detailed hyperparameters and block-group configurations, includes latency and memory profiling for structured sparsity, and ablates initialization, sparsity method, calibration size, and compression groups. The ViT results, in particular, are internally coherent and show a consistent advantage over AAFM/GFM baselines. However, the LLM-perplexity claim of 'negligible degradation' is not supported by the reported numbers for most models, and the abstract promises QAT results that do not appear in the full text.","major_comments":[{"comment":"The claim that FiPS compresses LLM MLPs by 40-50% 'while maintaining perplexity with negligible degradation' is contradicted by the paper's own Table 3. At a 50% parameter budget, Llama-3.2-3B rises from 6.19 to 9.70 WikiText-2 perplexity (+57% relative) and Llama-3.1-8B rises from 4.75 to 7.11 (+50% relative); only the two Gemma-2 models are within roughly 10%. The central framing in the introduction and conclusion must be revised to reflect that Llama compression at the advertised budget incurs substantial perplexity loss, or additional evidence (e.g., larger calibration sets, multiple seeds, or evaluation on additional held-out corpora) must be provided to show that these are conservative artifacts rather than genuine limitations.","section":"§4.2, Table 3; §1; §7"},{"comment":"The abstract claims that 'Combined with Quantization-Aware Training (QAT), 3-bit FiPS on Gemma-2-2B achieves lower perplexity than 2-bit QAT alone while matching the same 8x compression' and also claims downstream-benchmark improvements, but no QAT experiments, 3-bit results, or downstream-benchmark tables appear anywhere in the full text. This is a missing-results problem, not a wording issue: either the experiments must be added or the abstract must be reduced to what the paper actually demonstrates.","section":"Abstract; §1; §4.2"},{"comment":"All headline results are reported as single runs without error bars or seed variance, while the method's hyperparameters (learning rates, calibration length, sparsity schedule, and block-group size) are selected using the same validation split or test set used for the reported numbers. Given the sensitivity to calibration size and training length shown in Figure 4c, and the large Llama perplexity gaps in Table 3, the paper should either report variance across multiple runs or explicitly state that all numbers are point estimates from a single hyperparameter selection; as written, the reader cannot distinguish systematic degradation from hyperparameter or calibration sensitivity.","section":"§4.1, §4.2, §5"}],"minor_comments":[{"comment":"The text states that 'RigL consistently outperforms both Dense and Static Sparsity across parameter budgets ranging from 10% to 50%,' but Table 4 shows Static Sparsity slightly outperforming RigL at the 40% and 50% budgets (81.48 vs. 81.50 and 81.70 vs. 81.65). The sentence should be corrected to reflect the actual comparisons.","section":"§5, Table 4"},{"comment":"The paragraph beginning 'Sparsity Distribution and MSE-loss' is duplicated verbatim; one copy should be removed.","section":"§5"},{"comment":"The SVD baseline is reported only for the two smaller models (Gemma-2-2B and Llama-3.2-3B), so the claimed advantage of FiPS over SVD is not directly demonstrated for Gemma-2-9B and Llama-3.1-8B; adding those columns would strengthen the comparison.","section":"Table 3"},{"comment":"The definition of parameter budget in §2 is phrased awkwardly ('the fraction of retained parameters after truncated SVD minus the nonzero parameters preserved in the decomposed factors after sparsification'); it should be reworded to state clearly that the budget is the ratio of total nonzero parameters in the decomposed representation to the original module's parameter count.","section":"§2, parameter-budget definition"}],"recommendation":"major_revision","confidential_remarks":"The core decomposition idea is sound and the ViT experiments are convincing, but the paper's abstract and introduction overstate the LLM results and promise QAT evidence that is absent. These are fixable within the manuscript's scope, so I recommend major revision rather than rejection. I would also encourage the editor to ask the authors to clarify the relationship between the abstract's 'up to 20%' LLM compression and the introduction's '40-50%' claim, since the two statements are inconsistent as written."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: FiPS is a legit method for compressing transformer MLPs, and its ViT results are strong. The mechanism—concatenate weights across a group of blocks, factor into a shared dense basis U and sparse per-layer V, initialize by SVD, then optimize block-wise reconstruction—is a clean combination of AAFM/GFM and MiniViT, and the paper carefully investigates concatenation order and group size. At 40% budget, DeiT-B gets 81.69 vs AAFM's 80.33 and GFM's 81.28; Swin-L at 40% reaches 85.69, within a point of the original. The ablations (SVD vs random init, GMP vs RigL vs static sparsity, sparsity level, calibration size) are thorough and support the design choices. That part is worth a serious look.\n\nThe soft spot is the LLM section and the abstract. The abstract and conclusion claim 'negligible degradation' at 40–50% budgets, but Table 3 shows Llama-3.2-3B going from 6.19 to 9.70 at 50% (+57% relative) and Llama-3.1-8B from 4.75 to 7.11 (+50%). Gemma-2-2B goes from 6.66 to 7.47 (+12%), and Gemma-2-9B from 5.59 to 6.08 (+9%). Only Gemma-2-9B is arguably negligible. The body does hedge—it says Llama models are 'more challenging to compress'—but the qualitative claim in the abstract and conclusion overstates what Table 3 shows. Also, the abstract promises QAT results and downstream benchmark improvements that do not appear anywhere in the full text. That is a factual mismatch needing correction.\n\nThe reader's worry about the calibration proxy is fair: 163k tokens of SlimPajama, block-wise L2 only, hyperparameters selected on the same validation split, single runs, no error bars. That is a real limitation, and it matters more for the Llama results. But it does not invalidate the core method: at 50% budget FiPS gets 7.47 perplexity on Gemma-2-2B versus 41.03 for pure SVD, and at 60% budget Llama-3.2-3B drops to 7.22. The technique is clearly working; the characterization needs to match the numbers.\n\nBottom line: this deserves a serious referee. The ViT results alone justify the time, and the LLM claims are fixable with a revised abstract, error bars or repeated runs, and ideally downstream evaluation. I would cite it for the FiPS formulation and the ViT experiments. Bring it to reading group if you want a concrete example of an abstract-vs-body disconnect in a paper that is otherwise methodologically sound.","headline":"FiPS delivers strong ViT compression with a clean shared-basis/sparse-projection method, but the abstract's 'negligible' LLM perplexity claim is contradicted by the paper's own Table 3.","tokens_in":20747,"tokens_out":2860,"would_cite":true,"duration_ms":24312,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that fine-grained parameter sharing, implemented as a shared dense basis with sparse per-layer projections learned by block-wise activation reconstruction, compresses transformer MLPs by 50–75 percent with negligible…","keywords":["model compression","parameter sharing","tensor decomposition","sparsity","vision transformers","large language models","SVD initialization"],"falsifier":"Run the DeiT-B 25%-budget FiPS recipe with the calibration set replaced by 2,560 images drawn from a single ImageNet superclass; if the resulting top-1 accuracy drops by more than about one point below the reported 80.64, the local reconstruction proxy is not tracking global accuracy.","tokens_in":19630,"feed_emoji":"🧩","tokens_out":8726,"duration_ms":73659,"temperature":0.7,"pith_summary":"The paper tries to show that fine-grained parameter sharing is a practical way to compress transformers: instead of reusing whole blocks or pruning individual weights, it factorizes a group of MLP weight matrices into one shared dense basis and sparse per-layer projections. The claim is that this structure reduces the MLP parameter budget to 25–40 percent of the original for Vision Transformers while keeping top-1 accuracy within about a point, and to 50–60 percent for large language models with only small perplexity loss. If true, compression no longer has to be a per-layer decision: entire groups of layers can lean on a common low-dimensional substrate, and the same pipeline can be stacked with quantization. The paper also reports that this block-wise optimization is cheap enough to run on one GPU in under an hour for ViTs and a few hours for smaller LLMs.","feed_headline":"Basis sharing compresses transformer MLPs to 25-40%","feed_subtitle":"DeiT-B and Swin-L lose under a point of top-1 accuracy at those budgets.","key_machinery":"The central object is the factorization $W_i = U V_i$ over a group of MLP layers: $U$ is a dense shared basis in $\\mathbb{R}^{d \\times r}$ and $V_i$ are sparse layer-specific projections in $\\mathbb{R}^{r \\times p}$, produced by concatenating the $W_i$ along the long axis, applying truncated SVD, and gradually pruning the $V$ slices to about 75 percent sparsity. The basis $U$ stays dense and shared; the $V_i$ carry each layer's identity as a sparse combination of basis neurons. The work it does is to turn parameter sharing into a tensor-decomposition problem: sharing happens in the left factor, individual expression in the right factor, and block-wise L2 activation reconstruction supplies the training signal that makes the factorization track the original network's function rather than just its weights.","core_discovery":"On its own terms, the discovery is that sparsity in the projection factor is what makes cross-layer neuron sharing work. Given weights $W_i$, FiPS concatenates them along the long axis into $W_s = [W_1; \\dots; W_N]$, performs a truncated SVD $W_s = U\\Sigma \\hat{V}$ to get a shared basis $U$, and writes each layer as $W_i = U V_i$, where $V_i = \\Sigma \\hat{V}_i$ is sliced and then sparsified to about 75 percent. The $V$ factors are optimized by minimizing $\\sum_i \\lVert W_i X_i - U V_i X_i \\rVert_2^2$ on calibration activations, with $U$ updated from the aggregated per-layer gradients. The paper's evidence is that this beats SVD-only and feature-mimicking baselines at matched budgets: at a 40 percent budget DeiT-B keeps 81.69 top-1 accuracy versus 81.85 for the original, and Gemma-2-9B at a 50 percent budget reaches 6.08 WikiText-2 perplexity versus 5.59 for the original. The central insight is that allowing each layer a sparse combination of shared neurons gives far more representational capacity per parameter than one-hot sharing, so sparsity is not a side constraint but the enabling mechanism.","pith_inferences":["The paper only compresses MLP blocks; a natural extension is to apply the same shared-basis/sparse-projection factorization to attention projection matrices, where the paper's own conclusion says similar gains are likely but offers no measurements.","The use of a small calibration set means FiPS's reported numbers are conditional on that proxy; a stress test on out-of-distribution calibration data would clarify how robust the method is to domain shift, which the paper does not run.","Because $U$ is dense and shared while the $V_i$ are sparse, a deployment could keep the basis resident in fast memory and stream the per-layer projections; the paper's 2:4-sparsity latency benchmarks hint at this benefit but do not isolate cache reuse of the shared basis.","If the block-wise L2 objective is the right proxy, the method should extend to other layer types and architectures with alternating projections and sparsity schedules; the paper only tests two ViT families and four LLM families."],"forward_implications":["At 40–50 percent MLP budgets, DeiT-B and Swin-L remain within roughly 0.2 points of original ImageNet top-1 accuracy, so the shared basis is not merely a low-rank approximation but a functionally faithful replacement.","FiPS matches or exceeds the more expensive global feature-mimicking baseline (GFM) while doing only block-wise optimization, suggesting the local objective is sufficient for these budgets.","The same recipe transfers to LLMs: Gemma-2 and Llama-3 MLPs keep WikiText-2 perplexity within roughly 1 point at a 50 percent budget, and a compressed 9B/8B model can match the perplexity of an uncompressed smaller model.","FiPS composes with quantization: 3-bit FiPS on Gemma-2-2B at 8x compression achieves lower perplexity than 2-bit quantization-aware training alone, so sparsity-induced sharing and low-bit quantization capture different redundancies.","Transfer learning on CIFAR-100, Flowers102, Pets, and iNaturalist shows that compressed bases adapt to new tasks nearly as well as the original network, sometimes exceeding it."],"supporting_citations":[{"why":"Supplies the AAFM and GFM baselines and establishes block-wise error minimization as the prior state of the art that FiPS builds on and compares against.","marker":"Yu & Wu (2023)"},{"why":"Provides the gradual magnitude pruning schedule that FiPS uses to sparsify the projection factors $V_i$.","marker":"Zhu & Gupta (2017)"},{"why":"Provides RigL, the dynamic sparse training method used in FiPS's optional end-to-end fine-tuning stage.","marker":"Evci et al. (2021)"},{"why":"Provides the neuron-growing idea that FiPS adapts by appending zeros rather than splitting when the SVD rank exceeds the model dimension.","marker":"Chen et al. (2016)"},{"why":"Provides the weight-decomposed low-rank setup whose scaling vectors FiPS uses in its ablation of neuron scaling.","marker":"Liu et al. (2024)"},{"why":"Supplies the taxonomy of static and dynamic sparsity methods that frame FiPS's sparse training choices.","marker":"Hoefler et al. (2021)"},{"why":"Supplies the tensor-decomposition background that justifies treating the concatenated weight matrices as a factorization problem.","marker":"Kolda & Bader (2009)"},{"why":"Supports the choice of SlimPajama/RedPajama as calibration data for LLM pruning, on which FiPS's LLM calibration pipeline relies.","marker":"Bandari et al. (2024)"}],"fun_headline_variants":["Sparse sharing unlocks cross-layer MLP compression in FiPS","FiPS: sparsity turns one-hot sharing into fine-grained sharing","FiPS compresses ViTs 33% and LLMs 20% via sparse sharing","Shared basis, sparse projections: FiPS compresses transformers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that minimizing block-wise L2 activation error on a small calibration set (2,560 ImageNet images or 163,840 SlimPajama tokens) preserves full-model accuracy and perplexity for every tested model and budget.","fun_headline_variants_meta":{"raw":{"variants":["Sparse sharing unlocks cross-layer MLP compression in FiPS","FiPS: sparsity turns one-hot sharing into fine-grained sharing","FiPS compresses ViTs 33% and LLMs 20% via sparse sharing","Shared basis, sparse projections: FiPS compresses transformers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000844,"raw_usage":{"total_tokens":3749,"prompt_tokens":1092,"completion_tokens":2657,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":708,"completion_tokens_details":{"reasoning_tokens":2580}},"tokens_in":708,"tokens_out":2657,"duration_ms":18504,"temperature":1.0,"reasoning_tokens":2580,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T20:18:00.993492+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the DeiT-B 25%-budget FiPS recipe with the calibration set replaced by 2,560 images drawn from a single ImageNet superclass; if the resulting top-1 accuracy drops by more than about one point below the reported 80.64, the local reconstruction proxy is not tracking global accuracy.","supporting_citations":[],"review_version":1}