{"id":"d30c8153-c2cc-4aa4-b4c0-7f234aa21289","arxiv_id":"2508.02668","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"LOST pre-trains LLaMA models from 60M to 7B using an SVD-initialized low-rank plus channel-wise sparse decomposition and reports C4 perplexity at or below full-rank models with substantially smaller memory.","lead":"LOST is a pre-training method that splits each large weight matrix, using a one-time singular value decomposition, into a low-rank part plus a small channel-wise sparse part built from the discarded subspace. The authors report validation perplexity on C4 equal to or better than full-rank LLaMA training at 60M to 1B parameters while using roughly one quarter to one half of the memory.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central superiority claim rests on unreproduced full-rank baselines and single-seed runs, not on the frozen SVD mask.","rationale":"The paper is a coherent and well-ablated extension of sparse-plus-low-rank pre-training, with a released code repository and experiments across several model scales. The reader's fixed-mask concern is plausible but not the most load-bearing: the sparse component is only 1% of channels, and the paper's own Table 3 shows that the channel-selection strategy changes PPL by only about 0.1-0.2 at 60M and 130M. Even if the selected channels drift during training, the sparse component is a small complement, so the central result would not collapse. The more direct concern is the empirical comparison itself. The paper explicitly states that Full-Rank, LoRA, ReLoRA, GaLore, and SLTrain results are directly reported from prior work, and only LORO and CoLA were reproduced. The claimed margins over full-rank are small, and LOST runs appear to be single-seed. Under those conditions, the least-secure requirement for the central claim is that the inherited baselines are directly comparable and stable. The proposed check, rerunning the key baselines inside the LOST pipeline with multiple seeds, would settle this directly. If the baselines reproduce and the confidence intervals separate, the paper's central claim is credible; if not, the conclusion should be moderated. This does not change the reader's CONDITIONAL verdict; it sharpens the condition that should be imposed before the superiority claim is accepted.","tokens_in":13545,"tokens_out":19069,"duration_ms":219880,"concrete_test":"Using the released LOST repository, rerun Full-Rank, SLTrain, and LOST at 60M and 130M on C4 with identical token budgets, optimizer settings, and hardware, for three seeds each, and report mean and standard deviation of validation PPL. If the full-rank and SLTrain baselines reproduce the published values and LOST's mean lies outside their confidence intervals, the central claim stands; if the baselines shift or the confidence intervals overlap, the claim should be weakened to competitive rather than superior.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table 1's headline comparison is built on PPL numbers for Full-Rank, LoRA, ReLoRA, GaLore, and SLTrain that are reported as directly taken from prior work, with only LORO and CoLA reproduced in this paper. The claimed margins over full-rank are small: roughly 1.81 PPL at 60M, 0.31 at 130M, and 0.54 at 1B. Differences of this size are easily explained by tokenizer choice, batch size, learning-rate schedule, BF16 handling, or codebase differences between the inherited baselines and LOST's own runs. In addition, the LOST results appear to be single-seed, so no error bars are available for these margins. For the central claim that an SVD-initialized low-rank-plus-sparse model pre-trains as well as or better than full-rank, it must be true that the comparisons are matched and stable. That condition is the least secure point in the paper. The frozen-channel-mask concern identified by the reader is real but secondary: the sparse residual is only 1% of channels, and Table 3 shows that the SVD-selected mask beats a random mask by only about 0.1-0.2 PPL, so a drifted mask would not overturn the main result. The decisive uncertainty is whether the headline comparison to full-rank survives a fair, replicated rerun.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LOST, a method for pre-training LLMs from scratch using a sum of a low-rank component and a channel-wise sparse component. The low-rank factors are initialized from the top-r singular vectors/values of the initialized full-rank weight matrix, and the sparse channels are selected, once at initialization, from the residual subspace using an L2-norm importance score; the two branches are combined at the output level with a fixed trade-off coefficient gamma. The authors evaluate LOST on LLaMA-style models from 60M to 7B parameters trained on C4, reporting perplexity, parameter counts, and memory estimates, and they also test a fine-tuning variant on GLUE. The central claim is that LOST reaches lower or comparable validation perplexity than full-rank training while using fewer parameters and less memory.","tokens_in":13804,"tokens_out":4181,"duration_ms":45446,"significance":"If the central claim holds, LOST would be a practical contribution: it provides a parameter- and memory-efficient way to pre-train transformer LMs without a full-rank warm-up, with a clean SVD-based initialization that is performed once. The paper includes a thorough ablation study on the complementary matrix source, channel selection criterion, initialization scheme, activation placement, sparsity level, and trade-off coefficient, and it reports experiments up to 7B parameters plus fine-tuning results on GLUE. The method is clearly described and the code is promised to be released. However, the headline performance claim rests on a comparison whose full-rank and several baseline numbers are not reproduced in the same codebase, and no seeds or error bars are given; the margins at 130M and 1B are small enough that the central claim is not yet established by the evidence presented.","major_comments":[{"comment":"The headline comparison against full-rank training is not a matched comparison. The text states that only LORO and CoLA results are reproduced with their default scripts, while Full-Rank, LoRA, ReLoRA, GaLore, and SLTrain numbers are directly reported from [16, 20]. The claimed advantages over full-rank are 1.81 PPL at 60M, 0.31 PPL at 130M, and 0.54 PPL at 1B; at 350M LOST is actually 0.15 PPL worse than full-rank. These margins are comparable to typical cross-codebase differences in tokenizer, batch size, learning-rate schedule, and numerical format. To support the claim 'LOST achieves comparable or superior performance compared to full-rank models,' all baselines, especially full-rank, must be rerun in the same codebase and with the same training setup, or the comparison must be explicitly downgraded to a cross-paper reference.","section":"Section 4.1, Table 1"},{"comment":"No random seeds, replicates, or error bars are reported anywhere in Section 4. The main PPL differences over full-rank are small in absolute terms (0.31 at 130M, 0.54 at 1B), and without variance estimates it is impossible to know whether LOST's advantage is statistically meaningful. The authors should run the key comparisons (at least the 60M, 130M, and 1B models) with multiple seeds and report the mean and standard deviation, or at least the best-of-three with the spread.","section":"Section 4.1 / Experimental setup (no seeds)"},{"comment":"The 7B scaling claim is based on only 40K training steps (with LOST extended to 150K), while the paper's own Table 13 indicates a full epoch would require roughly 19.7B tokens. Moreover, in Table 11 the Full-Rank Adam baseline is run with batch size 4, whereas LOST runs with batch size 8, so the reported memory figures per GPU (49.53 GB vs 62.15 GB) are not comparable, and the full-rank run's instability after 12K steps is observed on a different batch size and a truncated schedule. A 40K-step comparison does not establish that LOST trains stably to convergence at 7B relative to full-rank. Please either extend the full-rank baseline to the same number of steps and batch size, or soften the scalability conclusion.","section":"Section 4.1, Table 2 and Appendix Table 11"},{"comment":"The sparse channel mask I is selected once from the SVD of the randomly initialized weight matrix and is never updated during training. The method therefore assumes that the channels that are most important at initialization remain the channels that are most important throughout training. The ablation in Table 3 shows that the specific choice of the complementary matrix changes PPL by only about 0.1–0.2 (e.g., 32.25 vs 32.35 at 60M), which undercuts the paper's emphasis on the 'complementary co-design' as the source of the improvement. The mechanism claim (contribution 2) would be strengthened by an experiment tracking how much the selected channels drift during training and whether an adaptive mask re-selection would improve performance.","section":"Section 3.2 / Algorithm 1, Step 2"}],"minor_comments":[{"comment":"The dimensions are inconsistent: the text defines W ∈ R^{m×n}, but Eq. (1) writes W ∈ R^{n×n} and the singular value matrix diag(σ_1,...,σ_n) ∈ R^{m×n}; please make the rectangular case explicit.","section":"Section 3.1, Eq. (1)"},{"comment":"The column header SVDrand_l2 appears twice; one of the entries is presumably intended to be a different variant (for example SVDrand_l1).","section":"Table 3"},{"comment":"The phrase 'low-rank LLM pertaining' should be 'pre-training'.","section":"Section 2.2"},{"comment":"The caption for Table 11 says 'for 40K steps' but the table reports results at 10K, 40K, 80K, 120K, and 150K; the caption should reflect the full step range.","section":"Section 4.1 / Table 11 caption"},{"comment":"The comparison between structured and unstructured sparsity is not completely fair because the unstructured variant has to be processed separately to accommodate the activation between low-rank factors; this extra requirement is described in the text, but the memory comparison would be clearer if the activation-free structured case were also reported.","section":"Appendix Table 12 and text"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable empirical contribution with a well-motivated initialization scheme, but the main 'surpasses full-rank' claim is not yet supported because the full-rank baseline and several other baselines are transcribed from other papers and the runs appear to be single-seed. This is fixable: rerun the baselines in the same codebase and add seeds. The 7B section is a short-run smoke test rather than a scaling study; the authors should either invest in a longer full-rank comparison or clearly label the 7B results as preliminary. I would not reject the paper, but it is not ready in its current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one thing to know: LOST is a solid, incremental contribution to efficient LLM pre-training. The new idea is to use the discarded singular subspace of the initial weight matrix to select which channels the sparse residual should keep, and to combine the low-rank and sparse components at the output level with an activation. That is a real difference from SLTrain, which initializes the two components independently, and from PiSSA/OATS, which target fine-tuning or compression. The ablations (Tables 3-9) are thoughtful and support the design choices: SVD-based channel selection beats random/top/bottom, SVD beats Kaiming for the low-rank factors, output-level combination plus activation helps, and the method is robust to the rank used for the complementary matrix.\n\nThe main result—that LOST matches or beats full-rank pre-training at 60M to 1B—is plausible but not fully established. The margins are small (0.31 PPL at 130M, 0.54 at 1B), the full-rank numbers are taken from prior papers rather than rerun, and there are no seeds or error bars. The stress-test note is right: the frozen SVD mask is a secondary concern because the sparse component is only 1% of channels and random channel selection costs only about 0.1-0.2 PPL. The decisive uncertainty is whether the comparison to full-rank survives a matched, replicated run. The 7B experiment is a 40K-step partial run, so the scaling claim is suggestive, not conclusive. The compute reduction claim is also asserted without wall-clock or FLOPs measurements, though the parameter and memory reductions are real and well-documented.\n\nThe paper is honest about its limitations and explicitly acknowledges the 7B ceiling. The hyperparameter choices (gamma=0.7, sparsity=0.01, Wcomp rank=256) were tuned on the same C4 validation for 60M/130M and then applied across scales; that is mild circularity, but it is a fixed choice, not a per-dataset fit.\n\nWho is this for? Anyone working on memory- or parameter-efficient pre-training, or on low-rank and sparse methods for transformers. It deserves a serious referee: the method is clearly presented, the ablations are careful, and the open questions (seeds, matched baselines, compute measurement) are fixable in revision.\n\nRecommendation: engage with it and send it to review. The central claim needs corroboration, but the work is not flawed at the load-bearing level.","headline":"A credible incremental method for low-rank plus sparse LLM pre-training with solid ablations, but the headline gains over full-rank rest on inherited baselines and single-seed runs.","tokens_in":14366,"tokens_out":3144,"would_cite":true,"duration_ms":31678,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Low-rank weights plus a frozen sparse channel mask can match full-rank LLM pretraining.","keywords":["LLM pretraining","low-rank training","sparse plus low-rank decomposition","SVD initialization","channel-wise sparsity","memory-efficient training","LLaMA"],"falsifier":"Train a 130M LOST model but re-run the SVD and channel selection every 5,000 steps using the current factored weights; if perplexity drops below the reported 24.05, the fixed mask is the load-bearing cost. A cheaper check is to replace the SVD-selected channel set with randomly chosen channels at initialization and compare perplexity: if random channels match, the complementary-spectrum selection is not the cause.","tokens_in":13330,"feed_emoji":"⚡","tokens_out":6834,"duration_ms":67739,"temperature":0.7,"pith_summary":"The paper sets out to show that large language models can be pre-trained from scratch without ever storing or updating a full-rank weight matrix: replace each weight with a low-rank factor plus a channel-wise sparse residual, choose both pieces from a single SVD of the initial matrix, and let gradient descent update them. Tested on LLaMA-style models from 60M to 1B on C4, LOST reports lower or equal perplexity than full-rank training at every scale except the 350M point, where it is close, while cutting memory nearly in half at 1B (3.66 GB vs 8.04 GB). A 7B run over 40K steps also reports lower perplexity than standard 8-bit Adam, and an 8-bit variant of the method halves memory again. If the claim holds, low-rank-plus-sparse weight decompositions are a viable substitute for full-rank weights during pre-training under tight memory budgets.","feed_headline":"SVD split makes low-rank LLM pretraining match full-rank","feed_subtitle":"Keeping top singular directions plus a few sparse channels halves memory at 1B scale.","key_machinery":"The load-bearing mechanism is a one-time SVD-based co-design of the two components. The low-rank factor is initialized from the top-$r$ singular vectors, so its product approximates the original weight in Frobenius norm; the sparse residual is initialized from the remaining singular subspace, so it carries directions orthogonal to the low-rank part instead of duplicating it. Channel-wise structure, keeping whole columns rather than element-wise entries, makes the store compact because only $k$ column indices need to be saved. The activation inserted between $A$ and $B$ adds nonlinearity without adding parameters. Together this preserves both the dominant subspace and a small set of high-energy output channels, and the frozen mask means the SVD cost is paid only once, before training.","core_discovery":"LOST's central discovery is that the information lost by truncating a randomly initialized weight matrix to rank $r$ does not need to be modeled densely: it can be captured by keeping $k$ whole output channels selected from the complementary subspace. Concretely, a single SVD of the Kaiming-initialized matrix $W$ gives $W_l = AB^T$ with $A = U_r \\Sigma_r^{1/2}$, $B = V_r \\Sigma_r^{1/2}$, with a SiLU activation between the two factors; the residual subspace $W_{\\text{comp}} = \\sum_{i>r} \\sigma_i u_i v_i^T$ is used to score each channel by its L2 norm, and the top $k = \\lceil \\rho n \\rceil$ channels are kept as $W_s = W[:, I]$. The forward pass is $o = \\gamma\\, \\sigma(xA)B^T + (1-\\gamma)\\, x[:,I] W_s^T$. With $\\gamma=0.7$, $\\rho=0.01$, and rank adjusted so the parameter count matches other low-rank baselines, LOST obtains 32.25 vs 34.06 perplexity at 60M, 24.05 vs 24.36 at 130M, 18.95 vs 18.80 at 350M, and 15.02 vs 15.56 at 1B, with estimated memory of 0.24, 0.57, 1.11, and 3.66 GB versus 0.35, 0.81, 2.21, and 8.04 GB for full-rank training.","pith_inferences":["Because the channel mask is frozen after initialization, the method implicitly assumes the SVD spectrum of the initial weight predicts which channels stay important through training; an adaptive re-selection schedule is a natural test, and if it helps, the fixed mask is a bottleneck rather than a feature.","The channel-wise structure may be a stronger design choice than the paper fully exploits: whole stored columns could combine directly with hardware-friendly structured sparsity kernels at inference time, something element-wise sparse methods cannot do without conversion.","The authors flag that models above 7B remain untested; if the one-time SVD split reproduces at 10B-plus scale, it would make from-scratch pretraining feasible on far smaller hardware, while longer training schedules would stress-test the frozen-mask assumption."],"forward_implications":["With the same token budget, LOST reports lower validation perplexity than full-rank training at 60M, 130M, and 1B, and effectively matches it at 350M.","The parameter count for a linear layer drops from $mn$ to $r(m+n) + mk$; at 1B this yields 3.66 GB estimated memory instead of 8.04 GB.","At 7B over 40K steps, LOST reports perplexity 16.48 versus 18.09 for 8-bit Adam, and 8-bit LOST uses 50.19 GB versus 72.59 GB.","Fine-tuning RoBERTa-base on GLUE with $r=7$ gives an average score of 86.51, above the full-size model's 86.28 in the paper's table, suggesting the decomposition transfers beyond pretraining.","Ablations show that selecting channels from the residual singular subspace outperforms alternatives derived from the largest, smallest, or random singular values."],"supporting_citations":[{"why":"Supplies the closest baseline, sparse-plus-low-rank pretraining with independent initialization, whose protocol and numbers LOST compares against.","marker":"[20]"},{"why":"Supplies the low-rank-gradient training baseline GaLore and the reported perplexity/memory numbers reused in Tables 1 and 2.","marker":"[16]"},{"why":"Supplies the source of inserting a nonlinear activation between low-rank factors and a baseline with reproduced results.","marker":"[37]"},{"why":"Supplies the low-rank decomposition paradigm and the LoRA baseline used in both pretraining and fine-tuning comparisons.","marker":"[2]"},{"why":"Supplies the warmup-based low-rank pretraining baseline ReLoRA.","marker":"[12]"},{"why":"Supplies the Riemannian low-rank pretraining baseline LORO with reproduced results.","marker":"[36]"},{"why":"Supplies the C4 dataset on which all pretraining runs are performed.","marker":"[41]"},{"why":"Supplies the LLaMA architecture that LOST modifies across all model scales.","marker":"[42]"}],"fun_headline_variants":["SVD-guided low-rank and sparse channels match full-rank LLM pretraining","Low-rank core plus sparse channels from SVD matches full-rank training","SVD split: keep low-rank core, sparse channels for cheaper LLM training","LOST: low-rank and sparse complement for full-rank LLM pretraining","Cheap LLM pretraining: SVD core plus sparse channels beats full-rank"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The sparse channel mask is fixed after initialization, so the whole method assumes that the channels that look important before training stay important all the way through.","fun_headline_variants_meta":{"raw":{"variants":["SVD-guided low-rank and sparse channels match full-rank LLM pretraining","Low-rank core plus sparse channels from SVD matches full-rank training","SVD split: keep low-rank core, sparse channels for cheaper LLM training","LOST: low-rank and sparse complement for full-rank LLM pretraining","Cheap LLM pretraining: SVD core plus sparse channels beats full-rank"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000797,"raw_usage":{"total_tokens":3620,"prompt_tokens":1173,"completion_tokens":2447,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":789,"completion_tokens_details":{"reasoning_tokens":2346}},"tokens_in":789,"tokens_out":2447,"duration_ms":15214,"temperature":1.0,"reasoning_tokens":2346,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:37:02.925815+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a 130M LOST model but re-run the SVD and channel selection every 5,000 steps using the current factored weights; if perplexity drops below the reported 24.05, the fixed mask is the load-bearing cost. A cheaper check is to replace the SVD-selected channel set with randomly chosen channels at initialization and compare perplexity: if random channels match, the complementary-spectrum selection is not the cause.","supporting_citations":[{"cited_title":"SLTrain: a sparse plus low rank approach for parameter and memory efficient pretraining","cited_arxiv_id":null,"evidence_quote":"Supplies the closest baseline, sparse-plus-low-rank pretraining with independent initialization, whose protocol and numbers LOST compares against."},{"cited_title":"Relora: High-rank training through low-rank updates","cited_arxiv_id":null,"evidence_quote":"Supplies the warmup-based low-rank pretraining baseline ReLoRA."},{"cited_title":"Parameter and memory efficient pretraining via low-rank riemannian optimization","cited_arxiv_id":null,"evidence_quote":"Supplies the Riemannian low-rank pretraining baseline LORO with reproduced results."}],"review_version":2}