{"id":"805fed96-726b-406a-be54-a7c3a9771b1a","arxiv_id":"2501.09767","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"LeMo reduces long-context fine-tuning memory by eliminating low-informativeness tokens, predicting sparsity patterns, and optimizing kernels, while keeping perplexity close to LoRA.","lead":"LeMo is a fine-tuning system that skips unimportant tokens during long-context training to cut GPU memory use by up to 1.93x and speed up training by up to 1.36x. It targets the growing need to fine-tune large language models on very long inputs without expensive hardware.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Predictor recall is measured only on the base model's attention patterns, not during fine-tuning; if the proxy degrades as LoRA updates shift the true sparsity pattern, the claimed accuracy preservation is unsupported.","rationale":"The reader's weakest_assumption is exactly the predictor staleness concern: 'no experiment tracks whether predictor recall stays at 95.13% as training progresses.' I agree. The central claim requires that the token sparsity pattern predicted at each layer remains a reliable proxy for token importance under LoRA weight updates. This is not established by the paper. I considered other candidate concerns: (1) mixed LongBench results could weaken the 'no meaningful accuracy loss' wording, but the paper's headline accuracy metric is perplexity, and the LongBench results are presented as comparable rather than uniformly better. (2) The 'first' claim is clouded by reference [65] from the same group, but that is a novelty defense issue, not a correctness risk. (3) Lack of code/data is serious for reproducibility but is already captured by the CONDITIONAL verdict. The predictor-proxy drift is more load-bearing because it targets the mechanism underlying both efficiency and accuracy: if the proxy is stale, the system's memory savings are real but the accuracy results are not explained. The proposed test is concrete and feasible: it reuses the paper's own definition of informativeness and recall metric, and it directly measures whether the proxy remains valid during the training process that the system is designed to support. Therefore the verdict should remain CONDITIONAL until the test is run; ACCEPT would require the evidence, and REJECT would overstate what we can conclude from the current paper.","tokens_in":21121,"tokens_out":1732,"duration_ms":19371,"concrete_test":"Run LeMo on Llama2-7B at sequence length 8K following the paper's protocol. At 0%, 25%, 50%, 75%, and 100% of fine-tuning steps, freeze the model and compute (a) true informativeness scores from Eq. 1 using current LoRA weights, and (b) predictor-predicted scores. Measure recall at the layer thresholds used by LeMo. If recall stays near 95% throughout, the concern is resolved. If recall drops materially while final perplexity is still reported as acceptable, then the system tolerates proxy drift. If recall drops and perplexity degrades beyond reported values, the central accuracy-preservation claim fails.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central accuracy claim—that eliminating low-informativeness tokens preserves model quality—depends on the predictor-based token elimination tracking the true optimum as the model is fine-tuned. Section 4.3 trains lightweight predictors to approximate informativeness scores I(S_attn) defined in Eq. 1, and the paper reports an average recall of 95.13% (Section 6.3). But that recall is computed against ground-truth patterns derived from the base model's attention scores. During LoRA fine-tuning, Q and K projections shift, so the true ground-truth sparsity pattern changes with training step. No experiment tracks whether predictor recall remains at 95.13% as training progresses, or whether the layer-specific thresholds learned offline remain valid later in fine-tuning. If the predicted pattern becomes stale, the elimination could discard tokens that have become informative, inflating the reported perplexity increases and undermining the 'no meaningful accuracy loss' claim. The ablation studies measure memory, speed, predictor size, and predictor training loss, but none measures the proxy gap over the course of fine-tuning. This is the most load-bearing concern because both the efficiency and the accuracy arguments pass through the same predictor.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces LeMo, a system for long-context LLM fine-tuning that reduces activation memory by eliminating low-information tokens from attention and MLP computation. Token informativeness is defined from attention-score column sums (Eq. 1), tokens are eliminated block-wise with layer-specific thresholds (Algorithm 1), and the resulting sparsity patterns are approximated by lightweight per-layer predictors (Section 4.3) so that full attention scores need not be materialized. Kernel optimizations include permutation-free token movement and a segment-based method for the final loss gradient. The evaluation reports up to 1.93x memory reduction and up to 1.36x speedups over LoRA/LongLoRA on OPT and Llama models, with perplexity and LongBench results presented as showing no meaningful accuracy loss.","tokens_in":21395,"tokens_out":8284,"duration_ms":82531,"significance":"If the accuracy claims hold, the paper makes a useful systems contribution: it identifies a token-level sparsity axis for fine-tuning that is complementary to hidden-dimension sparsity, and it validates memory and speed effects with direct measurements on several GPU platforms and model families. The kernel-level ablations, the scalability measurements, and the explicit overhead accounting are concrete strengths. However, the central accuracy-preservation claim currently rests on single-run, mixed task-level scores and on an unvalidated assumption that predictors trained on base-model attention remain accurate throughout LoRA fine-tuning, so the significance is conditional on additional evidence.","major_comments":[{"comment":"The LongBench evaluation consists of a single run per task and shows mixed task-level outcomes: gov_report drops from 27.44 to 25.92, qmsum from 22.64 to 20.33, and lcc from 71.28 to 70.32, while musique and 2wikimqa improve. No confidence intervals, significance tests, or aggregate statistics are reported, so the claim in §6.2 that LeMo achieves accuracy comparable to the original LoRA is not supported by the evidence as presented. Please report multiple seeds, error bars, and an aggregate summary (for example, mean and standard deviation across tasks, or per-category averages) so readers can assess whether the observed drops are within noise.","section":"§6.2, Table 6"},{"comment":"The reported predictor recall of 95.13% is computed against ground-truth informativeness derived from the base model's attention scores, but the predictors are used throughout LoRA fine-tuning while Q and K projections shift. The paper does not track whether recall, or the predicted sparsity pattern, stays accurate as training progresses, nor whether the layer-specific thresholds optimized in Algorithm 1 remain valid after weight updates. Because both the efficiency and the accuracy claims pass through this predictor, please add an experiment that re-measures predictor recall (or the overlap between predicted and true eliminated token sets) at several fine-tuning checkpoints, and report whether thresholds need to be re-tuned.","section":"§4.3, §6.3"},{"comment":"The definition of token informativeness as the column sum of raw pre-softmax attention scores is an assumption that is never directly validated. The paper shows that predicted scores approximate these raw scores, but it does not show that eliminating tokens with low scores preserves loss or downstream accuracy better than, say, random or fixed-position elimination at the same sparsity. An ablation with random or fixed baselines at matched token-elimination ratios would establish that the proposed score, rather than mere sparsity, is responsible for the reported accuracy behavior.","section":"§4.2, Eq. (1)"},{"comment":"The segment-based peak-cutting method is described as processing each segment independently and later aggregating gradients, which is ambiguous: if each segment's loss is computed from a forward pass that starts at the segment boundary, the autoregressive context from earlier tokens is lost and the fine-tuning objective changes; if instead the prefix is recomputed or a KV cache is carried over, the memory-savings accounting needs to state that explicitly. Please clarify the implementation and provide a semantic-equivalence or loss-match check against the non-segmented objective.","section":"§4.4"}],"minor_comments":[{"comment":"The paper uses both 'LeMo' (title, abstract) and 'LEMO' (body, figures); please standardize the capitalization.","section":"Title and Abstract"},{"comment":"The 'Comprehensive Overhead Analysis' discusses 'online inference', but the predictors are used during fine-tuning; the terminology should be 'runtime' or 'online fine-tuning'.","section":"§4.3"},{"comment":"No code or artifact link is provided; given the systems nature of the contribution, providing one would aid reproducibility.","section":"General"},{"comment":"References [68] and [69] are duplicate entries of the same paper; please remove one.","section":"References"},{"comment":"The phrase 'long-hanging fruit' should be 'low-hanging fruit'.","section":"§1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable fit for a systems/ML venue. The novelty claim relative to [65] (Long Exposure) is credible because that work addresses hidden-dimension sparsity, while LeMo targets token-level sparsity for fine-tuning. The main risk is the accuracy evidence, which needs strengthening before acceptance. I would not object to a revised version that adds the predictor-drift experiment, statistical reporting on LongBench, and a validation of the informativeness proxy; the efficiency measurements themselves are solid."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"LeMo is a credible systems contribution. The idea of exploiting token-level sparsity during long-context fine-tuning to cut activation memory is well-motivated, and the reported 1.93x memory reduction with doubled sequence length is consistent across OPT and Llama models. The kernel work—permutation-free token movement and segment-based peak cutting—looks solid, and the speedups, while modest, are believable. I'd give the authors credit for the careful memory breakdown and the MLP extension.\n\nThe soft spots are mostly on the accuracy side. LongBench results are single runs with no error bars, and several tasks drop noticeably (gov_report 27.44 to 25.92, qmsum 22.64 to 20.33). The predictor recall of 95.13% is computed against ground-truth patterns from the base model, before fine-tuning; nothing in the paper shows whether that recall holds as LoRA shifts the Q/K projections. If the sparsity pattern drifts, the elimination could discard tokens that matter. This is a missing experiment, not a demonstrated failure—the perplexity numbers on PG19 and Proof-Pile are reassuringly close to LoRA—but it's the load-bearing assumption, and it deserves a direct test.\n\nThe threshold tuning in Algorithm 1, using finite-difference gradients of validation accuracy, is a form of fitting to the validation set. Not necessarily disqualifying, but it means the 'no accuracy loss' result is partly calibrated on the evaluation data. The 'first' claim is also softer than stated: reference [65], with overlapping authors, already applies shadowy sparsity to PEFT, so the novelty boundary with that work needs to be drawn explicitly. No code or data are released, which makes independent verification harder.\n\nWho is this for? Systems and ML efficiency researchers working on long-context fine-tuning. The paper is worth a serious referee: the system is plausible, the evaluation is substantial, and the concerns I listed are addressable. I'd ask for code/data, confidence intervals on LongBench, a tracking of predictor recall and sparsity fidelity during fine-tuning, and a clearer comparison to Long Exposure.","headline":"Plausible systems paper with real memory savings, but the accuracy claim rests on an underexamined predictor; worth refereeing with requests for artifacts and a drift analysis.","tokens_in":21911,"tokens_out":2882,"would_cite":true,"duration_ms":27405,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that long-context fine-tuning can be made far cheaper by identifying and dropping low-information tokens, and presents LeMo, a system that turns this token-level sparsity into up to 1.93x memory savings and up to 1.36x…","keywords":["long-context fine-tuning","activation memory","token sparsity","contextual token sparsity","LoRA","attention sparsity","parameter-efficient fine-tuning","GPU memory optimization"],"falsifier":"Run a Llama2-7B fine-tuning run at 16K sequence length and, every 500 steps, compare the tokens LeMo's predictors eliminate against the tokens with the highest true informativeness scores under current weights; if the recall of eliminated-but-truly-informative tokens falls materially below the reported 95.13% as training progresses, or if a full-attention control model and LeMo diverge beyond the reported perplexity differences on LongBench, the claim that elimination is accuracy-preserving would be contradicted.","tokens_in":20917,"feed_emoji":"🧠","tokens_out":7395,"duration_ms":69287,"temperature":0.7,"pith_summary":"LeMo is a fine-tuning system built on the claim that long-context sequences contain tokens that are redundant for learning, and that those tokens can be excluded from attention and MLP computation without meaningful accuracy loss. The reason this matters is that activation memory, not model parameters, is the main bottleneck when extending context windows: every token that participates in a layer leaves an activation footprint even if attention is sparse, an effect the paper calls \"shadowy activation.\" LeMo directly reduces token involvement, which cuts activation memory and, as a side effect, speeds up training. The system identifies informative tokens with attention-score sums, predicts sparsity patterns with small neural networks, and uses kernel-level optimizations to make the elimination cheap. Evaluations on OPT and Llama families across three GPU platforms report memory reductions up to 1.93x, speedups up to 1.36x, and only small perplexity increases relative to LoRA.","feed_headline":"Dropping low-value tokens cuts fine-tuning memory 1.93x","feed_subtitle":"Pruning low-value tokens during training lets one GPU fine-tune sequences twice as long without hurting accuracy.","key_machinery":"The load-bearing object is the token informativeness score, $I(T_j) = \\sum_{i\\neq j} S_{ij} = \\sum_{i\\neq j} Q_i K_j$, the column sum of raw pre-softmax attention scores: a token is considered redundant when its aggregate interaction with all other tokens is low. LeMo applies this score block-wise, takes the maximum positive attention score within each block, sums across blocks by column, and compares the result to a layer-specific threshold; the same idea is applied to MLP activations. Because computing full attention scores is expensive, each layer gets a pair of small low-rank ReLU predictors that map token embeddings to approximate query and key block scores, and their product estimates the block informativeness; an elastic size transformation prunes predictor neurons that are always zero. The supporting system machinery is the permutation-free kernel, which fuses selection, padding, and residual addition to avoid global memory movement, and segment-based gradient computation, which splits loss-gradient computation to cut activation memory peaks.","core_discovery":"The paper's central claim is that token-level sparsity, which it names Contextual Token Sparsity, exists and is exploitable during long-context fine-tuning: the tokens that matter are a small subset of the sequence, and this subset shifts with the input text and with the model layer. It defines a token's informativeness as the column sum of raw pre-softmax attention scores, aggregates these scores in blocks, and eliminates the low-scoring blocks using thresholds tuned per layer. Two lightweight predictors per layer estimate the informative scores of query and key blocks from contextual embeddings, so the full attention matrix never has to be materialized. LeMo then makes the elimination practical with a permutation-free kernel that avoids global memory movement and a segment-based scheme that cuts the activation peak from loss gradient computation. The paper reports that this design reduces memory consumption by up to 1.93x and achieves up to 1.36x speedups over state-of-the-art fine-tuning, with perplexity on PG19 and Proof-Pile and LongBench accuracy staying close to standard LoRA.","pith_inferences":["Nothing in the paper tracks whether the predictor's 95.13% recall holds as LoRA weights drift during fine-tuning; a direct test would be to measure overlap between predicted and true sparsity patterns at every training step, and if recall decays as weights move far from the base model, the accuracy guarantees would need re-validation.","The same token-level sparsity could plausibly be applied during continued pre-training or instruction tuning on 128K-plus contexts, where activation memory grows linearly; the paper does not evaluate that regime, but the mechanism does not depend on LoRA specifically.","Because LeMo changes which training tokens the model sees, it may act as a regularizer or curriculum that alters what the model learns beyond the perplexity numbers reported; comparing downstream task behavior of LeMo-tuned and dense-tuned models would separate sparsity effects from efficiency effects.","Combining LeMo with activation compression or recomputation could give multiplicative reductions beyond the reported 1.93x; the authors state compatibility but do not measure those combinations."],"forward_implications":["LeMo doubles the maximum sequence length that fits on one GPU for the tested models: OPT 1.3B goes from 16K to 32K and OPT 350M from 32K to 64K without recomputation or offloading.","Activation memory savings grow with sequence length, averaging 38.2% at 4K and 50.5% at 8K against LoRA, because longer sequences are more redundant.","Fine-tuning becomes faster as well as cheaper, with up to 1.36x end-to-end speedup and up to 2.04x when token sparsity is combined with hidden-dimension sparsity.","The overhead of detecting sparsity is small enough to be practical: predictors reach 95.13% average recall, are pruned to about a third of their original size, and add negligible time and memory.","LeMo composes with existing memory techniques instead of competing with them: the paper notes compatibility with activation recomputation, offloading, and other optimization methods, so the gains can be stacked."],"supporting_citations":[{"why":"LoRA is the parameter-efficient baseline LeMo builds on and compares against; LeMo targets the activation memory that LoRA leaves untouched.","marker":"[28]"},{"why":"LongLoRA provides the shifted-sparse-attention baseline; its hidden-dimension sparsity is what LeMo contrasts with token-level sparsity.","marker":"[12]"},{"why":"FlashAttention is the attention kernel LeMo integrates its block-wise informativeness training into, yielding linear memory overhead.","marker":"[15, 16]"},{"why":"These prior studies are cited as evidence that lightweight predictors can approximate sparsity patterns quickly, supporting pattern prediction.","marker":"[44, 60, 65]"},{"why":"The shadowy-sparsity analysis in this prior work motivates the claim that hidden-dimension sparsity leaves activation memory unoptimized, and it supplies the 2D-sparsity extension.","marker":"[65]"},{"why":"These references ground the premise that natural language is redundant enough that a subset of tokens can carry the training signal.","marker":"[58, 66]"},{"why":"Prior attention-sparsity results show full attention can be approximated by a subset of token interactions, motivating contextual token sparsity.","marker":"[37, 41, 59, 61, 67, 77]"}],"fun_headline_variants":["Token sparsity cuts fine-tuning memory 1.93x","Smarter token pruning: 1.93x memory drop, 1.36x speedup","Contextual token sparsity enables longer context fine-tuning","LeMo: prune low-info tokens to double fine-tuning context"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The system assumes that a token's column sum of raw pre-softmax attention scores measures how much the model needs it, and that this ordering stays reliable as LoRA fine-tuning changes the weights; if the true set of informative tokens shifts during training, dropped tokens could take important learning signal with them.","fun_headline_variants_meta":{"raw":{"variants":["Token sparsity cuts fine-tuning memory 1.93x","Smarter token pruning: 1.93x memory drop, 1.36x speedup","Contextual token sparsity enables longer context fine-tuning","LeMo: prune low-info tokens to double fine-tuning context"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000167,"raw_usage":{"total_tokens":1287,"prompt_tokens":1008,"completion_tokens":279,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":624,"completion_tokens_details":{"reasoning_tokens":202}},"tokens_in":624,"tokens_out":279,"duration_ms":3423,"temperature":1.0,"reasoning_tokens":202,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:22:18.983978+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a Llama2-7B fine-tuning run at 16K sequence length and, every 500 steps, compare the tokens LeMo's predictors eliminate against the tokens with the highest true informativeness scores under current weights; if the recall of eliminated-but-truly-informative tokens falls materially below the reported 95.13% as training progresses, or if a full-attention control model and LeMo diverge beyond the reported perplexity differences on LongBench, the claim that elimination is accuracy-preserving would be contradicted.","supporting_citations":[{"cited_title":"Long expo- sure: Accelerating parameter-efficient fine-tuning for llms under shadowy sparsity","cited_arxiv_id":null,"evidence_quote":"The shadowy-sparsity analysis in this prior work motivates the claim that hidden-dimension sparsity leaves activation memory unoptimized, and it supplies the 2D-sparsity extension."}],"review_version":1}