{"id":"240b4f87-7d3d-47a2-82e6-fb1661a9f271","arxiv_id":"2412.14426","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"ATP jointly searches for pruning decisions and fine-tunes LLaMA models with LoRA in one stage, outperforming two-stage pruning on domain-specific tasks.","lead":"This paper introduces ATP, a method that combines structural pruning and fine-tuning of large language models into a single step, instead of pruning first and then fine-tuning. It reports better performance than two-stage pruning on legal and healthcare tasks, keeping up to 91% of the dense model's performance while removing 40% of parameters.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim hinges on pruned LoRA groups actually converging to zero, but no quantitative verification is provided; direct removal after training remains unvalidated.","rationale":"I read the paper in good faith. The core idea—jointly learning pruning decisions and LoRA adaptation—is plausible, and the experimental results are consistent with that plausibility. However, the method's key promise is that the final model can be obtained by directly removing pruned groups. That promise depends on the LoRA-aware sparsity regularization actually zeroing out the LoRA weights in pruned groups. The paper does not verify this: there are no norm measurements, no sparsity convergence plots for the LoRA groups, and no ablation comparing the model before vs. after physical removal. The loss curves in Fig. 5(b) show dynamics of Lgl but not its final magnitude relative to task loss. The text in Section 4.3 and Algorithm 1 line 13 take this convergence as given. This is a load-bearing gap because if the pruned LoRA weights are not negligible, the physically pruned model differs from the model that was optimized, and the reported task performance may overstate the compressed model's true quality. The reader's weakest_assumption identifies exactly this issue, and I agree. Other concerns (the large divergence from D-Pruner's published LLM-Pruner numbers, missing error bars, limited baselines) are real but secondary: they affect the fairness or strength of the comparison, not whether the method's own mechanism works as described. The appropriate verdict remains CONDITIONAL: the core idea deserves a conditional acceptance pending the missing quantitative verification. This does not change the reader's verdict, so verdict_should_be is UNCHANGED.","tokens_in":18868,"tokens_out":5175,"duration_ms":44675,"concrete_test":"Re-run ATP on LLaMA2-7B at p=0.4 in the HealthCare setting. After training, (1) measure the L2 norm of every LoRA row group in (I−Dprev)Wa and column group in Wb(I−D); report the maximum and mean over pruned groups relative to the norm of unpruned groups. (2) Evaluate the trained model before structural removal (i.e., with W masked by D and LoRA intact) on the four HealthCare tasks. (3) Physically remove the pruned groups and re-evaluate. If the after-removal relative performance drops by more than 1–2% compared to before-removal, or if the pruned-group norms are not orders of magnitude smaller than unpruned groups, the direct-removal assumption fails and the reported ATP results are not representative of the compressed model.","verdict_should_be":"UNCHANGED","load_bearing_attack":"ATP's defining guarantee is that the LoRA-aware sparsity regularization (Eq. 5) drives the LoRA weights associated with pruned structures to zero, so that the pruned groups can be directly removed (Algorithm 1, line 13) without further tuning. This is the linchpin of the one-stage advantage: if the pruned LoRA groups are not negligible, the physically pruned model after removal differs from the model used during training, and the reported task performance may not reflect the actual compressed model. The paper provides loss curves (Fig. 5b) but never quantitatively reports the L2 norms of the pruned LoRA groups, the final group sparsity achieved, or an ablation comparing the masked-training model (before removal) with the structurally pruned model (after removal). Section 4.3 asserts that freezing G and increasing β are 'crucial to facilitate the coverage of the pruned portions of LoRA weights towards zero,' but this is an assertion, not evidence. Without this verification, the central claim that ATP enables direct extraction of a compressed model with minimal performance loss is not established. All other concerns—baseline divergence, missing error bars, limited baselines—affect the strength of the comparison but not the internal validity of the method; this one affects whether the method does what it claims.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ATP, a one-stage method that jointly performs structured pruning and LoRA-based fine-tuning of domain-specific LLMs. A trainable pruning-decision generator G produces per-layer binary masks for attention and MLP dimensions, and the model is trained with a LoRA-aware masked forward pass plus group-lasso regularization on LoRA weights, so that pruned groups can be removed at the end without further tuning. Experiments on LLaMA2-7B and LLaMA3-8B in healthcare and legal domains compare ATP against LLM-Pruner and SliceGPT, reporting perplexity, NLI accuracy, QA macro-F1, summarization ROUGE scores, and a relative performance metric. The central claim is that dynamic one-stage pruning beats the two-stage pipeline, recovering 88% and 91% of dense-model performance at 40% sparsity on LLaMA2-7B and LLaMA3-8B, respectively.","tokens_in":19149,"tokens_out":6370,"duration_ms":49620,"significance":"ATP targets a genuine limitation of two-stage pruning pipelines: pruning decisions are made on pretrained weights and then frozen during fine-tuning, even though weight updates can change which substructures are important. The one-stage formulation with a trainable decision generator and LoRA-aware sparsity regularization is well motivated and potentially useful for deploying domain-adapted LLMs under limited data. The paper provides a complete algorithm, uses held-out evaluation corpora (HarrisonTextBook and MultiLegalPile) that are explicitly excluded from training and calibration, and includes sparsity sensitivity experiments and case studies. If the direct-removal mechanism is quantitatively validated, ATP would be a strong baseline for domain-specific LLM compression. At present, however, the key mechanism that makes the one-stage approach work is asserted rather than demonstrated, and the experimental comparison lacks uncertainty quantification.","major_comments":[{"comment":"The guarantee that pruned LoRA groups can be directly removed is asserted but not quantitatively verified. Eq. (5) penalizes rows of (I−Dprev)Wa and columns of Wb(I−D); Section 4.3 and Figure 5(b) show loss curves, but the paper never reports the post-training L2 norms of these groups, the fraction of groups that actually converge to zero, or the final achieved structural sparsity. Also absent is an ablation comparing the masked model used during training (forward pass fL, Eq. (4)) with the physically pruned model produced at line 13 of Algorithm 1. Without such evidence, the reader cannot tell whether the evaluated compressed model is the same function as the trained model, which is the defining premise of the method.","section":"§3.4–3.5, Eq. (5), Algorithm 1 line 13"},{"comment":"The forward-pass cross-references are inconsistent. Eq. (6) states that fG(sc; dall) applies Eq. (4) to all projections, and Eq. (7) states that fL(st; dall) applies Eq. (3), but Eq. (3) defines fG and Eq. (4) defines fL. If implemented literally, the generator would be trained with the LoRA-tuning forward pass and the LoRA weights would be updated with the generator forward pass, swapping the roles described in Section 3.4 and in Algorithm 2. The authors should correct these references and state explicitly, for each loss, which forward pass is used.","section":"§3.5, Eqs. (6)–(7)"},{"comment":"All sampled metrics are averaged over three evaluation runs, but Table 1 reports only point estimates with no standard deviations or confidence intervals. On several rows the differences between ATP and the better baseline are small or reversed on individual tasks (e.g., Table 1, LLaMA2-7B p=0.4, MedNLI accuracy 71.52 vs 72.29 for SliceGPT; LLaMA3-8B p=0.5, MedNLI accuracy 68.57 vs 75.04 for SliceGPT), so the headline claim that ATP greatly outperforms both baselines is not yet supported by uncertainty-aware comparisons. Reporting variances or significance tests is needed to support the comparative claim.","section":"Appendix A.2, Table 1"},{"comment":"The reported LLM-Pruner numbers diverge substantially from the values reported by D-Pruner (e.g., perplexity 13.67 vs 44.56 for HealthCare at p=0.5 on LLaMA2-7B). The authors attribute this to different loss modeling and hyperparameters, but no sensitivity analysis is provided to quantify how the baselines' scores change under the altered loss modeling or to justify that the modified baselines still represent the original methods fairly. Because the main comparative claim rests on these numbers, the divergence needs to be quantified rather than explained by assertion.","section":"Appendix A.4"}],"minor_comments":[{"comment":"Algorithm 1 uses Dc for the calibration dataset while Section 3.5 refers to a small calibration dataset Ds; please use one symbol consistently.","section":"Algorithm 1 / §3.5"},{"comment":"In Eq. (6), fG(dc; dall) uses a semicolon while fG is defined as a function of X in Eq. (3); the argument ordering should be aligned for clarity.","section":"§3.5, Eq. (6)"},{"comment":"Eq. (8) is not the standard Gumbel-Sigmoid parameterization; the role of the offset base b inside the sigmoid should be explained, and the chosen temperature and offset values (T=0.4, b=3) should be justified in the hyperparameter discussion.","section":"Appendix A.1, Eq. (8)"},{"comment":"The phrase 'state-of-the-art two-stage pruning methods' is stronger than the evidence: only LLM-Pruner and SliceGPT are compared, while several methods cited in Related Work (e.g., LLM-Surgeon, DisP-LLM) are not evaluated.","section":"§4.1"},{"comment":"Model sizes should be reported with a note on what is counted (e.g., embeddings included or excluded), since the ATP sizes at p=0.5 are not exactly 50% of the dense sizes and this is not explained.","section":"Table 1"},{"comment":"The periodic pattern in LLM_G is attributed to itertools.cycle over the calibration dataloader; this means calibration batches are always seen in the same order, which may bias G updates, and the authors should briefly discuss whether this affects the final pruning decisions.","section":"Appendix A.5"}],"recommendation":"major_revision","confidential_remarks":"No additional editorial concerns beyond those in the report. I would encourage the authors to release code and the exact pruning utilities, since the central ablation (masked model vs physically pruned model) is straightforward to check only with their implementation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper proposes ATP, a one-stage method that intertwines structural pruning decisions with LoRA fine-tuning, instead of the usual prune-then-tune pipeline. That is genuinely new for LLMs and a sensible idea, and the LoRA-aware forward pass plus group-lasso regularization are concrete design choices that make the one-stage idea workable. The empirical story is consistent: across LLaMA2-7B and LLaMA3-8B at 40–50% sparsity, ATP beats the two two-stage baselines on domain-specific perplexity and downstream tasks, with held-out evaluation sets that avoid obvious circularity.\n\nThe weak point is exactly the one the stress-test flagged. The paper's defining claim is that the pruned LoRA groups are driven to zero so that direct structural removal at line 13 of Algorithm 1 costs nothing. The text asserts this in Section 4.3 (\"freezing G and increasing β are crucial...\") and shows loss curves in Figure 5b, but never reports the L2 norm of pruned LoRA groups, the achieved group sparsity, or an ablation comparing the masked model before removal with the physically pruned model after removal. Without that, the reported task performance might be for a model that is not the one you actually deploy. This is a load-bearing gap, though fixable: one table of numbers would settle it.\n\nThe other concerns are proportionally smaller. Only two baselines (LLM-Pruner, SliceGPT) is thin but acceptable for a first domain-specific study. No error bars despite three runs is annoying but not fatal. The appendix's divergence from D-Pruner's published LLM-Pruner numbers is honestly disclosed and plausibly explained by the different loss modeling—though it does mean readers cannot cross-check against prior numbers. No code release makes all of this harder to verify.\n\nBottom line: the idea is worth taking seriously and the experiments support the comparative claim if the direct-removal step holds. The paper deserves a serious referee, not a desk reject, but the referee should demand the missing verification and ideally code and error bars. I'd bring it to reading group as a useful example of a one-stage pruning-and-tuning design, but I wouldn't cite it as a method until the zero-convergence claim is backed by numbers.","headline":"One-stage pruning with LoRA tuning is a real idea with promising results, but the paper's core guarantee—that pruned LoRA groups vanish so direct removal is safe—is asserted, not shown.","tokens_in":19694,"tokens_out":1725,"would_cite":false,"duration_ms":13967,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"One stage beats two for pruning domain-specific LLMs, claims ATP","keywords":["structural pruning","LoRA","domain-specific LLMs","one-stage tuning","pruning-decision generator","group lasso","LLM compression","parameter-efficient fine-tuning"],"falsifier":"Measure the ℓ2 norm of the LoRA weight groups (I−D_prev)W_a and W_b(I−D) at the positions the final pruning decision marks for removal, right after training. If these norms are not orders of magnitude smaller than the norms of retained groups, then the direct-removal step is not actually justified, and ATP's performance would be expected to drop if one instead re-ran fine-tuning with those groups kept.","tokens_in":18677,"feed_emoji":"✂️","tokens_out":3924,"duration_ms":29036,"temperature":0.7,"pith_summary":"The paper argues that the standard two-stage recipe for compressing domain-specific LLMs—first prune the pretrained model, then fine-tune it—is suboptimal because pruning decisions are frozen while the weights change. ATP instead runs pruning and LoRA-based fine-tuning together, with a trainable generator that revises pruning decisions as the LoRA weights evolve, and a sparsity regularizer that pushes the weights slated for removal toward zero. The result is a compact, domain-tuned model that needs no second-stage fine-tuning after pruning. On legal and healthcare benchmarks, ATP recovers up to 88% (LLaMA2-7B) and 91% (LLaMA3-8B) of dense-model performance at 40% parameter removal, beating the two-stage baselines LLM-Pruner and SliceGPT. If correct, this changes how domain-specific LLM compression should be done: the search for what to cut and the tuning of what remains belong in the same optimization loop.","feed_headline":"One stage beats two for pruning domain LLMs","feed_subtitle":"ATP couples pruning choices with LoRA tuning, recovering up to 91% of dense performance on legal and healthcare tasks.","key_machinery":"The load-bearing object is the pruning-decision generator G: a small Transformer-encoder network whose outputs are converted to binary decisions by Gumbel-Sigmoid sampling with a straight-through estimator. Alongside it, the LoRA-aware group-lasso regularizer (Eq. 5) forces the rows and columns of the LoRA matrices that correspond to pruned dimensions toward zero, so that direct removal of the pruned groups (Algorithm 1, line 13) is valid. The two forward passes (Eq. 3 for generator training, Eq. 4 for LoRA tuning) let the same decisions serve both the search for structure and the adaptation of weights.","core_discovery":"ATP establishes, on its own terms, that pruning decisions and LoRA tuning should be co-optimized rather than applied sequentially. The central mechanism is a trainable pruning-decision generator that outputs binary masks via Gumbel-Sigmoid sampling with a straight-through estimator, coupled with two LoRA-aware forward passes: one that simulates the pruned model for training the generator, and one that masks only the pretrained weights so the LoRA parameters stay fully trainable. A group-lasso term on the LoRA weights at the pruned positions drives those groups toward zero, so at the end the pruned structures can be removed directly without further fine-tuning. Experimentally, ATP achieves the best relative performance among structural-pruning methods on HealthCare and Legal tasks, with only minor exceptions on label-prediction tasks such as MedNLI and PubMedQA.","pith_inferences":["The same joint search-and-tune idea could extend to other parameter-efficient adapters (e.g., DoRA, prefix tuning) or to full fine-tuning, as the paper itself sketches; the key requirement is that the adapter has identifiable groups to regularize.","Because the generator's decisions shift by 20–55% when tuning is added (Fig. 3), this suggests that frozen-decision pruning generally leaves performance on the table for any fine-tuned deployment, not only domain-specific ones.","A testable extension: run ATP with the calibration set replaced by random samples from the general domain to see whether the decision evolution is driven by domain alignment or just by weight movement during tuning."],"forward_implications":["Domain-specific LLM compression can skip the separate recovery fine-tuning stage, saving compute and simplifying the deployment pipeline.","Pruning decisions should be treated as part of the optimization, not a precomputed artifact, whenever the weights are going to be updated afterward.","The 40–50% sparsity range becomes a practical operating point for legal and healthcare models, with relative performance of 75–91% of the dense model.","The framework transfers to any transformer whose linear projections can be grouped, since the decisions only select rows and columns."],"supporting_citations":[{"why":"Provides the LLM-Pruner baseline, the state-of-the-art two-stage structural pruning method that ATP must beat.","marker":"(Ma et al., 2023)"},{"why":"Provides the SliceGPT baseline, another two-stage structural pruning method compared against ATP.","marker":"(Ashkboos et al., 2024)"},{"why":"Introduces LoRA, the parameter-efficient fine-tuning method that ATP integrates with pruning.","marker":"(Hu et al., 2021)"},{"why":"Supplies the Gumbel-Sigmoid reparameterization used by the pruning-decision generator to produce discrete masks.","marker":"(Jang et al., 2016)"},{"why":"Supplies the straight-through estimator used for backpropagation through the binary pruning decisions.","marker":"(Bengio et al., 2013)"},{"why":"Sets the experimental setup (datasets, evaluation metrics, training schema) that ATP adopts for domain-specific pruning.","marker":"(Zhang et al., 2024b)"}],"fun_headline_variants":["ATP: joint pruning and LoRA tuning recovers up to 91% dense performance","Prune and tune in one go: ATP hits 91% of dense LLM on domain tasks","One-stage ATP beats two-stage pruning for domain LLMs","Prune and tune jointly: ATP recovers 91% of dense LLM"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim that after ATP ends, the LoRA weights in pruned positions are small enough that removing them directly causes no meaningful performance loss; the paper does not quantitatively verify the magnitude of those removed weights.","fun_headline_variants_meta":{"raw":{"variants":["ATP: joint pruning and LoRA tuning recovers up to 91% dense performance","Prune and tune in one go: ATP hits 91% of dense LLM on domain tasks","One-stage ATP beats two-stage pruning for domain LLMs","Prune and tune jointly: ATP recovers 91% of dense LLM"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000938,"raw_usage":{"total_tokens":4021,"prompt_tokens":968,"completion_tokens":3053,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":584,"completion_tokens_details":{"reasoning_tokens":2966}},"tokens_in":584,"tokens_out":3053,"duration_ms":18088,"temperature":1.0,"reasoning_tokens":2966,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:14:45.845638+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the ℓ2 norm of the LoRA weight groups (I−D_prev)W_a and W_b(I−D) at the positions the final pruning decision marks for removal, right after training. If these norms are not orders of magnitude smaller than the norms of retained groups, then the direct-removal step is not actually justified, and ATP's performance would be expected to drop if one instead re-ran fine-tuning with those groups kept.","supporting_citations":[],"review_version":1}