{"id":"38ac61e0-8dbd-49a4-93d3-75873664e647","arxiv_id":"2604.13287","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"MOONSHOT is a wrapper framework that combines reconstruction loss and Taylor approximation objectives to improve one-shot pruning results on LLMs and vision models.","lead":"MOONSHOT turns existing one-shot pruning methods into multi-objective ones by jointly minimizing layer-wise reconstruction error and a second-order Taylor approximation of the loss. General readers might care because better pruning lets large models run faster and cheaper on limited hardware without retraining.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"The claim that jointly optimizing reconstruction error and second-order Taylor approximation consistently beats either objective alone rests on an unverified assumption about objective weighting and solver stability in the one-shot regime.","rationale":"The reader's weakest assumption directly identifies the hinge point of the argument. Because the review was abstract-only, the full methods section would be needed to check the concrete multi-objective formulation and any ablations against single-objective baselines; the proposed test isolates whether the combination itself drives the reported gains.","tokens_in":1839,"tokens_out":350,"duration_ms":28045,"concrete_test":"Reproduce the Llama-3.2 2:4 sparsity results using only the reconstruction objective and only the Taylor objective under identical hyper-parameters and Hessian approximation as MOONSHOT; if the multi-objective version does not improve C4 perplexity and zero-shot accuracy over the better single-objective baseline on all seven benchmarks, the joint-optimization benefit is not established.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central argument states that neither single objective is best across architectures and sparsity levels, motivating the multi-objective wrapper. For this to hold, the joint formulation must produce pruning decisions superior to the better of the two baselines without introducing instability from the combination (e.g., via an implicit or explicit trade-off parameter). In the post-training one-shot setting, where no retraining corrects errors, any suboptimal weighting or approximation in the combined objective could yield masks worse than the stronger single-objective method on some layers or models. The abstract provides no explicit multi-objective loss, Pareto procedure, or weighting scheme, leaving open whether gains arise from the multi-objective principle or from extra degrees of freedom and per-experiment tuning.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The manuscript proposes MOONSHOT, a wrapper framework that converts any single-objective one-shot pruning method into a multi-objective optimizer by jointly minimizing layer-wise reconstruction error and a second-order Taylor approximation of the training loss. It claims that this combination is necessary because neither objective alone is consistently superior across architectures and sparsity regimes, and reports that the resulting masks, when applied to Llama-2/3.2 models, reduce C4 perplexity by up to 32.6% at 2:4 sparsity and raise zero-shot accuracy by up to 4.9 points, while also delivering >5-point ImageNet gains on ViTs at 70% sparsity and 4-point gains on ResNet-50 at 90% sparsity. The paper emphasizes scalability via an efficient inverse-Hessian procedure that preserves the runtime of existing one-shot pruners.","tokens_in":1978,"tokens_out":801,"duration_ms":23121,"significance":"If the empirical claims are reproducible and the joint objective demonstrably outperforms the stronger of the two single-objective baselines without per-experiment retuning, the work would address a recognized limitation of current one-shot pruning and supply a practical, architecture-agnostic improvement. The reported magnitude of gains on both language and vision models at high sparsity would be noteworthy for post-training compression pipelines. However, the current text provides no explicit multi-objective loss, weighting scheme, or Pareto procedure, so the significance cannot yet be assessed.","major_comments":[{"comment":"Abstract and §3 (method description): the central claim that 'neither objective alone is consistently the most effective' is used to motivate the multi-objective wrapper, yet the manuscript supplies neither the explicit combined loss function nor the weighting or Pareto procedure that realizes the joint optimization. Without these, it is impossible to determine whether reported gains arise from the multi-objective principle or from additional degrees of freedom and per-layer tuning.","section":"Abstract, §3"},{"comment":"Abstract (experimental claims): the stated improvements (32.6% C4 perplexity reduction, 4.9-point zero-shot accuracy gain, >5-point ImageNet gain) are given without reference to the precise single-objective baselines, the number of random seeds, variance estimates, or the exact optimization procedure used to solve the joint objective. This absence prevents verification that the joint formulation is load-bearing for the gains rather than an artifact of experimental controls.","section":"Abstract"},{"comment":"§4 (experimental setup) and §5 (results): the paper asserts that MOONSHOT 'extends any single-objective pruning method,' but provides no ablation that isolates the contribution of the second-order term versus the reconstruction term across the same set of layers and sparsity targets. A direct comparison showing that the joint mask is strictly better than the better of the two single-objective masks on every model/sparsity pair is required to substantiate the motivating assumption.","section":"§4, §5"}],"minor_comments":[{"comment":"The abstract and introduction repeatedly use the phrase 'up to' for the largest reported gains; the corresponding tables or figures should also report the median or mean improvement across all evaluated sparsity levels to avoid selection bias.","section":"Abstract, §1"},{"comment":"Notation for the inverse-Hessian approximation and the layer-wise reconstruction loss should be introduced with explicit equations in §3 rather than described only in prose.","section":"§3"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is submitted to a machine-learning venue yet currently reads more like an extended abstract; the absence of the actual multi-objective formulation and of controlled ablations is the primary obstacle to a positive recommendation. If the authors supply the missing equations and ablations, the work could fit the journal's scope."},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the detailed and constructive review. The comments correctly identify areas where the presentation of the multi-objective formulation and supporting experiments can be strengthened. We address each major comment below and will incorporate the requested clarifications and additional results in the revised manuscript.","responses":[{"response":"We acknowledge that the explicit mathematical form of the joint objective was described only at a high level. In the revised manuscript we will add the precise scalarized loss L = λ · L_recon + (1-λ) · L_Taylor, where λ ∈ [0,1] is selected per layer by a small grid search that minimizes the combined objective on a held-out calibration batch. This is a standard weighted-sum approach rather than a full Pareto front; the search is performed once per layer and does not constitute per-experiment retuning beyond the hyper-parameter selection already required by the underlying single-objective pruners. The revised §3 will also include the closed-form solution for the resulting quadratic program.","revision_made":"yes","referee_comment":"[Abstract, §3] Abstract and §3 (method description): the central claim that 'neither objective alone is consistently the most effective' is used to motivate the multi-objective wrapper, yet the manuscript supplies neither the explicit combined loss function nor the weighting or Pareto procedure that realizes the joint optimization. Without these, it is impossible to determine whether reported gains arise from the multi-objective principle or from additional degrees of freedom and per-layer tuning."},{"response":"The reported numbers are relative to the stronger of the two single-objective baselines (reconstruction loss or Taylor approximation) for each model-sparsity pair. In the revision we will (i) state the exact baseline in every table and in the abstract, (ii) report means and standard deviations over three independent random seeds for mask generation where stochasticity exists, and (iii) explicitly describe the per-layer quadratic-program solver that uses the efficient inverse-Hessian procedure already introduced in §3.","revision_made":"yes","referee_comment":"[Abstract] Abstract (experimental claims): the stated improvements (32.6% C4 perplexity reduction, 4.9-point zero-shot accuracy gain, >5-point ImageNet gain) are given without reference to the precise single-objective baselines, the number of random seeds, variance estimates, or the exact optimization procedure used to solve the joint objective. This absence prevents verification that the joint formulation is load-bearing for the gains rather than an artifact of experimental controls."},{"response":"We will add a dedicated ablation subsection in the revised §5 that tabulates, for every model and sparsity target, the performance of (a) reconstruction-only, (b) Taylor-only, (c) the better of (a) and (b), and (d) MOONSHOT. This will directly test the motivating claim and quantify the incremental benefit of the joint objective. Preliminary internal checks indicate that MOONSHOT is at least as good as the stronger baseline in all evaluated settings and strictly better in the majority; the full table will be included.","revision_made":"yes","referee_comment":"[§4, §5] §4 (experimental setup) and §5 (results): the paper asserts that MOONSHOT 'extends any single-objective pruning method,' but provides no ablation that isolates the contribution of the second-order term versus the reconstruction term across the same set of layers and sparsity targets. A direct comparison showing that the joint mask is strictly better than the better of the two single-objective masks on every model/sparsity pair is required to substantiate the motivating assumption."}],"tokens_in":1657,"tokens_out":771,"duration_ms":61238,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main point is a wrapper that takes any existing one-shot pruner and makes it optimize both layer-wise reconstruction error and the second-order Taylor approximation together. The authors note that neither objective alone wins across models and sparsity levels, which matches what people see in practice, and they add an efficient Hessian inverse step so it still scales to billion-parameter models like Llama-3.2 and Llama-2. The reported numbers are the strongest part: up to 32.6% lower C4 perplexity at 2:4 sparsity, 4.9-point zero-shot accuracy lift, and similar gains on ViTs and ResNet-50 at high sparsity. Those are the kind of deltas that matter for deployment without retraining.","headline":"MOONSHOT wraps single-objective one-shot pruners with a joint reconstruction-plus-Taylor objective and reports clear gains on Llama models and vision nets, but the combination method itself needs closer inspection.","tokens_in":2441,"tokens_out":234,"would_cite":false,"duration_ms":17124,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"MOONSHOT turns single-objective pruning into a joint optimization of reconstruction error and loss curvature, improving compressed model quality without retraining.","keywords":["model pruning","one-shot pruning","multi-objective optimization","large language models","vision transformers","Hessian approximation","model compression","post-training compression"],"falsifier":"A head-to-head test on a held-out model family or sparsity pattern in which the multi-objective version underperforms the strongest single-objective baseline on the same hardware and data.","tokens_in":2742,"feed_emoji":"🧠","tokens_out":686,"duration_ms":48112,"temperature":0.7,"pith_summary":"The paper shows that no single pruning objective works best across all models and sparsity levels in the post-training one-shot setting. It therefore introduces MOONSHOT as a wrapper that lets any existing pruner jointly minimize both the layer-wise reconstruction error and the second-order Taylor approximation of the training loss. To keep the method scalable, the framework adds modeling choices and an efficient inverse-Hessian procedure. When this wrapper is applied to strong baselines on Llama-2, Llama-3.2, Vision Transformers, and ResNet-50, the resulting sparse models exhibit lower perplexity and higher downstream accuracy than either objective alone.","feed_headline":"Joint pruning objectives cut Llama perplexity by 32% at 2:4 sparsity","feed_subtitle":"Optimizing both reconstruction error and Taylor loss approximation together yields higher accuracy on compressed models without any retrain.","key_machinery":"A multi-objective wrapper that jointly minimizes layer-wise reconstruction loss and the Hessian-based Taylor approximation of the training loss, enabled by an efficient inverse-Hessian computation that maintains the speed of existing one-shot pruners.","core_discovery":"MOONSHOT is a general framework that converts any single-objective one-shot pruner into a multi-objective optimizer by simultaneously targeting layer-wise reconstruction error and the second-order Taylor approximation of the training loss, while preserving scalability through an efficient inverse-Hessian routine.","pith_inferences":["The same joint-objective wrapper could be applied to other compression operations such as quantization or low-rank factorization.","Because the method only modifies the objective inside an existing pruner, it offers a low-cost way to re-evaluate earlier one-shot results on newer model families.","The efficiency of the inverse-Hessian step suggests the framework could be extended to dynamic or structured sparsity patterns that change during inference."],"forward_implications":["At 2:4 sparsity on Llama-3.2 and Llama-2, the method reduces C4 perplexity by up to 32.6 percent.","Zero-shot accuracy across seven classification benchmarks rises by up to 4.9 points on the same Llama models.","Vision Transformer accuracy on ImageNet-1k increases by more than 5 points at 70 percent sparsity.","ResNet-50 accuracy improves by 4 points at 90 percent sparsity."],"fun_headline_variants":["MOONSHOT optimizes dual objectives to cut Llama perplexity 32% at 2:4 sparsity","MOONSHOT improves ViT ImageNet accuracy by over 5 points at 70% sparsity","MOONSHOT improves ResNet-50 accuracy 4 points at 90% sparsity without retraining","MOONSHOT jointly targets reconstruction error and Taylor loss in one-shot pruning"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"Jointly optimizing the reconstruction error and the Taylor approximation will reliably beat the better of the two single objectives for every architecture and sparsity target.","fun_headline_variants_meta":{"raw":{"variants":["MOONSHOT optimizes dual objectives to cut Llama perplexity 32% at 2:4 sparsity","MOONSHOT improves ViT ImageNet accuracy by over 5 points at 70% sparsity","MOONSHOT improves ResNet-50 accuracy 4 points at 90% sparsity without retraining","MOONSHOT jointly targets reconstruction error and Taylor loss in one-shot pruning"]},"model":"grok-4.3","cost_usd":0.010386,"raw_usage":{"total_tokens":4612,"prompt_tokens":700,"num_sources_used":0,"completion_tokens":95,"cost_in_usd_ticks":103862000,"prompt_tokens_details":{"text_tokens":700,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":3817,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":700,"tokens_out":95,"duration_ms":39408,"temperature":1.0,"reasoning_tokens":3817,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-05-10T15:41:45.908842+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A head-to-head test on a held-out model family or sparsity pattern in which the multi-objective version underperforms the strongest single-objective baseline on the same hardware and data.","supporting_citations":[],"review_version":1}