{"id":"7fdc1d49-908e-470f-8bc7-4719cc6d7f33","arxiv_id":"2505.00350","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A preservation-set-guided quantization and pruning loop improved test accuracy by up to 2.5% over unquantized models while keeping about 60% of the original size on an MNIST CNN and a character-level name decoder.","lead":"The paper proposes a compression method that prunes and quantizes neural networks while keeping a small 'preservation set' of examples as a safety check. The authors report up to 2.5% higher test accuracy than the uncompressed model at about 60% of the original size on an MNIST CNN and a small name-prediction decoder.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The safety component is not isolated: comparisons against reference [12] cannot support the claim that preservation-set feedback causes the reported gains.","rationale":"The reader's weakest assumption concerned preservation-set representativeness. I agree that is relevant, but a more immediate logical gap is causal attribution: even a representative preservation set could be inert while other components (quantization schedule, L1 penalty, pruning) drive the gains. The paper compares against reference [12] rather than an internal ablation, so the novel mechanism is confounded. A single ablation would settle this. Because the authors themselves note manual loop modifications and the results are point estimates, the appropriate verdict remains conditional; I recommend no change to the reader's CONDITIONAL verdict.","tokens_in":7561,"tokens_out":7425,"duration_ms":82380,"concrete_test":"Run Algorithm 1 on MNIST and the decoder with the identical architecture, hyperparameters, and seeds, but set λ=0 and delete the precision-restoration block (Algorithm 1 lines 17–19) so that the preservation set has no influence; report test accuracy, model size, and standard errors over at least 5 seeds. If the no-preservation variant matches or beats the safety-driven variant, the reported improvement is not attributable to the preservation set, and the central claim must be downgraded.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that safety-driven quantization — i.e., Algorithm 1 with the preservation-set loss and the precision-restoration rule (lines 17–19) — outperforms unquantized and 'unsafely quantized' models. But 'Unsafe Quantization [12]' is a separate published method (self-compressing neural networks), not the proposed framework with the preservation mechanism removed. The two methods can differ in quantization function, bit-depth schedule, pruning criterion, and hyperparameters. Consequently, the 98.6% to 99.5% and 1.8 to 1.6 improvements in Tables I–II do not identify the preservation set as the causal ingredient; they only show that one full system beat another full system on two datasets. No ablation (λ=0 or threshold feedback disabled) is reported, and no preservation-set accuracy is reported to show that the feedback signal tracks test performance. Since the preservation set is the claimed novelty, this missing control is load-bearing: if removing it leaves performance unchanged, the central claim is not supported by the evidence.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes a training-time compression framework called safety-driven self-compression, in which a small \"preservation set\" (10% of the training data, selected by Grad-CAM, uncertainty sampling, and clustering) is used to guide pruning and quantization. The training objective combines the task loss with an L1 sparsity penalty, a quantization penalty, and a preservation loss (Section II, Eq. (2)). Algorithm 1 prunes zeroed kernels or attention heads and restores bit precision when preservation-set accuracy drops below a threshold. Experiments on an MNIST CNN and a decoder-based attention model compare the method against an unquantized model and a \"unsafe quantization\" baseline, reporting that the compressed models retain roughly 60% of the original size while improving test accuracy from 98.6% to 99.5% (CNN) and test loss from 1.8 to 1.6 (decoder). The abstract claims up to a 2.5% enhancement in test accuracy relative to the original unquantized models.","tokens_in":7765,"tokens_out":6071,"duration_ms":63184,"significance":"If the claims hold, the paper offers an interesting empirical observation: compression guided by a small preservation set can simultaneously reduce model size and slightly improve held-out performance. The paper has concrete strengths: the authors release code, test two architecture families, and propose a feedback mechanism that is easy to understand. However, the central claim is not yet established. The quantization equation is internally inconsistent, the reported metric Q is not an average bit depth, and the only baseline for the safety mechanism is a separate published method with no ablation isolating the preservation-set feedback. The evaluation on MNIST and a small n-gram dataset, with single accuracy values and no variance statistics, does not support the broader claims of generalization and variance reduction. With corrections and additional experiments, the core idea could be a useful contribution, but the current evidence is insufficient.","major_comments":[{"comment":"Equation (1) as written is not a quantizer: the expression is a scaled clamp that multiplies by 2^e, clamps to [-2^(b-1), 2^(b-1)-1], and scales back, with no rounding, flooring, or binning operation. The output is continuous rather than discrete. The text invokes the Straight-Through Estimator to handle the non-differentiability of rounding, but the formula contains no rounding step. Because the bit-depth parameters b are central to the compression objective and to the Q metric, please replace Eq. (1) with a true quantization operation (e.g., floor/round inside the clamp) and specify the STE bypass precisely.","section":"Section II, Eq. (1)"},{"comment":"Q as defined is not an average bit depth. Equation (4) multiplies per-channel bit depths by the number of output channels and the spatial dimensions, yielding a quantity with units of bits per layer, and Eq. (3) then averages this over layers. The text calls Q the \"average bit depth\" and uses it as a quantization penalty. Please provide a dimensionally correct per-weight average bit depth, or reinterpret Q as total encoded size and adjust the surrounding claims accordingly.","section":"Section II, Eqs. (3)-(4)"},{"comment":"The central claim that preservation-set feedback causes the observed gains is not isolated by the experiments. The only compressed baseline is \"Unsafe Quantization [12]\", which is a separate published method and can differ in quantization function, bit-depth schedule, and pruning rule. No ablation with the preservation loss removed (e.g., lambda=0) or with the restoration threshold disabled is reported, and no preservation-set accuracy is reported to show that the feedback signal tracks test performance. Tables I-II therefore do not identify the preservation set as the causal ingredient. Please add ablations that toggle the preservation loss and the restoration rule, and report preservation-set accuracy over training.","section":"Section IV, Tables I and II"},{"comment":"The preservation set is a 10% subset of the training data selected by Grad-CAM, uncertainty sampling, and clustering. If this subset is not representative of the test distribution, the restoration rule can overfit to the preservation set and the reported test improvement might not generalize. The manuscript provides no analysis of preservation-set representativeness, no cross-validation, and no seed-to-seed variance; Tables I-II report single values. Please add repeatability statistics (e.g., standard deviations or confidence intervals over seeds), report the preservation-set selection hyperparameters, and evaluate on at least one additional split or dataset per architecture.","section":"Section III, Preservation set; Algorithm 1, lines 17-19"}],"minor_comments":[{"comment":"Equation (3) sums over L layers while N is defined as the total number of layers; please make the notation consistent.","section":"Section II, Eq. (3)"},{"comment":"Equation (4) is defined only for convolutional layers, but the paper applies quantization to attention heads in the decoder; please provide the analogous formula for the attention-based model.","section":"Section II, Eq. (4)"},{"comment":"The abstract claims \"up to a 2.5% enhancement in test accuracy\" while Table I shows a 0.9 percentage-point gain over the unquantized model (98.6% to 99.5%); please clarify whether the 2.5% is relative to the unsafe baseline and whether it is an absolute or relative increase.","section":"Abstract and Section IV"},{"comment":"The experimental setup lists ranges for batch sizes, learning rates, and bit depths, but does not report the actual values used for alpha, gamma, lambda, the preservation-accuracy threshold, or the preservation-set size; without these, the experiments are not fully reproducible from the text.","section":"Section III, Hyperparameters"},{"comment":"Figure 1 lacks axis labels and a clear explanation of the \"solid black line best fit\" and the gray dots, making the claimed variance reduction difficult to verify from the figure.","section":"Figure 1"},{"comment":"The limitation paragraph notes that the approach requires manual adjustments to the training loop and specialized hardware for feed-forward layers; please state these conditions in the introduction or experimental setup as explicit scope limitations.","section":"Conclusion, Limitations"}],"recommendation":"major_revision","confidential_remarks":"For the editor: the manuscript is not ready in its current form, but the core idea is worth a revision. The main risk is that the reported gains may be artifacts of the baseline choice or of overfitting to the preservation set, rather than evidence for the safety mechanism. I would ask for corrected quantization definitions, an ablation isolating the preservation feedback, and variability statistics before further consideration. The fit to this venue would also be substantially improved by moving beyond MNIST and the small n-gram dataset, and by comparing with standard quantization-aware training and post-training quantization baselines."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The preservation-set idea is genuinely new relative to the self-compressing network work it builds on, and the paper is honestly written about its limitations. But the central causal claim — that safety-guided feedback causes the accuracy improvement — is not supported as presented, because there is no ablation that removes the preservation mechanism. That is the main thing you should know.\n\nWhat is actually new: combining Grad-CAM, uncertainty sampling, and clustering to build a small 'preservation set' and using a preservation loss plus a precision-restoration rule to guide quantization. That is a real extension. The method is demonstrated on MNIST and a character-level name decoder, with sizes dropping to about 60% and test accuracy reported to improve slightly. As a modest compression recipe, it is worth testing.\n\nWhere it falls short, in order of importance. First, the missing control: Tables I–II compare the full proposed method against 'Unsafe Quantization [12]', which is a different published system, not the proposed method with the preservation set removed. Differences in quantization function, bit-depth schedule, and pruning mean the reported gains could come from anywhere. A λ=0 run or a run with the threshold feedback disabled would settle it. Second, Eq. (1) is written as a scaled clamp, with no rounding or flooring, even though the text invokes the Straight-Through Estimator to handle rounding. As written, the function does not quantize. Third, the Q metric in Eqs. (3)–(4) is a total bit count, not an average bit depth, despite the label. Fourth, the tables show single numbers with no variance or repetition count, so the claim that results were averaged across hyperparameters is unverifiable. The preservation set is only 10% of the training data, and the paper gives no evidence that its accuracy tracks test performance, so the overfitting channel the reader flagged is real. Finally, the abstract promises a GitHub link that does not appear in the text.\n\nThe citation pattern is fine; the related work is appropriately referenced. The limitations paragraph is honest about hardware and training-loop constraints.\n\nWho gets value: readers working on practical compression for edge devices may want to test the preservation-set idea, but they should treat the empirical claims as preliminary. The paper deserves a serious referee, because the idea is testable and the authors need the missing controls to be requested. I would not cite the current version's numbers. It would be a good workshop paper after revision.\n\nRecommendation: send to peer review if you are the editor; the flaws are fixable and the core idea is worth a careful look. If you are just deciding where to spend your reading time, skim the method and skip the results.","headline":"Promising preservation-set extension, but no ablation isolates it, so the headline accuracy gain is not attributable to the claimed mechanism.","tokens_in":8286,"tokens_out":4035,"would_cite":false,"duration_ms":40337,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A safety-driven compression procedure using a small preservation set of training data can shrink neural networks to about 60% of their original size while improving test accuracy by up to 2.5 percentage points.","keywords":["model compression","quantization","pruning","preservation set","grad-cam","uncertainty sampling","generalization","attention models"],"falsifier":"Train the same CNN and decoder with the same pipeline but replacing the preservation set with a uniformly random 10% subset of the training data. If the random-subset models match or beat the Grad-CAM/uncertainty-selected ones in test accuracy and size, then the selection criteria are not the cause of the reported gains; if they fall short, the selection criteria are doing the work.","tokens_in":7361,"feed_emoji":"⚙️","tokens_out":4955,"duration_ms":46057,"temperature":0.7,"pith_summary":"This paper tries to establish that a compression procedure guided by a 'preservation set' can shrink a neural network to roughly 60% of its original size while improving test accuracy by up to 2.5 percentage points and reducing variance, beating both the unquantized model and conventionally quantized models. The claim is demonstrated on a CNN trained on MNIST and an attention-based decoder trained on a names dataset. If correct, it means that pruning and quantization guided by a small, carefully selected subset of training data can act as a regularizer, producing smaller models that generalize better rather than worse.","feed_headline":"Safety-guided pruning shrinks models 40% and lifts accuracy","feed_subtitle":"A 10% preservation set keeps CNNs and transformers at 60% size while beating the original models.","key_machinery":"The preservation set: a roughly 10% subset of training data chosen via Grad-CAM, uncertainty sampling, and diversity-enforcing clustering, used as a proxy for the features the model must retain. Around it, the method builds a differentiable quantization function with a Straight-Through Estimator, a composite loss that adds an L1 sparsity penalty, a quantization-size penalty, and a preservation-set loss, plus a training loop (Algorithm 1) that prunes zeroed-out components and restores bit precision whenever preservation-set performance dips below a threshold.","core_discovery":"The central claim is that safety-driven quantization—pruning zeroed-out kernels or attention heads and adaptively adjusting per-component bit depth in response to preservation-set accuracy—turns compression into a performance improvement, not just a size reduction. In the reported experiments, the CNN's test accuracy rises from 98.6% without quantization to 99.5% with safety-driven quantization, while model size falls from 326,192 to 214,730 bytes; the attention decoder's test loss improves from 1.8 to 1.6 while size falls from 831,846 to 505,381 bytes. The authors interpret these results as evidence that preserving critical features while removing parameter noise improves generalization and reduces train-test variance.","pith_inferences":["If the preservation set is doing what the paper claims, it functions as a data-dependent regularizer; the 10% size becomes a tunable knob, and smaller or larger sets should change the compression-generalization frontier in predictable ways.","The feedback loop could be applied to other compression actions, such as layer dropping or head merging, and combined with structured pruning to see whether size savings can go well below 60%.","A sharper test would move to larger-scale datasets like CIFAR-10 or ImageNet, where a 10% preservation set is harder to build; if gains vanish there, the method's value is confined to small-scale settings."],"forward_implications":["A 40% smaller model can match or beat its unquantized parent, so compression no longer has to be traded against accuracy.","Bit precision can be restored on the fly when preservation accuracy drops, giving an explicit guardrail against destructive pruning.","The same technique is shown on a CNN and an attention-based decoder, indicating it transfers across vision and language architectures.","Averaging across learning rates, batch sizes, bit depths, and hardware indicates the gains are not artifacts of one training configuration."],"supporting_citations":[{"why":"Supplies the differentiable quantization and self-compression approach the authors compare against as 'unsafe quantization' in the experiments.","marker":"[12]"},{"why":"Provides the Straight-Through Estimator that makes the quantization function differentiable during backpropagation.","marker":"[13]"},{"why":"Used to identify high-activation regions when constructing the preservation set for both vision and language models.","marker":"[10]"},{"why":"Used to select high-uncertainty data points for the preservation set, capturing challenging cases prone to errors.","marker":"[11]"},{"why":"Provides the MNIST dataset on which the CNN experiment is run.","marker":"[9]"}],"fun_headline_variants":["Safety-driven compression: 40% smaller, 2.5% more accurate","Shrink neural nets 40%, boost accuracy with safety pruning","Safety-guided quantization: smaller, sharper models","Compression that beats the original: safety-guided pruning","Safety sets enable 40% smaller AI with better accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The preservation set—a 10% subset of training data selected by Grad-CAM, uncertainty sampling, and clustering—is a reliable stand-in for the features the model needs to generalize to unseen test data, so that guarding performance on it guards performance everywhere.","fun_headline_variants_meta":{"raw":{"variants":["Safety-driven compression: 40% smaller, 2.5% more accurate","Shrink neural nets 40%, boost accuracy with safety pruning","Safety-guided quantization: smaller, sharper models","Compression that beats the original: safety-guided pruning","Safety sets enable 40% smaller AI with better accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000523,"raw_usage":{"total_tokens":2489,"prompt_tokens":867,"completion_tokens":1622,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":483,"completion_tokens_details":{"reasoning_tokens":1539}},"tokens_in":483,"tokens_out":1622,"duration_ms":12787,"temperature":1.0,"reasoning_tokens":1539,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:44:34.202002+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same CNN and decoder with the same pipeline but replacing the preservation set with a uniformly random 10% subset of the training data. If the random-subset models match or beat the Grad-CAM/uncertainty-selected ones in test accuracy and size, then the selection criteria are not the cause of the reported gains; if they fall short, the selection criteria are doing the work.","supporting_citations":[{"cited_title":"Self-Compressing Neural Networks","cited_arxiv_id":"2301.13142","evidence_quote":"Supplies the differentiable quantization and self-compression approach the authors compare against as 'unsafe quantization' in the experiments."},{"cited_title":"Understanding straight-through estimator in training activation quantized neural nets,","cited_arxiv_id":null,"evidence_quote":"Provides the Straight-Through Estimator that makes the quantization function differentiable during backpropagation."}],"review_version":1}