{"id":"56a3613b-8255-4eec-b0b2-a26dcc479153","arxiv_id":"2501.04287","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"ElasticZO uses zeroth-order gradients for most layers and backpropagation only for the last few, improving accuracy over pure ZO training with tiny memory overhead, and its INT8 variant runs integer-only.","lead":"ElasticZO trains a neural network by using backpropagation only on the last one or two layers and cheap random-noise gradients everywhere else, cutting training memory to near inference levels. An 8-bit integer version, ElasticZO-INT8, does the same training without floating point, aiming at phones and microcontrollers.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The Eq. 12 floor-based sign estimator is the load-bearing component for the INT8* claims, and the paper provides only a single empirical ~95% correctness rate with no error analysis for other batch sizes or loss scales.","rationale":"The reader's weakest assumption identifies exactly the component I would stress-test: the Eq. 12 sign estimator. I agree with their assessment, but I want to make the failure mode more precise than '~5% wrong signs.' The floor operation is applied per sample before summation, so the approximation error is not a random zero-mean perturbation; it is a systematic discarding of fractional parts that can accumulate linearly in the batch size. For B=256 the error budget is large relative to small per-sample loss differences, and the sign can flip. The paper gives one empirical correctness rate, not a sensitivity analysis. The proposed test would settle this by measuring agreement across batch sizes and by ablating the approximation itself. I do not think this concern overturns the paper's FP32 contribution; the hybrid idea is simple and the reported accuracy gaps are large. But the INT8* integer-only claim is the paper's distinctive novelty, and it is exactly the part that depends on Eq. 12. Therefore the existing CONDITIONAL verdict is appropriate: the quantitative INT8* claims should not be taken at face value until the sign-estimator robustness is demonstrated or the claim is weakened. No code or seeds are provided, which amplifies the need for this check.","tokens_in":20245,"tokens_out":9071,"duration_ms":86275,"concrete_test":"Instrument the training loop of ElasticZO-INT8 to compute, at every step, both the exact FP32 batch loss difference Δ and the Eq. 12 integer sign for B ∈ {1, 8, 32, 256} on MNIST, Fashion-MNIST, and ModelNet40. Record the sign-agreement rate and the distribution of |Δ| at each B. Then rerun training with the true FP32 sign substituted for the approximate sign (keeping all other integer arithmetic unchanged) and compare end-to-end accuracy. If sign agreement drops below ~95% for any tested B or the end-to-end accuracy changes by more than ~1% absolute when the true sign is used, the Eq. 12 approximation is not a reliable load-bearing component and the INT8* transfer claims need to be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.3, Eq. 12 replaces the true batch loss difference Δ = Σ_b ln(A_b/B_b) with ln 2 · Σ_b (⌊log2 A_b⌋ − ⌊log2 B_b⌋), where A_b and B_b are the sums of powers of two in Eq. 10. This discards the fractional parts of each log2 term before summation. Since each fractional part is in [0,1), the accumulated error is O(B) in nats, while |Δ| can be arbitrarily small; the sign of Δ can therefore flip whenever the true per-sample log-ratios are small. The paper reports only a single empirical estimate (~95% correct signs) and no analysis of how these sign errors interact with the BP-trained head. The authors explicitly concede in §4.3 that the floor 'may lead to incorrect results.' Because the entire 'integer-arithmetic-only ZO training' contribution and the ElasticZO-INT8 accuracy rows in Tables 1–2 rest on this estimator, it is the most load-bearing assumption in the paper. The FP32 hybrid claim is comparatively secure; the risk is concentrated in the INT8* results, which are exactly the paper's headline novelty.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ElasticZO, a hybrid training method that updates the first C layers of a DNN with zeroth-order (SPSA-style) gradient estimates and the last L−C layers with backpropagation, and ElasticZO-INT8, an 8-bit integer-only variant that approximates the ZO gradient direction by the sign of the loss difference computed with floor-based integer arithmetic. The authors derive memory formulas for both variants, report classification experiments on MNIST and Fashion-MNIST with LeNet-5 and on ModelNet40 with PointNet, and measure wall-clock training time on a Raspberry Pi Zero 2. The main claims are that ElasticZO reaches accuracy close to full BP with only 0.072–1.7% memory overhead over full ZO, and that ElasticZO-INT8 performs integer-only training with an additional 1.46–1.60x memory reduction and 1.38–1.42x speedup without compromising accuracy.","tokens_in":20515,"tokens_out":4838,"duration_ms":45994,"significance":"The hybrid partition idea is simple, clearly presented, and likely to be useful for on-device learning: training only the last one or two layers with BP while keeping the rest ZO is an intuitive way to trade a small memory overhead for a large accuracy gain. The memory accounting in Sections 4.1 and 4.4 is careful arithmetic, and the paper strengthens its practical claims with a C++ implementation and timing measurements on an edge board. These are genuine strengths. However, the integer-only sign estimator in Section 4.3 (Eq. 12) is the load-bearing component for the headline INT8* results, and the paper provides only a single empirical correctness rate (~95%) with no robustness analysis. The absence of error bars and the omission of final hyperparameter values also make the numerical comparisons difficult to evaluate. If the estimator is shown to be robust across batch sizes and loss scales, and if the experiments are repeated with error bars and full hyperparameter reporting, this would be a solid contribution to the on-device learning literature.","major_comments":[{"comment":"The floor-based sign estimator is load-bearing for the INT8* accuracy claims in Tables 1 and 2, but its error behavior is not analyzed. Replacing each per-sample log term by its floor discards fractional parts in [0,1) before summation, so the accumulated error is O(B) nats while the true batch loss difference can be arbitrarily small. The paper reports only a single empirical correctness rate of about 95% and does not say under which batch size, perturbation scale, or loss magnitude that rate was obtained, nor how wrong signs interact with the BP-trained head. Please provide an error analysis or systematic experiments varying batch size, perturbation scale, and dataset; without this, the integer-only training claim is not sufficiently supported.","section":"Section 4.3, Eq. (12)"},{"comment":"The claim that ElasticZO-INT8 'reduces the memory usage and training time ... without compromising the accuracy' is not supported by the reported numbers. On Fashion-MNIST, INT8* ZO-Feat-Cls1 is 81.60% versus 86.60% for the FP32 ElasticZO counterpart, a 5.0-point drop, and ZO-Feat-Cls2 drops from 82.28% to 77.93%. Even compared with the INT8 (floating-point ZO gradient) version, the INT8* accuracy is lower by 2.4–3.1 points. Please either quantify the accuracy trade-off in the claim or compare ElasticZO-INT8 against a matched baseline under the same memory reduction.","section":"Table 1 and Abstract"},{"comment":"The selected hyperparameters are not reported. The text states that the learning rate is tuned in [1e-4, 5e-2], the perturbation scale epsilon in [1,3,7,15,31,63], and that gclip is used for ZO, but the final values chosen for each configuration are omitted. Because ZO training is sensitive to these settings, the accuracy results in Tables 1 and 2 are not reproducible without the final values.","section":"Section 5.1.1"},{"comment":"All accuracy results appear to be single runs with no error bars or repeated seeds. Given that many reported differences between configurations are 1–5 accuracy points, confidence intervals or standard deviations over multiple seeds are needed to establish that the hybrid and integer-only improvements are not due to random variation.","section":"Tables 1 and 2"}],"minor_comments":[{"comment":"The memory overhead range is inconsistent: the Abstract reports 0.072–1.7%, the Conclusion reports 0.072–0.17%, and Section 5.3 reports values up to 2.4% (for B=32). Please reconcile these numbers.","section":"Abstract and Conclusion"},{"comment":"The memory numbers in these figures are computed from Equations 2–4 and 13–15, which allocate all buffers for the whole training process; the text acknowledges this assumption, but the figure captions should state it explicitly since the abstract's percentages inherit this conservative accounting.","section":"Figures 4–6"},{"comment":"The configurations ZO-Feat-Cls1 and ZO-Feat-Cls2 are used throughout the tables and figures without a definition in the captions; please define which layers are trained by BP and which by ZO in the captions of Tables 1 and 2.","section":"Table captions"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the scope of the journal and the FP32 hybrid result is credible and well presented. My main concern is that the INT8* contribution, which is the paper's headline novelty, rests on a sign estimator whose robustness is not established; this is fixable with additional experiments and analysis, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea here is solid and worth engaging with: train most of a network with zeroth-order optimization and reserve backprop for the last one or two layers. The memory formulas in Secs. 4.1 and 4.4 are careful and the breakdowns in Figs. 4–6 actually explain where the bytes go. The empirical trend is consistent — ZO-Feat-Cls1 and Cls2 close most of the accuracy gap to full BP with negligible memory overhead — and the fine-tuning experiments add useful evidence. The C++ implementation on Raspberry Pi Zero 2 with timing breakdowns is real engineering work, and the paper is honest about its limits (it explicitly says choosing C is an open problem and that there is still an accuracy gap). Give credit where due: this is a new combination, not a rehash.\n\nThe soft spot is exactly where the stress test points. Eq. 12 replaces the true batch loss difference with a floor-based sum of log2 terms, and the paper concedes in Sec. 4.3 that the floor 'may lead to incorrect results.' The single empirical ~95% sign-correctness rate is not enough. The error accumulates with batch size and the sign can flip when per-sample log-ratios are small, so the INT8* rows in Tables 1 and 2 — the headline novelty — rest on an uncharacterized approximation. The FP32 hybrid claims are comparatively safe; the risk is concentrated in the integer-only part. Also missing: no error bars or repeated seeds, no code release, and the final hyperparameter values (learning rate, gclip, epsilon) are not reported, which makes the quantitative claims hard to verify. These are fixable, but they matter.\n\nWho is this for? People working on on-device learning, edge inference, or ZO optimization. The hybrid partition is a useful design point and the integer-only training direction is worth exploring. I would send this to a serious referee: the idea is new, the engineering is real, and the evidentiary holes are addressable rather than fatal. My recommendation is conditional accept — require a proper error analysis of Eq. 12, repeated seeds, and code release before the quantitative claims can be trusted.","headline":"The hybrid ZO/BP partition is a genuinely useful idea with careful memory accounting, but the integer-only sign-gradient trick — the paper's headline — rests on a heuristic that needs error analysis before the INT8* accuracy claims can be taken at face value.","tokens_in":21010,"tokens_out":1488,"would_cite":true,"duration_ms":16562,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By backpropagating only the last one or two layers and using zeroth-order perturbation for everything else, ElasticZO closes most of the gap to full backprop while adding less than 1.7% memory over inference; its INT8 variant does the…","keywords":["zeroth-order optimization","on-device learning","backpropagation","integer-only training","quantization","memory-efficient training","edge devices","hybrid optimization"],"falsifier":"A concrete falsifier: measure the empirical sign accuracy of Eq. 12 against the true sign of the loss difference across batch sizes (e.g., 32 vs 256) and architectures; if it drops well below 95% or the wrong signs systematically oppose the BP-trained head's gradients, the INT8* accuracy claims would not transfer. Alternatively, train a deeper or wider network (e.g., a ResNet or small transformer) from scratch with ElasticZO-INT8 and check whether the accuracy gap to full BP stays within the 1.4–5.7 points reported for LeNet-5.","tokens_in":20036,"feed_emoji":"📱","tokens_out":12541,"duration_ms":94464,"temperature":0.7,"pith_summary":"This paper tries to establish that a hybrid optimizer — zeroth-order perturbation for the bulk of a network and backpropagation for the last one or two layers — recovers most of full-backprop accuracy while keeping memory near inference level. The companion 8-bit variant, ElasticZO-INT8, shows that the same hybrid can run with integer-only arithmetic by replacing the ZO gradient with the sign of an integer-computed loss difference, and that this cuts memory and runtime further without accuracy loss. This matters because it would make on-device training feasible on low-cost edge hardware without floating-point units, for both full training and fine-tuning.","feed_headline":"Backprop the head, perturb the rest: near-BP accuracy at ZO memory","feed_subtitle":"Training only the last one or two layers with backprop closes most of the accuracy gap at near-inference memory.","key_machinery":"The load-bearing mechanism is the partition point $C$: it divides the network into a ZO-trained body (layers $1..C$) and a BP-trained head (layers $C+1..L$). ElasticZO uses the SPSA estimator from Eq. 1 with a seed-replay trick to avoid storing the perturbation vector, and keeps activations only for the head to backprop. ElasticZO-INT8 replaces the scalar $g$ with the ternary sign $\\mathrm{sgn}(\\ell_+ - \\ell_-)$, evaluated by rescaling integer logits, approximating exponentials as powers of two, and using $\\lfloor \\log_2 \\rfloor$ of the sums to decide the sign (Eq. 12). The integer framework from NITI represents all variables as (int8, exponent) pairs, so every operation is 8-bit integer arithmetic.","core_discovery":"The central claim is that the accuracy gap between zeroth-order (ZO) and backpropagation (BP) training is mostly a head problem: training the final one or two fully-connected layers with BP and everything before them with ZO moves accuracy from 32–90% (Full ZO) to 70–98% (ElasticZO), within 1.6–4.8 points of Full BP on LeNet-5 and PointNet, while adding only 0.072–1.7% memory over pure ZO. For the integer-only version, the paper claims that a ternary ZO update based on the sign of the loss difference — computed in integer arithmetic via a floor-log approximation that is correct about 95% of the time — preserves accuracy within a few points and cuts memory by 1.46–1.60x and time by 1.38–1.42x. The memory formulas derived in the paper bracket ElasticZO's footprint between Full ZO and Full BP, with activations dominating the total.","pith_inferences":["The paper does not explore adaptive partition selection; a layer-wise sensitivity measure could find cheaper partitions on larger models than the fixed $C = L-1$ or $L-2$ choices.","Since activations dominate memory (93.5–97.6% at batch 256), the next bottleneck is activation memory rather than parameter memory; activation checkpointing inside the ZO body could shrink the footprint beyond the paper's stated formulas.","The sign-based estimator could be combined with signSGD-style majority voting or variance reduction to mitigate the ~5% wrong signs, which the paper does not analyze.","If the hybrid's success reflects a real property of deep networks (the head needs first-order gradients, the body can be trained by noisy function evaluations), it suggests that ZO can be applied to self-supervised or contrastive objectives where only the projection head is BP-trained."],"forward_implications":["Training from scratch and fine-tuning both work: ElasticZO improves rotated-MNIST and rotated-Fashion-MNIST accuracy by 15.6–53.5 points over no fine-tuning.","The memory formulas (Eqs. 2–4 and 13–15) let a practitioner pick the number of BP-trained layers to hit a target memory budget, since the footprint lies between Full ZO and Full BP.","Because forward passes dominate wall-clock time (84–97%), existing inference engines can be reused for on-device training, and INT8 arithmetic gives a 1.38–1.42x speedup on a Raspberry Pi Zero 2.","The sign-based ternary gradient inherits the robustness rationale of ZO-signSGD, so wrong signs (~5%) degrade accuracy by only 0.4–3.1 points relative to floating-point ZO on the tested datasets.","Both hybrid variants (ZO-Feat-Cls1 and ZO-Feat-Cls2) outperform Full ZO by 5.2–9.5% (FP32) and 6.4–10.7% (INT8) while staying within 1.7% memory overhead."],"supporting_citations":[{"why":"Supplies the seed-replay trick that lets ZO training run at inference-level memory, which ElasticZO inherits for its ZO-updated layers.","marker":"[6]"},{"why":"NITI framework provides the integer-only representation ((int8, exponent)) and rounding procedures that ElasticZO-INT8 builds on for forward and backward passes.","marker":"[42]"},{"why":"SPSA estimator in Eq. 1 is the zeroth-order gradient approximation that ElasticZO uses for the ZO-trained layers.","marker":"[41]"},{"why":"ZO-signSGD motivates using the sign of the ZO gradient, which ElasticZO-INT8 adopts for its integer-only ternary update.","marker":"[25]"},{"why":"ZO-SGD is the baseline zeroth-order update rule that the hybrid replaces for the body of the network.","marker":"[27]"}],"fun_headline_variants":["Backprop the head, ZO the body: near-BP accuracy, tiny memory","ElasticZO: hybrid training that shrinks the accuracy gap at near-zero cost","One backprop layer closes the ZO gap: minimal memory overhead","ZO training gets a brain: BP last layers, perturb the rest","Integer-only ZO training: first time, with near-BP accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the sign of the loss difference computed from integer logits (correct roughly 95% of the time) is a reliable enough ternary gradient for the ZO-trained layers, and that the accuracy gains measured on LeNet-5 and PointNet transfer to other architectures and datasets.","fun_headline_variants_meta":{"raw":{"variants":["Backprop the head, ZO the body: near-BP accuracy, tiny memory","ElasticZO: hybrid training that shrinks the accuracy gap at near-zero cost","One backprop layer closes the ZO gap: minimal memory overhead","ZO training gets a brain: BP last layers, perturb the rest","Integer-only ZO training: first time, with near-BP accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000216,"raw_usage":{"total_tokens":1498,"prompt_tokens":1079,"completion_tokens":419,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":695,"completion_tokens_details":{"reasoning_tokens":320}},"tokens_in":695,"tokens_out":419,"duration_ms":4601,"temperature":1.0,"reasoning_tokens":320,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:36:41.371480+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete falsifier: measure the empirical sign accuracy of Eq. 12 against the true sign of the loss difference across batch sizes (e.g., 32 vs 256) and architectures; if it drops well below 95% or the wrong signs systematically oppose the BP-trained head's gradients, the INT8* accuracy claims would not transfer. Alternatively, train a deeper or wider network (e.g., a ResNet or small transformer) from scratch with ElasticZO-INT8 and check whether the accuracy gap to full BP stays within the 1.4–5.7 points reported for LeNet-5.","supporting_citations":[{"cited_title":"Lee, Danqi Chen, and Sanjeev Arora","cited_arxiv_id":null,"evidence_quote":"Supplies the seed-replay trick that lets ZO training run at inference-level memory, which ElasticZO inherits for its ZO-updated layers."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"NITI framework provides the integer-only representation ((int8, exponent)) and rounding procedures that ElasticZO-INT8 builds on for forward and backward passes."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"SPSA estimator in Eq. 1 is the zeroth-order gradient approximation that ElasticZO uses for the ZO-trained layers."},{"cited_title":"signSGD via Zeroth-Order Oracle","cited_arxiv_id":null,"evidence_quote":"ZO-signSGD motivates using the sign of the ZO gradient, which ElasticZO-INT8 adopts for its integer-only ternary update."},{"cited_title":"Stochastic First- and Zeroth-Order Methods for Nonconvex Stochastic Pro- gramming","cited_arxiv_id":null,"evidence_quote":"ZO-SGD is the baseline zeroth-order update rule that the hybrid replaces for the body of the network."}],"review_version":1}