Pith. sign in

REVIEW 4 major objections 3 minor 50 references

ElasticZO: A Memory-Efficient On-Device Learning with Combined Zeroth- and First-Order Optimization

T0 review · 4 major / 3 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read By backpropagating only the last one or two layers and using zeroth-order perturbation for everything else, ElasticZO closes most of the gap to full backprop while adding less than 1.7% memory over inference; its INT8 variant does the…

desk verdict The hybrid ZO/BP partition is a genuinely useful idea with careful memory accounting, but the integer-only sign-gradient trick — the paper's headline — rests on a heuristic that needs error analysis before the INT8* accuracy claims can be taken at face value. read the letter →

arxiv 2501.04287 v1 pith:UOHKWKB7 submitted 2025-01-08 cs.LG

Keisuke Sugiura , Hiroki Matsutani This is my paper · ORCID
classification cs.LG
keywords zeroth-orderoptimizationon-devicelearningbackpropagationinteger-onlytrainingquantizationmemory-efficientedgedeviceshybrid
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a hybrid optimizer — zeroth-order perturbation for the bulk of a network and backpropagation for the last one or two layers — recovers most of full-backprop accuracy while keeping memory near inference level. The companion 8-bit variant, ElasticZO-INT8, shows that the same hybrid can run with integer-only arithmetic by replacing the ZO gradient with the sign of an integer-computed loss difference, and that this cuts memory and runtime further without accuracy loss. This matters because it would make on-device training feasible on low-cost edge hardware without floating-point units, for both full training and fine-tuning.

What carries the argument

The load-bearing mechanism is the partition point $C$: it divides the network into a ZO-trained body (layers $1..C$) and a BP-trained head (layers $C+1..L$). ElasticZO uses the SPSA estimator from Eq. 1 with a seed-replay trick to avoid storing the perturbation vector, and keeps activations only for the head to backprop. ElasticZO-INT8 replaces the scalar $g$ with the ternary sign $\mathrm{sgn}(\ell_+ - \ell_-)$, evaluated by rescaling integer logits, approximating exponentials as powers of two, and using $\lfloor \log_2 \rfloor$ of the sums to decide the sign (Eq. 12). The integer framework from NITI represents all variables as (int8, exponent) pairs, so every operation is 8-bit integer arithmetic.

What would settle it

A concrete falsifier: measure the empirical sign accuracy of Eq. 12 against the true sign of the loss difference across batch sizes (e.g., 32 vs 256) and architectures; if it drops well below 95% or the wrong signs systematically oppose the BP-trained head's gradients, the INT8* accuracy claims would not transfer. Alternatively, train a deeper or wider network (e.g., a ResNet or small transformer) from scratch with ElasticZO-INT8 and check whether the accuracy gap to full BP stays within the 1.4–5.7 points reported for LeNet-5.

Watch

Extended reading notes

Core claim

The central claim is that the accuracy gap between zeroth-order (ZO) and backpropagation (BP) training is mostly a head problem: training the final one or two fully-connected layers with BP and everything before them with ZO moves accuracy from 32–90% (Full ZO) to 70–98% (ElasticZO), within 1.6–4.8 points of Full BP on LeNet-5 and PointNet, while adding only 0.072–1.7% memory over pure ZO. For the integer-only version, the paper claims that a ternary ZO update based on the sign of the loss difference — computed in integer arithmetic via a floor-log approximation that is correct about 95% of the time — preserves accuracy within a few points and cuts memory by 1.46–1.60x and time by 1.38–1.42x. The memory formulas derived in the paper bracket ElasticZO's footprint between Full ZO and Full BP, with activations dominating the total.

Load-bearing premise

The load-bearing premise is that the sign of the loss difference computed from integer logits (correct roughly 95% of the time) is a reliable enough ternary gradient for the ZO-trained layers, and that the accuracy gains measured on LeNet-5 and PointNet transfer to other architectures and datasets.

Editorial extensions

If this is right

  • Training from scratch and fine-tuning both work: ElasticZO improves rotated-MNIST and rotated-Fashion-MNIST accuracy by 15.6–53.5 points over no fine-tuning.
  • The memory formulas (Eqs. 2–4 and 13–15) let a practitioner pick the number of BP-trained layers to hit a target memory budget, since the footprint lies between Full ZO and Full BP.
  • Because forward passes dominate wall-clock time (84–97%), existing inference engines can be reused for on-device training, and INT8 arithmetic gives a 1.38–1.42x speedup on a Raspberry Pi Zero 2.
  • The sign-based ternary gradient inherits the robustness rationale of ZO-signSGD, so wrong signs (~5%) degrade accuracy by only 0.4–3.1 points relative to floating-point ZO on the tested datasets.
  • Both hybrid variants (ZO-Feat-Cls1 and ZO-Feat-Cls2) outperform Full ZO by 5.2–9.5% (FP32) and 6.4–10.7% (INT8) while staying within 1.7% memory overhead.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not explore adaptive partition selection; a layer-wise sensitivity measure could find cheaper partitions on larger models than the fixed $C = L-1$ or $L-2$ choices.
  • Since activations dominate memory (93.5–97.6% at batch 256), the next bottleneck is activation memory rather than parameter memory; activation checkpointing inside the ZO body could shrink the footprint beyond the paper's stated formulas.
  • The sign-based estimator could be combined with signSGD-style majority voting or variance reduction to mitigate the ~5% wrong signs, which the paper does not analyze.
  • If the hybrid's success reflects a real property of deep networks (the head needs first-order gradients, the body can be trained by noisy function evaluations), it suggests that ZO can be applied to self-supervised or contrastive objectives where only the projection head is BP-trained.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. The paper proposes ElasticZO, a hybrid training method that updates the first C layers of a DNN with zeroth-order (SPSA-style) gradient estimates and the last L−C layers with backpropagation, and ElasticZO-INT8, an 8-bit integer-only variant that approximates the ZO gradient direction by the sign of the loss difference computed with floor-based integer arithmetic. The authors derive memory formulas for both variants, report classification experiments on MNIST and Fashion-MNIST with LeNet-5 and on ModelNet40 with PointNet, and measure wall-clock training time on a Raspberry Pi Zero 2. The main claims are that ElasticZO reaches accuracy close to full BP with only 0.072–1.7% memory overhead over full ZO, and that ElasticZO-INT8 performs integer-only training with an additional 1.46–1.60x memory reduction and 1.38–1.42x speedup without compromising accuracy.

Significance. The hybrid partition idea is simple, clearly presented, and likely to be useful for on-device learning: training only the last one or two layers with BP while keeping the rest ZO is an intuitive way to trade a small memory overhead for a large accuracy gain. The memory accounting in Sections 4.1 and 4.4 is careful arithmetic, and the paper strengthens its practical claims with a C++ implementation and timing measurements on an edge board. These are genuine strengths. However, the integer-only sign estimator in Section 4.3 (Eq. 12) is the load-bearing component for the headline INT8* results, and the paper provides only a single empirical correctness rate (~95%) with no robustness analysis. The absence of error bars and the omission of final hyperparameter values also make the numerical comparisons difficult to evaluate. If the estimator is shown to be robust across batch sizes and loss scales, and if the experiments are repeated with error bars and full hyperparameter reporting, this would be a solid contribution to the on-device learning literature.

major comments (4)
  1. [Section 4.3, Eq. (12)] The floor-based sign estimator is load-bearing for the INT8* accuracy claims in Tables 1 and 2, but its error behavior is not analyzed. Replacing each per-sample log term by its floor discards fractional parts in [0,1) before summation, so the accumulated error is O(B) nats while the true batch loss difference can be arbitrarily small. The paper reports only a single empirical correctness rate of about 95% and does not say under which batch size, perturbation scale, or loss magnitude that rate was obtained, nor how wrong signs interact with the BP-trained head. Please provide an error analysis or systematic experiments varying batch size, perturbation scale, and dataset; without this, the integer-only training claim is not sufficiently supported.
  2. [Table 1 and Abstract] The claim that ElasticZO-INT8 'reduces the memory usage and training time ... without compromising the accuracy' is not supported by the reported numbers. On Fashion-MNIST, INT8* ZO-Feat-Cls1 is 81.60% versus 86.60% for the FP32 ElasticZO counterpart, a 5.0-point drop, and ZO-Feat-Cls2 drops from 82.28% to 77.93%. Even compared with the INT8 (floating-point ZO gradient) version, the INT8* accuracy is lower by 2.4–3.1 points. Please either quantify the accuracy trade-off in the claim or compare ElasticZO-INT8 against a matched baseline under the same memory reduction.
  3. [Section 5.1.1] The selected hyperparameters are not reported. The text states that the learning rate is tuned in [1e-4, 5e-2], the perturbation scale epsilon in [1,3,7,15,31,63], and that gclip is used for ZO, but the final values chosen for each configuration are omitted. Because ZO training is sensitive to these settings, the accuracy results in Tables 1 and 2 are not reproducible without the final values.
  4. [Tables 1 and 2] All accuracy results appear to be single runs with no error bars or repeated seeds. Given that many reported differences between configurations are 1–5 accuracy points, confidence intervals or standard deviations over multiple seeds are needed to establish that the hybrid and integer-only improvements are not due to random variation.
minor comments (3)
  1. [Abstract and Conclusion] The memory overhead range is inconsistent: the Abstract reports 0.072–1.7%, the Conclusion reports 0.072–0.17%, and Section 5.3 reports values up to 2.4% (for B=32). Please reconcile these numbers.
  2. [Figures 4–6] The memory numbers in these figures are computed from Equations 2–4 and 13–15, which allocate all buffers for the whole training process; the text acknowledges this assumption, but the figure captions should state it explicitly since the abstract's percentages inherit this conservative accounting.
  3. [Table captions] The configurations ZO-Feat-Cls1 and ZO-Feat-Cls2 are used throughout the tables and figures without a definition in the captions; please define which layers are trained by BP and which by ZO in the captions of Tables 1 and 2.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claims rest on external baselines, direct measurement, and arithmetic accounting; the Eq. 12 sign estimator is an acknowledged approximation, not a fitted input.

full rationale

The paper's derivation chain is self-contained. ElasticZO's accuracy gains are measured against Full ZO and Full BP on MNIST, Fashion-MNIST, and ModelNet40; no accuracy quantity is defined in terms of a parameter fitted to that same quantity. The memory formulas (Eqs. 2-4 and 13-15) are arithmetic decompositions of buffer sizes under an explicitly stated no-reuse accounting convention, not predictions that reduce to their inputs. ElasticZO-INT8 adopts integer arithmetic building blocks from the external NITI framework [42] and evaluates the resulting method empirically; there are no self-citations by the present authors in the reference list, and no uniqueness theorem is invoked. The one load-bearing approximation, Eq. 12's floor-based sign estimator, is explicitly acknowledged in Section 4.3 ('the use of floor operation floor(.) may lead to incorrect results'), and its roughly 95 percent sign-correctness rate is an empirical measurement, not a fitted parameter used to define the reported INT8* accuracies; the INT8* rows are measured outcomes, so the approximation weakens robustness for other batch sizes or loss scales but does not make the derivation circular. The conclusion's admitted 'open problem' of choosing the number of BP-trained layers is a limitation, not a circular step.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

No new physical entities are introduced. The method's support comes from standard SPSA, the MeZO seed trick, NITI, and an empirically calibrated sign approximation; the free hyperparameters are tuned but not reported at final values.

free parameters (6)
  • learning rate eta = not reported; tuned in [1e-4, 5e-2] with 0.8 decay per 10 epochs
    Chosen per model/dataset (Sec. 5.1.1); final values are omitted, so accuracy claims depend on unlisted tuning.
  • ZO gradient clipping gclip = not reported
    Used in FP32 ZO training to stabilize updates (Sec. 5.1.1); value not given.
  • INT8 perturbation scale epsilon = tuned from {1,3,7,15,31,63}
    Selected per experiment for ElasticZO-INT8 (Sec. 5.1.1); final value per table not stated.
  • ZO sparsity pzero schedule = 0.33 -> 0.5 -> 0.9 at epochs 20/50
    Sparse perturbation mask probability is increased during training and improves INT8 accuracy (Sec. 5.1.1, Sec. 5.2).
  • gradient bitwidth schedule bBP/bZO = bBP 5->4->3; bZO=1
    Bitwidth schedule follows NITI and is fixed for ZO; it affects update magnitude and accuracy.
  • hybrid partition point C = C = L-1 (ZO-Feat-Cls1) or L-2 (ZO-Feat-Cls2)
    The number of BP-trained layers is chosen by hand; the main results depend on this choice, and the paper says choosing it optimally is open.
assumptions (5)
  • standard math The SPSA estimator in Eq. 1 gives an unbiased estimate of the gradient as epsilon goes to zero.
    Used to justify all ZO gradient updates; Section 3.1.
  • domain assumption A random seed can regenerate the same Gaussian perturbation vector z exactly, eliminating the need to store z.
    From MeZO [6], relied on in Algorithm 1 lines 3-9 and Section 3.2.
  • domain assumption NITI's integer arithmetic, exponent scaling, and pseudo-stochastic rounding preserve trainability of 8-bit networks.
    ElasticZO-INT8 inherits the NITI framework [42] wholesale for forward/backward passes (Section 4.2).
  • domain assumption The sign of the loss difference is a useful ternary gradient direction for ZO updates.
    Adopted from ZO-signSGD [25] and used to make INT8 gradients quantized (Section 4.3).
  • ad hoc to paper The floor-based batch estimator in Eq. 12 gives the correct sign with high probability (about 95%).
    Authors state empirical correctness; no bound or analysis is provided (Section 4.3).

how reviews work

0 comments
Cite this review

Pith. "Pith review of ElasticZO: A Memory-Efficient On-Device Learning with Combined Zeroth- and First-Order Optimization." pith.science (2026). https://pith.science/paper/UOHKWKB7

@misc{pith2026250104287,
  author       = {Pith},
  title        = {Pith review of: ElasticZO: A Memory-Efficient On-Device Learning with Combined Zeroth- and First-Order Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UOHKWKB7}},
  note         = {Machine review of arXiv:2501.04287}
}
read the original abstract

Zeroth-order (ZO) optimization is being recognized as a simple yet powerful alternative to standard backpropagation (BP)-based training. Notably, ZO optimization allows for training with only forward passes and (almost) the same memory as inference, making it well-suited for edge devices with limited computing and memory resources. In this paper, we propose ZO-based on-device learning (ODL) methods for full-precision and 8-bit quantized deep neural networks (DNNs), namely ElasticZO and ElasticZO-INT8. ElasticZO lies in the middle between pure ZO- and pure BP-based approaches, and is based on the idea to employ BP for the last few layers and ZO for the remaining layers. ElasticZO-INT8 achieves integer arithmetic-only ZO-based training for the first time, by incorporating a novel method for computing quantized ZO gradients from integer cross-entropy loss values. Experimental results on the classification datasets show that ElasticZO effectively addresses the slow convergence of vanilla ZO and shrinks the accuracy gap to BP-based training. Compared to vanilla ZO, ElasticZO achieves 5.2-9.5% higher accuracy with only 0.072-1.7% memory overhead, and can handle fine-tuning tasks as well as full training. ElasticZO-INT8 further reduces the memory usage and training time by 1.46-1.60x and 1.38-1.42x without compromising the accuracy. These results demonstrate a better tradeoff between accuracy and training cost compared to pure ZO- and BP-based approaches, and also highlight the potential of ZO optimization in on-device learning.

Figures

Figures reproduced from arXiv: 2501.04287 by the authors.

Figure 1
Figure 1. Overview of ElasticZO (top: LeNet-5, bottom: PointNet). ElasticZO trains the first [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Training and test loss curves of LeNet-5 (FP32; left: MNIST, right: Fashion-MNIST). [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗
Figure 3
Figure 3. Training and test loss curves of LeNet-5 (INT8; left: MNIST, right: Fashion-MNIST). [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Memory usage breakdown of LeNet-5 (FP32; left: [PITH_FULL_IMAGE:figures/full_fig_p012_4.png]
Figure 5
Figure 5. Figure 5: Memory usage breakdown of LeNet-5 (INT8; left: [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]
Figure 6
Figure 6. Figure 6: Memory usage breakdown of PointNet (FP32, [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 7
Figure 7. Figure 7: Execution time breakdown of the C++ implementation of Full ZO and ElasticZO on Raspberry Pi Zero 2 [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

50 extracted references · 40 canonical work pages

  1. [1]

    Rumelhart, Geoffrey E

    David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. Learning Representations by Back-propagating Errors. Nature, 323(6088):533–536, October 1986

    work page 1986

  2. [2]

    Howard, Wayne Hubbard, and Lawrence Jackel

    Yann LeCun, Bernhard Boser, John Denker, Donnie Henderson, R. Howard, Wayne Hubbard, and Lawrence Jackel. Handwritten Digit Recognition with a Back-Propagation Network. In Proceedings of the Advances in Neural Information Processing Systems (NIPS), pages 396–404, November 1989

    work page 1989

  3. [3]

    LeCun, B

    Y . LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, and W. Hubbard. Backpropagation Applied to Handwritten Zip Code Recognition. Neural Computation, 1(4):541–551, December 1989

    work page 1989

  4. [4]

    Adaptive Subgradient Methods for Online Learning and Stochastic Optimization

    John Duchi, Elad Hazan, and Yoram Singer. Adaptive Subgradient Methods for Online Learning and Stochastic Optimization. Journal of Machine Learning Research, 12(61):2121–2159, July 2011

    work page 2011

  5. [5]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In Proceedings of the International Conference on Learning Representations (ICLR), pages 1–15, May 2015

    work page 2015

  6. [6]

    Lee, Danqi Chen, and Sanjeev Arora

    Sadhika Malladi, Tianyu Gao, Eshaan Nichani, Alex Damian, Jason D. Lee, Danqi Chen, and Sanjeev Arora. Fine-Tuning Large Language Models with Just Forward Passes. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), pages 53038–53075, December 2023

    work page 2023

  7. [7]

    Hero III, and Pramod K

    Sijia Liu, Pin-Yu Chen, Bhavya Kailkhura, Gaoyuan Zhang, Alfred O. Hero III, and Pramod K. Varshney. A Primer on Zeroth-Order Optimization in Signal Processing and Machine Learning: Principals, Recent Advances, and Applications. IEEE Signal Processing Magazine, 37(5):43–54, September 2020

    work page 2020

  8. [8]

    Lee, Wotao Yin, Mingyi Hong, Zhangyang Wang, Sijia Liu, and Tianlong Chen

    Yihua Zhang, Pingzhi Li, Junyuan Hong, Jiaxiang Li, Yimeng Zhang, Wenqing Zheng, Pin-Yu Chen, Jason D. Lee, Wotao Yin, Mingyi Hong, Zhangyang Wang, Sijia Liu, and Tianlong Chen. Revisiting Zeroth-Order Opti- mization for Memory-Efficient LLM Fine-Tuning: A Benchmark. In Proceedings of the International Confer- ence on Machine Learning (ICML), pages 59173–...

    work page 2024

Show all 50 references
  1. [9]

    ZOO: Zeroth Order Optimization Based Black-box Attacks to Deep Neural Networks without Training Substitute Models

    Pin-Yu Chen, Huan Zhang, Yash Sharma, Jinfeng Yi, and Cho-Jui Hsieh. ZOO: Zeroth Order Optimization Based Black-box Attacks to Deep Neural Networks without Training Substitute Models. In Proceedings of the ACM Workshop on Artificial Intelligence and Security (AISec), pages 15–...

    2017

  2. [10]

    Hessian-Aware Zeroth-Order Opti- mization for Black-Box Adversarial Attack

    Haishan Ye, Zhichao Huang, Cong Fang, Chris Junchi Li, and Tong Zhang. Hessian-Aware Zeroth-Order Opti- mization for Black-Box Adversarial Attack. arXiv preprint arXiv:1812.11377, December 2018

    2018 arXiv

  3. [11]

    AutoZOOM: Autoencoder-Based Zeroth Order Optimization Method for Attacking Black-Box Neural Networks

    Chun-Chen Tu, Paishun Ting, Pin-Yu Chen, Sijia Liu, Huan Zhang, Jinfeng Yi, Cho-Jui Hsieh, and Shin-Ming Cheng. AutoZOOM: Autoencoder-Based Zeroth Order Optimization Method for Attacking Black-Box Neural Networks. In Proceedings of the AAAI Conference on Artificial Intelligenc...

    2019

  4. [12]

    How to Ro- bustify Black-Box ML Models? A Zeroth-Order Optimization Perspective

    Yimeng Zhang, Yuguang Yao, Jinghan Jia, Jinfeng Yi, Mingyi Hong, Shiyu Chang, and Sijia Liu. How to Ro- bustify Black-Box ML Models? A Zeroth-Order Optimization Perspective. In Proceedings of the International Conference on Learning Representations (ICLR), pages 1–15, April 2022

    2022

  5. [13]

    DeepZero: Scaling up Zeroth-Order Optimization for Deep Model Training

    Aochuan Chen, Yimeng Zhang, Jinghan Jia, James Diffenderfer, Jiancheng Liu, Konstantinos Parasyris, Yihua Zhang, Zheng Zhang, Bhavya Kailkhura, and Sijia Liu. DeepZero: Scaling up Zeroth-Order Optimization for Deep Model Training. In Proceedings of the International Conference...

    2024

  6. [14]

    ZARTS: On Zero-order Optimiza- tion for Neural Architecture Search

    Xiaoxing Wang, Wenxuan Guo, Jianlin Su, Xiaokang Yang, and Junchi Yan. ZARTS: On Zero-order Optimiza- tion for Neural Architecture Search. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), pages 12868–12880, November 2022

    2022

  7. [15]

    ZO-DARTS: Differentiable Architecture Search with Zeroth-Order Approximation

    Lunchen Xie, Kaiyu Huang, Fan Xu, and Qingjiang Shi. ZO-DARTS: Differentiable Architecture Search with Zeroth-Order Approximation. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5, June 2023

    2023

  8. [16]

    Zeroth-Order Online Alternating Direction Method of Mul- tipliers: Convergence Analysis and Applications

    Sijia Liu, Jie Chen, Pin-Yu Chen, and Alfred Hero. Zeroth-Order Online Alternating Direction Method of Mul- tipliers: Convergence Analysis and Applications. In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS), pages 288–297, April 2018

    2018

  9. [17]

    A Zeroth-Order Block Coordinate Gra- dient Descent Method For Cellular Network Optimization

    Pengcheng He, Siyuan Lu, Xin Guan, Yibin Kang, and Qingjiang Shi. A Zeroth-Order Block Coordinate Gra- dient Descent Method For Cellular Network Optimization. In Proceedings of the International Symposium on Wireless Communication Systems (ISWCS), pages 1–6, October 2022

    2022

  10. [18]

    A Parallel Zeroth-Order Framework for Efficient Cellular Network Optimization

    Pengcheng He, Siyuan Lu, Fan Xu, Yibin Kang, Qi Yan, and Qingjiang Shi. A Parallel Zeroth-Order Framework for Efficient Cellular Network Optimization. IEEE Transactions on Wireless Communications, 23(11):17522– 17538, November 2024. 14 ElasticZO: Memory-Efficient On-Device Lea...

    2024

  11. [19]

    Zeroth-Order Optimization Meets Human Feedback: Prov- able Learning via Ranking Oracles

    Zhiwei Tang, Dmitry Rybin, and Tsung-Hui Chang. Zeroth-Order Optimization Meets Human Feedback: Prov- able Learning via Ranking Oracles. In Proceedings of the International Conference on Learning Representations (ICLR), pages 1–31, May 2024

    2024

  12. [20]

    Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inference

    Qining Zhang and Lei Ying. Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inference. arXiv preprint arXiv:2409.17401, September 2024

    2024 arXiv

  13. [21]

    Variance-reduced Zeroth- Order Methods for Fine-Tuning Language Models

    Tanmay Gautam, Youngsuk Park, Hao Zhou, Parameswaran Raman, and Wooseok Ha. Variance-reduced Zeroth- Order Methods for Fine-Tuning Language Models. arXiv preprint arXiv:2404.08080, April 2024

    2024 arXiv

  14. [22]

    ZO-AdaMU Optimizer: Adapting Perturbation by the Momentum and Uncertainty in Zeroth-order Optimization

    Shuoran Jiang, Qingcai Chen, Youchen Pan, Yang Xiang, Yukang Lin, Xiangping Wu, Chuanyi Liu, and Xi- aobao Song. ZO-AdaMU Optimizer: Adapting Perturbation by the Momentum and Uncertainty in Zeroth-order Optimization. In Proceedings of the AAAI Conference on Artificial Intellig...

    2024

  15. [23]

    Sparse MeZO: Less Pa- rameters for Better Performance in Zeroth-Order LLM Fine-Tuning

    Yong Liu, Zirui Zhu, Chaoyu Gong, Minhao Cheng, Cho-Jui Hsieh, and Yang You. Sparse MeZO: Less Pa- rameters for Better Performance in Zeroth-Order LLM Fine-Tuning. arXiv preprint arXiv:2402.15751, February 2024

    2024

  16. [24]

    Gardner, Osbert Bastani, Christopher De Sa, Xiaodong Yu, Beidi Chen, and Zhaozhuo Xu

    Wentao Guo, Jikai Long, Yimeng Zeng, Zirui Liu, Xinyu Yang, Yide Ran, Jacob R. Gardner, Osbert Bastani, Christopher De Sa, Xiaodong Yu, Beidi Chen, and Zhaozhuo Xu. Zeroth-Order Fine-Tuning of LLMs with Extreme Sparsity. arXiv preprint arXiv:2406.02913, June 2024

    2024 arXiv

  17. [25]

    signSGD via Zeroth-Order Oracle

    Sijia Liu, Pin-Yu Chen, Xiangyi Chen, and Mingyi Hong. signSGD via Zeroth-Order Oracle. In Proceedings of the International Conference on Learning Representations (ICLR), pages 1–24, May 2019

    2019

  18. [26]

    Random Gradient-Free Minimization of Convex Functions

    Yurii Nesterov and Vladimir Spokoiny. Random Gradient-Free Minimization of Convex Functions. Foundations of Computational Mathematics, 17(1):527–566, April 2017

    2017

  19. [27]

    Stochastic First- and Zeroth-Order Methods for Nonconvex Stochastic Pro- gramming

    Saeed Ghadimi and Guanghui Lan. Stochastic First- and Zeroth-Order Methods for Nonconvex Stochastic Pro- gramming. SIAM Journal on Optimization, 23(4):2341–2368, December 2013

    2013

  20. [28]

    A Comprehensive Linear Speedup Analysis for Asynchronous Stochastic Parallel Optimization from Zeroth-Order to First-Order

    Xiangru Lian, Huan Zhang, Cho-Jui Hsieh, Yijun Huang, and Ji Liu. A Comprehensive Linear Speedup Analysis for Asynchronous Stochastic Parallel Optimization from Zeroth-Order to First-Order. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), page...

    2016

  21. [29]

    Zeroth-Order Stochas- tic Variance Reduction for Nonconvex Optimization

    Sijia Liu, Bhavya Kailkhura, Pin-Yu Chen, Paishun Ting, Shiyu Chang, and Lisa Amini. Zeroth-Order Stochas- tic Variance Reduction for Nonconvex Optimization. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), pages 1–11, December 2018

    2018

  22. [30]

    Improved Zeroth-Order Variance Reduced Algorithms and Analysis for Nonconvex Optimization

    Kaiyi Ji, Zhe Wang, Yi Zhou, and Yingbin Liang. Improved Zeroth-Order Variance Reduced Algorithms and Analysis for Nonconvex Optimization. In Proceedings of the International Conference on Machine Learning (ICML), pages 3100–3109, June 2019

    2019

  23. [31]

    Reddi, Ahmed Hefny, Suvrit Sra, Barnabas Poczos, and Alex Smola

    Sashank J. Reddi, Ahmed Hefny, Suvrit Sra, Barnabas Poczos, and Alex Smola. Stochastic Variance Reduction for Nonconvex Optimization. In Proceedings of the International Conference on Machine Learning (ICML) , pages 314–323, June 2016

    2016

  24. [32]

    signSGD: Com- pressed Optimisation for Non-Convex Problems

    Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadenesheli, and Animashree Anandkumar. signSGD: Com- pressed Optimisation for Non-Convex Problems. In Proceedings of the International Conference on Machine Learning (ICML), pages 560–569, July 2018

    2018

  25. [33]

    Accelerated Zeroth-Order and First-Order Momen- tum Methods from Mini to Minimax Optimization

    Feihu Huang, Shangqian Gao, Jian Pei, and Heng Huang. Accelerated Zeroth-Order and First-Order Momen- tum Methods from Mini to Minimax Optimization. The Journal of Machine Learning Research , 23(36):1–70, February 2022

    2022

  26. [34]

    ZO-AdaMM: Zeroth- Order Adaptive Momentum Method for Black-Box Optimization

    Xiangyi Chen, Sijia Liu, Kaidi Xu, Xingguo Li, Xue Lin, Mingyi Hong, and David Cox. ZO-AdaMM: Zeroth- Order Adaptive Momentum Method for Black-Box Optimization. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), pages 7204–7215, December 2019

    2019

  27. [35]

    AdaZeta: Adaptive Zeroth- Order Tensor-Train Adaption for Memory-Efficient Large Language Models Fine-Tuning

    Yifan Yang, Kai Zhen, Ershad Banijamal, Athanasios Mouchtaris, and Zheng Zhang. AdaZeta: Adaptive Zeroth- Order Tensor-Train Adaption for Memory-Efficient Large Language Models Fine-Tuning. InProceedings of the Conference on Empirical Methods in Natural Language Processing (EM...

    2024

  28. [36]

    Private Fine-tuning of Large Language Models with Zeroth-order Optimization

    Xinyu Tang, Ashwinee Panda, Milad Nasr, Saeed Mahloujifar, and Prateek Mittal. Private Fine-tuning of Large Language Models with Zeroth-order Optimization. arXiv preprint arXiv:2401.04343, January 2024

    2024 arXiv

  29. [37]

    Federated Full- Parameter Tuning of Billion-Sized Language Models with Communication Cost under 18 Kilobytes

    Zhen Qin, Daoyuan Chen, Bingchen Qian, Bolin Ding, Yaliang Li, and Shuiguang Deng. Federated Full- Parameter Tuning of Billion-Sized Language Models with Communication Cost under 18 Kilobytes. In Pro- ceedings of the International Conference on Machine Learning (ICML), pages 4...

    2024

  30. [38]

    On the Convergence of Zeroth-Order Federated Tuning for Large Language Models

    Zhenqing Ling, Daoyuan Chen, Liuyi Yao, Yaliang Li, and Ying Shen. On the Convergence of Zeroth-Order Federated Tuning for Large Language Models. In Proceedings of the ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), pages 1827–1838, August 2024

    2024

  31. [39]

    Poor Man’s Training on MCUs: A Memory-Efficient Quantized Back-Propagation-Free Approach

    Yequan Zhao, Hai Li, Ian Young, and Zheng Zhang. Poor Man’s Training on MCUs: A Memory-Efficient Quantized Back-Propagation-Free Approach. arXiv preprint arXiv:2411.05873, November 2024

    2024 arXiv

  32. [40]

    PocketLLM: Enabling On-Device Fine-Tuning for Personalized LLMs

    Dan Peng, Zhihui Fu, and Jun Wang. PocketLLM: Enabling On-Device Fine-Tuning for Personalized LLMs. In Proceedings of the Workshop on Privacy in Natural Language Processing (PrivateNLP) , pages 91–96, August 2024

    2024

  33. [41]

    J.C. Spall. Multivariate Stochastic Approximation using a Simultaneous Perturbation Gradient Approximation. IEEE Transactions on Automatic Control, 37(3):332–341, March 1992

    1992

  34. [42]

    Maolin Wang, Seyedramin Rasoulinezhad, Philip H. W. Leong, and Hayden K.-H. So. NITI: Training Inte- ger Neural Networks Using Integer-Only Arithmetic. IEEE Transactions on Parallel and Distributed Systems (TPDS), 33(11):3249–3261, November 2022

    2022

  35. [43]

    Gradient-Based Learning Applied to Document Recognition

    Yann Lecun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-Based Learning Applied to Document Recognition. Proceedings of the IEEE, 86(11):2278–2324, November 1998

    1998

  36. [44]

    Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms

    Han Xiao, Kashif Rasul, and Roland V ollgraf. Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms. arXiv preprint arXiv:1708.07747, September 2017

    2017 arXiv

  37. [45]

    3D ShapeNets: A Deep Representation for V olumetric Shapes

    Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3D ShapeNets: A Deep Representation for V olumetric Shapes. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1912–1920, June 2015

    1912

  38. [46]

    Qi, Hao Su, Kaichun Mo, and Leonidas J

    Charles R. Qi, Hao Su, Kaichun Mo, and Leonidas J. Guibas. PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 652–660, July 2017

    2017

  39. [47]

    Efficient Domain Generalization via Common-Specific Low-Rank Decomposition

    Vihari Piratla, Praneeth Netrapalli, and Sunita Sarawagi. Efficient Domain Generalization via Common-Specific Low-Rank Decomposition. In Proceedings of the International Conference on Machine Learning (ICML), pages 7728–7738, July 2020

    2020

  40. [48]

    PRIOT: Pruning-Based Integer-Only Transfer Learning for Embedded Systems

    Honoka Anada, Sefutsu Ryu, Masayuki Usui, Tatsuya Kaneko, and Shinya Takamaeda-Yamazaki. PRIOT: Pruning-Based Integer-Only Transfer Learning for Embedded Systems. IEEE Embedded Systems Letters, Octo- ber 2024. Early Access

    2024

  41. [49]

    LoRA: Low-Rank Adaptation of Large Language Models

    Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-Rank Adaptation of Large Language Models. InProceedings of the International Conference on Learning Representations (ICLR), pages 1–13, April 2022

    2022

  42. [50]

    QLoRA: Efficient Finetuning of Quan- tized LLMs

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. QLoRA: Efficient Finetuning of Quan- tized LLMs. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), pages 10088– 10115, December 2023. 16

    2023

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.