Pith. sign in

REVIEW 4 major objections 6 minor 52 references

FZOO: Fast Zeroth-Order Optimizer for Fine-Tuning Large Language Models towards Adam-Scale Speed

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read FZOO, a forward-only zeroth-order optimizer, claims Adam-scale fine-tuning speed at inference-level memory, reporting 18× fewer forward passes than MeZO on RoBERTa-large.

desk verdict The batched forward pass in Algorithm 1 does not compute the estimator that the theory analyzes, so the convergence guarantees apply to a different algorithm than the one being tested. read the letter →

arxiv 2506.09034 v2 pith:2L5ZGK7N submitted 2025-06-10 cs.LG cs.AI

Sizhe Dang , Yangyang Guo , Yanjun Zhao , Haishan Ye , Xiaodong Zheng , Guang Dai , Ivor Tsang This is my paper
classification cs.LGcs.AI MSC 68T0790C56
keywords zeroth-orderoptimizationLLMfine-tuningnormalizedSGDRademacherperturbationbatchedforwardpassesinference-levelmemoryforward-only
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Fine-tuning large language models with Adam multiplies memory cost by more than ten because the backward pass stores activations; zeroth-order optimizers avoid that by using only forward passes, but they have been much slower to converge. FZOO is a zeroth-order optimizer that tries to remove this speed penalty. It estimates the gradient from a batch of one-sided function-value differences along Rademacher ($\pm1$) directions and divides the estimate by the standard deviation of the batch losses, a step the paper proves to be, up to higher-order terms, a normalized stochastic gradient in the sense of normalized-SGD. The $\pm1$ directions also let the $N$ perturbed forward passes be fused into single parallel kernels. The paper reports 3\% average accuracy gain over MeZO with 3$\times$ fewer forward passes, an 18$\times$ reduction on RoBERTa-large with convergence comparable to Adam, and inference-level memory throughout. If right, this makes single-GPU full-parameter fine-tuning of models like OPT-30B practical today.

What carries the argument

Three mechanisms carry the argument. The first is the batched one-sided estimator with Rademacher directions: because $u_i \in \{+1,-1\}^d$, the layer-wise perturbation $P^{(j)} = \epsilon (u \odot Y^{(j-1)})$ is an element-wise sign flip and addition rather than a second matrix multiplication, so all $N$ perturbed forwards can be concatenated along the batch axis and executed in fused CUDA kernels instead of sequential passes. The second is the variance-normalized update rule $g_t/\sigma_t$: Proposition 3.2 and Remark 3.3 identify the ratio $\mathbb{E}[\|g_t\|^2]/\mathbb{E}[\sigma_t^2]$ with the constant $\frac{N+d-1}{N}\epsilon^{-2}$, which is what licenses reading the update as normalized stochastic gradient descent, with the $\sigma_t$ denominator supplying Adam-like adaptivity at no memory cost. The third is the seed-replay device inherited from MeZO: perturbation vectors are regenerated from stored random seeds during the parameter update, so the full $N \times d$ perturbation matrix never has to reside in GPU memory.

What would settle it

Fine-tune the same model and task twice with identical seeds and hyperparameters: once with the batched activation-perturbation forward of Algorithm 1, and once with a genuine parameter-perturbation forward that evaluates $L(\theta + \epsilon u)$ for the same Rademacher vectors (the paper's own Algorithm 3 implements exactly this). If the per-step loss estimates and converged accuracies differ systematically, the batched forward is not computing the perturbed-parameter loss, and the normalized-SGD analysis of Proposition 3.2 does not govern the update that actually runs.

Watch

Extended reading notes

Core claim

The paper's claim is that the apparent speed/memory trade-off of zeroth-order fine-tuning is not fundamental. FZOO uses the update $\theta_{t+1} = \theta_t - \eta_t g_t/\sigma_t$, where $g_t = \frac{1}{\epsilon N}\sum_{i=1}^N (L(\theta_t + \epsilon u_i; B_t) - L(\theta_t; B_t)) u_i$ is a batched one-sided gradient estimate along $N$ i.i.d. Rademacher ($\pm 1$) directions $u_i$, and $\sigma_t$ is the sample standard deviation of the $N$ perturbed losses. Proposition 3.2 computes $\mathbb{E}[\|g_t\|^2] = \frac{N+d-1}{N}\|\nabla L(\theta_t, B_t)\|^2 + O(\epsilon)$ and $\mathbb{E}[\sigma_t^2] = \epsilon^2 \|\nabla L(\theta_t, B_t)\|^2 + O(\epsilon^3)$, so up to higher-order terms the ratio of the two expectations is the iteration-independent constant $\frac{N+d-1}{N}\epsilon^{-2}$; Remark 3.3 uses this to regard $g_t/\sigma_t$ as a normalized stochastic gradient. The paper argues this gives Adam-style adaptivity, larger steps on flat loss regions and smaller steps on steep ones, without Adam's momentum memory, and Theorem 3.6 supplies a convergence bound for smooth objectives with bounded variance. Empirically, FZOO is reported to beat MeZO by 3\% average accuracy with 3$\times$ fewer forward passes across models up to OPT-66B, to converge comparably to Adam on RoBERTa-large with an 18$\times$ reduction in forward passes, to hold memory at the inference level, to optimize non-differentiable objectives directly, and to compose with parameter-efficient fine-tuning.

Load-bearing premise

The method's speedups are measured with a batched forward pass that nudges each layer's activations by a $\pm1$ pattern instead of nudging the model's weights themselves; the paper assumes these two procedures give the same losses, and that unproved equivalence is the only bridge between the gradient theory and the update that actually runs.

Editorial extensions

If this is right

  • Zeroth-order fine-tuning can reach Adam-scale wall-clock convergence on models like RoBERTa-large, so the assumption that forward-only methods are inherently slow stops holding for well-engineered batched estimators.
  • Full-parameter fine-tuning of a model like OPT-30B becomes feasible on a single GPU, since FZOO keeps memory at inference level while converging in far fewer steps than MeZO.
  • FZOO optimizes non-differentiable objectives directly, because it only queries function values; the paper reports a 5.53\% average F1 gain over MeZO on SQuAD across OPT scales.
  • FZOO is an update rule rather than a parameter-selection strategy, so it composes with PEFT methods such as prefix-tuning, yielding even larger memory savings.
  • Theorem 3.6's bound shows the variance-normalized zeroth-order update is provably convergent for smooth objectives under bounded variance, with the average squared gradient norm decreasing in the number of steps.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The theory and the implementation are joined by an unproved equivalence: the batched forward pass (Algorithm 1) adds $\epsilon(u \odot Y)$ to each layer's activations, which the paper treats as producing the same losses as perturbing every model parameter by $\epsilon u$; a head-to-head comparison of the batched path against genuine parameter perturbation (the paper's own Algorithm 3) would show w
  • The $\sigma_t$ normalization ties step size to the spread of losses across the perturbation batch; a natural extension would test whether the same normalization accelerates other zeroth-order estimators, or whether it functions mainly as an adaptive learning-rate schedule.
  • The wall-clock advantage depends on kernel fusion being available for batched forwards; on runtimes without such fusion, the gain would shrink to the step-count reduction alone, which the paper's released non-parallel variant could quantify.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes FZOO, a zeroth-order optimizer for fine-tuning large language models. FZOO uses batched Rademacher perturbations, a one-sided loss-difference estimator, and a step size normalized by the standard deviation of batch losses. The authors claim that FZOO is equivalent to a normalized-SGD update (Section 3.4), provide a convergence guarantee (Theorem 3.6), and report experiments on RoBERTa-large, OPT, Phi-2, and Llama3 showing that FZOO outperforms MeZO in accuracy while using far fewer forward passes, approaching Adam-scale convergence speed. The theoretical analysis is built on the estimator g_t in Eq. (2), defined through losses L(θ_t + εu_i; B_t), while the accelerated implementation in Algorithm 1 perturbs activations rather than parameters. This discrepancy is the central issue addressed in the major comments.

Significance. If the claims were correct, FZOO would be a practically valuable contribution: it would show that zeroth-order fine-tuning can come close to first-order methods in step count while keeping inference-level memory, and the batched Rademacher implementation would be a useful engineering trick for ZO optimization. The paper has genuine strengths: a broad empirical study spanning models from 350M to 66B parameters, experiments with non-differentiable objectives, memory profiling, and a non-parallel control variant (Algorithm 3) that helps isolate the effect of batching. However, the load-bearing inconsistency between the objective analyzed in Section 3.4 and the algorithm implemented in Section 3.3 means that the theoretical guarantees do not apply to the executed update, and the headline empirical speedup numbers conflate step-count speedup with wall-clock speedup. The central claim is therefore not supported as stated.

major comments (4)
  1. [§3.3, Algorithm 1 (lines 11–19) vs. §3.4, Eq. (2)] The batched forward pass in Algorithm 1 computes Y^(j) = F^(j) + ε(u ⊙ Y^(j-1)), which is equivalent to adding ε diag(u) to the weight matrix of each layer, i.e., a diagonal weight perturbation. It does not compute L(θ + εu_i; B), the quantity used to define g_t in Eq. (2) and analyzed in Proposition 3.2 and Theorem 3.6. For a dense weight matrix W^(j), a full parameter perturbation (W^(j) + εU)X differs from W^(j)X + ε(u ⊙ (W^(j)X)) unless U is diagonal, so the gradient estimated by Algorithm 1 is not the one-sided gradient of the true parameter-perturbed loss. Algorithm 3, the non-parallel variant, correctly perturbs θ, which confirms that the discrepancy is introduced specifically by the 'accelerated' implementation used in the experiments. This is an internal inconsistency between the theory and the implemented algorithm, and it undermines the convergence claims for the method as executed.
  2. [§3.4, Remark 3.3 and Proposition 3.2] The claimed 'formal equivalence to normalized-SGD' is not established. Equations (6) and (7) only relate E[||g_t||^2] to E[σ_t^2] up to O(ε) remainder terms. The actual update uses the ratio g_t/σ_t, where σ_t is a random variable, and the proof does not show that g_t/σ_t is close to g_t/||g_t|| in any relevant sense. This is a heuristic scaling relation, not an equivalence of update rules. Since the introduction lists 'a formal proof of its equivalence to a normalized-SGD update rule' as a contribution, this claim should either be proven or substantially softened.
  3. [§3.5, Theorem 3.6 and Appendix B.4] The proof requires the step-size condition η_t = η/σ_t ≤ N/(16dL) for every t, but the theorem's proposed choice η = sqrt((L(θ_1)-L(θ*))/(4dLV^2 Σ_k σ_k^{-2})) does not enforce this bound; σ_t is random and can be arbitrarily small. Without an additional condition or a high-probability lower bound on σ_t, the descent inequality used to telescope the sum is not justified, so the convergence guarantee as stated is incomplete.
  4. [Algorithm 1, line 4 vs. Eq. (2)] Algorithm 1 computes projected_grad = (ℓ - L(θ; B))/(N·std), whereas Eq. (2) defines g_t = (1/(εN)) Σ_i (l_i - l0) u_i. The factor 1/ε is missing from the implemented update. Unless ε = 1 or the learning rate is rescaled to absorb this factor, the update executed by Algorithm 1 is not the update analyzed in Theorem 3.6. The pseudocode should be aligned with Eq. (2), or the learning-rate rescaling should be stated explicitly.
minor comments (6)
  1. [§3.2.1 heading] The heading 'Movtivation of FZOO' contains a typo and should read 'Motivation of FZOO'.
  2. [Algorithm 1, line 4] The pseudocode uses L(θ; B) in line 4, but BatchPerturbParameters does not return or compute the unperturbed loss; the algorithm should specify how l0 is obtained.
  3. [References] Reference [2] is cited for normalized-SGD, but the cited Bernstein et al. paper is about signSGD and sign-based compression, not normalized-SGD; an appropriate reference for normalized gradient descent should be provided.
  4. [Table 7 header] The table header says 'Roberts-Large' instead of 'RoBERTa-large'.
  5. [Abstract, Figure 1, and Table 6] The speedup metrics are inconsistent across the abstract, Figure 1, and Table 6: '18× reduction in forward passes', '18× speedup', and '20× speed-up in total steps' are different quantities, and Table 5 shows that FZOO's per-step wall-clock time is actually higher than MeZO's. The paper should state explicitly which quantity each headline number refers to and should distinguish step-count speedup from wall-clock speedup.
  6. [§3.3, notation for U] The text introduces U = diag(u_1, ..., u_N) as a block-diagonal sign matrix, but the subsequent line P^(j) = ε(U ⊙ Y^(j-1)) suggests elementwise broadcasting; the shape and broadcasting convention for U should be clarified.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the normalized-SGD equivalence is a derived consistency result and the headline claims are benchmarked externally.

full rationale

The paper's main theoretical claim is that FZOO's update theta_{t+1} = theta_t - eta_t g_t / sigma_t (Eq. 4) is a normalized-SGD-type rule. This is not circular: g_t and sigma_t are defined from loss queries (Eqs. 2-3), and Proposition 3.2 derives the relation E[||g_t||^2] approximately ((N+d-1)/N) epsilon^{-2} E[sigma_t^2] from a Taylor expansion, so the normalization is a proven consequence rather than an assumed equality. The empirical speed and accuracy claims are benchmarked against external baselines MeZO and Adam, and the learning-rate/mu grids are standard hyperparameter selection, not fitted predictions of the reported metrics. The only overlapping-author citation, HiZOO [52], is used as a related-work baseline and is not load-bearing for FZOO's convergence or speed claims. Note: the reviewer-identified mismatch between Algorithm 1's batched activation perturbation and the exact L(theta + epsilon u) queried in Eq. 2 is a correctness/validity concern about whether the implementation matches the theory, not a circularity of the derivation chain; therefore it does not affect the circularity score here.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical entities. The central method depends on tuned hyperparameters (epsilon, N, learning rate) and on an unproven, likely false equivalence between activation perturbation and parameter perturbation, which is the most serious burden in the axiom ledger.

free parameters (3)
  • perturbation scale epsilon (mu) = 1e-4 for RoBERTa-large; grid 1e-5 to 1e-4 for OPT
    Tuned per model and task; controls the magnitude of the loss spread that is used for normalization and step sizing.
  • perturbation batch size N = 8
    Selected from the ablation grid in Appendix G.1 (Table 14) as the best trade-off; not derived from theory.
  • learning rate eta = 1e-4 for RoBERTa-large; 1e-5 to 5e-4 for OPT
    Grid-searched per model and dataset; appears directly in the parameter update and in the convergence theorem's step-size bound.
assumptions (4)
  • standard math The loss is L-smooth (Assumption 3.4)
    Standard smoothness assumption used in the convergence proof and in the Taylor-expansion bounds of Lemmas B.3-B.5.
  • standard math Bounded stochastic gradient variance (Assumption 3.5)
    Standard assumption for stochastic gradient convergence proofs.
  • standard math Taylor expansion remainder is small enough to treat high-order terms as negligible in Proposition 3.2
    Used in Remark 3.3 to claim approximate equivalence to normalized-SGD.
  • ad hoc to paper Perturbed activations in Algorithm 1 represent parameter perturbation of the loss L(theta + epsilon u)
    This equivalence is not proven and is generally false for matrix multiplications; it is the bridge between the theoretical analysis and the implemented algorithm. The paper provides no argument for why activation perturbation equals parameter perturbation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FZOO: Fast Zeroth-Order Optimizer for Fine-Tuning Large Language Models towards Adam-Scale Speed." pith.science (2026). https://pith.science/paper/2L5ZGK7N

@misc{pith2026250609034,
  author       = {Pith},
  title        = {Pith review of: FZOO: Fast Zeroth-Order Optimizer for Fine-Tuning Large Language Models towards Adam-Scale Speed},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2L5ZGK7N}},
  note         = {Machine review of arXiv:2506.09034}
}
read the original abstract

Fine-tuning large language models (LLMs) often faces GPU memory bottlenecks: the backward pass of first-order optimizers like Adam increases memory usage to more than 10 times the inference level (e.g., 633 GB for OPT-30B). Zeroth-order (ZO) optimizers avoid this cost by estimating gradients only from forward passes, yet existing methods like MeZO usually require many more steps to converge. Can this trade-off between speed and memory in ZO be fundamentally improved? Normalized-SGD demonstrates strong empirical performance with greater memory efficiency than Adam. In light of this, we introduce FZOO, a Fast Zeroth-Order Optimizer toward Adam-Scale Speed. FZOO reduces the total forward passes needed for convergence by employing batched one-sided estimates that adapt step sizes based on the standard deviation of batch losses. It also accelerates per-batch computation through the use of Rademacher random vector perturbations coupled with CUDA's parallel processing. Extensive experiments on diverse models, including RoBERTa-large, OPT (350M-66B), Phi-2, and Llama3, across 11 tasks validate FZOO's effectiveness. On average, FZOO outperforms MeZO by 3 percent in accuracy while requiring 3 times fewer forward passes. For RoBERTa-large, FZOO achieves average improvements of 5.6 percent in accuracy and an 18 times reduction in forward passes compared to MeZO, achieving convergence speeds comparable to Adam. We also provide theoretical analysis proving FZOO's formal equivalence to a normalized-SGD update rule and its convergence guarantees. FZOO integrates smoothly into PEFT techniques, enabling even larger memory savings. Overall, our results make single-GPU, high-speed, full-parameter fine-tuning practical and point toward future work on memory-efficient pre-training.

Figures

Figures reproduced from arXiv: 2506.09034 by the authors.

Figure 1
Figure 1. Performance of MeZO, Adam and FZOO on different tasks when fine-tuning RoBERTa [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Structure of the of FZOO. The bottom half depicts the toy example of the efficient [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Training loss curves when using MeZO and FZOO to fine-tune different LLMs on the [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: GPU memory consumption with different OPT models and tuning methods on MultiRC [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: FZOO FT vs PEFT (prefix) on Roberta-large. We use FZOO (prefix) to fine￾tune Roberta-large on multiple datasets. More results can be found in Appendix D.1 [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 7
Figure 7. Figure 7: Loss curves on opt-125m between FZOO and FZOO-R. [PITH_FULL_IMAGE:figures/full_fig_p022_7.png]
Figure 8
Figure 8. Figure 8: Loss curves on RoBERTa-large between MeZO and FZOO. [PITH_FULL_IMAGE:figures/full_fig_p023_8.png]
Figure 9
Figure 9. Figure 9: Loss curves on OPT. D.2 More results on various LLMs We use the hyperparameters in [PITH_FULL_IMAGE:figures/full_fig_p024_9.png]
Figure 10
Figure 10. Figure 10: Loss curves on Phi-2 between MeZO and FZOO. [PITH_FULL_IMAGE:figures/full_fig_p025_10.png]
Figure 11
Figure 11. Figure 11: Loss curves on Llama3 between MeZO and FZOO. [PITH_FULL_IMAGE:figures/full_fig_p025_11.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

52 extracted references · 29 canonical work pages

  1. [1]

    The fine-grained complexity of gradient computation for training large language models

    Josh Alman and Zhao Song. The fine-grained complexity of gradient computation for training large language models. arXiv preprint arXiv:2402.04497, 2024

    arXiv 2024

  2. [2]

    signsgd: Compressed optimisation for non-convex problems

    Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadenesheli, and Animashree Anandkumar. signsgd: Compressed optimisation for non-convex problems. In International Conference on Machine Learning, pages 560–569. PMLR, 2018

    work page 2018

  3. [3]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-V oss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwi...

    2020

  4. [4]

    A zeroth-order block coordinate descent algorithm for huge-scale black-box optimization

    Hanqin Cai, Yuchen Lou, Daniel Mckenzie, and Wotao Yin. A zeroth-order block coordinate descent algorithm for huge-scale black-box optimization. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 1193–1203. PMLR, 18–24 Jul 2021

    work page 2021

  5. [5]

    Towards efficient low-order hybrid optimizer for language model fine-tuning

    Minping Chen, You-Liang Huang, and Zeyi Wen. Towards efficient low-order hybrid optimizer for language model fine-tuning. 2025

    work page 2025

  6. [6]

    Zoo: Zeroth order optimization based black-box attacks to deep neural networks without training substitute models

    Pin-Yu Chen, Huan Zhang, Yash Sharma, Jinfeng Yi, and Cho-Jui Hsieh. Zoo: Zeroth order optimization based black-box attacks to deep neural networks without training substitute models. New York, NY , USA, 2017. Association for Computing Machinery

    work page 2017

  7. [7]

    Enhancing zeroth-order fine-tuning for language models with low-rank structures

    Yiming Chen, Yuan Zhang, Liyuan Cao, Kun Yuan, and Zaiwen Wen. Enhancing zeroth-order fine-tuning for language models with low-rank structures. arXiv preprint arXiv:2410.07698, 2024

    arXiv 2024

  8. [8]

    A memory efficient ran- domized subspace optimization method for training large language models

    Yiming Chen, Yuan Zhang, Yin Liu, Kun Yuan, and Zaiwen Wen. A memory efficient ran- domized subspace optimization method for training large language models. arXiv preprint arXiv:2502.07222, 2025. 10

    arXiv 2025

Show all 52 references
  1. [9]

    GPT3.int8(): 8-bit matrix multiplication for transformers at scale

    Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. GPT3.int8(): 8-bit matrix multiplication for transformers at scale. In Advances in Neural Information Processing Systems, 2022

    2022

  2. [10]

    8-bit optimizers via block-wise quantization

    Tim Dettmers, Mike Lewis, Sam Shleifer, and Luke Zettlemoyer. 8-bit optimizers via block-wise quantization. In International Conference on Learning Representations, 2022

    2022

  3. [11]

    Qlora: Efficient finetuning of quantized llms, 2023

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms, 2023

    2023

  4. [12]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In North American Chapter of the Association for Computational Linguistics, 2019

    2019

  5. [13]

    Adaptive subgradient methods for online learning and stochastic optimization

    John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. J. Mach. Learn. Res., 12(null):2121–2159, jul 2011

    2011

  6. [14]

    Textbooks are all you need, 2023

    Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, Sébastien Bubeck, Ronen Eldan, Adam Tau- man Kalai,...

    2023

  7. [15]

    Zeroth-order fine-tuning of llms with extreme sparsity

    Wentao Guo, Jikai Long, Yimeng Zeng, Zirui Liu, Xinyu Yang, Yide Ran, Jacob R Gardner, Osbert Bastani, Christopher De Sa, Xiaodong Yu, et al. Zeroth-order fine-tuning of llms with extreme sparsity. arXiv preprint arXiv:2406.02913, 2024

    2024 arXiv

  8. [16]

    Suchin Gururangan, Ana Marasovi´c, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. Don’t stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 834...

    2020

  9. [17]

    Zavlanos

    Davood Hajinezhad and Michael M. Zavlanos. Gradient-free multi-agent nonconvex nonsmooth optimization. 2018 IEEE Conference on Decision and Control (CDC), pages 4939–4944, 2018

    2018

  10. [18]

    LoRA: Low-rank adaptation of large language models

    Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022

    2022

  11. [19]

    Zo-adamu optimizer: Adapting perturbation by the momentum and uncertainty in zeroth-order optimization, 2023

    Shuoran Jiang, Qingcai Chen, Youchen Pan, Yang Xiang, Yukang Lin, Xiangping Wu, Chuanyi Liu, and Xiaobao Song. Zo-adamu optimizer: Adapting perturbation by the momentum and uncertainty in zeroth-order optimization, 2023

    2023

  12. [20]

    Adam: A method for stochastic optimization

    Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), San Diega, CA, USA, 2015

    2015

  13. [21]

    Albert: A lite bert for self-supervised learning of language representations

    Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. In International Conference on Learning Representations, 2020

    2020

  14. [22]

    Prefix-tuning: Optimizing continuous prompts for generation

    Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International J...

    2021

  15. [23]

    Textbooks are all you need ii: phi-1.5 technical report, 2023

    Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks are all you need ii: phi-1.5 technical report, 2023

    2023

  16. [24]

    signsgd via zeroth-order oracle

    Sijia Liu, Pin-Yu Chen, Xiangyi Chen, and Mingyi Hong. signsgd via zeroth-order oracle. In International Conference on Learning Representations, 2019

    2019

  17. [25]

    Roberta: A robustly optimized bert pretraining approach

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019. 11

    1907 arXiv

  18. [26]

    Sparse mezo: Less parameters for better performance in zeroth-order llm fine-tuning

    Yong Liu, Zirui Zhu, Chaoyu Gong, Minhao Cheng, Cho-Jui Hsieh, and Yang You. Sparse mezo: Less parameters for better performance in zeroth-order llm fine-tuning. arXiv preprint arXiv:2402.15751, 2024

    2024

  19. [27]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019

    2019

  20. [28]

    Lee, Danqi Chen, and Sanjeev Arora

    Sadhika Malladi, Tianyu Gao, Eshaan Nichani, Alex Damian, Jason D. Lee, Danqi Chen, and Sanjeev Arora. Fine-tuning language models with just forward passes. In Thirty-seventh Conference on Neural Information Processing Systems, 2023

    2023

  21. [29]

    Simple random search provides a competitive approach to reinforcement learning, 2018

    Horia Mania, Aurelia Guy, and Benjamin Recht. Simple random search provides a competitive approach to reinforcement learning, 2018

    2018

  22. [30]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:2773...

    2022

  23. [31]

    Lisa: Layerwise importance sampling for memory-efficient large language model fine-tuning, 2024

    Rui Pan, Xiang Liu, Shizhe Diao, Renjie Pi, Jipeng Zhang, Chi Han, and Tong Zhang. Lisa: Layerwise importance sampling for memory-efficient large language model fine-tuning, 2024

    2024

  24. [32]

    Language models are unsupervised multitask learners

    Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019

    2019

  25. [33]

    J.C. Spall. Multivariate stochastic approximation using a simultaneous perturbation gradient approximation. IEEE Transactions on Automatic Control, 37(3):332–341, 1992

    1992

  26. [34]

    BBTv2: Towards a gradient-free future with large language models

    Tianxiang Sun, Zhengfu He, Hong Qian, Yunhua Zhou, Xuanjing Huang, and Xipeng Qiu. BBTv2: Towards a gradient-free future with large language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3916–3930, 2022

    2022

  27. [35]

    Black-box tuning for language-model-as-a-service

    Tianxiang Sun, Yunfan Shao, Hong Qian, Xuanjing Huang, and Xipeng Qiu. Black-box tuning for language-model-as-a-service. In International Conference on Machine Learning, pages 20841–20855, 2022

    2022

  28. [36]

    Tezo: Empowering the low-rankness on the temporal dimension in the zeroth-order optimization for fine-tuning llms

    Yan Sun, Tiansheng Huang, Liang Ding, Li Shen, and Dacheng Tao. Tezo: Empowering the low-rankness on the temporal dimension in the zeroth-order optimization for fine-tuning llms. arXiv preprint arXiv:2501.19057, 2025

    2025 arXiv

  29. [37]

    Harmony in divergence: Towards fast, accurate, and memory-efficient zeroth-order llm fine-tuning

    Qitao Tan, Jun Liu, Zheng Zhan, Caiwei Ding, Yanzhi Wang, Jin Lu, and Geng Yuan. Harmony in divergence: Towards fast, accurate, and memory-efficient zeroth-order llm fine-tuning. arXiv preprint arXiv:2502.03304, 2025

    2025

  30. [38]

    Effectively learning from data and generating data in differentially private machine learning

    Xinyu Tang. Effectively learning from data and generating data in differentially private machine learning. PhD thesis, Princeton University, 2024

    2024

  31. [39]

    Distributed zero-order algorithms for nonconvex multiagent optimization

    Yujie Tang, Junshan Zhang, and Na Li. Distributed zero-order algorithms for nonconvex multiagent optimization. IEEE Transactions on Control of Network Systems, 8(1):269–281, 2021

    2021

  32. [40]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation langua...

    2023 arXiv

  33. [41]

    ReLIZO: Sample reusable linear interpolation-based zeroth-order optimization

    Xiaoxing Wang, Xiaohan Qin, Xiaokang Yang, and Junchi Yan. ReLIZO: Sample reusable linear interpolation-based zeroth-order optimization. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  34. [42]

    The advancement in stochastic zeroth-order optimization: Mechanism of accelerated convergence of gaussian direction on objectives with skewed hessian eigenvalues

    Yilong Wang, Haishan Ye, Yong Liu, Guang Dai, Ivor Tsang, and Jingdong Wang. The advancement in stochastic zeroth-order optimization: Mechanism of accelerated convergence of gaussian direction on objectives with skewed hessian eigenvalues. 12

  35. [43]

    Roofline: an insightful visual performance model for multicore architectures

    Samuel Williams, Andrew Waterman, and David Patterson. Roofline: an insightful visual performance model for multicore architectures. Communications of the ACM, 52(4):65–76, 2009

    2009

  36. [44]

    Hessian-aware zeroth-order optimization for black-box adversarial attack, 2019

    Haishan Ye, Zhichao Huang, Cong Fang, Chris Junchi Li, and Tong Zhang. Hessian-aware zeroth-order optimization for black-box adversarial attack, 2019

    2019

  37. [45]

    Large batch optimization for deep learning: Training bert in 76 minutes

    Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large batch optimization for deep learning: Training bert in 76 minutes. In International Conference on Learning Representations, 2020

    2020

  38. [46]

    Matthew D. Zeiler. Adadelta: An adaptive learning rate method, 2012

    2012

  39. [47]

    Opt: Open pre-trained transformer language models, 2022

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. O...

    2022

  40. [48]

    Zavlanos

    Yan Zhang, Yi Zhou, Kaiyi Ji, and Michael M. Zavlanos. A new one-point residual-feedback oracle for black-box learning and control. Automatica, 136(C), feb 2022

    2022

  41. [49]

    Lee, Wotao Yin, Mingyi Hong, Zhangyang Wang, Sijia Liu, and Tianlong Chen

    Yihua Zhang, Pingzhi Li, Junyuan Hong, Jiaxiang Li, Yimeng Zhang, Wenqing Zheng, Pin-Yu Chen, Jason D. Lee, Wotao Yin, Mingyi Hong, Zhangyang Wang, Sijia Liu, and Tianlong Chen. Revisiting zeroth-order optimization for memory-efficient LLM fine-tuning: A benchmark. In Ruslan S...

    2024

  42. [50]

    Helene: Hessian layer-wise clipping and gradient annealing for accelerating fine-tuning llm with zeroth-order optimization

    Huaqin Zhao, Jiaxi Li, Yi Pan, Shizhe Liang, Xiaofeng Yang, Wei Liu, Xiang Li, Fei Dou, Tianming Liu, and Jin Lu. Helene: Hessian layer-wise clipping and gradient annealing for accelerating fine-tuning llm with zeroth-order optimization. arXiv preprint arXiv:2411.10696, 2024

    2024 arXiv

  43. [51]

    Galore: Memory-efficient llm training by gradient low-rank projection

    Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memory-efficient llm training by gradient low-rank projection. In International Conference on Learning Representations, 2024

    2024

  44. [52]

    Yanjun Zhao, Sizhe Dang, Haishan Ye, Guang Dai, Yi Qian, and Ivor W. Tsang. Second-order fine-tuning without pain for llms:a hessian informed zeroth-order optimizer. In International Conference on Learning Representations, 2025. 13 A Related Works A.1 First-order Optimizer Use...

    2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.