Pith. sign in

REVIEW 4 major objections 3 minor 62 references

KerZOO: Kernel Function Informed Zeroth-Order Optimization for Accurate and Accelerated LLM Fine-Tuning

T0 review · 4 major / 3 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read KerZOO claims that weighting zeroth-order gradient estimates with a Legendre-polynomial kernel removes the lower-order bias that slows forward-only LLM fine-tuning, cutting GPU training time by up to 74% on WSC and 44% on MultiRC while…

desk verdict A promising empirical prototype undercut by a theory-algorithm mismatch: the r-shrink heuristic breaks the bias-removal condition, and the speedups depend on an undefined convergence threshold. read the letter →

arxiv 2505.18886 v1 pith:6H4SEO2C submitted 2025-05-24 cs.LG

Zhendong Mi , Qitao Tan , Xiaodong Yu , Zining Zhu , Geng Yuan , Shaoyi Huang This is my paper · ORCID
classification cs.LG
keywords zeroth-orderoptimizationLLMfine-tuningkernelfunctiongradientestimationbiasLegendrepolynomialsmomentconditionsmemory-efficienttrainingconvergenceacceleration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

KerZOO claims that the slow convergence of zeroth-order (forward-only) fine-tuning of large language models comes largely from a lower-order bias in the gradient estimate, and that this bias can be removed by weighting the symmetric perturbation with a kernel function chosen to satisfy moment conditions. The paper derives the bias term from a Taylor expansion, constructs kernel functions from shifted Legendre polynomials so that the first moment is a nonzero constant while the third moment is zero, and shows that the resulting estimator has only fourth-order bias. On OPT-2.7B this reduces GPU training hours by up to 74% on WSC and 44% on MultiRC while exceeding the baseline accuracy by 2.9% and 2.6%. The method works in both full-parameter and LoRA-based fine-tuning and remains memory-efficient, using only forward passes.

What carries the argument

The key object is the kernel function $K_\beta(r) = C \sum_{m=0}^{\beta} p'_m(0) p_m(r)$, where $p_m$ are normalized Legendre polynomials. The design principle is a pair of moment conditions: $E[rK(r)] = C$ keeps the estimator aligned with the gradient, and $E[r^3K(r)] = 0$ removes the leading second-order bias; higher-order kernels also zero $E[r^5K(r)]$ and beyond. In the algorithm, the random direction $u$ is sampled from the unit sphere and the scalar $r$ is sampled uniformly in $[-1,1]$, with the range of $r$ shrunk as iterations increase. The Legendre construction is what makes the moment conditions hold exactly for $r \sim U[-1,1]$, so the Taylor-expansion bias analysis carries through.

What would settle it

Compute the third moment $E[r^3 K_3(r)]$ under the actual shrinking-$r$ schedule used in the algorithm (for example, with $r$ uniform on $[-a_t, a_t]$ for an announced schedule $a_t < 1$); whenever it is nonzero, the leading second-order bias is not removed in the deployed method. Separately, rerun the OPT-2.7B WSC and MultiRC comparisons with an explicitly stated convergence threshold and verify that the 74% and 44% GPU-hour reductions persist.

Watch

Extended reading notes

Core claim

The central claim is that a kernel-weighted symmetric-difference estimator, $\hat{g}_K = \frac{L(\theta+\epsilon r u)-L(\theta-\epsilon r u)}{2\epsilon} K(r) u$, eliminates the $O(\epsilon^2)$ bias of standard zeroth-order gradient estimates. Taylor-expanding the loss shows that the leading bias comes from the third directional derivative; weighting by a kernel with $E[rK(r)] = C$ and $E[r^3K(r)] = 0$ cancels that term, so $E[\hat{g}_K] = \frac{C}{d}\nabla L(\theta) + O(\epsilon^4)$. The construction draws $r$ uniformly from $[-1,1]$ and uses $K_3(r) = C\cdot \frac{15}{4}r(5-7r^2)$, built from Legendre polynomials; higher-order kernels such as $K_5$ can also remove higher moments. Empirically, with only three perturbations per step, the paper reports that KerZOO reaches comparable or better accuracy than the standard zeroth-order baseline while cutting training iterations by more than 70% on RoBERTa-large and reducing GPU hours on OPT-2.7B by up to 74% on WSC and 44% on MultiRC.

Load-bearing premise

The bias-removal guarantee depends on $r$ being drawn from a fixed uniform distribution on $[-1,1]$, where the kernel's moment conditions hold; the practical algorithm shrinks the range of $r$ as iterations increase, changing that distribution and reintroducing the lower-order bias, and the paper does not provide a schedule or analysis for this tradeoff.

Editorial extensions

If this is right

  • On OPT-2.7B, KerZOO reduces GPU hours to convergence by 74% on WSC and 44% on MultiRC compared with the standard forward-only baseline, while improving accuracy by 2.9% and 2.6%.
  • On RoBERTa-large few-shot and many-shot classification, KerZOO reduces training iterations by over 70% on average with only three perturbation directions, and improves accuracy over the baseline by 1.7 to 7.4 percentage points.
  • The same kernel estimator works under LoRA parameter-efficient fine-tuning, keeping memory close to the forward-only baseline (e.g., 9.7G versus 8.1G on OPT-2.7B SQuAD) while using about a third of the baseline's GPU hours.
  • The design principle is general: any kernel satisfying $E[rK(r)] = C$ and $E[r^3K(r)] = 0$ removes the leading second-order bias, and higher-order Legendre kernels can remove further moments.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the argument only requires Taylor smoothness, the same kernel-weighting trick could transfer to other zeroth-order settings, such as black-box adversarial attacks, hyperparameter search, or reinforcement-learning fine-tuning, wherever function evaluations are cheaper than gradient computation.
  • The practical shrinking of $r$'s range breaks the $r \sim U[-1,1]$ assumption behind the moment conditions; a schedule that either keeps the moments zero under the changing distribution or explicitly trades bias against variance would close the gap between the theory and the implementation.
  • The paper defines $K_5$ but experiments use $K_3$; testing $K_5$ would show whether removing the fourth-order bias further accelerates convergence or merely raises estimator variance.
  • The reported GPU-hour reductions are relative to unspecified convergence thresholds; fixing explicit thresholds would make the speedup numbers reproducible across implementations.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. The paper proposes KerZOO, a zeroth-order (ZO) gradient estimator for LLM fine-tuning that weights perturbation samples by a kernel function K(r). The authors derive a Taylor expansion showing that the expectation of the standard symmetric-difference estimator has an O(epsilon^2) bias term proportional to E[r^3K(r)], and design Legendre-polynomial kernels (K1, K3, K5) that satisfy E[rK(r)]=C and E[r^3K(r)]=0, reducing the bias to O(epsilon^4). They validate the approach on RoBERTa-large, OPT-2.7B/6.7B, and LLaMA-3-3B/8B against MeZO and HiZOO, reporting higher accuracy and reduced GPU hours, e.g., 74% and 44% reductions on WSC and MultiRC. The appendix includes a variance discussion, hyperparameters, and a convergence-analysis sketch.

Significance. If the bias-removal guarantee held for the implemented algorithm, KerZOO would be a useful, simple addition to the ZO fine-tuning toolbox: the moment-condition design is clearly stated, the K3 moment calculations are correct, and the experimental coverage is broad. The idealized estimator's O(epsilon^4)-bias property is a legitimate theoretical observation. However, the transfer from theory to practice is not established: the implemented r-shrinking contradicts the distributional assumption, and the convergence proof contains an incorrect identity. The contribution therefore needs substantial rework before the claims are supported.

major comments (4)
  1. [Section 3.3 / Appendix 9.1 / Algorithm 1] The bias-removal guarantee (Eq. 19) requires r to be drawn from the fixed distribution used to construct K, here Uniform[-1,1]. The paper states that in practice "we can limit the r in a smaller range as the iteration step increases" (Section 3.3) and Table 5 lists "r Shrink as iteration step increases", yet Algorithm 1 line 5 still samples r_i ~ Uniform[-1,1] and no schedule is given. For K3(r)=C*(15/4)r(5-7r^2) and r uniform on [-a,a], E[r^3K3(r)] = (15C/4)a^4(1-a^2), which is nonzero for every 0<a<1; hence the O(epsilon^2) bias that the method is designed to remove is present in the shrinking-range regime. Appendix 9.1 acknowledges that restricting r "may result in gradient estimates that are not strictly unbiased", but no quantitative bias analysis or fixed-range experiment is provided. The reported GPU-hour speedups therefore come from an algorithm not covered by the theoretical guarantee.
  2. [Appendix 9.4, Eq. (36)] The step labeled "3: a version of Stokes' theorem" asserts E[(1/(2epsilon))L(theta+epsilon r u)K(r)u] = E[(nabla L(theta+epsilon r u)) r K(r)]. This identity is false: for L(theta)=a^T theta, the left expectation is E[rK(r)](1/d)a = C a/d, while the right expectation is C a. The missing factor 1/d means the bias bound b <= kappa_beta L epsilon^{beta-1} is not established by the displayed derivation. Since this bound is the basis for the convergence theorem's iteration complexity, the convergence proof is invalid as written.
  3. [Section 4.3 / Tables 4, 8, 9 / Figure 2] The headline GPU-hour reductions (e.g., 74% on WSC, 44% on MultiRC) and the normalized "Iteration step" / "GPU hours" numbers in Tables 4, 8, and 9 are not accompanied by any definition of the convergence threshold used to stop training. Without specifying the stopping criterion, "time to convergence" is not a well-defined quantity, and the reported reductions cannot be verified or compared across methods. The authors should state the threshold (e.g., validation accuracy or loss level, or a fixed step budget) used for each table.
  4. [Appendix 9.4, Theorem] The convergence theorem is only sketched. The proof invokes a "biased oracle" bound from [61] without verifying its conditions for the kernel estimator, uses an undefined constant R in the condition epsilon <= (psi/(kappa_beta L R))^{1/(beta-1)}, and the "Sketchup for convergence" simply bounds each term by psi without justifying the simultaneous choice of N and epsilon. The theorem is not a complete proof; if it is intended as a formal contribution, the missing steps must be supplied. This is separate from the incorrect Stokes step in Eq. (36), which already invalidates the bias bound.
minor comments (3)
  1. [Section 3.4, Eq. (23)] The coefficient 195C/64 in the displayed integral for E[r^3K3(r)] should be 15C/4; the stated constant belongs to K5. The integral still vanishes, but the typo obscures the calculation.
  2. [Table 5] "r Shrink as iteration step increases" is not a hyperparameter value; a concrete schedule (e.g., a_t = 1 - t/N) and its theoretical effect on the moment conditions are needed.
  3. [Section 7] The Limitations section discusses only future application domains and does not mention the bias-variance trade-off from r-shrinking that is acknowledged in Appendix 9.1; this tension should be discussed explicitly.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the kernel-moment construction makes the O(ε²)-bias cancellation definitional, and the reported speedups are empirical and externally benchmarked.

full rationale

The central derivation (Section 3.3, Eq. 19) shows E[ĝ_K] = E[rK(r)](1/d)∇L + E[r³K(r)]E[ε²/6 D³L[u,u,u]u] + O(ε⁴). The kernel is then explicitly designed in Section 3.4 to satisfy E[rK]=C and E[r³K]=0 (Eqs. 22–23), so the disappearance of the O(ε²) term is a direct consequence of the defining moment conditions, not a prediction extracted from data. This is a legitimate construction, and the constant C is absorbed by the learning rate rather than fit to achieve the reported accuracies. The claimed GPU-hour reductions are empirical comparisons to MeZO/HiZOO on standard benchmarks, not outputs of Eq. 19; no fitted parameter is renamed as a prediction. There is one self-citation ([4], Qitao Tan and Geng Yuan of this paper) for the background statement that ZO methods converge slowly, but that claim is not load-bearing for the bias-removal derivation or the experimental conclusions. The convergence analysis (Appendix 9.4) states standard smoothness and noise assumptions and applies the same kernel moment conditions; it does not import any uniqueness theorem from the authors' prior work. The admitted limitation that shrinking the range of r (Appendix 9.1) 'may result in gradient estimates that are not strictly unbiased' is a mismatch between the idealized theory and the implemented schedule, not a circularity: it undermines the transfer of the unbiasedness guarantee without making any equation equal to its own input. Therefore the derivation is self-contained and no circular step is present.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The method introduces no new physical entities. All free parameters are tuning choices; the most consequential is the unspecified r-shrink schedule, which breaks the theoretical moment conditions. The axioms are standard smoothness assumptions plus an unproved integration identity used only in the convergence sketch.

free parameters (4)
  • Kernel constant C = 4
    Hand-chosen constant in K(r) that scales the effective gradient; appears in Eq (20) and Appendix 9.2. It acts as a global learning-rate multiplier.
  • r-shrink schedule = unspecified
    Appendix 9.1 says 'we can limit the r in a smaller range as the iteration step increases', but no schedule is given; this hand-designed bias-variance tradeoff changes the estimator's distribution.
  • Gradient clip constant R = unspecified
    Algorithm 1 line 8 projects the update onto a ball of radius R; R is never specified in the text.
  • Number of perturbations n = 3
    Chosen for all experiments (Appendix 9.1: 'we use only three random perturbations'); it triples per-step forward passes.
assumptions (5)
  • domain assumption Loss function is at least third-order differentiable
    Taylor expansion in Section 3.3 (Eqs 13-16) requires third-order differentiability; not guaranteed for ReLU-based networks.
  • standard math E[uu^T] = (1/d) I for unit Gaussian or uniform-sphere u
    Used in Eq (11) and Eq (19) to relate the expectation of <grad, u>u to the gradient.
  • domain assumption Independence of r and u
    Used to factor expectations in Eq (19); the algorithm samples them independently, so this holds.
  • domain assumption Assumptions 1-3 for the convergence theorem (L-smoothness, higher-order smoothness, overparameterized variance bound)
    Appendix 9.4 states these assumptions; they are standard but not verified for LLM losses.
  • ad hoc to paper Stokes' theorem identity used in the bias bound
    The step labeled 'a version of the Stokes theorem [62]' in Eq (36) is not stated or proven, and appears to replace the expectation of the difference estimator with a gradient expression; this is a gap in the convergence proof.

how reviews work

0 comments
Cite this review

Pith. "Pith review of KerZOO: Kernel Function Informed Zeroth-Order Optimization for Accurate and Accelerated LLM Fine-Tuning." pith.science (2026). https://pith.science/paper/6H4SEO2C

@misc{pith2026250518886,
  author       = {Pith},
  title        = {Pith review of: KerZOO: Kernel Function Informed Zeroth-Order Optimization for Accurate and Accelerated LLM Fine-Tuning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6H4SEO2C}},
  note         = {Machine review of arXiv:2505.18886}
}
read the original abstract

Large language models (LLMs) have demonstrated impressive capabilities across numerous NLP tasks. Nevertheless, conventional first-order fine-tuning techniques impose heavy memory demands, creating practical obstacles to real-world applications. Zeroth-order (ZO) optimization has recently emerged as a promising memory-efficient alternative, as it circumvents the need for backpropagation by estimating gradients solely through forward passes--making it particularly suitable for resource-limited environments. Despite its efficiency, ZO optimization suffers from gradient estimation bias, which significantly hinders convergence speed. To address this, we analytically identify and characterize the lower-order bias introduced during ZO-based gradient estimation in LLM fine-tuning. Motivated by tools in mathematical physics, we introduce a kernel-function-based ZO framework aimed at mitigating this bias and improving optimization stability. KerZOO achieves comparable or superior performance to existing ZO baselines in both full-parameter and parameter-efficient fine-tuning settings of LLMs, while significantly reducing the number of iterations required to reach convergence. For example, KerZOO reduces total GPU training hours by as much as 74% and 44% on WSC and MultiRC datasets in fine-tuning OPT-2.7B model and can exceed the MeZO baseline by 2.9% and 2.6% in accuracy. We show that the kernel function is an effective avenue for reducing estimation bias in ZO methods.

Figures

Figures reproduced from arXiv: 2505.18886 by the authors.

Figure 1
Figure 1. Training loss comparison of MeZO and KerZOO (Ours) on RoBERTa-large [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 3
Figure 3. Results on LLaMA3-3B and LLaMA3-8B We evaluate KerZOO on OPT and LLaMA series models across classification and generation tasks. The experiments span OPT-2.7B, OPT-6.7B, and LLaMA3-3B/8B models. Results are reported in Ta￾7 [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 2
Figure 2. Comparison of GPU hours for convergence across different datasets on OPT-2.7B between [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Training loss curves of MeZO with different perturbations and KerZOO [PITH_FULL_IMAGE:figures/full_fig_p016_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

62 extracted references · 33 canonical work pages

  1. [24]

    Optimal order of accuracy of search algorithms in stochastic optimization

    Boris Teodorovich Polyak and Aleksandr Borisovich Tsybakov. Optimal order of accuracy of search algorithms in stochastic optimization. Problemy Peredachi Informatsii, 26(2):45–53, 1990

    work page 1990

  2. [19]

    Accelerated zero-order sgd under high-order smoothness and overparameterized regime

    Georgii Bychkov, Darina Dvinskikh, Anastasia Antsiferova, Alexander Gasnikov, and Aleksandr Lobanov. Accelerated zero-order sgd under high-order smoothness and overparameterized regime. arXiv preprint arXiv:2411.13999, 2024

    arXiv 2024

  3. [61]

    An even more optimal stochastic optimization algo- rithm: minibatching and interpolation learning

    Blake E Woodworth and Nathan Srebro. An even more optimal stochastic optimization algo- rithm: minibatching and interpolation learning. Advances in neural information processing systems, 34:7333–7345, 2021

    work page 2021

  4. [1]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021

    arXiv 2021

  5. [2]

    Qlora: Efficient finetuning of quantized llms

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances in neural information processing systems, 36:10088– 10115, 2023

    2023

  6. [3]

    Efficient on-chip learning for optical neural networks through power-aware sparse zeroth-order optimization

    Jiaqi Gu, Chenghao Feng, Zheng Zhao, Zhoufeng Ying, Ray T Chen, and David Z Pan. Efficient on-chip learning for optical neural networks through power-aware sparse zeroth-order optimization. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 7583–7591, 2021

    work page 2021

  7. [4]

    Harmony in divergence: Towards fast, accurate, and memory-efficient zeroth-order llm fine-tuning

    Qitao Tan, Jun Liu, Zheng Zhan, Caiwei Ding, Yanzhi Wang, Jin Lu, and Geng Yuan. Harmony in divergence: Towards fast, accurate, and memory-efficient zeroth-order llm fine-tuning. arXiv preprint arXiv:2502.03304, 2025

    arXiv 2025

  8. [5]

    Second-order fine-tuning without pain for llms: A hessian informed zeroth-order optimizer

    Yanjun Zhao, Sizhe Dang, Haishan Ye, Guang Dai, Yi Qian, and Ivor W Tsang. Second-order fine-tuning without pain for llms: A hessian informed zeroth-order optimizer. arXiv preprint arXiv:2402.15173, 2024

    arXiv 2024

Show all 62 references
  1. [6]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022

    2022

  2. [7]

    Prefix-tuning: Optimizing continuous prompts for generation

    Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190, 2021

    2021 arXiv

  3. [8]

    Galore: Memory-efficient llm training by gradient low-rank projection

    Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memory-efficient llm training by gradient low-rank projection. arXiv preprint arXiv:2403.03507, 2024

    2024 arXiv

  4. [9]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  5. [10]

    Adam: A method for stochastic optimization

    Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

  6. [11]

    Revisiting zeroth-order optimization for memory-efficient llm fine-tuning: A benchmark

    Yihua Zhang, Pingzhi Li, Junyuan Hong, Jiaxiang Li, Yimeng Zhang, Wenqing Zheng, Pin-Yu Chen, Jason D Lee, Wotao Yin, Mingyi Hong, et al. Revisiting zeroth-order optimization for memory-efficient llm fine-tuning: A benchmark. arXiv preprint arXiv:2402.11592, 2024

    2024 arXiv

  7. [12]

    Fine-tuning language models with just forward passes

    Sadhika Malladi, Tianyu Gao, Eshaan Nichani, Alex Damian, Jason D Lee, Danqi Chen, and Sanjeev Arora. Fine-tuning language models with just forward passes. Advances in Neural Information Processing Systems, 36:53038–53075, 2023

    2023

  8. [13]

    Eigenvalues of the hessian in deep learning: Singularity and beyond

    Levent Sagun, Leon Bottou, and Yann LeCun. Eigenvalues of the hessian in deep learning: Singularity and beyond. arXiv preprint arXiv:1611.07476, 2016

    2016 arXiv

  9. [14]

    An investigation into neural net opti- mization via hessian eigenvalue density

    Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao. An investigation into neural net opti- mization via hessian eigenvalue density. In International Conference on Machine Learning, pages 2232–2241. PMLR, 2019

    2019

  10. [15]

    Why are adaptive methods good for attention models? Advances in Neural Information Processing Systems, 33:15383–15393, 2020

    Jingzhao Zhang, Sai Praneeth Karimireddy, Andreas Veit, Seungyeon Kim, Sashank Reddi, Sanjiv Kumar, and Suvrit Sra. Why are adaptive methods good for attention models? Advances in Neural Information Processing Systems, 33:15383–15393, 2020. 10

    2020

  11. [16]

    Variance-reduced zeroth-order methods for fine-tuning language models

    Tanmay Gautam, Youngsuk Park, Hao Zhou, Parameswaran Raman, and Wooseok Ha. Variance-reduced zeroth-order methods for fine-tuning language models. arXiv preprint arXiv:2404.08080, 2024

    2024 arXiv

  12. [17]

    overparametrization

    Aleksandr Lobanov and Alexander Gasnikov. Accelerated zero-order sgd method for solving the black box optimization problem under “overparametrization” condition. In International Conference on Optimization and Applications, pages 72–83. Springer, 2023

    2023

  13. [18]

    Gradient-free optimization of highly smooth functions: improved analysis and a new algorithm

    Arya Akhavan, Evgenii Chzhen, Massimiliano Pontil, and Alexandre B Tsybakov. Gradient-free optimization of highly smooth functions: improved analysis and a new algorithm. Journal of Machine Learning Research, 25(370):1–50, 2024

    2024

  14. [20]

    Highly-smooth zero-th order online optimization

    Francis Bach and Vianney Perchet. Highly-smooth zero-th order online optimization. In Conference on Learning Theory, pages 257–283. PMLR, 2016

    2016

  15. [21]

    Towards understanding convergence and generalization of adamw

    Pan Zhou, Xingyu Xie, Zhouchen Lin, and Shuicheng Yan. Towards understanding convergence and generalization of adamw. IEEE transactions on pattern analysis and machine intelligence, 2024

    2024

  16. [22]

    Stochastic first-and zeroth-order methods for nonconvex stochastic programming

    Seyed Ghadimi and Guanghui Lan. Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization, 23(4):2341–2368, 2013

    2013

  17. [23]

    Random gradient-free minimization of convex functions

    Yurii Nesterov and Vladimir Spokoiny. Random gradient-free minimization of convex functions. In Foundations of Computational Mathematics, pages 527–566. Springer, 2017

    2017

  18. [25]

    Roberta: A robustly optimized bert pretraining approach

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019

    1907 arXiv

  19. [26]

    Opt: Open pre-trained transformer language models

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022

    2022 arXiv

  20. [27]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  21. [28]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018

    2018 arXiv

  22. [29]

    Visualgpt: Data-efficient adaptation of pretrained language models for image captioning

    Jun Chen, Han Guo, Kai Yi, Boyang Li, and Mohamed Elhoseiny. Visualgpt: Data-efficient adaptation of pretrained language models for image captioning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18030–18040, 2022

    2022

  23. [30]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pa...

    2021

  24. [31]

    Revisiting weakly supervised pre-training of visual perception models

    Mannat Singh, Laura Gustafson, Aaron Adcock, Vinicius de Freitas Reis, Bugra Gedik, Raj Pra- teek Kosaraju, Dhruv Mahajan, Ross Girshick, Piotr Dollár, and Laurens Van Der Maaten. Revisiting weakly supervised pre-training of visual perception models. In Proceedings of the IEEE...

    2022

  25. [32]

    Dora: Weight-decomposed low-rank adaptation

    Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. In Forty-first International Conference on Machine Learning, 2024. 11

    2024

  26. [33]

    Smoothquant: Accurate and efficient post-training quantization for large language models

    Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pages 38087–38099. PMLR, 2023

    2023

  27. [34]

    Outlier suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling

    Xiuying Wei, Yunchen Zhang, Yuhang Li, Xiangguo Zhang, Ruihao Gong, Jinyang Guo, and Xianglong Liu. Outlier suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling. arXiv preprint arXiv:2304.09145, 2023

    2023 arXiv

  28. [35]

    Zoo: Zeroth order optimization based black-box attacks to deep neural networks without training substitute models

    Pin-Yu Chen, Huan Zhang, Yash Sharma, Jinfeng Yi, and Cho-Jui Hsieh. Zoo: Zeroth order optimization based black-box attacks to deep neural networks without training substitute models. In Proceedings of the 10th ACM workshop on artificial intelligence and security, pages 15–26, 2017

    2017

  29. [36]

    Hessian-aware zeroth-order optimization for black-box adversarial attack

    Haishan Ye, Zhichao Huang, Cong Fang, Chris Junchi Li, and Tong Zhang. Hessian-aware zeroth-order optimization for black-box adversarial attack. arXiv preprint arXiv:1812.11377, 2018

    2018 arXiv

  30. [37]

    Certified zeroth-order black-box defense with robust unet denoiser

    Astha Verma, Siddhesh Bangar, A Venkata Subramanyam, Naman Lal, Rajiv Ratn Shah, and Shin’ichi Satoh. Certified zeroth-order black-box defense with robust unet denoiser. arXiv preprint arXiv:2304.06430, 2023

    2023 arXiv

  31. [38]

    Deepzero: Scaling up zeroth-order optimization for deep model training

    Aochuan Chen, Yimeng Zhang, Jinghan Jia, James Diffenderfer, Jiancheng Liu, Konstantinos Parasyris, Yihua Zhang, Zheng Zhang, Bhavya Kailkhura, and Sijia Liu. Deepzero: Scaling up zeroth-order optimization for deep model training. arXiv preprint arXiv:2310.02025, 2023

    arXiv

  32. [39]

    Sparse mezo: Less parameters for better performance in zeroth-order llm fine-tuning

    Yong Liu, Zirui Zhu, Chaoyu Gong, Minhao Cheng, Cho-Jui Hsieh, and Yang You. Sparse mezo: Less parameters for better performance in zeroth-order llm fine-tuning. arXiv preprint arXiv:2402.15751, 2024

    2024

  33. [40]

    Pri- vate fine-tuning of large language models with zeroth-order optimization

    Xinyu Tang, Ashwinee Panda, Milad Nasr, Saeed Mahloujifar, and Prateek Mittal. Pri- vate fine-tuning of large language models with zeroth-order optimization. arXiv preprint arXiv:2401.04343, 2024

    2024 arXiv

  34. [41]

    Zeroth- order stochastic variance reduction for nonconvex optimization.Advances in Neural Information Processing Systems, 31, 2018

    Sijia Liu, Bhavya Kailkhura, Pin-Yu Chen, Paishun Ting, Shiyu Chang, and Lisa Amini. Zeroth- order stochastic variance reduction for nonconvex optimization.Advances in Neural Information Processing Systems, 31, 2018

    2018

  35. [42]

    Accelerating stochastic gradient descent using predictive variance reduction

    Rie Johnson and Tong Zhang. Accelerating stochastic gradient descent using predictive variance reduction. Advances in neural information processing systems, 26, 2013

    2013

  36. [43]

    Zeroth-order optimization with trajectory-informed derivative estimation

    Yao Shu, Zhongxiang Dai, Weicong Sng, Arun Verma, Patrick Jaillet, and Bryan Kian Hsiang Low. Zeroth-order optimization with trajectory-informed derivative estimation. In The Eleventh International Conference on Learning Representations, 2023

    2023

  37. [44]

    Pretrained optimization model for zero-shot black box optimization

    Xiaobin Li, Kai Wu, Xiaoyu Zhang, Handing Wang, Jing Liu, et al. Pretrained optimization model for zero-shot black box optimization. Advances in Neural Information Processing Systems, 37:14283–14324, 2024

    2024

  38. [45]

    Recursive deep models for semantic compositionality over a sentiment treebank

    Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP, 2013

    2013

  39. [46]

    A large annotated corpus for learning natural language inference

    Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. A large annotated corpus for learning natural language inference. In EMNLP, 2015

    2015

  40. [47]

    Building a question answering test collection

    Ellen M V oorhees and Dawn M Tice. Building a question answering test collection. InSIGIR, 2000

    2000

  41. [48]

    Improving retrieval-based sentence generation with context-aware answer selection

    Xiangru Yao and Jimmy Lin. Improving retrieval-based sentence generation with context-aware answer selection. In AAAI, 2020

    2020

  42. [49]

    The pascal recognising textual entailment challenge

    Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment challenge. In MLCW, 2005. 12

    2005

  43. [50]

    The second pascal recognising textual entailment challenge

    Roy Bar-Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. The second pascal recognising textual entailment challenge. In PASCAL Challenges Workshop, 2006

    2006

  44. [51]

    The fifth pascal recognizing textual entailment challenge

    Luisa Bentivogli, Ido Dagan, Hoa T Dang, Danilo Giampiccolo, and Bernardo Magnini. The fifth pascal recognizing textual entailment challenge. In TAC, 2009

    2009

  45. [52]

    The third pascal recogniz- ing textual entailment challenge

    Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. The third pascal recogniz- ing textual entailment challenge. In ACL Workshop on Textual Entailment and Paraphrasing, 2007

    2007

  46. [53]

    Superglue: A stickier benchmark for general-purpose language understanding systems

    Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. arXiv preprint arXiv:1905.00537, 2019

    1905 arXiv

  47. [54]

    The shared task on implicit and explicit hate speech detection

    Marie-Catherine De Marneffe, Nicolas Simard, Wanrong Xu, et al. The shared task on implicit and explicit hate speech detection. In W-NUT, 2019

    2019

  48. [55]

    Boolq: Exploring the surprising difficulty of natural yes/no questions

    Christopher Clark and Kenton Lee. Boolq: Exploring the surprising difficulty of natural yes/no questions. NAACL, 2019

    2019

  49. [56]

    Wic: The word-in-context dataset for evaluating context-sensitive meaning representations

    Mohammad Taher Pilehvar and Jose Camacho-Collados. Wic: The word-in-context dataset for evaluating context-sensitive meaning representations. In EMNLP, 2018

    2018

  50. [57]

    The winograd schema challenge

    Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In KR, 2012

    2012

  51. [58]

    Looking beyond the surface: A challenge set for reading comprehension over multiple sentences

    Daniel Khashabi, Snigdha Chaturvedi, and Dan Roth. Looking beyond the surface: A challenge set for reading comprehension over multiple sentences. In NAACL, 2018

    2018

  52. [59]

    Squad: 100,000+ questions for machine comprehension of text

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. In EMNLP, 2016

    2016

  53. [60]

    Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs

    Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gard- ner. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In NAACL, 2019

    2019

  54. [62]

    Mathematical analysis II, volume 220

    Vladimir Antonovich Zorich and Octavio Paniagua. Mathematical analysis II, volume 220. Springer, 2016. 13 9 Appendix 9.1 Variance Analysis of the Zeroth-Order Estimator with Kernel Function When applying a kernel function K(r) and a scalar random variable r in the zeroth-order...

    2016

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.