A general framework for floating point error analysis of simplex derivatives

Amy Wiebe; Warren Hare; Yiwen Chen

arxiv: 2406.04530 · v1 · pith:NDSH3SH4new · submitted 2024-06-06 · 🧮 math.NA · cs.NA· math.OC

A general framework for floating point error analysis of simplex derivatives

Yiwen Chen , Warren Hare , Amy Wiebe This is my paper

Pith reviewed 2026-05-24 00:04 UTC · model grok-4.3

classification 🧮 math.NA cs.NAmath.OC

keywords simplex derivativesfloating point error analysisderivative-free optimizationgradient approximationsimplex gradientnumerical optimization

0 comments

The pith

A general framework analyzes floating point errors uniformly for any simplex derivative that fits one general form.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper establishes a framework for floating point error analysis that applies to simplex derivatives in derivative-free optimization as long as they match a specified general form. This covers the generalized simplex gradient, the generalized centred simplex gradient, and a newly defined generalized adapted centred simplex gradient. The authors review definitions, derive accuracy results under the standard floating point model, and use those results to recommend a minimal approximate diameter for the sample set. A sympathetic reader would care because these approximations are widely used when gradients are unavailable, yet floating point effects can degrade their reliability without a uniform way to quantify the damage.

Core claim

Floating point error analysis of simplex derivatives can be performed in a way that does not depend on which specific simplex derivative is chosen, provided the derivative satisfies the paper's general form; the framework is then applied to three generalized variants to obtain accuracy statements and to suggest how small the sample-set diameter should be.

What carries the argument

The general form satisfied by simplex derivatives, which permits a single floating-point error analysis to cover multiple variants under the standard model of floating point arithmetic.

If this is right

The same error bounds apply to the generalized simplex gradient.
The same error bounds apply to the generalized centred simplex gradient.
The same error bounds apply to the generalized adapted centred simplex gradient.
Concrete suggestions follow for the smallest useful approximate diameter of the sample set.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

If other gradient approximations can be rewritten to match the general form, the framework would immediately supply their floating point error bounds as well.
The diameter recommendations could be checked by measuring actual optimization performance on test problems with known gradients.
The analysis might guide when to switch from these approximations to higher-precision arithmetic in large-scale derivative-free solvers.

Load-bearing premise

The simplex derivatives under study must satisfy the general form defined in the paper so that the same error analysis applies to all of them.

What would settle it

Numerical tests with a simplex derivative that meets the general form but produces observed errors larger than the derived bounds would show the framework does not hold.

read the original abstract

Gradient approximations are a class of numerical approximation techniques that are of central importance in numerical optimization. In derivative-free optimization, most of the gradient approximations, including the simplex gradient, centred simplex gradient, and adapted centred simplex gradient, are in the form of simplex derivatives. Owing to machine precision, the approximation accuracy of any numerical approximation technique is subject to the influence of floating point errors. In this paper, we provide a general framework for floating point error analysis of simplex derivatives. Our framework is independent of the choice of the simplex derivative as long as it satisfies a general form. We review the definition and approximation accuracy of the generalized simplex gradient and generalized centred simplex gradient. We define and analyze the accuracy of a generalized version of the adapted centred simplex gradient. As examples, we apply our framework to the generalized simplex gradient, generalized centred simplex gradient, and generalized adapted centred simplex gradient. Based on the results, we give suggestions on the minimal choice of approximate diameter of the sample set.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

The paper supplies a unified floating-point error framework for simplex derivatives that fit a stated general form, introduces a generalized adapted centred simplex gradient, and derives advice on minimal sample-set diameter from the bounds.

read the letter

The main contribution is a single error analysis that applies to any simplex derivative matching their general form, rather than separate treatments for each variant. They review the generalized simplex gradient and generalized centred simplex gradient, define the generalized adapted centred simplex gradient, apply the framework to all three, and close with suggestions on how small the approximate diameter of the sample set can be while controlling floating-point effects. This unification is the clearest new element and makes the error analysis more reusable for derivative-free optimization work. The structure is straightforward: state the general form, derive the error bound once under the standard floating-point model, then instantiate it on the examples. No circularity shows up, and the stress-test note correctly flags that the uniformity claim stands if the form is stated correctly and the cases satisfy it. The central argument therefore holds up on its own terms. The soft spots are limited. The framework only covers derivatives inside the general form, so anything outside needs its own analysis; that is a scope limit rather than a flaw. The results rest on the usual floating-point model, which is standard but means the bounds are not pushed to more refined arithmetic models. The paper is mostly theoretical, so the tightness of the bounds in actual computation is not shown here. This is for readers working on numerical gradient approximations in optimization who need to manage machine-precision effects when choosing sample sets. Someone who already uses simplex gradients or similar methods would get direct value from the reusable bounds and the diameter recommendation. It deserves a serious referee because it packages a methodological tool that can be applied beyond the three cases shown.

Referee Report

0 major / 3 minor

Summary. The paper claims to provide a general framework for floating-point error analysis of simplex derivatives (used in derivative-free optimization) that applies uniformly to any such derivative satisfying a stated general form, independent of the specific choice. It reviews the approximation accuracy of the generalized simplex gradient and generalized centred simplex gradient, defines and analyzes a generalized adapted centred simplex gradient, instantiates the framework on all three cases, and uses the resulting error bounds to recommend a minimal approximate diameter for the sample set.

Significance. If the central uniformity claim holds, the framework would supply a reusable error-analysis template for simplex-based gradient approximations under the standard floating-point model, directly informing practical choices of sample-set diameter to control total error. The manuscript's structure—deriving the general analysis once and then specializing it—avoids redundant case-by-case derivations and supplies explicit suggestions grounded in the derived bounds.

minor comments (3)

[§2] §2, after Eq. (3): the general form for the simplex derivative is introduced without an explicit statement of the floating-point model (e.g., whether it is the standard model with machine epsilon or a more refined one); adding a displayed equation for the model would make the subsequent error propagation steps easier to follow.
[§4.2] §4.2, Theorem 4.1: the bound on the adapted centred simplex gradient contains an O(δ) term whose coefficient is left symbolic; a short remark on whether this coefficient can be bounded independently of the simplex geometry would strengthen the later diameter recommendation.
[§5] Figure 1 and the accompanying text in §5: the plotted error curves are not accompanied by the exact parameter values (h, δ, machine epsilon) used to generate them; adding a caption table or a sentence listing these values would improve reproducibility.

Simulated Author's Rebuttal

0 responses · 0 unresolved

We thank the referee for their careful reading and positive assessment of the manuscript. The recommendation for minor revision is noted. No specific major comments were listed in the report, so we have no individual points to address at this time. We are prepared to incorporate any minor suggestions that may arise during the revision process.

Circularity Check

0 steps flagged

No significant circularity; framework is self-contained

full rationale

The paper derives a uniform floating-point error analysis from the standard model of floating-point arithmetic applied to any simplex derivative satisfying the stated general form. It reviews prior simplex gradient definitions, introduces the generalized adapted centred simplex gradient by direct extension, instantiates the framework on the three cases, and extracts diameter recommendations from the resulting bounds. No step reduces by construction to a fitted input, self-citation chain, or renamed ansatz; the central uniformity claim rests on the general form and external arithmetic model rather than internal redefinition. This matches the expected non-circular outcome for a derivation grounded in established numerical analysis.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

The framework rests on the assumption that simplex derivatives fit a general mathematical form and on the standard model of floating point arithmetic with bounded relative errors.

axioms (2)

domain assumption Simplex derivatives satisfy a general form that allows uniform error analysis independent of specific choice.
Explicitly stated as the basis for the framework in the abstract.
domain assumption Floating point arithmetic follows the standard model with machine epsilon bounding relative rounding errors.
Standard assumption invoked for all floating point error analyses in numerical methods.

pith-pipeline@v0.9.0 · 5700 in / 1285 out tokens · 25559 ms · 2026-05-24T00:04:56.455972+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

16 extracted references · 16 canonical work pages

[1]

Audet and W

C. Audet and W. Hare , Derivative-free and blackbox optimization , Springer, 2017

work page 2017
[2]

A. S. Berahas, R. H. Byrd, and J. Nocedal , Derivative-free optimization of noisy functions via quasi - Newton methods , SIAM Journal on Optimization, 29 (2019), pp. 965–993

work page 2019
[3]

Bj ¨orck, Numerical methods for least squares problems , SIAM, 1996

˚ A. Bj ¨orck, Numerical methods for least squares problems , SIAM, 1996

work page 1996
[4]

Byers and H

R. Byers and H. Xu , A new scaling for Newton ’s iteration for the polar decomposi tion and its backward stability, SIAM Journal on Matrix Analysis and Applications, 30 (2008 ), pp. 822–843

work page 2008
[5]

Chen and W

Y. Chen and W. Hare , Adapting the centred simplex gradient to compensate for mis aligned sample points , IMA Journal of Numerical Analysis, (2023)

work page 2023
[6]

A. R. Conn, K. Scheinberg, and L. N. Vicente , Geometry of sample sets in derivative-free optimization: polynomial regression and underdetermined interpolation , IMA Journal of Numerical Analysis, 28 (2008), pp. 721–748

work page 2008
[7]

A. R. Conn, K. Scheinberg, and L. N. Vicente , Introduction to derivative-free optimization , SIAM, 2009

work page 2009
[8]

A. L. Cust ´odio, J. E. Dennis, and L. N. Vicente , Using simplex gradients of nonsmooth functions in direct search methods, IMA Journal of Numerical Analysis, 28 (2008), pp. 770–784

work page 2008
[9]

A. L. Cust ´odio and L. N. Vicente , Using sampling and simplex derivatives in pattern search me thods, SIAM Journal on Optimization, 18 (2007), pp. 537–555

work page 2007
[10]

W. Hare, G. Jarry-Bolduc, and C. Planiden , Error bounds for overdetermined and underdetermined generalized centred simplex gradients , IMA Journal of Numerical Analysis, 42 (2022), pp. 744–770

work page 2022
[11]

N. J. Higham , Accuracy and stability of numerical algorithms , SIAM, 2002

work page 2002
[12]

C. T. Kelley , Detection and remediation of stagnation in the Nelder–Mead algorithm using a suﬃcient decrease condition, SIAM Journal on Optimization, 10 (1999), pp. 43–55. 13. , Iterative methods for optimization , SIAM, 1999

work page 1999
[13]

Liuzzi, S

G. Liuzzi, S. Lucidi, F. Rinaldi, and L. N. Vicente , Trust-region methods for the derivative-free optimiza- tion of nonsmooth black-box functions , SIAM Journal on Optimization, 29 (2019), pp. 3012–3035

work page 2019
[14]

Penrose, A generalized inverse for matrices , in Mathematical Proceedings of the Cambridge Philosophic al Society, vol

R. Penrose, A generalized inverse for matrices , in Mathematical Proceedings of the Cambridge Philosophic al Society, vol. 51(3), Cambridge University Press, 1955, pp. 406–413

work page 1955
[15]

R. G. Regis , The calculus of simplex gradients , Optimization Letters, 9 (2015), pp. 845–865

work page 2015
[16]

Smoktunowicz and I

A. Smoktunowicz and I. Wr ´obel, Numerical aspects of computing the Moore-Penrose inverse o f full column rank matrices, BIT Numerical Mathematics, 52 (2012), pp. 503–524

work page 2012

[1] [1]

Audet and W

C. Audet and W. Hare , Derivative-free and blackbox optimization , Springer, 2017

work page 2017

[2] [2]

A. S. Berahas, R. H. Byrd, and J. Nocedal , Derivative-free optimization of noisy functions via quasi - Newton methods , SIAM Journal on Optimization, 29 (2019), pp. 965–993

work page 2019

[3] [3]

Bj ¨orck, Numerical methods for least squares problems , SIAM, 1996

˚ A. Bj ¨orck, Numerical methods for least squares problems , SIAM, 1996

work page 1996

[4] [4]

Byers and H

R. Byers and H. Xu , A new scaling for Newton ’s iteration for the polar decomposi tion and its backward stability, SIAM Journal on Matrix Analysis and Applications, 30 (2008 ), pp. 822–843

work page 2008

[5] [5]

Chen and W

Y. Chen and W. Hare , Adapting the centred simplex gradient to compensate for mis aligned sample points , IMA Journal of Numerical Analysis, (2023)

work page 2023

[6] [6]

A. R. Conn, K. Scheinberg, and L. N. Vicente , Geometry of sample sets in derivative-free optimization: polynomial regression and underdetermined interpolation , IMA Journal of Numerical Analysis, 28 (2008), pp. 721–748

work page 2008

[7] [7]

A. R. Conn, K. Scheinberg, and L. N. Vicente , Introduction to derivative-free optimization , SIAM, 2009

work page 2009

[8] [8]

A. L. Cust ´odio, J. E. Dennis, and L. N. Vicente , Using simplex gradients of nonsmooth functions in direct search methods, IMA Journal of Numerical Analysis, 28 (2008), pp. 770–784

work page 2008

[9] [9]

A. L. Cust ´odio and L. N. Vicente , Using sampling and simplex derivatives in pattern search me thods, SIAM Journal on Optimization, 18 (2007), pp. 537–555

work page 2007

[10] [10]

W. Hare, G. Jarry-Bolduc, and C. Planiden , Error bounds for overdetermined and underdetermined generalized centred simplex gradients , IMA Journal of Numerical Analysis, 42 (2022), pp. 744–770

work page 2022

[11] [11]

N. J. Higham , Accuracy and stability of numerical algorithms , SIAM, 2002

work page 2002

[12] [12]

C. T. Kelley , Detection and remediation of stagnation in the Nelder–Mead algorithm using a suﬃcient decrease condition, SIAM Journal on Optimization, 10 (1999), pp. 43–55. 13. , Iterative methods for optimization , SIAM, 1999

work page 1999

[13] [13]

Liuzzi, S

G. Liuzzi, S. Lucidi, F. Rinaldi, and L. N. Vicente , Trust-region methods for the derivative-free optimiza- tion of nonsmooth black-box functions , SIAM Journal on Optimization, 29 (2019), pp. 3012–3035

work page 2019

[14] [14]

Penrose, A generalized inverse for matrices , in Mathematical Proceedings of the Cambridge Philosophic al Society, vol

R. Penrose, A generalized inverse for matrices , in Mathematical Proceedings of the Cambridge Philosophic al Society, vol. 51(3), Cambridge University Press, 1955, pp. 406–413

work page 1955

[15] [15]

R. G. Regis , The calculus of simplex gradients , Optimization Letters, 9 (2015), pp. 845–865

work page 2015

[16] [16]

Smoktunowicz and I

A. Smoktunowicz and I. Wr ´obel, Numerical aspects of computing the Moore-Penrose inverse o f full column rank matrices, BIT Numerical Mathematics, 52 (2012), pp. 503–524

work page 2012