BoLT is a benchmark of surrogate models fitted to real LLM experiment data that enables evaluation of Bayesian and black-box optimization methods on multi-fidelity, multi-objective, high-dimensional LLM tasks.
hub Canonical reference
A Tutorial on Bayesian Optimization
Canonical reference. 90% of citing Pith papers cite this work as background.
57
Pith papers citing it
Background
90% of classified citations
abstract
Bayesian optimization is an approach to optimizing objective functions that take a long time (minutes or hours) to evaluate. It is best-suited for optimization over continuous domains of less than 20 dimensions, and tolerates stochastic noise in function evaluations. It builds a surrogate for the objective and quantifies the uncertainty in that surrogate using a Bayesian machine learning technique, Gaussian process regression, and then uses an acquisition function defined from this surrogate to decide where to sample. In this tutorial, we describe how Bayesian optimization works, including Gaussian process regression and three common acquisition functions: expected improvement, entropy search, and knowledge gradient. We then discuss more advanced techniques, including running multiple function evaluations in parallel, multi-fidelity and multi-information source optimization, expensive-to-evaluate constraints, random environmental conditions, multi-task Bayesian optimization, and the inclusion of derivative information. We conclude with a discussion of Bayesian optimization software and future research directions in the field. Within our tutorial material we provide a generalization of expected improvement to noisy evaluations, beyond the noise-free setting where it is more commonly applied. This generalization is justified by a formal decision-theoretic argument, standing in contrast to previous ad hoc modifications.
hub tools
citation-role summary
background 9
dataset 1
citation-polarity summary
representative citing papers
EFE-based planning is formulated as variational free energy minimization with epistemic priors, decomposing into expected plan costs plus a complexity term.
EFE-based active inference planning is characterized as VFE on an augmented model plus entropy and planning corrections, with a derived message-passing implementation and grid-world validation.
First experimental realization of LEO-based adiabatic speedup on superconducting quantum processors, with enhanced fidelity using ideal pulses but limited gains from noise-model optimization.
Bayesian optimization identifies cement-salt hydrate composites achieving up to five times higher specific energy than prior cement-based TCES materials, with LiCl-based formulations reaching 458 kJ/kg.
A cost-aware value-of-information acquisition function is derived to balance direct observations against noisy pairwise human comparisons in Bayesian optimization, approaching the convex hull of the individual information sources' performance trajectories.
Proposes a vector-valued RKHS framework for Bayesian optimization with structured measurements, deriving concentration bounds and UCB-based regret guarantees that recover sublinear rates.
A myopic MINMPC framework learns a value function offline via inverse optimization from expert data, allowing short horizons with near-optimal performance and strict integer feasibility online for hybrid systems.
PALM-Mean combines sign-aware piecewise-linear relaxations of locally important kernel terms with closed-form analytic bounds on the rest inside a reduced-space branch-and-bound framework, yielding valid lower bounds and ε-global convergence for GP posterior mean optimization.
CCBO enables collaborative contextual Bayesian optimization across clients with sublinear regret guarantees and shows substantial gains over non-collaborative methods in simulations and a hot rolling application even under heterogeneity.
Proposes a trajectory-oriented Bayesian optimization method using Gaussian process surrogates on parameters and seeds with adaptive Thompson sampling for efficient discovery of data-consistent trajectories in stochastic epidemic models.
Reformulates constrained black-box optimization as posterior inference in latent space of flow-based models amortized by outsourced diffusion models, claiming superior performance on synthetic and real tasks.
Introduces BOBa, a multi-armed bandit method for scalable surrogate optimization that adaptively allocates inference and evaluations to promising partitions of ultra-large chemical libraries.
MatFormBench introduces a synthetic data generator with five difficulty levels and MatFormScore metric to benchmark 39 inverse design algorithms for target-driven materials formulation.
A framework learns invariant symbolic reward functions from few demonstrations that generalize zero-shot to variations in robotic manipulation tasks.
Recasts sampling-based nonconvex optimization as smoothed gradient descent to obtain non-asymptotic convergence guarantees and introduces the DIDA annealed algorithm that converges to the global optimum.
BACO replaces direct black-box calls in collaborative optimization with Gaussian process surrogates at both subsystem and system levels, achieving lower objectives and near-zero constraint violations on MDO benchmarks and a CRM wing problem within limited evaluations.
Proposes GP-Perm kernel and DKL-DS model for permutation-invariant Bayesian optimization applied to well placement in CCS, evaluated on synthetic benchmarks and one real formation case.
HASOD is a hybrid adaptive framework that unifies factor screening via a new CWESS statistic and response optimization using Gaussian processes, achieving 97% detection accuracy in simulations with asymptotic consistency guarantees.
Non-interacting electron density features with two-point correlations and active learning achieve <2% NMAE on bulk modulus using 10 samples and enable zero-shot transfer to unseen 7-component refractory BCC alloys.
POGPN-JPSS integrates partially observable Gaussian process networks with joint parameter and state-space modeling to leverage expert-derived low-dimensional features from high-dimensional intermediate observations for faster optimization of multi-stage manufacturing processes.
LILO integrates LLMs to translate natural language feedback into preference signals for Gaussian process-based Bayesian optimization, outperforming standard preference BO and LLM-only methods on benchmarks.
Comparative evaluation of Bayesian Neural Network surrogates versus Gaussian Processes in Bayesian Optimization applied to Carbon Capture and Storage operations, presented as the first such application in reservoir engineering.
Bilevel-optimized implicit neural representation with Gaussian process hyperparameter tuning enables scan-specific accelerated MRI reconstruction without training data.
citing papers explorer
-
BoLT: A Benchmark to Democratize Black-box Optimization Research for Expensive LLM Tasks
BoLT is a benchmark of surrogate models fitted to real LLM experiment data that enables evaluation of Bayesian and black-box optimization methods on multi-fidelity, multi-objective, high-dimensional LLM tasks.
-
Expected Free Energy-based Planning as Variational Inference
EFE-based planning is formulated as variational free energy minimization with epistemic priors, decomposing into expected plan costs plus a complexity term.
-
What Type of Inference is Active Inference?
EFE-based active inference planning is characterized as VFE on an augmented model plus entropy and planning corrections, with a derived message-passing implementation and grid-world validation.
-
Realizing leakage elimination operator-based adiabatic speedup on a superconducting quantum processor
First experimental realization of LEO-based adiabatic speedup on superconducting quantum processors, with enhanced fidelity using ideal pulses but limited gains from noise-model optimization.
-
High-Throughput Bayesian Optimization of Cement-Salt Hydrates Composites for Seasonal Thermochemical Energy Storage
Bayesian optimization identifies cement-salt hydrate composites achieving up to five times higher specific energy than prior cement-based TCES materials, with LiCl-based formulations reaching 458 kJ/kg.
-
Elicitation-Augmented Bayesian Optimization
A cost-aware value-of-information acquisition function is derived to balance direct observations against noisy pairwise human comparisons in Bayesian optimization, approaching the convex hull of the individual information sources' performance trajectories.
-
Bayesian Optimization with Structured Measurements: A Vector-Valued RKHS Framework
Proposes a vector-valued RKHS framework for Bayesian optimization with structured measurements, deriving concentration bounds and UCB-based regret guarantees that recover sublinear rates.
-
Learning myopic mixed-integer nonlinear model predictive control from expert demonstrations
A myopic MINMPC framework learns a value function offline via inverse optimization from expert data, allowing short horizons with near-optimal performance and strict integer feasibility online for hybrid systems.
-
An Efficient Spatial Branch-and-Bound Algorithm for Global Optimization of Gaussian Process Posterior Mean Functions
PALM-Mean combines sign-aware piecewise-linear relaxations of locally important kernel terms with closed-form analytic bounds on the rest inside a reduced-space branch-and-bound framework, yielding valid lower bounds and ε-global convergence for GP posterior mean optimization.
-
Collaborative Contextual Bayesian Optimization
CCBO enables collaborative contextual Bayesian optimization across clients with sublinear regret guarantees and shows substantial gains over non-collaborative methods in simulations and a hot rolling application even under heterogeneity.
-
Staying on Track: Efficient Trajectory Discovery with Adaptive Batch Sampling
Proposes a trajectory-oriented Bayesian optimization method using Gaussian process surrogates on parameters and seeds with adaptive Thompson sampling for efficient discovery of data-consistent trajectories in stochastic epidemic models.
-
Posterior Inference in Latent Space for Scalable Constrained Black-box Optimization
Reformulates constrained black-box optimization as posterior inference in latent space of flow-based models amortized by outsourced diffusion models, claiming superior performance on synthetic and real tasks.
-
Target-Aware Bandit Allocation for Scalable Surrogate Optimization in Chemical Space
Introduces BOBa, a multi-armed bandit method for scalable surrogate optimization that adaptively allocates inference and evaluations to promising partitions of ultra-large chemical libraries.
-
MatFormBench: A Benchmarking Evaluation Framework for Target-Driven Materials Formulation
MatFormBench introduces a synthetic data generator with five difficulty levels and MatFormScore metric to benchmark 39 inverse design algorithms for target-driven materials formulation.
-
Beyond Pixels: Learning Invariant Rewards for Real-World Robotics From a Few Demonstrations
A framework learns invariant symbolic reward functions from few demonstrations that generalize zero-shot to variations in robotic manipulation tasks.
-
Global Convergence of Sampling-Based Nonconvex Optimization through Diffusion-Style Smoothing
Recasts sampling-based nonconvex optimization as smoothed gradient descent to obtain non-asymptotic convergence guarantees and introduces the DIDA annealed algorithm that converges to the global optimum.
-
Bayesian Algorithm for Collaborative Optimization with Application to Aircraft Design
BACO replaces direct black-box calls in collaborative optimization with Gaussian process surrogates at both subsystem and system levels, achieving lower objectives and near-zero constraint violations on MDO benchmarks and a CRM wing problem within limited evaluations.
-
Inducing Permutation Invariant Priors in Bayesian Optimization for Carbon Capture and Storage Applications
Proposes GP-Perm kernel and DKL-DS model for permutation-invariant Bayesian optimization applied to well placement in CCS, evaluated on synthetic benchmarks and one real formation case.
-
HASOD: A Hybrid Adaptive Screening-Optimization Design for High-Dimensional Industrial Experiments
HASOD is a hybrid adaptive framework that unifies factor screening via a new CWESS statistic and response optimization using Gaussian processes, achieving 97% detection accuracy in simulations with asymptotic consistency guarantees.
-
Electronic manifolds for extrapolative alloy discovery
Non-interacting electron density features with two-point correlations and active learning achieve <2% NMAE on bulk modulus using 10 samples and enable zero-shot transfer to unseen 7-component refractory BCC alloys.
-
Joint Parameter and State-Space Bayesian Optimization: Using Process Expertise to Accelerate Manufacturing Optimization
POGPN-JPSS integrates partially observable Gaussian process networks with joint parameter and state-space modeling to leverage expert-derived low-dimensional features from high-dimensional intermediate observations for faster optimization of multi-stage manufacturing processes.
-
LILO: Bayesian Optimization with Natural Language Feedback
LILO integrates LLMs to translate natural language feedback into preference signals for Gaussian process-based Bayesian optimization, outperforming standard preference BO and LLM-only methods on benchmarks.
-
Bayesian Neural Network Surrogates for Bayesian Optimization of Carbon Capture and Storage Operations
Comparative evaluation of Bayesian Neural Network surrogates versus Gaussian Processes in Bayesian Optimization applied to Carbon Capture and Storage operations, presented as the first such application in reservoir engineering.
-
Bilevel Optimized Implicit Neural Representation for Scan-Specific Accelerated MRI Reconstruction
Bilevel-optimized implicit neural representation with Gaussian process hyperparameter tuning enables scan-specific accelerated MRI reconstruction without training data.
-
Bayesian Parameter Shift Rule in Variational Quantum Eigensolvers
Bayesian PSR with Gaussian processes and GradCoRe accelerates VQE SGD by reusing observations and minimizing per-step costs while reducing to standard PSR in special cases.
-
Bayesian Optimization for Repeater Protocols
Bayesian optimization reliably identifies optimal protocols that maximize secret-key rates for heterogeneous quantum repeater chains with arbitrary nodes.
-
Global Optimization of Gaussian Process Acquisition Functions Using a Piecewise-Linear Kernel Approximation
Introduces PK-MIQP, a piecewise-linear kernel approximation that converts Gaussian process acquisition function optimization into a solvable MIQP for any stationary or dot-product kernel, with regret bounds and tests on synthetic and real tasks.
-
Approximate and Weighted Data Reconstruction Attack in Federated Learning
An interpolation approximation plus layer-weighted loss enables stronger data reconstruction attacks on FedAvg federated learning, with experiments showing gains on image data over prior methods.
-
Optimization of randomized neural networks for transfer operator approximation
Optimizing the activation function in randomized neural networks provides a more suitable dictionary for transfer operator approximation in stochastic differential equations and random walks on graphons.
-
Agentic Discovery of Cryomicroneedle Formulations
A closed-loop workflow using Gaussian process surrogate modeling and Bayesian optimization, updated over ten iterations with 106 wet-lab tests, adapted from literature data to identify a cryoprotectant formulation achieving 95.15% post-thaw viability for cryomicroneedles.
-
On the Tradeoffs of On-Device Generative Models in Federated Predictive Maintenance Systems
Experiments on real industrial time series show that partial model sharing improves diffusion model performance in bandwidth-limited non-IID settings, while full sharing stabilizes GAN training but offers less robustness than VAE or DDPM alternatives.
-
ORTHOBO: Orthogonal Bayesian Hyperparameter Optimization
OrthoBO introduces an orthogonal acquisition estimator subtracting an optimally weighted score-function control variate to reduce Monte Carlo variance, preserve the acquisition target, and improve ranking stability in Bayesian hyperparameter optimization.
-
Harnessing a 256-qubit Neutral Atom Simulator for Graph Classification
A 256-qubit neutral atom simulator computes Quantum Evolution Kernels for graph classification on the PROTEINS dataset, achieving slightly better performance than classical kernels.
-
Caliper-in-the-Loop: Black-Box Optimization for Hyperledger Fabric Performance Tuning
Bayesian optimization with dimensionality reduction improves Hyperledger Fabric throughput by up to 12% in a 317-dimensional configuration space via an automated Caliper benchmarking loop.
-
AgentOpt v0.1 Technical Report: Client-Side Optimization for LLM-Based Agent
AgentOpt introduces a framework-agnostic package that uses algorithms like UCB-E to find cost-effective model assignments in multi-step LLM agent pipelines, cutting evaluation budgets by 62-76% while maintaining near-optimal accuracy on benchmarks.
-
Physics-informed automated surface reconstructing via low-energy electron diffraction based on Bayesian optimization
A trust-region Bayesian optimization framework integrates LEED multiple scattering models to jointly optimize structural and experimental parameters for automated surface reconstruction.
-
AI4S-SDS: A Neuro-Symbolic Solvent Design System via Sparse MCTS and Differentiable Physics Alignment
AI4S-SDS uses sparse MCTS and differentiable physics alignment to generate valid solvent mixtures and identifies a competitive photoresist developer formulation.
-
Choosing a Suitable Acquisition Function for Batch Bayesian Optimization: Comparison of Serial and Monte Carlo Approaches
Empirical comparison finds qUCB outperforms qlogEI and matches or exceeds UCB/LP for convergence in noiseless and noisy 6D optimization, recommending it as default for unknown landscapes.
-
Understanding High-Dimensional Bayesian Optimization
Vanishing gradients from GP initialization explain HDBO failures; MLE of length scales suffices for SOTA performance, enabling the MSR variant on real-world tasks.
-
Adaptive Compression-based Lifelong Learning
Bayesian optimization enables adaptive network pruning rates in lifelong learning, performing heavier pruning on small/simple tasks and milder on large/complex ones.
-
A Human-in-the-Loop Bayesian Optimization Framework for Constraint-Aware Bioprocess Development
An extension of PFGS adds posterior probability of constraint satisfaction and Monte Carlo robustness estimation as Pareto objectives for interactive candidate selection in Bayesian optimization, demonstrated on an 8D CHO cell culture simulator.
-
Improving Evaluation of Recombination-based Cartesian Genetic Programming
Hyperparameter optimization yields performance improvements for recombination-based Cartesian Genetic Programming on SRBench.
-
Enhancing Model Based Derivative Free Optimization using Direct Search
A hybrid switching approach integrates Direct Search into model-based derivative-free optimization, with a convergence proof for single-objective cases and empirical gains on ML tasks and CUTEr benchmarks.
-
BayMOTH: Bayesian optiMizatiOn with meTa-lookahead -- a simple approacH
BayMOTH unifies meta-Bayesian optimization with a usefulness-based fallback to lookahead, demonstrating competitive results on function optimization tasks even under low task relatedness.
-
Derivative-free optimization is competitive for aerodynamic design optimization in moderate dimensions
Systematic benchmarks on NACA0012, RAE2822, and ONERA M6 cases show derivative-free optimizers competitive with adjoint-based methods and stronger in higher dimensions.
-
Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey
A comprehensive survey of PEFT algorithms for large models, covering their performance, overhead, applications, and real-world system implementations.
-
NUBO: A Transparent Python Package for Bayesian Optimization
NUBO is a transparent, modular Python package implementing established Bayesian optimization techniques for bounded, constrained, and mixed input spaces.
-
Two-stage Optimization for Machine Learning Workflow
Two-stage optimization for ML workflows that prioritizes data pipeline search over hyperparameter tuning, with time-allocation policies and a specificity metric for pruning.
-
Black-box optimization using factorization and Ising machines
FMQA uses factorization machines as surrogates for black-box optimization, converting them directly into QUBO problems solvable by Ising machines for faster acquisition function optimization.
-
Multi-Variable Batch Bayesian Optimization in Materials Research: Synthetic Data Analysis of Noise Sensitivity and Problem Landscape Effects
Synthetic simulations show noise hurts needle-in-haystack optimization far more than smooth landscapes with local optima, and prior domain knowledge of noise and structure is needed for effective BO in materials research.