Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

BOFormer: Learning to Solve Multi-Objective Bayesian Optimization via Non-Markovian RL

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read BOFormer, a Transformer Q-network trained on synthetic functions, is presented as the first RL-learned acquisition function for multi-objective Bayesian optimization, beating handcrafted rules and prior learned methods without fine-tuning.

desk verdict First RL-based acquisition function for MOBO with a genuinely novel Q-augmented history representation and broad zero-shot experiments, but the abstract overclaims and the paper's central identifiability fix rests on a heuristic rather than a proof. read the letter →

arxiv 2505.21974 v2 pith:DZG2KGYK submitted 2025-05-28 cs.LG

Yu-Heng Hung , Kai-Jie Lin , Yu-Heng Lin , Chien-Yi Wang , Cheng Sun , Ping-Chun Hsieh This is my paper · ORCID
classification cs.LG
keywords multi-objectiveBayesianoptimizationacquisitionfunctionlearningnon-MarkovianreinforcementdeepQ-learningTransformersequencemodelinghypervolumeimprovementzero-shottransfer3DGaussianSplattinghyperparameter
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that multi-objective Bayesian optimization can be recast as a non-Markovian reinforcement-learning problem, and that a Transformer-based deep Q-network trained only on synthetic Gaussian-process functions learns a general-purpose acquisition function that beats handcrafted rules and prior learned methods on synthetic benchmarks and on real hyperparameter tuning for 3D Gaussian Splatting rendering. The obstacle it targets is the hypervolume identifiability issue: the benefit of a query, measured by the increase in hypervolume (the volume of objective space dominated by the samples collected so far), depends on the whole history of previous samples, so a policy fed only current posterior statistics cannot tell whether a move is good. BOFormer feeds a Transformer the sequence of past observation-action pairs, describes each candidate point by its posterior mean and uncertainty, the best values seen so far, the remaining budget, and the network's own predicted Q-value, and selects the point with the maximal Q-value. If the claim holds, non-myopic multi-objective acquisition becomes a single forward pass of a learned network, with inference cost nearly flat in the number of objectives and zero-shot transfer across domains and dimensions.

What carries the argument

The load-bearing object is the generalized optimal Q-function on histories, $Q^*(h,a)$, defined by the non-Markovian Bellman optimality equation $Q(h,a) = \mathbb{E}_{o \sim p(\cdot|h,a)}[r(h,a,o) + \gamma \max_{a'} Q(h', a')]$ with $h' = (h,a,o)$, which the paper imports from the general-RL literature and turns into a temporal-difference loss with a frozen target network. BOFormer approximates this Q-function with a GPT-2-style Transformer that consumes the sequence of past per-point observation-action tokens; the crucial augmentation is that each observation carries its own Q-value from the target network, which acts as a learned summary of the prospective hypervolume improvement and keeps the token size constant as the domain grows. The reward is the normalized hypervolume improvement $r_t = (HV(X_t) - HV(X_{t-1}))/(HV(X^{*}) - HV(X_t))$, which keeps the learning signal strong late in an episode, and the fixed history window ($w = 31$) supplies the context that posterior statistics alone provably lack. The action-selection rule carries the argument: at each step the query is $x_t = \arg\max_{x \in X} Q_{\hat{\theta}}(h_t, o_t(x))$.

What would settle it

Build the pair of scenarios from the paper's motivating example: a candidate point whose posterior statistics and best-observed values are identical in the two settings, but whose hypervolume improvement upon sampling differs because the earlier histories differ, and ask BOFormer to score that point in both settings. If the learned Q-values coincide, the representation has not encoded history in exactly the case the paper says it fixes; a complementary stress test would check whether BOFormer's margin over qNEHVI collapses when the training lengthscales are drawn from a wider range than the paper's [0.1, 0.4].

Watch

Extended reading notes

Core claim

The paper's central claim is that the history-dependence of hypervolume improvement, which makes multi-objective Bayesian optimization non-Markovian in a way single-objective BO is not, can be handled by learning a generalized optimal Q-function over histories rather than over states, and that this yields a working acquisition function. It formalizes the idea as Generalized DQN, built on the non-Markovian Bellman optimality equation, and implements it as BOFormer, a GPT-2-style Transformer that consumes a sequence of per-step observation-action tokens and outputs Q-values. The decisive design choice is the Q-augmented representation: each candidate point $x$ is encoded as $o_t(x) = (\mu_t^{(i)}(x), \sigma_t^{(i)}(x), y_t^{(i)*}, t/T)$ over objectives $i$, augmented with the Q-value produced by a frozen target network, and the chosen point is $x_t = \arg\max_{x \in X} Q_{\hat{\theta}}(h_t, o_t(x))$ with $h_t$ the history of past observation-action pairs. Because each token describes one candidate point and its size does not grow with the domain size, the model transfers across domains and dimensions and avoids the memory blow-up of a direct full-state implementation. Trained on randomly drawn GP functions with a normalized hypervolume-improvement reward, guided partly by an expected-hypervolume-improvement demo policy and replayed from a prioritized trajectory buffer, BOFormer is deployed without fine-tuning; the authors report that it attains the largest or near-largest hypervolume at the final sampling step on the synthetic benchmarks and on a 3D Gaussian Splatting hyperparameter-tuning dataset.

Load-bearing premise

The whole scheme rests on the assumption that a fixed-size summary — each candidate point's posterior statistics, a window of the last 31 decisions, and the network's own predicted Q-value — captures enough of the sampling history to tell the policy what the next query is worth; the paper verifies this empirically on its benchmarks but provides no guarantee that this summary identifies the hypervolume improvement in general.

Editorial extensions

If this is right

  • Multi-objective acquisition becomes a cheap forward pass: reported per-step inference time is roughly constant (about 14 ms) for 2, 3, and 4 objectives, whereas qNEHVI grows from about 6 ms to 224 ms, so the learned rule is the cheaper option as the number of objectives grows.
  • Zero-shot and cross-domain deployment is claimed: the same weights trained only on synthetic GP functions produce competitive hypervolumes on 10-, 30-, 40-, and 100-dimensional test problems and on the real 3D Gaussian Splatting tuning task, without fine-tuning or task metadata.
  • The appended Q-value is the load-bearing piece of the representation: Q-Transformer, which uses the same temporal-difference sequence modeling without the Q-augmented observation, performs noticeably worse, isolating the paper's key design choice.
  • History is doing real work: the ablation across sequence lengths shows that variants with $w > 1$ beat the Markovian variant with $w = 1$, supporting the diagnosis that the improvement signal is history-dependent.
  • The paper positions BOFormer as the first RL-based acquisition function for multi-objective Bayesian optimization, extending the learned-acquisition paradigm from the single-objective setting where FSAF-style methods worked to the multi-objective setting where their direct extension fails.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The Q-augmented observation is self-referential: the appended value is an output of the same network being trained, so it is informative only to the extent the policy has already learned, and early in training it cannot help identify history-dependent improvements. A testable consequence is that the method's edge over expected-hypervolume-improvement baselines should shrink if the Q-augmentation i
  • The window length is a fixed hyperparameter ($w = 31$), making the method approximate, not exact, in its non-Markovianity; a natural stress test is to check whether the window must grow with the number of objectives or with the GP lengthscale, since longer-range posterior correlations should make more distant history relevant.
  • The training-time reward normalizes by the optimal hypervolume $HV(X^{*})$ of the synthetic training functions, a quantity unavailable at deployment; this makes the training distribution the carrier of that information, and the policy's behavior on tasks whose Pareto front is hard to estimate is an open question the benchmarks do not fully settle.
  • The Generalized DQN machinery is not specific to hypervolume: other acquisition problems with history-dependent utility, such as batch acquisition where the value of a point depends on the other points chosen alongside it, or constrained BO where past queries affect future feasibility, could be cast in the same sequence-modeling form.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The manuscript proposes BOFormer, a Transformer-based deep Q-network for multi-objective Bayesian optimization (MOBO). It formulates MOBO as non-Markovian RL, uses a generalized DQN loss over histories, and compresses each history into a fixed-dimensional sequence of posterior statistics, best-so-far values, rewards, and target-network Q-values. The policy selects the point maximizing the learned Q-value. BOFormer is trained solely on synthetic GP functions and evaluated zero-shot on synthetic benchmarks and a new 3D Gaussian Splatting hyperparameter optimization dataset. The paper claims to be the first RL-based acquisition function for MOBO and claims to systematically resolve the hypervolume identifiability issue while consistently outperforming rule-based and learning-based baselines.

Significance. The contribution is potentially significant: a learning-based, domain-agnostic, non-myopic acquisition function for MOBO with zero-shot transfer would be a useful advance, and the paper provides broad empirical coverage, public code, and a new HPO-3DGS dataset. The Generalized DQN framing is sound, and the ablations on the demo policy, sequence length, and transfer across numbers of objectives are informative. However, the central theoretical claim about resolving the identifiability issue is not established for the compressed representation, and the empirical claim of consistent superiority is not supported by the reported tables. The paper warrants a major revision rather than acceptance in its current form.

major comments (4)
  1. [Abstract and §5, Tables 1–2] The abstract's claim that BOFormer 'constantly outperforms' the benchmark algorithms is contradicted by the reported numbers. In Table 1, BOFormer attains 0.4476 on BC versus 0.4773 for qNEHVI, 0.9461 on DRa versus 0.9573 for NSGA-II, and 0.8642 on Matern versus 0.8696 for qNEHVI; in Table 2, BOFormer attains 0.8816 on Mic versus 0.8871 for FSAF. The text later hedges to 'largest or among the largest' (Section 5). The load-bearing claim of consistent outperformance should be replaced with a precise statement of where and by how much BOFormer wins, together with the associated statistical evidence.
  2. [§5, Tables 1–2 and Figure 3] The main results are reported as point estimates without error bars or confidence intervals, despite being averages over 100 evaluation episodes. This is not a purely cosmetic issue: on many tasks the differences between BOFormer and qNEHVI are around 1% (e.g., AR, RBF, BCD in Table 1; Chairs, Lego, Materials in Table 2), which is precisely the threshold used for bold/underline formatting. Performance profiles are shown only for a subset of tasks in Figure 3 and the appendix. The authors should report standard errors or bootstrap intervals for all tasks and perform a paired comparison (e.g., Wilcoxon signed-rank or performance-profile analysis) to support the claimed superiority.
  3. [§3.2–§4.2, Eq. (5), Figure 1] The theoretical justification only covers full histories: Proposition 3.1 characterizes Q* for histories H_t, and Figure 1 demonstrates the identifiability failure for an observation representation without the Q-augmented token. However, BOFormer's policy is a function of the fixed-dimensional summary h_t in Eq. (5), which stores posterior means and standard deviations at past selected points, the best-so-far vector, normalized rewards, and a Q-bar token. It does not store the queried locations x_j or the observed values y_j, and Q-bar is itself computed by the target network from the same compressed inputs, so it cannot inject information absent from h_t. No theorem, invariant, or bound establishes that this compressed representation preserves all information relevant to future hypervolume improvement. Two runs with different GP posteriors can in principle yield identical BOFormer inputs but different optimal future improvements, in which case no function of the BOFormer input can represent Q*. The claim that BOFormer 'systematically addresses' the identifiability issue is therefore not supported; at present this resolution is heuristic, and its success may depend on the training distribution and window size. The authors should either prove sufficiency for a well-defined class of environments, or explicitly downgrade the claim and provide evidence that the failure cases are rare in the tested distribution.
  4. [§5, Figure 4 and Appendix A.2] The history window w=31 is selected by an ablation on DRa, RBF, and Matern52 (Figure 4), which are the same test functions used in Table 1. Selecting hyperparameters on the evaluation tasks can inflate the reported performance and makes the zero-shot claim less clean. Please move the window-size selection to a separate validation set drawn from the training distribution (e.g., held-out synthetic GP functions) and report the final configuration with that protocol.
minor comments (6)
  1. [Abstract and §5] The abstract's 'constantly outperforms' conflicts with Section 5's 'largest or among the largest'; please harmonize the wording.
  2. [§4.2, Eq. (5)] The definition of y(i)*_t contains a typo ('y(i)_i' instead of 'y(i)_j'), and the history in Eq. (5) uses y(i)*_{j-1}; make the index convention explicit.
  3. [Abstract and §5] The abstract states 68,000 parameters for HPO-3DGS while the main text says 'dense-grid searching 1440 hyperparameters'; clarify whether 1440 refers to hyperparameter configurations per scene and how the 68,000 figure is derived.
  4. [Appendix C.1, Algorithm 1] The loss in Algorithm 1 uses the notation Qb_bθ and Qb_¯θ without defining the superscript b; define it in the caption or surrounding text.
  5. [§2.2] The text says 'first learning-based solution to MOBO' while the introduction claims 'first RL-based AF for MOBO'; the former is broader and should be qualified.
  6. [Table 3] The heading 'per-sample inference time' is ambiguous; specify that it is time per acquisition-function evaluation or query.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity; the core RL derivation is self-contained, with only minor non-load-bearing self-citations and a methodological confound that falls short of a definitional circularity.

full rationale

The paper's central training objective, Eq. (4), is the standard temporal-difference residual of deep Q-learning adapted to history-dependent value functions, and Proposition 3.1 is imported from the external result of Dong et al. (2022), not from the authors' own prior work. The empirical claims are tested against external rule-based and learning-based baselines on synthetic GP functions and an independently constructed 3DGS dataset, so the main reported predictions are not forced by the training inputs. The only self-citations (FSAF, Sobol-grid maximization) are used as a benchmark or as a standard implementation detail and are not load-bearing for the claimed derivation. Two concerns raised by the reader and skeptic are better classified as correctness or methodological risks rather than circularity. First, the Q-augmented representation in Eq. (5) is bootstrapped: Q-bar is computed recursively from the same compressed history tokens, so it is not established that appending Q-bar restores the raw information omitted from the summary; this is a representational-sufficiency gap, not a definitional equivalence. Second, the demo policy used during training is qNEHVI, which also appears as a headline baseline; this is a benchmark-contamination confound, but BOFormer's advantage over qNEHVI is not statistically forced by that construction. Accordingly, no step reduces by construction to its own inputs, and the appropriate finding is no significant circularity.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

BOFormer postulates no new physical entity and no new mathematical axiom; it relies on GPs as surrogates, the non-Markov Bellman equations of Dong et al., and several hand-tuned training choices. The main ledger entries are the window size, demo rate, discount factor, and model hyperparameters, plus the domain assumption that GP training distributions cover deployment tasks.

free parameters (4)
  • Window size w = 31
    Sequence length of the history fed to the Transformer. The ablation in Figure 4 shows w=1 is worse and w=31 is used in the main experiments; the choice is made on the same test functions DRa, RBF, and Matern52.
  • Demo policy mix rate r_demo = 0.01
    Probability of collecting a training trajectory with an EHVI or qNEHVI demo policy. Appendix A.2 and D.2 show that the demo policy affects performance.
  • Discount factor gamma = not reported in paper
    Appears in loss Eq. (4) and Algorithms 1 and 2 but is absent from the hyperparameter table; it controls how non-myopic the learned Q-function is and is unverifiable from the text.
  • Architecture and training hyperparameters = hidden size 128, 8 layers, 4 heads, learning rate 1e-5, batch size 8, buffer size 64, target sync every 5 episodes
    Manually chosen values in Appendix A.2; performance of a learned acquisition function can depend on them, and no sensitivity study is reported.
assumptions (5)
  • standard math Non-Markov Bellman optimality: (V*, Q*) is the unique solution of Eqs. (2)-(3).
    Invoked in Section 3.2, Proposition 3.1, and used as the Generalized DQN loss in Section 4.1; the proof belongs to Dong et al. 2022.
  • domain assumption GP posteriors are sufficient surrogates for the true black-box functions.
    All training and testing use GP surrogate models per Appendix A.1; deployment on 3D Gaussian Splatting assumes the GP model captures the objective landscape for hyperparameter tuning.
  • ad hoc to paper Training lengthscale distribution Uniform(0.1, 0.4) covers the test functions.
    Section 5 says all learning-based methods are trained with lengthscales in [0.1, 0.4] for fairness; zero-shot transfer to unseen functions and to 3DGS relies on this distributional match, and the paper notes BC is not expected to work well because its lengthscales are larger.
  • domain assumption Approximate action maximization over Sobol grids is sufficient to realize the policy.
    Continuous domains are handled by Sobol grids in Remark 4.2 and Appendix C.2, so the policy is not evaluated at the exact argmax and may differ from the learned Q-function's true maximizer.
  • domain assumption HV(X*) is known or cheaply approximated during training.
    Reward normalization in Eq. (7) uses the optimal hypervolume HV(X*); Remark C.1 says it can be precomputed or approximated, but for arbitrary black-box functions this is nontrivial.

how reviews work

0 comments
Cite this review

Pith. "Pith review of BOFormer: Learning to Solve Multi-Objective Bayesian Optimization via Non-Markovian RL." pith.science (2026). https://pith.science/paper/DZG2KGYK

@misc{pith2026250521974,
  author       = {Pith},
  title        = {Pith review of: BOFormer: Learning to Solve Multi-Objective Bayesian Optimization via Non-Markovian RL},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DZG2KGYK}},
  note         = {Machine review of arXiv:2505.21974}
}
read the original abstract

Bayesian optimization (BO) offers an efficient pipeline for optimizing black-box functions with the help of a Gaussian process prior and an acquisition function (AF). Recently, in the context of single-objective BO, learning-based AFs witnessed promising empirical results given its favorable non-myopic nature. Despite this, the direct extension of these approaches to multi-objective Bayesian optimization (MOBO) suffer from the \textit{hypervolume identifiability issue}, which results from the non-Markovian nature of MOBO problems. To tackle this, inspired by the non-Markovian RL literature and the success of Transformers in language modeling, we present a generalized deep Q-learning framework and propose \textit{BOFormer}, which substantiates this framework for MOBO via sequence modeling. Through extensive evaluation, we demonstrate that BOFormer constantly outperforms the benchmark rule-based and learning-based algorithms in various synthetic MOBO and real-world multi-objective hyperparameter optimization problems. We have made the source code publicly available to encourage further research in this direction.

Figures

Figures reproduced from arXiv: 2505.21974 by the authors.

Figure 1
Figure 1. Left: In SOBO, an RL-based AF (e.g., FSAF (Hsieh et al., 2021)) takes the posterior mean and standard deviation (µt(x), σt(x)) and the best function value observed so far y ∗ t as input and then outputs the AF value Υt(x). An direct extension to MOBO simply takes into account the same set of information about all the K objective functions. Right: The hypervolume identifiability issue can be illustrated by comparing … view at source ↗
Figure 2
Figure 2. BOFormer comprises two distinct networks as [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Performance profiles of hypervolume at the final step. [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: Attained hypervolumes of BOFormer under various sequence lengths. [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Training loss for 3 variants of BOFormer models Handling Continuous Domains Under BOFormer. As BOFormer is a learning-based acquisition function (AF) implemented as a neural network, it can nicely handle continuous domains by leveraging the common ways of finding an (a…
Figure 6
Figure 6. Figure 6: Averaged attained hypervolume under synthetic functions with two objectives. [PITH_FULL_IMAGE:figures/full_fig_p024_6.png]
Figure 7
Figure 7. Figure 7: Averaged attained hypervolume under HPO-3DGS with two objectives. [PITH_FULL_IMAGE:figures/full_fig_p024_7.png]
Figure 3
Figure 3. Figure 3: Intuitively, we would expect that the trajectories contributed by qNEHVI can [PITH_FULL_IMAGE:figures/full_fig_p024_3.png]
Figure 8
Figure 8. Figure 8: Averaged attained hypervolume under synthetic and HPO-3DGS with three objectives. [PITH_FULL_IMAGE:figures/full_fig_p025_8.png]
Figure 9
Figure 9. Figure 9: Performance profiles in terms of hypervolume at the final step under synthetic functions [PITH_FULL_IMAGE:figures/full_fig_p025_9.png]
Figure 10
Figure 10. Figure 10: Averaged attained hypervolume under synthetic functions with two objectives. [PITH_FULL_IMAGE:figures/full_fig_p026_10.png]
Figure 11
Figure 11. Figure 11: Averaged attained hypervolume under more challenging GP functions with higher [PITH_FULL_IMAGE:figures/full_fig_p026_11.png]
Figure 12
Figure 12. Figure 12: Averaged attained hypervolume under various challenging functions, including synthetic [PITH_FULL_IMAGE:figures/full_fig_p027_12.png]
Figure 13
Figure 13. Figure 13: Averaged attained hypervolume under more challenging problems with [PITH_FULL_IMAGE:figures/full_fig_p028_13.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Model-free Reinforcement Learning for Model-based Control: Towards Safe, Interpretable and Sample-efficient Agents

    cs.LG 2025-07 conditional novelty 3.0 of 10

    A perspective paper argues that model predictive control can be used as a learned policy in model-free reinforcement learning and reviews the methods and open problems.

Reference graph

Works this paper leans on

77 extracted references · 69 canonical work pages · cited by 1 Pith paper

  1. [1]

    Deep reinforcement learning at the edge of the statistical precipice

    Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in Neural Information Processing Systems, 34: 0 29304--29320, 2021

    work page 2021

  2. [2]

    BoTorch: A framework for efficient Monte-Carlo Bayesian optimization

    Maximilian Balandat, Brian Karrer, Daniel Jiang, Samuel Daulton, Ben Letham, Andrew G Wilson, and Eytan Bakshy. BoTorch: A framework for efficient Monte-Carlo Bayesian optimization . Advances in Neural Information Processing Systems, 33: 0 21524--21538, 2020

    work page 2020

  3. [3]

    Max-value entropy search for multi-objective Bayesian optimization

    Syrine Belakaria, Aryan Deshwal, and Janardhan Rao Doppa. Max-value entropy search for multi-objective Bayesian optimization . Advances in Neural Information Processing Systems, 32, 2019

    work page 2019

  4. [4]

    Uncertainty-aware search framework for multi-objective Bayesian optimization

    Syrine Belakaria, Aryan Deshwal, Nitthilan Kannappan Jayakodi, and Janardhan Rao Doppa. Uncertainty-aware search framework for multi-objective Bayesian optimization . In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp.\ 10044--10052, 2020

    work page 2020

  5. [5]

    Output space entropy search framework for multi-objective Bayesian optimization

    Syrine Belakaria, Aryan Deshwal, and Janardhan Rao Doppa. Output space entropy search framework for multi-objective Bayesian optimization . Journal of Artificial Intelligence Research, 72: 0 667--715, 2021

    work page 2021

  6. [6]

    SMS-EMOA: Multiobjective selection based on dominated hypervolume

    Nicola Beume, Boris Naujoks, and Michael Emmerich. SMS-EMOA: Multiobjective selection based on dominated hypervolume . European Journal of Operational Research, 181 0 (3): 0 1653--1669, 2007

    work page 2007

  7. [7]

    Settling the reward hypothesis

    Michael Bowling, John D Martin, David Abel, and Will Dabney. Settling the reward hypothesis. In International Conference on Machine Learning, pp.\ 3003--3020, 2023

    work page 2023

  8. [8]

    Q-Transformer: Scalable offline reinforcement learning via autoregressive q-functions

    Yevgen Chebotar, Quan Vuong, Karol Hausman, Fei Xia, Yao Lu, Alex Irpan, Aviral Kumar, Tianhe Yu, Alexander Herzog, Karl Pertsch, et al. Q-Transformer: Scalable offline reinforcement learning via autoregressive q-functions . In Conference on Robot Learning, pp.\ 3909--3928, 2023

    work page 2023

Show all 77 references
  1. [9]

    Decision Transformer: Reinforcement learning via sequence modeling

    Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision Transformer: Reinforcement learning via sequence modeling . Advances in Neural Information Processing Systems, 34: 0 15084--15097, 2021

    2021

  2. [10]

    Towards learning universal hyperparameter optimizers with transformers

    Yutian Chen, Xingyou Song, Chansoo Lee, Zi Wang, Richard Zhang, David Dohan, Kazuya Kawakami, Greg Kochanski, Arnaud Doucet, Marc'aurelio Ranzato, et al. Towards learning universal hyperparameter optimizers with transformers. Advances in Neural Information Processing Systems, ...

    2022

  3. [11]

    A reference vector guided evolutionary algorithm for many-objective optimization

    Ran Cheng, Yaochu Jin, Markus Olhofer, and Bernhard Sendhoff. A reference vector guided evolutionary algorithm for many-objective optimization . IEEE Transactions on Evolutionary Computation, 20 0 (5): 0 773--791, 2016

    2016

  4. [12]

    Unifying PAC and regret: Uniform PAC bounds for episodic reinforcement learning

    Christoph Dann, Tor Lattimore, and Emma Brunskill. Unifying PAC and regret: Uniform PAC bounds for episodic reinforcement learning . Advances in Neural Information Processing Systems, 30, 2017

    2017

  5. [13]

    Hypervolume knowledge gradient: A lookahead approach for multi-objective Bayesian optimization with partial information

    Sam Daulton, Maximilian Balandat, and Eytan Bakshy. Hypervolume knowledge gradient: A lookahead approach for multi-objective Bayesian optimization with partial information . In International Conference on Machine Learning, pp.\ 7167--7204, 2023

    2023

  6. [14]

    Differentiable expected hypervolume improvement for parallel multi-objective Bayesian optimization

    Samuel Daulton, Maximilian Balandat, and Eytan Bakshy. Differentiable expected hypervolume improvement for parallel multi-objective Bayesian optimization . Advances in Neural Information Processing Systems, 33: 0 9851--9864, 2020

    2020

  7. [15]

    Parallel Bayesian optimization of multiple noisy objectives with expected hypervolume improvement

    Samuel Daulton, Maximilian Balandat, and Eytan Bakshy. Parallel Bayesian optimization of multiple noisy objectives with expected hypervolume improvement . Advances in Neural Information Processing Systems, 34: 0 2187--2200, 2021

    2021

  8. [16]

    Multi-objective Bayesian optimization over high-dimensional search spaces

    Samuel Daulton, David Eriksson, Maximilian Balandat, and Eytan Bakshy. Multi-objective Bayesian optimization over high-dimensional search spaces . In Uncertainty in Artificial Intelligence, pp.\ 507--517, 2022

    2022

  9. [17]

    A fast and elitist multiobjective genetic algorithm: NSGA-II

    Kalyanmoy Deb, Amrit Pratap, Sameer Agarwal, and TAMT Meyarivan. A fast and elitist multiobjective genetic algorithm: NSGA-II . IEEE Transactions on Evolutionary Computation, 6 0 (2): 0 182--197, 2002

    2002

  10. [18]

    Simple agent, complex environment: Efficient reinforcement learning with agent states

    Shi Dong, Benjamin Van Roy, and Zhengyuan Zhou. Simple agent, complex environment: Efficient reinforcement learning with agent states . Journal of Machine Learning Research, 23 0 (1): 0 11627--11680, 2022

    2022

  11. [19]

    The computation of the expected improvement in dominated hypervolume of Pareto front approximations

    Michael Emmerich and Jan-willem Klinkenberg. The computation of the expected improvement in dominated hypervolume of Pareto front approximations . Rapport technique, Leiden University, 2008

    2008

  12. [20]

    Hypervolume-based expected improvement: Monotonicity properties and exact computation

    Michael TM Emmerich, Andr \'e H Deutz, and Jan Willem Klinkenberg. Hypervolume-based expected improvement: Monotonicity properties and exact computation . In IEEE Congress of Evolutionary Computation (CEC), pp.\ 2147--2154, 2011

    2011

  13. [21]

    Generalized decision transformer for offline hindsight information matching

    Hiroki Furuta, Yutaka Matsuo, and Shixiang Shane Gu. Generalized decision transformer for offline hindsight information matching. In International Conference on Learning Representations, 2021

    2021

  14. [22]

    Conditional neural processes

    Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Rezende, and SM Ali Eslami. Conditional neural processes. In International Conference on Machine Learning, pp.\ 1704--1713, 2018

    2018

  15. [23]

    Predictive entropy search for multi-objective Bayesian optimization with constraints

    Eduardo C Garrido-Merch \'a n and Daniel Hern \'a ndez-Lobato. Predictive entropy search for multi-objective Bayesian optimization with constraints . Neurocomputing, 361: 0 50--68, 2019

    2019

  16. [24]

    Automatic chemical design using a data-driven continuous representation of molecules

    Rafael G \'o mez-Bombarelli, Jennifer N Wei, David Duvenaud, Jos \'e Miguel Hern \'a ndez-Lobato, Benjam \' n S \'a nchez-Lengeling, Dennis Sheberla, Jorge Aguilera-Iparraguirre, Timothy D Hirzel, Ryan P Adams, and Al \'a n Aspuru-Guzik. Automatic chemical design using a data-...

    2018

  17. [25]

    Deep recurrent Q-learning for partially observable MDPs

    Matthew Hausknecht and Peter Stone. Deep recurrent Q-learning for partially observable MDPs . In AAAI Fall Symposium Series, 2015

    2015

  18. [26]

    Predictive entropy search for multi-objective Bayesian optimization

    Daniel Hern \'a ndez-Lobato, Jose Hernandez-Lobato, Amar Shah, and Ryan Adams. Predictive entropy search for multi-objective Bayesian optimization . In International Conference on Machine Learning, pp.\ 1492--1501, 2016

    2016

  19. [27]

    Reinforced few-shot acquisition function learning for Bayesian optimization

    Bing-Jing Hsieh, Ping-Chun Hsieh, and Xi Liu. Reinforced few-shot acquisition function learning for Bayesian optimization . Advances in Neural Information Processing Systems, 34: 0 7718--7731, 2021

    2021

  20. [28]

    Faster exact algorithms for computing expected hypervolume improvement

    Iris Hupkens, Andr \'e Deutz, Kaifeng Yang, and Michael Emmerich. Faster exact algorithms for computing expected hypervolume improvement. In International Conference on Evolutionary Multi-Criterion Optimization, pp.\ 65--79. Springer, 2015

    2015

  21. [29]

    Joint entropy search for maximally-informed Bayesian optimization

    Carl Hvarfner, Frank Hutter, and Luigi Nardi. Joint entropy search for maximally-informed Bayesian optimization . Advances in Neural Information Processing Systems, 35: 0 11494--11506, 2022

    2022

  22. [30]

    Reinforcement learning algorithm for partially observable Markov decision problems

    Tommi Jaakkola, Satinder Singh, and Michael Jordan. Reinforcement learning algorithm for partially observable Markov decision problems . Advances in Neural Information Processing systems, 7, 1994

    1994

  23. [31]

    Offline reinforcement learning as one big sequence modeling problem

    Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. Advances in Neural Information Processing Systems, 34: 0 1273--1286, 2021

    2021

  24. [32]

    3d gaussian splatting for real-time radiance field rendering

    Bernhard Kerbl, Georgios Kopanas, Thomas Leimk \"u hler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42 0 (4): 0 1--14, 2023

    2023

  25. [33]

    Attentive neural processes

    Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Garnelo, Ali Eslami, Dan Rosenbaum, Oriol Vinyals, and Yee Whye Teh. Attentive neural processes. In International Conference on Learning Representations, 2018

    2018

  26. [34]

    Adam: A method for stochastic optimization

    Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization . In International Conference for Learning Representations, 2015

    2015

  27. [35]

    Fast Bayesian optimization of machine learning hyperparameters on large datasets

    Aaron Klein, Stefan Falkner, Simon Bartels, Philipp Hennig, and Frank Hutter. Fast Bayesian optimization of machine learning hyperparameters on large datasets . In International Conference on Artificial Intelligence and Statistics, pp.\ 528--536, 2017

    2017

  28. [36]

    ParEGO: A hybrid algorithm with on-line landscape approximation for expensive multiobjective optimization problems

    Joshua Knowles. ParEGO: A hybrid algorithm with on-line landscape approximation for expensive multiobjective optimization problems . IEEE Transactions on Evolutionary Computation, 10 0 (1): 0 50--66, 2006

    2006

  29. [37]

    Stabilizing off-policy Q-learning via bootstrapping error reduction

    Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy Q-learning via bootstrapping error reduction . Advances in Neural Information Processing Systems, 32, 2019

    2019

  30. [38]

    The sample-complexity of general reinforcement learning

    Tor Lattimore, Marcus Hutter, and Peter Sunehag. The sample-complexity of general reinforcement learning. In International Conference on Machine Learning, pp.\ 28--36, 2013

    2013

  31. [39]

    Multi-game decision transformers

    Kuang-Huei Lee, Ofir Nachum, Mengjiao Sherry Yang, Lisa Lee, Daniel Freeman, Sergio Guadarrama, Ian Fischer, Winnie Xu, Eric Jang, Henryk Michalewski, et al. Multi-game decision transformers. Advances in Neural Information Processing Systems, 35: 0 27921--27936, 2022

    2022

  32. [40]

    Nonparametric general reinforcement learning

    Jan Leike. Nonparametric general reinforcement learning. PhD thesis, The Australian National University (Australia), 2016

    2016

  33. [41]

    SMAC3: A versatile Bayesian optimization package for hyperparameter optimization

    Marius Lindauer, Katharina Eggensperger, Matthias Feurer, Andr \'e Biedenkapp, Difan Deng, Carolin Benjamins, Tim Ruhkopf, Ren \'e Sass, and Frank Hutter. SMAC3: A versatile Bayesian optimization package for hyperparameter optimization . Journal of Machine Learning Research, 2...

    2022

  34. [42]

    Reinforcement learning, bit by bit

    Xiuyuan Lu, Benjamin Van Roy, Vikranth Dwaracherla, Morteza Ibrahimi, Ian Osband, Zheng Wen, et al. Reinforcement learning, bit by bit. Foundations and Trends in Machine Learning , 16 0 (6): 0 733--865, 2023

    2023

  35. [43]

    Batch Bayesian optimization via multi-objective acquisition ensemble for automated analog circuit design

    Wenlong Lyu, Fan Yang, Changhao Yan, Dian Zhou, and Xuan Zeng. Batch Bayesian optimization via multi-objective acquisition ensemble for automated analog circuit design . In International Conference on Machine Learning, pp.\ 3306--3314, 2018

    2018

  36. [44]

    Abstractions of general reinforcement learning: An inquiry into the scalability of generally intelligent agents

    Sultan J Majeed. Abstractions of general reinforcement learning: An inquiry into the scalability of generally intelligent agents . PhD thesis, The Australian National University (Australia), 2021

    2021

  37. [45]

    End-to-end meta-Bayesian optimisation with Transformer neural processes

    Alexandre Maraval, Matthieu Zimmer, Antoine Grosnit, and Haitham Bou Ammar. End-to-end meta-Bayesian optimisation with Transformer neural processes . Advances in Neural Information Processing Systems, 2023

    2023

  38. [46]

    Nonlinear multiobjective optimization, volume 12

    Kaisa Miettinen. Nonlinear multiobjective optimization, volume 12. Springer Science & Business Media, 1999

    1999

  39. [47]

    NeRF: Representing scenes as neural radiance fields for view synthesis

    Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing scenes as neural radiance fields for view synthesis . Communications of the ACM, 65 0 (1): 0 99--106, 2021

    2021

  40. [49]

    A survey of partially observable Markov decision processes: Theory, models, and algorithms

    George E Monahan. A survey of partially observable Markov decision processes: Theory, models, and algorithms . Management Science, 28 0 (1): 0 1--16, 1982

    1982

  41. [50]

    A unifying view of optimism in episodic reinforcement learning

    Gergely Neu and Ciara Pike-Burke. A unifying view of optimism in episodic reinforcement learning. Advances in Neural Information Processing Systems, 33: 0 1392--1403, 2020

    2020

  42. [51]

    A flexible framework for multi-objective Bayesian optimization using random scalarizations

    Biswajit Paria, Kirthevasan Kandasamy, and Barnab \'a s P \'o czos. A flexible framework for multi-objective Bayesian optimization using random scalarizations . In Uncertainty in Artificial Intelligence, pp.\ 766--776, 2020

    2020

  43. [52]

    Multiobjective optimization using Gaussian process emulators via stepwise uncertainty reduction

    Victor Picheny. Multiobjective optimization using Gaussian process emulators via stepwise uncertainty reduction . Statistics and Computing, 25 0 (6): 0 1265--1280, 2015

    2015

  44. [53]

    Multiobjective optimization on a limited budget of evaluations using model-assisted-metric selection

    Wolfgang Ponweiser, Tobias Wagner, Dirk Biermann, and Markus Vincze. Multiobjective optimization on a limited budget of evaluations using model-assisted-metric selection. In International Conference on Parallel Problem Solving From Nature, pp.\ 784--794. Springer, 2008

    2008

  45. [54]

    Prioritized experience replay

    Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. In International Conference for Learning Representations, 2016

    2016

  46. [55]

    Reinforcement learning upside down: Don't predict rewards--Just map them to actions

    Juergen Schmidhuber. Reinforcement learning upside down: Don't predict rewards--Just map them to actions . arXiv preprint arXiv:1912.02875, 2019

    1912 arXiv

  47. [56]

    RTDK-BO: High dimensional Bayesian optimization with reinforced transformer deep kernels

    Alexander Shmakov, Avisek Naug, Vineet Gundecha, Sahand Ghorbanpour, Ricardo Luna Gutierrez, Ashwin Ramesh Babu, Antonio Guillen, and Soumyendu Sarkar. RTDK-BO: High dimensional Bayesian optimization with reinforced transformer deep kernels . In IEEE International Conference o...

    2023

  48. [57]

    Practical Bayesian optimization of machine learning algorithms

    Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical Bayesian optimization of machine learning algorithms . Advances in Neural Information Processing Systems, 25, 2012

    2012

  49. [58]

    Multi-objective Bayesian optimization using Pareto-frontier entropy

    Shinya Suzuki, Shion Takeno, Tomoyuki Tamura, Kazuki Shitara, and Masayuki Karasuyama. Multi-objective Bayesian optimization using Pareto-frontier entropy . In International Conference on Machine Learning, pp.\ 9279--9288, 2020

    2020

  50. [59]

    MuJoCo: A physics engine for model-based control

    Emanuel Todorov, Tom Erez, and Yuval Tassa. MuJoCo: A physics engine for model-based control . In IEEE/RSJ International Conference on Intelligent Robots and Systems, pp.\ 5026--5033, 2012

    2012

  51. [60]

    Joint entropy search for multi-objective Bayesian optimization

    Ben Tu, Axel Gandy, Nikolas Kantas, and Behrang Shafei. Joint entropy search for multi-objective Bayesian optimization . Advances in Neural Information Processing Systems, 35: 0 9922--9938, 2022

    2022

  52. [61]

    COMBO: An Efficient Bayesian Optimization Library for Materials Science

    Tsuyoshi Ueno, Trevor David Rhone, Zhufeng Hou, Teruyasu Mizoguchi, and Koji Tsuda. COMBO: An Efficient Bayesian Optimization Library for Materials Science . Materials Discovery, 4: 0 18--21, 2016

    2016

  53. [62]

    Meta-learning acquisition functions for transfer learning in Bayesian optimization

    Michael Volpp, Lukas P Fr \"o hlich, Kirsten Fischer, Andreas Doerr, Stefan Falkner, Frank Hutter, and Christian Daniel. Meta-learning acquisition functions for transfer learning in Bayesian optimization . In International Conference on Learning Representations, 2020

    2020

  54. [63]

    Bayesian optimization for multi-objective optimization and multi-point search

    Takashi Wada and Hideitsu Hino. Bayesian optimization for multi-objective optimization and multi-point search . arXiv preprint arXiv:1905.02370, 2019

    1905 arXiv

  55. [64]

    Max-value entropy search for efficient Bayesian optimization

    Zi Wang and Stefanie Jegelka. Max-value entropy search for efficient Bayesian optimization . In International Conference on Machine Learning, pp.\ 3627--3635, 2017

    2017

  56. [65]

    Gaussian processes for machine learning, volume 2

    Christopher KI Williams and Carl Edward Rasmussen. Gaussian processes for machine learning, volume 2. MIT Press, 2006

    2006

  57. [66]

    Multi-objective Bayesian global optimization using expected hypervolume improvement gradient

    Kaifeng Yang, Michael Emmerich, Andr \'e Deutz, and Thomas B \"a ck. Multi-objective Bayesian global optimization using expected hypervolume improvement gradient . Swarm and Evolutionary Computation, 44: 0 945--956, 2019

    2019

  58. [67]

    Bayesian model-agnostic meta-learning

    Jaesik Yoon, Taesup Kim, Ousmane Dia, Sungwoong Kim, Yoshua Bengio, and Sungjin Ahn. Bayesian model-agnostic meta-learning. Advances in Neural Information Processing Systems, 31, 2018

    2018

  59. [68]

    MVImgNet: A large-scale dataset of multi-view images

    Xianggang Yu, Mutian Xu, Yidan Zhang, Haolin Liu, Chongjie Ye, Yushuang Wu, Zizheng Yan, Chenming Zhu, Zhangyang Xiong, Tianyou Liang, et al. MVImgNet: A large-scale dataset of multi-view images . In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni...

    2023

  60. [69]

    MOEA/D: A multiobjective evolutionary algorithm based on decomposition

    Qingfu Zhang and Hui Li. MOEA/D: A multiobjective evolutionary algorithm based on decomposition . IEEE Transactions on Evolutionary Computation, 11 0 (6): 0 712--731, 2007

    2007

  61. [70]

    Expensive multiobjective optimization by MOEA/D with Gaussian process model

    Qingfu Zhang, Wudong Liu, Edward Tsang, and Botond Virginas. Expensive multiobjective optimization by MOEA/D with Gaussian process model . IEEE Transactions on Evolutionary Computation, 14 0 (3): 0 456--474, 2009

    2009

  62. [71]

    Online decision transformer

    Qinqing Zheng, Amy Zhang, and Aditya Grover. Online decision transformer. In International Conference on Machine Learning, pp.\ 27042--27059, 2022

    2022

  63. [72]

    A trust-region parallel Bayesian optimization method for simulation-driven antenna design

    Jinzhu Zhou, Zhanbiao Yang, Yu Si, Le Kang, Haitao Li, Mei Wang, and Zhiya Zhang. A trust-region parallel Bayesian optimization method for simulation-driven antenna design . IEEE Transactions on Antennas and Propagation, 69 0 (7): 0 3966--3981, 2020

    2020

  64. [73]

    Multiobjective evolutionary algorithms: A comparative case study and the strength Pareto approach

    Eckart Zitzler and Lothar Thiele. Multiobjective evolutionary algorithms: A comparative case study and the strength Pareto approach . IEEE Transactions on Evolutionary Computation, 3 0 (4): 0 257--271, 1999

    1999

  65. [74]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  66. [75]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  67. [76]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  68. [77]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  69. [78]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

    2019 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.