{"id":"58823d63-2798-4db1-b8c7-1fd5f21959ca","arxiv_id":"2412.18518","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"BILBAO models the lower-level objective as a joint GP over leader and follower decisions and uses a multi-task acquisition function to learn the follower's best-response map efficiently.","lead":"Researchers present BILBAO, a Bayesian optimization method for bilevel problems where both the leader's and follower's objective functions are expensive black boxes. It learns the follower's best-response map with a linked Gaussian-process model and reports faster convergence than nested Bayesian optimization on synthetic benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"BILBAO's upper-level search is driven by the same estimated response map whose error at the optimum the authors report does not converge, so the reported oracle-evaluated optimality gap may not reflect the algorithm's actual knowledge; this needs a direct test.","rationale":"The paper proposes a plausible alternating BO framework, and the central claim is an empirical one: BILBAO is highly sample-efficient and outperforms existing methods on fully black-box bilevel problems. For that claim to hold, the estimated lower-level response map Phi_n must be accurate enough at the upper-level actions that the algorithm actually proposes. The reader identified this same weakest assumption. My stress-test sharpens it: the map error can be self-reinforcing, because the lower-level acquisition budget is directed by the same upper-level model whose quality depends on the map. The authors' Appendix explicitly reports non-convergence of the action gap, including at the optimal upper action, which is direct evidence that the weakest assumption is not guaranteed. The reported optimality gap evaluates the proposed x_u with the oracle Phi*, so it does not expose this internal inconsistency. A single oracle-map ablation would settle whether the map error actually changes the proposed argmax. Because this concern reinforces the reader's conditional verdict rather than overturning the framework, I recommend no change to the CONDITIONAL verdict.","tokens_in":14856,"tokens_out":10484,"duration_ms":108524,"concrete_test":"On SMD1-SMD4 and the Camel-Branin/Dixon-Branin tests, compare BILBAO with an oracle-map ablation: use the true Phi* only in the upper-level Thompson sampling and in the final selection (Algorithm 2 lines 8-9 and 20), while the lower-level acquisition remains GP-based. If the oracle variant's optimality gap is materially smaller than the reported BILBAO gap, then Phi_n error is the bottleneck and the headline sample-efficiency claim needs qualification. If the gaps are comparable, the non-convergence in Appendix Figure 6 is harmless.","verdict_should_be":"UNCHANGED","load_bearing_attack":"BILBAO's information flow is a closed loop: upper-level Thompson sampling and final selection use the estimated lower-level response map Phi_n (Algorithm 2, lines 7-12 and 20), and the lower-level acquisition REVI/REVITS concentrates queries on the upper-level set X_TS derived from that same upper-level model (Equation 5, lines 13-15). If Phi_n is inaccurate near the true optimum, X_TS will tend to miss that region, and no independent mechanism forces lower-level evaluations there. The authors' own Appendix (Figures 5-6) reports exactly this symptom: the action gap over the domain does not converge, and the action gap at the optimal upper-level solution 'seems not fully confirmed' to decrease. Meanwhile, the reported optimality gap (Section 4) evaluates the proposed x_u with the oracle response Phi*(x_u). That is the correct leader's objective, but it means the headline curves can look good even if the algorithm's internal model at the proposed point is poor. The load-bearing, untested assumption is that the error in Phi_n does not change the argmax of the upper-level restricted posterior; the paper provides no evidence for this, and its own appendix raises doubt. This is a self-referential limitation that should be weighed in the verdict.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes BILBAO, a Bayesian optimization framework for bilevel problems where both upper- and lower-level objectives are black-box and expensive. Upper and lower-level functions are modeled with separate GPs over the joint decision space, and the lower-level response map Phi(x_u) is estimated by maximizing the lower-level GP posterior mean. The upper-level is optimized by Thompson sampling of the restricted posterior mean F(x_u, Phi(x_u)). The paper introduces REVI and REVITS acquisition functions for lower-level sampling, which weight the lower-level query by a set of Thompson-sampled important upper-level points. The algorithm is compared against a from-scratch lower-level BO benchmark on two 2D and four 4D synthetic problems, and the paper reports optimality gaps over evaluation budget.","tokens_in":15190,"tokens_out":8323,"duration_ms":68809,"significance":"If the empirical claims hold, BILBAO addresses a real gap: existing BO bilevel methods either assume a known lower level or solve the lower level from scratch, which is sample-inefficient. The joint GP modeling and the REVI/REVITS acquisition functions are a reasonable instantiation of multi-task BO for this problem, and the authors are commendably transparent in the appendix about the non-convergence of action-gap metrics. The absence of theoretical guarantees and the lack of released code are not fatal for an empirical paper, but the evidence as presented does not yet fully support the claim of 'highly sample-efficient and outperforms existing methods' because of the oracle-based evaluation and limited benchmark budgets.","major_comments":[{"comment":"The reported optimality gap is computed with the true lower-level response Phi*(x_u), while all internal decisions (Algorithm 2, lines 8-9 and 20, and Equation 5) use the estimated response Phi_n(x_u). The appendix (Figures 5-6) explicitly shows that the action gap over the domain does not converge and that the action gap at the optimal upper-level solution 'seems not fully confirmed' to decrease. This means the headline curves can look good even when the algorithm's internal model at the returned point is inaccurate. Please report the optimality gap evaluated with the internal estimate F(x_u^n, Phi_n(x_u^n)) and track the action gap at the returned solution; this would directly test whether BILBAO actually knows the returned solution is good.","section":"Section 4, optimality gap definition"},{"comment":"The benchmark algorithm is allocated only M=4 or M=2 lower-level BO iterations per upper-level point in the 2D tests and M=10 or M=5 in the 4D tests, and the paper explicitly does not implement the re-optimization step recommended by Wang et al. (2021). These budgets are very small for a from-scratch lower-level solver, so the comparison may be systematically handicapped. Please also run the benchmark with a larger lower-level budget (e.g., comparable to BILBAO's lower-level evaluations) or with the final re-optimization, and report those results.","section":"Sections 4.1-4.2, benchmark comparison"},{"comment":"The comparison reports means and standard errors over only 10 runs and no significance tests. The claim that BILBAO 'outperforms existing methods' and 'converges faster' is load-bearing; please provide paired statistical tests (e.g., Wilcoxon signed-rank tests or bootstrap confidence intervals) at representative evaluation budgets, especially where the standard-error bands overlap as in the early iterations of Figures 4a and 4b.","section":"Section 4, statistical analysis"},{"comment":"The acquisition function in Equation (5) weights lower-level queries by the set X_TS of Thompson-sampled upper-level points, which is itself derived from the upper-level GP that uses Phi_n. This closed loop means that if Phi_n is inaccurate near the true optimum, X_TS will tend to miss that region and lower-level evaluations will not be placed there, allowing the error to persist. The paper does not analyze this risk, and the non-converging action gap in the appendix is consistent with it. Please add a diagnostic experiment that breaks this loop, for example by seeding the lower-level GP with a few evaluations near the true response at x_u^*, and show whether the optimality gap or action gap improves.","section":"Section 3, Equation (5), and Appendix"}],"minor_comments":[{"comment":"The action-gap formula in Equation (6) and the similar expression in the appendix have unbalanced absolute-value brackets (e.g., |F(x_u^*, Phi^*(x_u^*) - F(x_u^*, Phi_n(x_u^*)|) and an inconsistent use of f versus F. Please correct these expressions.","section":"Equation (6) and Appendix"},{"comment":"The GP kernel family, hyperparameter priors, and fitting details (e.g., Matern 5/2 with automatic relevance determination, length-scale bounds) are not specified; please state them for reproducibility.","section":"Section 4"},{"comment":"The test functions are said to be normalized to unit bounds, but the direction of optimization (maximization vs minimization) for Six-Hump Camel, Branin, and Dixon-Price is not stated. Since the optimality gap is an absolute value, please clarify whether the functions are negated to formulate a maximization problem.","section":"Section 4.1"},{"comment":"Line 13 of Algorithm 2 uses the notation F^{n+1}_Phi before it is defined; earlier in the text and in line 7 the restricted sample path is denoted F^n_Phi. Please align the notation or define the superscripted version.","section":"Algorithm 2"},{"comment":"Figure 2 is discussed only globally; the individual subplots (a)-(f) and the markers (red cross, yellow star, red star) are not systematically described. Please annotate the subplots or refer to them individually in the text.","section":"Figure 2"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"BILBAO is a sensible, incremental method for expensive black-box bilevel optimization, and the paper is more honest than most in reporting where it struggles. The new bit is the combination: a joint-space GP for the lower level with REVI reweighted by Thompson-sampled upper-level interest points. That is a reasonable way to transfer information across upper-level tasks, and on the small synthetic testbed it does converge faster than the nested-from-scratch benchmark. The appendix deserves credit for showing the action-gap metric flat-lining and admitting that the map at the true optimum 'seems not fully confirmed' to improve.\n\nThe soft spots are real. The closed-loop concern lands: upper-level selection and the final recommendation both use the estimated response map Phi_n, and lower-level acquisition focuses on interest points drawn from that same map. If Phi_n is wrong near the true optimum, there is no independent mechanism to pull evaluations there. The reported optimality gap is computed with the oracle response Phi*, which is the right leader's objective, but it can look good even when the algorithm's internal model at the proposed point is poor. The paper's own appendix suggests this is not just hypothetical. That disconnect needs a direct test: report the objective value of the recommended point using the algorithm's own Phi_n, or at least track the error in Phi_n at the argmax of the upper-level posterior.\n\nAlso missing is a comparison against function-network BO or conditional BO (Dogan and Prestwich), which already use a joint-space GP. The benchmark is handicapped with a very small lower-level budget, and there are no significance tests. No code is released. These are all fixable.\n\nThis is not a takedown. The central idea is plausible, the writing is clear, and the limitations are disclosed rather than hidden. It just doesn't yet support 'highly sample-efficient' at face value. A serious referee should send it out, ask for code and the missing baselines, and push the authors to address the response-map error at the optimum. I'd put it in a reading group as a good example of how a self-referential surrogate loop can quietly undermine headline results.","headline":"Solid incremental BO-for-bilevel work with an honest appendix, but the headline efficiency claim is weakened by a closed-loop response-map assumption and missing baselines.","tokens_in":15677,"tokens_out":3077,"would_cite":false,"duration_ms":28003,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Bilevel black-box optimization can be much more sample-efficient by modeling both levels with one shared Gaussian process and learning a single lower-level response map instead of re-solving the lower level for every upper-level candidate.","keywords":["bilevel optimization","Bayesian optimization","Gaussian processes","multi-task learning","knowledge gradient","response map","black-box optimization","sample efficiency"],"falsifier":"Run BILBAO on a synthetic bilevel problem with a known, strongly multimodal lower-level response map, such as two crossing modes whose heights swap as $x_u$ varies, and compare the final returned upper-level solution against the true bilevel optimum; if the surrogate-smoothed map misses the true mode at the returned $x_u$, the algorithm fails despite a large budget.","tokens_in":14652,"feed_emoji":"🎯","tokens_out":8665,"duration_ms":65924,"temperature":0.7,"pith_summary":"The paper proposes BILBAO, a Bayesian optimization framework for bilevel problems in which both the upper- and lower-level objective functions are black boxes and expensive to evaluate. Instead of re-solving the lower-level problem from scratch for every candidate upper-level action, BILBAO models both functions as Gaussian processes over the joint space of upper- and lower-level decisions and learns the lower-level response map once, transferring information across sub-problems. A new acquisition function, REVI, targets lower-level queries at regions that matter for the upper level. If the framework works as reported, it would cut the total number of expensive evaluations needed to find high-quality bilevel solutions.","feed_headline":"Shared GP makes bilevel black-box search sample-efficient","feed_subtitle":"BILBAO learns one response map instead of re-solving the lower level for each candidate, cutting costly evaluations.","key_machinery":"The machinery is the estimated lower-level response map $\\Phi^n(x_u) = \\arg\\max_{x_l} \\mu_l^n(x_u, x_l)$, computed by optimizing the lower-level GP posterior mean with L-BFGS-B restarts, together with the REVI acquisition function (a task-weighted knowledge gradient) that selects lower-level queries by averaging KG over a Thompson-sampled set of promising upper-level actions. The response map couples the two GPs: the upper level is optimized over the restricted surface $F^n_\\Phi(x_u) = F^n(x_u, \\Phi^n(x_u))$, and REVI steers lower-level evaluations to shrink the map where the upper level cares most.","core_discovery":"The core claim is that a bilevel problem with two expensive black-box functions can be solved far more sample-efficiently by treating it as a coupled pair of Gaussian-process learning tasks rather than as a sequence of independent lower-level optimizations. BILBAO maintains $\\mathrm{GP}_U$ over $F(x_u, x_l)$ and $\\mathrm{GP}_L$ over $f(x_u, x_l)$, derives an estimated response map $\\Phi^n(x_u) = \\arg\\max_{x_l} \\mu_l^n(x_u, x_l)$, and then optimizes the upper-level objective restricted to that map. Lower-level sampling is driven by REVI, which weights knowledge-gradient gains at upper-level points that Thompson sampling identifies as promising. The experiments report faster decline of optimality and action gaps than the benchmark that solves the lower level fresh for each upper-level point.","pith_inferences":["The paper does not analyze what happens when the lower level is multimodal or discontinuous; a natural extension is a response-map estimator that keeps multiple modes rather than a single posterior-mean maximizer.","Casting bilevel optimization as multi-task BO suggests a transfer-learning extension: one lower-level function shared across several upper-level objectives could be solved jointly with a single $\\mathrm{GP}_L$, amortizing the cost of learning the response map.","A testable extension is to compare BILBAO against semi-black-box baselines where the lower level is known and solvable; if BILBAO remains competitive, joint GP modeling could substitute for exact lower-level solvers in some applications."],"forward_implications":["If BILBAO's empirical claims hold, expensive bilevel applications such as gas-market modeling and adversarial network design become tractable with far fewer function evaluations than per-candidate lower-level solving.","The joint-space GP design means lower-level data collected for one upper-level action improves the response estimate for nearby actions, so the total evaluation budget no longer grows multiplicatively with the number of lower-level iterations per upper-level point.","REVI gives a principled rule for spending lower-level budget where the upper level is most likely to act, which should be useful in any bilevel problem where the response map is expensive and only partially learnable.","The reported sensitivity of the benchmark to its lower-level budget, contrasted with BILBAO's stable performance, implies that users can rely on a single configuration rather than hand-tuning the inner-loop budget."],"supporting_citations":[{"why":"Supplies the benchmark algorithm that BILBAO is compared against; it solves the lower level from scratch with BO for each upper-level point.","marker":"Wang et al. (2021)"},{"why":"Provides the REVI acquisition function and the continuous multi-task BO formulation that BILBAO adapts.","marker":"Pearce & Branke (2018)"},{"why":"Defines the knowledge-gradient quantity whose per-task values REVI averages.","marker":"Frazier et al. (2009)"},{"why":"Introduces multi-task GPs, the modeling idea behind sharing one lower-level GP across upper-level actions.","marker":"Swersky et al. (2013)"},{"why":"Supplies the SMD test suite used for the 4D experiments.","marker":"Sinha et al. (2014)"}],"fun_headline_variants":["BILBAO learns one map, slashes bilevel black-box cost","Bayesian bilevel breaks bottleneck with shared Gaussian process","One GP for both levels: cheaper bilevel black-box search","BILBAO: Bayesian bilevel optimization that shares its GP","Sample-efficient bilevel black-box via shared GP learning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that maximizing the lower-level Gaussian process's posterior mean gives a lower-level response map close enough to the true optimal response that the upper-level search is not misled.","fun_headline_variants_meta":{"raw":{"variants":["BILBAO learns one map, slashes bilevel black-box cost","Bayesian bilevel breaks bottleneck with shared Gaussian process","One GP for both levels: cheaper bilevel black-box search","BILBAO: Bayesian bilevel optimization that shares its GP","Sample-efficient bilevel black-box via shared GP learning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00032,"raw_usage":{"total_tokens":1742,"prompt_tokens":823,"completion_tokens":919,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":439,"completion_tokens_details":{"reasoning_tokens":833}},"tokens_in":439,"tokens_out":919,"duration_ms":6192,"temperature":1.0,"reasoning_tokens":833,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T04:42:22.415319+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run BILBAO on a synthetic bilevel problem with a known, strongly multimodal lower-level response map, such as two crossing modes whose heights swap as $x_u$ varies, and compare the final returned upper-level solution against the true bilevel optimum; if the surrogate-smoothed map misses the true mode at the returned $x_u$, the algorithm fails despite a large budget.","supporting_citations":[],"review_version":1}