{"id":"a6cf451a-722d-48c3-b245-e876a11bd148","arxiv_id":"2411.10841","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"ALPHA adaptively selects among non-hierarchical low-fidelity models during reinforcement learning, using policy alignment with the high-fidelity policy to train a high-quality design policy.","lead":"This paper introduces ALPHA, a reinforcement learning method that lets a design agent switch between several cheap, imperfect simulators and one expensive, accurate simulator without needing to rank the cheap ones in a fixed order. It matters for engineering design because it promises faster and more consistent searches through complex design spaces by using each simulator only where it is useful.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim depends on an unvalidated proxy: cosine similarity between policy action means does not guarantee low-fidelity reward accuracy in the regions whose data are augmented into the HF buffer; without a direct test, the efficiency and quality results are only conditionally supported.","rationale":"The reader's weakest_assumption identifies exactly the condition that must hold for the central claim: action-distribution alignment must be a reliable signal that low-fidelity rewards are safe to use in training the high-fidelity policy. My reading agrees that this is the most load-bearing assumption, and that the paper does not test it. The mechanism in Algorithm 1 is not a conservative use of LF data: aligned LF subsequences are copied into BHF with no bias correction, so the HF policy receives LF rewards as if they were HF rewards. If those rewards are wrong in an aligned region, the final policy can be optimized toward LF minima, and the claimed 'unified high-fidelity policy' is not actually a high-fidelity policy. The paper's own results show that the LF surrogates in the octocopter case have validation accuracies of only 0.51 and 0.56 and much lower accuracy on the full space, which makes the unvalidated alignment proxy particularly risky. The concrete test I propose would directly measure whether the cosine-similarity gate selects states with low reward error on the paper's own analytical benchmark. If the correlation is weak, the central claim should be downgraded to a behavioral demonstration that the heuristic works on two examples, not a generally supported mechanism. Since the reader's conditional verdict already reflects this uncertainty, no change to the verdict is needed.","tokens_in":19962,"tokens_out":4474,"duration_ms":50807,"concrete_test":"Use the Ackley benchmark to measure whether the alignment gate actually selects reward-accurate regions. Sample 10,000 states uniformly in the scaled domain; at each state, compute the cosine similarity of each LF policy against the final trained HF policy, and compute the local reward error between that LF model and the HF model for the action recommended by the HF policy, averaged over at least 10 action samples. Then produce a threshold-precision curve: for the alignment threshold schedule used in Algorithm 2, determine what fraction of states passing the gate have reward error below 10% of the HF reward range, separately for LF1 and LF2. If the precision is near chance, the proxy is not selecting reward-accurate regions and Algorithm 1's augmentation is unjustified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing step is the data augmentation in Algorithm 1: when an LF policy is chosen exploitatively and its mean-action cosine similarity to the HF policy exceeds a threshold, contiguous aligned subsequences of LF experience are appended to the HF buffer and used to train the HF policy. Section 3.1 states that this keeps 'the learning precision of the HF policy' because LF rewards are used only in aligned regions. The central efficiency and quality claims collapse if this proxy is wrong: if the LF reward is biased in an aligned region, the HF policy is trained on biased targets while still being called a high-fidelity policy. Cosine similarity between action-distribution means measures whether two policies recommend similar actions at a state; it does not measure whether the two models' reward functions or value estimates agree there. The paper provides no test of that correspondence. The Ackley analysis shows only that LF1 is used near (-0.3,-0.3), near a HF local optimum, and LF2 near (0.3,0.3), near the HF global optimum; that is post hoc behavioral correlation, not evidence that action alignment implies reward accuracy. The future-work paragraph explicitly lists 'different alignment metrics' as open, which is an admitted gap. This is an internal-validity objection, not a disagreement with any external consensus: the training signal injected into BHF is not demonstrated to be the reward signal the algorithm assumes.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ALPHA, a multi-fidelity reinforcement learning framework for engineering design that adaptively combines one high-fidelity (HF) model with several non-hierarchical low-fidelity (LF) models. Each model has its own policy; at each design step, the framework computes the cosine similarity between the mean actions of each LF policy and the HF policy, uses an epsilon-greedy rule with a cosine-scheduled alignment threshold to select which model to step, and then augments the HF replay buffer with aligned contiguous subsequences of LF experience. The method is evaluated on a modified Ackley optimization problem and an octocopter design problem, with two LF models in each case. The central claims are that ALPHA eliminates the need for a predefined model schedule, learns a unified HF policy that yields high-quality solutions at reduced computational expense, and shows superior convergence behavior compared to hierarchical multi-fidelity RL baselines.","tokens_in":20278,"tokens_out":4359,"duration_ms":48527,"significance":"If the claims are substantiated, ALPHA would be a useful contribution to multi-fidelity RL in engineering design: it relaxes the hierarchy assumption that underlies most existing multi-fidelity RL methods, and it introduces a concrete mechanism for leveraging heterogeneous LF models via policy alignment. The paper is clearly written, the algorithm is explicit, and the two case studies provide qualitative evidence of adaptive model usage across time and design space, including Moran's I analysis. However, the central efficiency and quality claims currently rest on an unvalidated proxy (cosine similarity of policy action means implying reward accuracy in aligned regions) and on results from what appear to be single training runs without statistical replication. These issues, along with the low global accuracy of the octocopter surrogates, make the empirical support conditional rather than conclusive.","major_comments":[{"comment":"The load-bearing assumption is that cosine similarity between the means of the action distributions of an LF policy and the HF policy is a reliable signal that LF rewards can be used to train the HF policy. The text states that 'rewards from LF models are used in aligned regions of a trajectory' and that this 'maintains the learning precision of the HF policy,' but cosine similarity measures policy agreement, not reward or value accuracy. If LF rewards are biased in an aligned region, the HF policy is trained on biased targets while still being called a high-fidelity policy. The paper provides no direct test of this correspondence. Please add a validation: for example, measure LF reward error (or value error) in aligned versus non-aligned regions, and/or compare ALPHA against an ablation that does not augment LF experience into the HF buffer. Without such evidence, the efficiency and quality results are only conditionally supported.","section":"§3.1, Algorithm 1 and data augmentation paragraph"},{"comment":"The results appear to come from a single training run per condition; the violin plots show variability over seed designs, not over stochastic training runs with different random seeds. The conclusion claims that ALPHA 'consistently yields high quality solutions,' but with one run per condition there is no statistical basis for consistency. Please report multiple independent training runs (e.g., 5–10 random seeds) and apply appropriate statistical tests on final HF-evaluated solution quality, such as Mann–Whitney U tests or bootstrap confidence intervals. Also report the random seeds and the full set of PPO and environment hyperparameters to enable reproducibility.","section":"§5.1 and §5.2, Figures 9 and 15"},{"comment":"The low-fidelity surrogates LF1 and LF2 have global prediction accuracies of 0.23 and 0.38 on the entire dataset (presumably R-squared, but the metric is not defined). These values indicate that the surrogates are very inaccurate globally, which makes it especially important to demonstrate that the LF rewards used in aligned regions are locally accurate. The paper only reports validation accuracies on the respective training subsets (0.51 and 0.56) and does not report local accuracy in the regions where each LF model is actually selected or aligned. Please report local reward error or R-squared in the aligned regions, or otherwise show that the injected LF rewards are not systematically biased in those regions.","section":"§4.2, octocopter design problem"},{"comment":"The cosine schedule for the alignment threshold and the epsilon-greedy parameter are free hyperparameters that directly control the four regimes R1–R4 observed in the model-usage plots. Without a sensitivity analysis or ablation (e.g., fixed threshold, different schedule slopes, different epsilon values), it is unclear how much of the claimed adaptivity is due to the method itself versus the hand-chosen schedule. Please add an ablation or sensitivity study over a small grid of these parameters, at least for the analytical problem, to support the generality of the adaptive behavior.","section":"§3.1, Algorithm 2 and Figures 11, 18"}],"minor_comments":[{"comment":"The subsection numbering is inconsistent: the octocopter problem is labeled '4.1' but should be '4.2' after the analytical problem.","section":"§4"},{"comment":"The text refers inconsistently to Figures 15 and 16: for example, the caption is 'FIGURE 16' but the text calls it 'Fig. 16(a)' in some places and 'Fig. 15(c)' in others. Please unify the figure numbering and cross-references.","section":"§5.2, Figures 15–17"},{"comment":"The definition of 'aligned' is terse: 'aligned = [model == (argmax p_m)]'. This means that if the epsilon-greedy exploration selects a non-argmax model, aligned is False, but the data augmentation paragraph only says 'chosen exploitatively.' Please state this explicitly in the main text to avoid ambiguity.","section":"Algorithm 2"},{"comment":"The HF simulator cost in the octocopter problem has a heavy-tailed distribution (25th percentile 0 s, median 1.78 s, 75th percentile 22.04 s). This variability is likely to affect the efficiency comparison; consider reporting total compute with confidence intervals or using a cost metric that is robust to outliers.","section":"§5.2, computational cost"},{"comment":"Reference [21] lacks year and venue information.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper extends the authors' own hierarchical framework [3] and would benefit from a sharper positioning against control-variate multi-fidelity RL [4], which also uses LF experience without a fixed hierarchy. The submitted version does not mention code or data release, which limits reproducibility. If the authors add the requested validation of the alignment proxy and repeated-run statistics, the paper could become acceptable for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"ALPHA is a real extension of the multi-fidelity RL toolkit: per-model agents with a cosine-similarity alignment gate for mixing low- and high-fidelity experience, rather than a fixed model schedule. The two case studies show the adaptive behavior the paper claims, and the writing is refreshingly concrete — Algorithms 1 and 2 are specific enough to reimplement.\n\nCredit where it is due: the Moran's I analysis is a nice way to show spatial structure in model usage, and the authors disclose the poor global accuracy of the octocopter surrogates (R^2 around 0.23 and 0.38). That ALPHA still lands on competitive solutions despite those surrogates is interesting, but it also raises the bar for explaining why the alignment proxy works.\n\nThe stress-test concern is on target. In Algorithm 1, aligned low-fidelity experience is appended to the high-fidelity buffer, and Section 3.1 says this \"maintains the learning precision of the HF policy.\" But alignment is cosine similarity between policy action means, not agreement of reward landscapes or value estimates. A low-fidelity policy can recommend the same action as the high-fidelity policy while its reward for that action is biased. The paper offers no direct test of that correspondence. The Ackley analysis is post hoc behavioral correlation, not validation. And the future-work paragraph explicitly lists \"different alignment metrics\" as open, which reads as an admission that the current metric is not yet grounded.\n\nAlso missing: repeated runs with different seeds, statistical tests on final solution quality, ablations of the epsilon-greedy and threshold schedule, and code/data. The claim that ALPHA \"consistently\" yields higher quality at lower cost rests on single-run comparisons. None of these omissions are fatal to the core idea, but they make the central claim conditional rather than demonstrated.\n\nIf you work on multi-fidelity RL for engineering design, this is worth reading as a novel heuristic and a clear baseline for future work. I would not yet build on it as a proven mechanism. Send it to a serious referee, but expect major revision: repeated runs, a direct validation of the alignment-to-reward assumption (e.g., measuring true reward error in aligned vs unaligned regions), and at least one ablation removing the data augmentation to isolate its contribution.","headline":"ALPHA is a genuinely new adaptive multi-fidelity RL scheme, clearly described and worth a serious referee, but its core alignment-to-reward assumption is unvalidated and the empirical evidence lacks repeated runs and statistics.","tokens_in":683,"tokens_out":1630,"would_cite":true,"duration_ms":40793,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"No model ladder: adaptive RL borrows cheap simulators in place","keywords":["multi-fidelity reinforcement learning","heterogeneous models","policy alignment","adaptive methods","non-hierarchical frameworks","engineering design","octocopter design","proximal policy optimization"],"falsifier":"In a two-dimensional test problem, compute the true per-region difference between each low-fidelity model's reward and the high-fidelity model's reward, then compare it against the cosine alignment used by ALPHA; if regions with high alignment show no smaller reward error than unaligned regions, the central efficiency claim collapses. A simpler version: build a low-fidelity model whose action guidance agrees with the high-fidelity policy but whose reward values systematically overestimate performance, and observe whether ALPHA's final policy degrades.","tokens_in":19784,"feed_emoji":"🛸","tokens_out":4773,"duration_ms":44103,"temperature":0.7,"pith_summary":"This paper proposes ALPHA, a multi-fidelity reinforcement learning framework that learns a high-fidelity design policy by adaptively borrowing from a set of non-hierarchical, heterogeneous low-fidelity models. Instead of following a fixed low-to-high schedule, ALPHA watches how closely each low-fidelity policy's action choices match the high-fidelity policy's choices, and uses the cheap models mainly in the design regions where they agree. The claim is that the resulting unified high-fidelity policy matches or beats hierarchical multi-fidelity RL baselines at lower computational expense, and that adaptive agents take more direct paths to high-performance designs. The paper demonstrates this in an Ackley-style analytical optimization problem and an octocopter design task with two low-fidelity surrogates and one high-fidelity flight simulator.","feed_headline":"No model ladder: adaptive RL borrows cheap simulators in place","feed_subtitle":"Cheap simulators are used only where they agree with the expensive model, cutting design cost without losing quality.","key_machinery":"The load-bearing mechanism is Algorithm 1, ALPHA, built around a policy-alignment gate. For each low-fidelity agent, cosine similarity between the mean of its action distribution and the mean of the high-fidelity policy's action distribution at the current state is computed and compared with a threshold theta that follows a cosine schedule from 90 to 0 degrees over training. Algorithm 2 converts the two similarity comparisons into an epsilon-greedy categorical choice over LF1, LF2, and HF; after each episode, contiguous run-lengths of low-fidelity experience in which the chosen model was aligned and exploitatively selected are appended to the HF buffer, so the HF policy trains on LF rewards only in the regions the alignment gate certifies.","core_discovery":"The central discovery is that the alignment between low-fidelity and high-fidelity policies, measured by the cosine similarity of the means of their action distributions at a given state, is a sufficient guide for deciding when a low-fidelity model's rewards can safely contribute to training the high-fidelity policy. ALPHA maintains one policy per analysis model, picks a model at each design step with an epsilon-greedy rule whose probabilities depend on this alignment against a cosine-annealed threshold, and augments the high-fidelity replay buffer with contiguous subsequences of low-fidelity experience collected in aligned regions. On the two case studies, the trained high-fidelity policy attains consistently high solution quality and lower evaluation cost than training on the high-fidelity model alone, while avoiding the ordering sensitivity that afflicts hierarchical schedules. The paper concludes that adaptive policy alignment eliminates the need for a predefined model schedule and yields more direct convergence paths.","pith_inferences":["A direct test the paper does not run: compare the alignment score against the actual per-region reward error of each low-fidelity model; if trajectories with high cosine similarity do not have lower reward error, the data-augmentation step is training on biased rewards.","Because model choice probabilities are not cost-weighted, replacing them with cost-aware weights when several low-fidelity models align could further improve the quality-efficiency tradeoff.","The alignment threshold schedule is hand-tuned via a cosine rule; an adaptive threshold driven by measured regional agreement could make the method robust to problems where alignment evolves non-monotonically.","The framework assumes every model shares a common design representation; extending it to models with different state and action spaces would require a representation alignment step beyond policy alignment."],"forward_implications":["A designer can combine arbitrary low-fidelity models with a high-fidelity one and obtain a single high-fidelity policy without deciding a model schedule in advance.","The learned policy's convergence does not depend on the order in which the low-fidelity models are presented, unlike hierarchical multi-fidelity RL baselines.","Low-fidelity models are used for broad exploration while the high-fidelity model is reserved for critical regions such as the global optimum, concentrating computation where accuracy matters.","If the alignment signal is reliable, the same framework should extend to any number of non-hierarchical models with heterogeneous error distributions.","Solution quality is maintained while evaluation time is reduced relative to high-fidelity-only training, and variance across seeds is lower than in the baselines."],"supporting_citations":[{"why":"Supplies the base hierarchical multi-fidelity RL framework that ALPHA extends and the principal comparative baseline.","marker":"[3]"},{"why":"Proximal policy optimization is the algorithm used to train all agents in ALPHA.","marker":"[47]"},{"why":"The cosine annealing schedule for the alignment threshold is taken from this work.","marker":"[46]"},{"why":"Informs the multi-fidelity simulator-switching background and provides a contrast for non-adaptive fidelity transfer.","marker":"[42]"},{"why":"The control-variate multi-fidelity RL baseline that relies on reward-error correlations, against which ALPHA's alignment approach is positioned.","marker":"[4]"},{"why":"A surrogate-guided multi-fidelity RL approach that still uses a fixed low-to-high schedule, the assumption ALPHA removes.","marker":"[5]"},{"why":"Motivates non-hierarchical multi-model fusion with heterogeneous spatial errors, which ALPHA operationalizes in RL.","marker":"[40]"},{"why":"Moran's I is used to measure spatial clustering of model usage in the design-space analysis.","marker":"[49]"}],"fun_headline_variants":["Adaptive RL aligns cheap models to cut design cost","Policy alignment picks simulators only when they agree","No fixed ladder: RL borrows cheap simulators adaptively","ALPHA: Letting cheap sims help only when they align","Adaptive multi-fidelity RL skips the hierarchy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that when a low-fidelity policy's action distribution points in the same direction as the high-fidelity policy's, the low-fidelity model's rewards are trustworthy enough to train the high-fidelity policy; if action agreement does not imply reward accuracy, the high-fidelity policy can be trained on biased low-fidelity rewards.","fun_headline_variants_meta":{"raw":{"variants":["Adaptive RL aligns cheap models to cut design cost","Policy alignment picks simulators only when they agree","No fixed ladder: RL borrows cheap simulators adaptively","ALPHA: Letting cheap sims help only when they align","Adaptive multi-fidelity RL skips the hierarchy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000164,"raw_usage":{"total_tokens":1255,"prompt_tokens":962,"completion_tokens":293,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":578,"completion_tokens_details":{"reasoning_tokens":212}},"tokens_in":578,"tokens_out":293,"duration_ms":3813,"temperature":1.0,"reasoning_tokens":212,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T19:14:22.576246+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"In a two-dimensional test problem, compute the true per-region difference between each low-fidelity model's reward and the high-fidelity model's reward, then compare it against the cosine alignment used by ALPHA; if regions with high alignment show no smaller reward error than unaligned regions, the central efficiency claim collapses. A simpler version: build a low-fidelity model whose action guidance agrees with the high-fidelity policy but whose reward values systematically overestimate performance, and observe whether ALPHA's final policy degrades.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Moran's I is used to measure spatial clustering of model usage in the design-space analysis."},{"cited_title":"Reinforcement Learning for Efficient Design Space Exploration With Variable Fidelity Analysis Models,","cited_arxiv_id":null,"evidence_quote":"Supplies the base hierarchical multi-fidelity RL framework that ALPHA extends and the principal comparative baseline."},{"cited_title":"Multifidelity Reinforcement Learning with Control Variates,","cited_arxiv_id":null,"evidence_quote":"The control-variate multi-fidelity RL baseline that relies on reward-error correlations, against which ALPHA's alignment approach is positioned."},{"cited_title":"Nonhierarchical Multi‐model Fusion Using Spatial Random Processes,","cited_arxiv_id":null,"evidence_quote":"Motivates non-hierarchical multi-model fusion with heterogeneous spatial errors, which ALPHA operationalizes in RL."}],"review_version":1}