{"id":"785bc824-685d-43bd-a80f-7fcb3606303b","arxiv_id":"2412.13492","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"ROSKA co-evolves reward functions and policies, using Bayesian optimization to fuse the previous best policy with random weights, reporting 95.3% average normalized improvement over Eureka on six Isaac Gym tasks.","lead":"The paper proposes ROSKA, a framework that co-evolves LLM-designed reward functions and policies by reusing the previous best policy instead of retraining from scratch. The authors report large gains over Eureka on six simulated robot tasks while using fewer training samples.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim hinges on an unvalidated early-stop ranking: SC-BO picks fusion ratios from 200-update scores but reports final 2500-update performance, and no experiment isolates whether co-evolution or warm-starting drives the gains.","rationale":"The paper proposes a plausible and useful framework: warm-starting policies while evolving LLM-designed reward functions is a natural direction, and Table 1 does show consistent MTS improvements over Eureka on all six tasks. The reader's weakest assumption correctly identifies the SC-BO early-stop premise in Eqs. (8)-(10): that a 200-update score predicts which fusion ratio and reward-policy combination will be best after 2500 updates. This premise is load-bearing because SC-BO is the mechanism that is supposed to make co-evolution efficient and robust; without it, the reported gains could be attributed to warm-starting and broader candidate sampling rather than to the proposed selection method. I agree with the reader's assessment that this is the central concern. I would only add that the headline 95.3% HNS improvement is also not directly reproducible from Table 1 using the stated HNS formula, and the TTS comparison excludes the repeated first-round LLM queries acknowledged in the Limitations section. These reinforce the conditional verdict but do not by themselves overturn it. The requested check—measuring early-vs-final rank agreement and comparing against warm-start-only baselines—would settle whether the central mechanism works as claimed. Since the reader already recommends CONDITIONAL and my concern supports that same conclusion, no verdict change is needed.","tokens_in":17933,"tokens_out":8482,"duration_ms":76018,"concrete_test":"Run ROSKA on ShadowHand and Humanoid for the same five rounds and six generated reward functions, with seeds fixed. For each round, evaluate all 12 fusion-ratio candidates under both TBO=200 and the full Tmax=2500, then compute the Kendall rank correlation between s(α, TBO) and V(I(R, θf(α), 2500)), and compare the α chosen by SC-BO with the α that maximizes final 2500-update performance. If the rank correlation is not significantly positive, or the chosen α's final return is not within noise of the best final α, the SC-BO premise fails; the Table 1 gains would then need re-attribution against warm-start-only (α=1) and random-α baselines at matched total samples.","verdict_should_be":"UNCHANGED","load_bearing_attack":"ROSKA's headline gain rests on Eq. (8)-(10): SC-BO maximizes s(α; θf(α), TBO=200) and then reports V(I(R, θf(αSC-BO), Tmax=2500)). For the 95.3% improvement to be evidence for reward-policy co-evolution, the 200-update ranking must substantially agree with the 2500-update ranking. The paper states this divergence premise once, with no validation: no correlation, no selected α values, no seed-level curves. If early ranking diverges, SC-BO is selecting α essentially by a noisy proxy, and the measured gains could come from warm-starting at α·θbest + (1−α)·θ0 or from evaluating 12 candidate policies per reward function, not from identifying a capable-and-plastic combination. This also affects sample-efficiency claims: the 89% TTS calculation is predicated on the 200-update evaluations being informative; otherwise they are just extra sampling with a different budget. The comparison to ROSKA-U (11 uniform α's trained 3000 epochs) does not test the shortcut because it changes both search method and budget. A clean validation is missing and is directly load-bearing.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes ROSKA, a reward-policy co-evolution framework for LLM-based reward design in high-dimensional robotic control. Reward evolution prompts GPT-4o with the previous best reward code and policy statistics to generate new candidate reward functions, while policy evolution fuses the previous best policy parameters with random initialization through a fusion ratio alpha chosen by a short-cut Bayesian optimization (SC-BO) that uses only TBO=200 updates per evaluation. The best reward-policy pair is then trained for Tmax=2500 updates and carried to the next round. Experiments on six Isaac Gym tasks compare ROSKA with sparse reward, human reward, Eureka, and a uniform-search variant ROSKA-U, reporting Max Training Success (MTS), Human Normalized Score (HNS), and Total Training Samples (TTS). The paper claims an average normalized improvement of 95.3% over Eureka while using roughly 89% of Eureka's training samples.","tokens_in":18095,"tokens_out":10597,"duration_ms":85661,"significance":"If the central claims are supported, the contribution is valuable: co-evolving rewards with policies and using Bayesian optimization to tune parameter inheritance addresses a real inefficiency in current LLM-based reward design, namely the need to retrain policies from scratch for every candidate reward. The evaluation is anchored to external sparse-reward task scores rather than to the method's own reward functions, so there is no definitional circularity. I credit the authors for including a uniform-search ablation, TTS calculations, and a limitations section that candidly notes first-round instability. However, the load-bearing SC-BO early-stop assumption is not experimentally validated, the dynamic-population mechanism is not reflected in the reported algorithm or experiments, and several numerical/descriptive claims are internally inconsistent. The contribution is therefore plausible but not yet established at the level claimed.","major_comments":[{"comment":"The short-cut premise is load-bearing for the paper's central claims, but it is never validated. Eq. (8) defines s(alpha; theta_f(alpha), TBO) with TBO=200, Eq. (9) selects alpha by maximizing this 200-update score, and Eq. (10) reports final performance after Tmax=2500 updates. For the 95.3% improvement and the 89% TTS claim to support reward-policy co-evolution, the 200-update ranking must be predictive of the 2500-update ranking. The paper contains no correlation analysis, no selected alpha values per task/round, and no seed-level curves showing that early divergence tracks final divergence. The ROSKA-U comparison (11 uniformly sampled alpha values trained for 3000 epochs) changes both the search strategy and the training budget, so it cannot isolate the early-stop effect. Please add a direct validation: report 200-update versus 2500-update rankings for all alpha values and seeds, or run an ablation in which SC-BO selects alpha but total training budget is held equal to a no-shortcut search. Without this, the gains could come from warm-starting at the previous best policy or from evaluating 12 candidate configurations per reward, rather than from identifying an actually capable-and-plastic reward-policy combination.","section":"Policy Evolution via Bayesian Optimization, Eqs. (8)-(10)"},{"comment":"The headline 'average normalized improvement of 95.3%' is not reproducible from the reported tables. Recomputing HNS from Table 1 via Eq. (11) gives approximate Eureka/ROSKA HNS pairs of (0.97, 1.46) on Ant, (1.17, 2.20) on Humanoid, (1.60, 4.09) on ShadowHand, (1.26, 2.01) on AllegroHand, (3.11, 8.55) on FrankaCabinet, and (0.56, 1.47) on ShadowHand-U; the average relative HNS improvement is about 115%, not 95.3%. If the intended statistic is instead the average relative MTS improvement, the value is about 99%. Please state the exact formula and per-task values used for the 95.3% figure, or correct the number.","section":"Abstract and Experimental Results (Table 1, Fig. 4)"},{"comment":"The dynamic population mechanism is described as generating K reward functions repeatedly until a reward function outperforms the previous round's benchmark, but it is not implemented or reported in the experiments. Algorithm 1 loops over N=5 fixed rounds with K=6 reward functions, and the experimental settings state that each task uses N=5 rounds and K=6 functions. No DP-round sizes, no variable query counts, and no comparisons between fixed and dynamic population sizes are reported. The claim that the dynamic population acts as a filter and guarantees continual refinement is therefore untested. Either perform a dynamic-population ablation or revise the text to describe the fixed-size rounds honestly.","section":"Reward Evolution with Dynamic Population vs. Algorithm 1"},{"comment":"The text states that 'ROSKA achieved better results in Ant, AllegroHand, and ShadowHand-U tasks' when compared with ROSKA-U, but Table 1 shows the opposite direction on three of the six tasks: ROSKA-U scores 12.52 vs 12.07 on Ant, 26.80 vs 23.22 on AllegroHand, and 23.72 vs 21.82 on ShadowHand-U. Since the ROSKA-U comparison is used to argue that SC-BO recovers uniform-search performance at lower sample cost, the table reading must be corrected and the conclusion restated to reflect that ROSKA wins only on Humanoid, ShadowHand, and FrankaCabinet by the reported means.","section":"Comparison to Baseline Methods (Table 1)"},{"comment":"The Limitations section concedes that for complex tasks the first-round LLM may produce no workable reward and that the authors 'have to query the LLM multiple times' until at least one reward works. This extra querying is not counted in the round totals, the TTS ratios, or the cost comparison with Eureka. The claim of 'using less data' is therefore at best a training-sample claim, not a total-query or total-cost claim. Please report the actual number of LLM calls per task and per seed and state this limitation explicitly in the data-efficiency discussion.","section":"Limitations and TTS calculation"}],"minor_comments":[{"comment":"Since the HNS values in Fig. 4 are hard to read from the figure, please provide a numerical table of HNS values alongside the bar chart.","section":"Evaluation Metrics, Eq. (11)"},{"comment":"In the ROSKA-U TTS calculation, '6(rounds) × 3000' should read '6(reward functions) × 3000' for consistency with the first-round description.","section":"Appendix, ROSKA-U TTS calculation"},{"comment":"The note justifies using the mean sparse reward as a proxy for return; please provide a fuller justification or report actual discounted returns to ensure that MTS is not sensitive to this approximation.","section":"Table 4 note"},{"comment":"The subsection contains unpolished phrasing and a sign inconsistency in the final reward formula; please edit for clarity and consistency with the earlier reward formulas.","section":"Discussion on Evolution of total reward computation"},{"comment":"Several pairwise comparisons have overlapping standard deviations (e.g., Ant ROSKA 12.07 ± 0.60 vs ROSKA-U 12.52 ± 1.03, and ShadowHand-U ROSKA 21.82 ± 5.87 vs ROSKA-U 23.72 ± 4.96); please report the number of seeds and, where appropriate, paired-seed results to support the 'consistently achieves superior performance' statement.","section":"Table 1 error bars"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for this venue. The main uncertainty is not circularity, because the evaluation uses external sparse reward scores; it is the unvalidated early-stop ranking in SC-BO and the inconsistent reporting of several quantitative claims. I believe these are fixable with additional experiments and a careful rewrite: an early-ranking-vs-final-ranking validation, a corrected headline statistic, an implemented or honestly described dynamic population, and a corrected ROSKA-U comparison. The first-round extra-LLM-query issue disclosed in the Limitations section should also be incorporated into the cost comparison."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one thing to know: this paper is a legitimate extension of Eureka. The new idea is to co-evolve the LLM-designed reward and the policy by reusing the previous round's best policy, fusing its weights with random weights via Eq. (7), and using Bayesian optimization to pick the fusion ratio. That loop is new, and the empirical comparison across six Isaac Gym tasks shows ROSKA beating Eureka on both MTS and HNS in every task while using about 89% of Eureka's training samples. That is a real result worth building on.\n\nThe soft spots are proportional. First, the headline \"95.3% average normalized improvement\" does not reproduce from the tables. Recomputing from Table 1 and Fig. 4 gives roughly 99% average relative improvement on MTS and 115% on HNS, depending on the averaging convention. That discrepancy matters for a central claim. Second, the SC-BO early-stop premise in Eqs. (8)-(10) is load-bearing and unvalidated: alpha is chosen from 200-update scores, but final performance is reported after 2500 updates. The paper gives no correlation, no seed-level curves, and no selected alpha values. If the early ranking diverges from the final ranking, the gains could come from evaluating 12 short runs per reward function rather than from finding a capable-and-plastic combination. The ROSKA-U comparison does not settle this because it changes both the search method and the budget. Third, the dynamic population mechanism described in the Method is not used in Algorithm 1 or the experiments; all rounds use a fixed K=6. The limitations section also admits repeated first-round LLM queries, which are not counted in the efficiency comparison. Fourth, the paper does not isolate warm-starting from co-evolution: Table 2 shows that ROSKA-50% and ROSKA-100% already beat Eureka on two of three tasks, so the adaptive alpha helps, but the \"co-evolution\" framing is stronger than the evidence.\n\nThat said, the paper is honest about its limitations and the experimental trend is consistent across six diverse tasks. This is not a fatal flaw; it is an engineering contribution with a few unproven assumptions. The right reader is someone working on LLM-based reward design or sample-efficient RL. It deserves a serious referee, not a desk reject. The referee should ask for code and seeds, selected alpha values, a correlation between early and final rankings, and an ablation that controls for the number of policy evaluations. With those, the paper could be solid.","headline":"A plausible Eureka variant that reuses policies via a BO-selected fusion ratio, with consistent gains across six tasks, but the headline number and the early-stop assumption need work before the quantitative claims are reliable.","tokens_in":18693,"tokens_out":9059,"would_cite":true,"duration_ms":68305,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that LLM-designed rewards should co-evolve with the policy rather than be validated by retraining from scratch, and reports a 95.3% average improvement in normalized score while using about 89% of the baseline's training…","keywords":["reward-policy co-evolution","LLM reward design","Bayesian optimization","sample efficiency","robotic skill acquisition","policy fusion","Isaac Gym","reinforcement learning"],"falsifier":"Run a given co-evolution round's candidate fusion ratios through both 200 and 2,500 updates, then compare the rank order of their final task scores: if the ratio ranked best at 200 updates fails to be best at 2,500 in a substantial share of rounds across the six tasks, SC-BO's early-stop selection criterion is unreliable and the central efficiency claim is unsupported.","tokens_in":17662,"feed_emoji":"🤖","tokens_out":9890,"duration_ms":76761,"temperature":0.7,"pith_summary":"ROSKA tries to establish that, for language-instructed robot skill learning, reward functions and policies should be improved together rather than treating the reward as a fixed target that a freshly initialized policy must chase. The method alternates an LLM-driven reward evolution, which generates a growing population of reward functions guided by the previous best, with a policy evolution that fuses the previous best policy parameters with random parameters in a proportion selected by a fast Bayesian optimization. The paper reports that this co-evolution outperforms the from-scratch LLM reward-design baseline on all six tested Isaac Gym tasks with only about 89% of the baseline's training samples, and that it improves the human-normalized score by 95.3% on average. The underlying argument is that a policy-dependent reward, refined as the robot improves, is more practical than a universal reward expected to work throughout training.","feed_headline":"Reward-policy co-evolution beats from-scratch training on less data","feed_subtitle":"On six robot tasks, co-evolving reward and policy lifts normalized score 95.3% over the from-scratch baseline.","key_machinery":"The central object is a co-evolution cycle coupling two search processes. Reward evolution uses the LLM, prompted with the previous best reward code, task and environment descriptions, and policy performance statistics, to generate a dynamic reward population that keeps expanding until a candidate beats the previous best. Policy evolution parameterizes inheritance through the fusion ratio $\\alpha \\in [0,1]$ in $\\theta^m_f(\\alpha)=\\alpha\\theta^{m-1}_{\\mathrm{best}}+(1-\\alpha)\\theta_0$, blending the previous best policy with a random initialization to balance retained skill and plasticity; Short-Cut Bayesian Optimization, a Gaussian-process surrogate with expected improvement and an early stop at $T_{BO}=200$ updates, selects $\\alpha$ so that reward candidates can be compared without full retraining.","core_discovery":"The central claim is that reward-policy co-evolution is both more sample-efficient and higher-performing than the conventional loop in which each candidate reward function is verified by training a policy from scratch. In each co-evolution cycle, the previous round's best reward code and performance statistics are fed to an LLM, which returns a dynamically sized set of new reward functions; each candidate is then evaluated not from random initialization but from a fused policy $\\theta^m_f(\\alpha)=\\alpha\\theta^{m-1}_{\\mathrm{best}}+(1-\\alpha)\\theta_0$, where $\\alpha$ is chosen by Short-Cut Bayesian Optimization after only $T_{BO}=200$ updates. The best-performing reward-policy pair receives an extended training run and is carried into the next cycle. On six high-dimensional Isaac Gym tasks, the paper reports higher Max Training Success and Human Normalized Scores than the from-scratch baseline in every task, with an average normalized improvement of 95.3% and a total training-sample budget about 89% of the baseline.","pith_inferences":["If the early-stop ranking at 200 updates is stable, the same Short-Cut Bayesian Optimization trick could be applied to other RL design choices, such as network width, learning-rate schedules, or reward component weights, turning each into a cheap inner loop of the co-evolution outer loop.","The reward-evolution side is effectively a Lamarckian search: improvements acquired by the policy are written back into the LLM prompt that generates the next reward. A testable extension would feed richer policy statistics, such as value estimates or feature activations, back to the LLM instead of reward-component traces.","A head-to-head comparison against other warm-start or transfer-learning reward-design methods would clarify how much of the reported gain comes from co-evolution itself versus from reusing pretrained policy parameters.","On tasks where random parameter fusion disrupts a delicate policy, the method could stall; a safeguard would be to adapt the fusion distribution based on observed candidate divergence at the early-stop horizon rather than assuming one horizon works for all tasks."],"forward_implications":["LLM-generated rewards no longer need to be universally effective; each reward only has to improve the current policy, which should widen the set of usable reward candidates.","With the same five LLM reward-design rounds, the method reports the best Max Training Success on all six tasks and surpasses expert-designed rewards on all six, with the largest margins on the ShadowHand and FrankaCabinet tasks.","The total training-sample budget can be cut to about 89% of the from-scratch baseline without sacrificing the gain, and a reduced variant using 56% of the budget still matches or beats the baseline on the reported tasks.","The rule that only reward-policy pairs outperforming the previous best survive ensures monotonic improvement across rounds, a property the from-scratch baseline does not have.","Ablations with fixed fusion ratios (0%, 50%, 100%) all underperform the Bayesian-selected ratio, supporting the claim that choosing $\\alpha$ matters for the gain."],"supporting_citations":[{"why":"Supplies the from-scratch LLM reward-design baseline that ROSKA compares against and whose reward-evolution prompt structure it extends.","marker":"Ma et al. 2024"},{"why":"Provides the Isaac Gym simulator and the six task environments used for all experiments.","marker":"Makoviychuk et al. 2021"},{"why":"Provides the PPO algorithm used for all policy training and evaluation.","marker":"Schulman et al. 2017"},{"why":"Supplies the Bayesian optimization and expected-improvement technique that SC-BO adapts for fusion-ratio search.","marker":"Astudillo and Frazier 2022"},{"why":"Motivates the random parameter fusion by documenting plasticity loss in converged policies.","marker":"Dohare, Hernandez-Garcia, and Rahman 2023"},{"why":"Motivates partial rather than full parameter inheritance by showing that naive transfer can slow or prevent convergence.","marker":"Parisotto, Ba, and Salakhutdinov 2016"}],"fun_headline_variants":["Co-evolving rewards and policies beats retraining from scratch","LLM co-evolution lifts robot skills 95% on less data","Reward-policy co-evolution: more skill, less data than retraining","From-scratch training outperformed by reward-policy co-evolution","Co-evolving rewards with policies yields 95% better robot skills"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework assumes that a fused policy's score after only 200 updates reveals which fusion ratio and reward function will perform best after 2,500 updates; if early rankings diverge from final rankings, the selected combination is not actually the best and the reported gains could come from evaluating extra candidates.","fun_headline_variants_meta":{"raw":{"variants":["Co-evolving rewards and policies beats retraining from scratch","LLM co-evolution lifts robot skills 95% on less data","Reward-policy co-evolution: more skill, less data than retraining","From-scratch training outperformed by reward-policy co-evolution","Co-evolving rewards with policies yields 95% better robot skills"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000144,"raw_usage":{"total_tokens":1225,"prompt_tokens":1048,"completion_tokens":177,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":664,"completion_tokens_details":{"reasoning_tokens":85}},"tokens_in":664,"tokens_out":177,"duration_ms":2254,"temperature":1.0,"reasoning_tokens":85,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:04:16.331517+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a given co-evolution round's candidate fusion ratios through both 200 and 2,500 updates, then compare the rank order of their final task scores: if the ratio ranked best at 200 updates fails to be best at 2,500 in a substantial share of rounds across the six tasks, SC-BO's early-stop selection criterion is unreliable and the central efficiency claim is unsupported.","supporting_citations":[],"review_version":1}