REVIEW 5 major objections 5 minor 15 references
Efficient Language-instructed Skill Acquisition via Reward-Policy Co-Evolution
T0 review · 5 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read This paper argues that LLM-designed rewards should co-evolve with the policy rather than be validated by retraining from scratch, and reports a 95.3% average improvement in normalized score while using about 89% of the baseline's training…
desk verdict A plausible Eureka variant that reuses policies via a BO-selected fusion ratio, with consistent gains across six tasks, but the headline number and the early-stop assumption need work before the quantitative claims are reliable. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is a co-evolution cycle coupling two search processes. Reward evolution uses the LLM, prompted with the previous best reward code, task and environment descriptions, and policy performance statistics, to generate a dynamic reward population that keeps expanding until a candidate beats the previous best. Policy evolution parameterizes inheritance through the fusion ratio $\alpha \in [0,1]$ in $\theta^m_f(\alpha)=\alpha\theta^{m-1}_{\mathrm{best}}+(1-\alpha)\theta_0$, blending the previous best policy with a random initialization to balance retained skill and plasticity; Short-Cut Bayesian Optimization, a Gaussian-process surrogate with expected improvement and an early stop at $T_{BO}=200$ updates, selects $\alpha$ so that reward candidates can be compared without full retraining.
What would settle it
Run a given co-evolution round's candidate fusion ratios through both 200 and 2,500 updates, then compare the rank order of their final task scores: if the ratio ranked best at 200 updates fails to be best at 2,500 in a substantial share of rounds across the six tasks, SC-BO's early-stop selection criterion is unreliable and the central efficiency claim is unsupported.
Extended reading notes
Core claim
The central claim is that reward-policy co-evolution is both more sample-efficient and higher-performing than the conventional loop in which each candidate reward function is verified by training a policy from scratch. In each co-evolution cycle, the previous round's best reward code and performance statistics are fed to an LLM, which returns a dynamically sized set of new reward functions; each candidate is then evaluated not from random initialization but from a fused policy $\theta^m_f(\alpha)=\alpha\theta^{m-1}_{\mathrm{best}}+(1-\alpha)\theta_0$, where $\alpha$ is chosen by Short-Cut Bayesian Optimization after only $T_{BO}=200$ updates. The best-performing reward-policy pair receives an extended training run and is carried into the next cycle. On six high-dimensional Isaac Gym tasks, the paper reports higher Max Training Success and Human Normalized Scores than the from-scratch baseline in every task, with an average normalized improvement of 95.3% and a total training-sample budget about 89% of the baseline.
Load-bearing premise
The framework assumes that a fused policy's score after only 200 updates reveals which fusion ratio and reward function will perform best after 2,500 updates; if early rankings diverge from final rankings, the selected combination is not actually the best and the reported gains could come from evaluating extra candidates.
Editorial extensions
If this is right
- LLM-generated rewards no longer need to be universally effective; each reward only has to improve the current policy, which should widen the set of usable reward candidates.
- With the same five LLM reward-design rounds, the method reports the best Max Training Success on all six tasks and surpasses expert-designed rewards on all six, with the largest margins on the ShadowHand and FrankaCabinet tasks.
- The total training-sample budget can be cut to about 89% of the from-scratch baseline without sacrificing the gain, and a reduced variant using 56% of the budget still matches or beats the baseline on the reported tasks.
- The rule that only reward-policy pairs outperforming the previous best survive ensures monotonic improvement across rounds, a property the from-scratch baseline does not have.
- Ablations with fixed fusion ratios (0%, 50%, 100%) all underperform the Bayesian-selected ratio, supporting the claim that choosing $\alpha$ matters for the gain.
Reading between the lines
- If the early-stop ranking at 200 updates is stable, the same Short-Cut Bayesian Optimization trick could be applied to other RL design choices, such as network width, learning-rate schedules, or reward component weights, turning each into a cheap inner loop of the co-evolution outer loop.
- The reward-evolution side is effectively a Lamarckian search: improvements acquired by the policy are written back into the LLM prompt that generates the next reward. A testable extension would feed richer policy statistics, such as value estimates or feature activations, back to the LLM instead of reward-component traces.
- A head-to-head comparison against other warm-start or transfer-learning reward-design methods would clarify how much of the reported gain comes from co-evolution itself versus from reusing pretrained policy parameters.
- On tasks where random parameter fusion disrupts a delicate policy, the method could stall; a safeguard would be to adapt the fusion distribution based on observed candidate divergence at the early-stop horizon rather than assuming one horizon works for all tasks.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript proposes ROSKA, a reward-policy co-evolution framework for LLM-based reward design in high-dimensional robotic control. Reward evolution prompts GPT-4o with the previous best reward code and policy statistics to generate new candidate reward functions, while policy evolution fuses the previous best policy parameters with random initialization through a fusion ratio alpha chosen by a short-cut Bayesian optimization (SC-BO) that uses only TBO=200 updates per evaluation. The best reward-policy pair is then trained for Tmax=2500 updates and carried to the next round. Experiments on six Isaac Gym tasks compare ROSKA with sparse reward, human reward, Eureka, and a uniform-search variant ROSKA-U, reporting Max Training Success (MTS), Human Normalized Score (HNS), and Total Training Samples (TTS). The paper claims an average normalized improvement of 95.3% over Eureka while using roughly 89% of Eureka's training samples.
Significance. If the central claims are supported, the contribution is valuable: co-evolving rewards with policies and using Bayesian optimization to tune parameter inheritance addresses a real inefficiency in current LLM-based reward design, namely the need to retrain policies from scratch for every candidate reward. The evaluation is anchored to external sparse-reward task scores rather than to the method's own reward functions, so there is no definitional circularity. I credit the authors for including a uniform-search ablation, TTS calculations, and a limitations section that candidly notes first-round instability. However, the load-bearing SC-BO early-stop assumption is not experimentally validated, the dynamic-population mechanism is not reflected in the reported algorithm or experiments, and several numerical/descriptive claims are internally inconsistent. The contribution is therefore plausible but not yet established at the level claimed.
major comments (5)
- [Policy Evolution via Bayesian Optimization, Eqs. (8)-(10)] The short-cut premise is load-bearing for the paper's central claims, but it is never validated. Eq. (8) defines s(alpha; theta_f(alpha), TBO) with TBO=200, Eq. (9) selects alpha by maximizing this 200-update score, and Eq. (10) reports final performance after Tmax=2500 updates. For the 95.3% improvement and the 89% TTS claim to support reward-policy co-evolution, the 200-update ranking must be predictive of the 2500-update ranking. The paper contains no correlation analysis, no selected alpha values per task/round, and no seed-level curves showing that early divergence tracks final divergence. The ROSKA-U comparison (11 uniformly sampled alpha values trained for 3000 epochs) changes both the search strategy and the training budget, so it cannot isolate the early-stop effect. Please add a direct validation: report 200-update versus 2500-update rankings for all alpha values and seeds, or run an ablation in which SC-BO selects alpha but total training budget is held equal to a no-shortcut search. Without this, the gains could come from warm-starting at the previous best policy or from evaluating 12 candidate configurations per reward, rather than from identifying an actually capable-and-plastic reward-policy combination.
- [Abstract and Experimental Results (Table 1, Fig. 4)] The headline 'average normalized improvement of 95.3%' is not reproducible from the reported tables. Recomputing HNS from Table 1 via Eq. (11) gives approximate Eureka/ROSKA HNS pairs of (0.97, 1.46) on Ant, (1.17, 2.20) on Humanoid, (1.60, 4.09) on ShadowHand, (1.26, 2.01) on AllegroHand, (3.11, 8.55) on FrankaCabinet, and (0.56, 1.47) on ShadowHand-U; the average relative HNS improvement is about 115%, not 95.3%. If the intended statistic is instead the average relative MTS improvement, the value is about 99%. Please state the exact formula and per-task values used for the 95.3% figure, or correct the number.
- [Reward Evolution with Dynamic Population vs. Algorithm 1] The dynamic population mechanism is described as generating K reward functions repeatedly until a reward function outperforms the previous round's benchmark, but it is not implemented or reported in the experiments. Algorithm 1 loops over N=5 fixed rounds with K=6 reward functions, and the experimental settings state that each task uses N=5 rounds and K=6 functions. No DP-round sizes, no variable query counts, and no comparisons between fixed and dynamic population sizes are reported. The claim that the dynamic population acts as a filter and guarantees continual refinement is therefore untested. Either perform a dynamic-population ablation or revise the text to describe the fixed-size rounds honestly.
- [Comparison to Baseline Methods (Table 1)] The text states that 'ROSKA achieved better results in Ant, AllegroHand, and ShadowHand-U tasks' when compared with ROSKA-U, but Table 1 shows the opposite direction on three of the six tasks: ROSKA-U scores 12.52 vs 12.07 on Ant, 26.80 vs 23.22 on AllegroHand, and 23.72 vs 21.82 on ShadowHand-U. Since the ROSKA-U comparison is used to argue that SC-BO recovers uniform-search performance at lower sample cost, the table reading must be corrected and the conclusion restated to reflect that ROSKA wins only on Humanoid, ShadowHand, and FrankaCabinet by the reported means.
- [Limitations and TTS calculation] The Limitations section concedes that for complex tasks the first-round LLM may produce no workable reward and that the authors 'have to query the LLM multiple times' until at least one reward works. This extra querying is not counted in the round totals, the TTS ratios, or the cost comparison with Eureka. The claim of 'using less data' is therefore at best a training-sample claim, not a total-query or total-cost claim. Please report the actual number of LLM calls per task and per seed and state this limitation explicitly in the data-efficiency discussion.
minor comments (5)
- [Evaluation Metrics, Eq. (11)] Since the HNS values in Fig. 4 are hard to read from the figure, please provide a numerical table of HNS values alongside the bar chart.
- [Appendix, ROSKA-U TTS calculation] In the ROSKA-U TTS calculation, '6(rounds) × 3000' should read '6(reward functions) × 3000' for consistency with the first-round description.
- [Table 4 note] The note justifies using the mean sparse reward as a proxy for return; please provide a fuller justification or report actual discounted returns to ensure that MTS is not sensitive to this approximation.
- [Discussion on Evolution of total reward computation] The subsection contains unpolished phrasing and a sign inconsistency in the final reward formula; please edit for clarity and consistency with the earlier reward formulas.
- [Table 1 error bars] Several pairwise comparisons have overlapping standard deviations (e.g., Ant ROSKA 12.07 ± 0.60 vs ROSKA-U 12.52 ± 1.03, and ShadowHand-U ROSKA 21.82 ± 5.87 vs ROSKA-U 23.72 ± 4.96); please report the number of seeds and, where appropriate, paired-seed results to support the 'consistently achieves superior performance' statement.
Circularity Check
No significant circularity: ROSKA's reward and fusion-ratio selections are anchored to external sparse-reward task scores, not to the method's own reward functions.
full rationale
The paper's central claims are not definitionally circular. Reward candidates are selected via Eq. (6) using V(·), the return under the ground-truth sparse reward, and all reported metrics (MTS, HNS) are computed against the same external sparse-reward fitness functions (Table 4), not against the LLM-generated dense rewards. The co-evolution loop selects reward-policy pairs by measured task performance, which is the intended mechanism rather than a self-referential derivation. The Short-Cut Bayesian Optimization step (Eqs. 8-10) does rest on an unvalidated premise that 200-update scores rank fusion ratios similarly to 2500-update final performance, and the paper does not report correlation or seed-level evidence for this premise. However, this is an empirical/correctness risk about whether SC-BO selects the best fusion ratio, not a circularity: the final numbers are measured, not derived from the early-stop scores by construction. The limitation section candidly notes LLM-generated reward instability and the need for repeated LLM queries in the first round, which is a robustness concern but does not indicate that any prediction reduces to its input. No load-bearing self-citation chain or imported uniqueness theorem is present; the comparison baseline Eureka is external prior work. Overall, the evaluation is self-contained against external benchmarks, so the circularity score is 0.
Assumptions & free parameters
free parameters (1)
- Fusion ratio alpha (per task per round) =
not reported
assumptions (4)
- domain assumption Early policy performance after TBO=200 updates predicts final performance after 2500 updates.
- domain assumption Fusing previous best policy weights with random weights at ratio alpha preserves useful skills and restores plasticity.
- domain assumption Evaluating a reward by fine-tuning the previous best policy selects rewards that lead to the best final performance.
- standard math Gaussian process and expected improvement assumptions for the response surface s(alpha).
Cite this review
Pith. "Pith review of Efficient Language-instructed Skill Acquisition via Reward-Policy Co-Evolution." pith.science (2026). https://pith.science/paper/43UYQUII
@misc{pith2026241213492,
author = {Pith},
title = {Pith review of: Efficient Language-instructed Skill Acquisition via Reward-Policy Co-Evolution},
year = {2026},
howpublished = {\url{https://pith.science/paper/43UYQUII}},
note = {Machine review of arXiv:2412.13492}
}
read the original abstract
The ability to autonomously explore and resolve tasks with minimal human guidance is crucial for the self-development of embodied intelligence. Although reinforcement learning methods can largely ease human effort, it's challenging to design reward functions for real-world tasks, especially for high-dimensional robotic control, due to complex relationships among joints and tasks. Recent advancements large language models (LLMs) enable automatic reward function design. However, approaches evaluate reward functions by re-training policies from scratch placing an undue burden on the reward function, expecting it to be effective throughout the whole policy improvement process. We argue for a more practical strategy in robotic autonomy, focusing on refining existing policies with policy-dependent reward functions rather than a universal one. To this end, we propose a novel reward-policy co-evolution framework where the reward function and the learned policy benefit from each other's progressive on-the-fly improvements, resulting in more efficient and higher-performing skill acquisition. Specifically, the reward evolution process translates the robot's previous best reward function, descriptions of tasks and environment into text inputs. These inputs are used to query LLMs to generate a dynamic amount of reward function candidates, ensuring continuous improvement at each round of evolution. For policy evolution, our method generates new policy populations by hybridizing historically optimal and random policies. Through an improved Bayesian optimization, our approach efficiently and robustly identifies the most capable and plastic reward-policy combination, which then proceeds to the next round of co-evolution. Despite using less data, our approach demonstrates an average normalized improvement of 95.3% across various high-dimensional robotic skill learning tasks.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
You may find it helpful to normalize the reward to a fixed range by applying transformations like torch.exp to the overall reward or its compo- nents
-
[3]
Make sure the type of each input variable is correctly specified; a float input variable should not be speci- fied as torch.Tensor
-
[4]
Under no circumstance can you in- troduce new input variables
Most importantly, the reward code’s input variables must contain only attributes of the provided envi- ronment class definition (namely, variables that have prefix self.). Under no circumstance can you in- troduce new input variables. Feedback Prompt: It consist of the code and the mea- surement of the best-performed reward function. As shown in the follo...
-
[6]
Each transformed reward component should have its own temperature variable
If you choose to transform a reward component, then you must also introduce a temperature parame- ter inside the transformation function; this parameter must be a named variable in the reward function and it must not be an input variable. Each transformed reward component should have its own temperature variable
-
[9]
If the success rates are always near zero, then you must rewrite the entire reward function
-
[10]
If the values for a certain reward component are near iden- tical throughout, then this means RL is not able to optimize this component as it is written. You may consider: (a) Changing its scale or the value of its temperature pa- rameter (b) Re-writing the reward component (c) Discarding the reward component
-
[11]
If some reward components’ magnitude is significantly larger, then you must re-scale its value to a proper range. Please analyze each existing reward component in the sug- gested manner above first, and then write the reward function code. The output of the reward function should consist of two items:
-
[12]
a dictionary of each individual reward component. The code output should be formatted as a python code string: "‘‘‘python ... ‘‘‘" . Some helpful tips for writing the reward function code:
Show all 15 references
-
[13]
You may find it helpful to normalize the reward to a fixed range by applying transformations like torch.exp to the overall reward or its components
-
[14]
Each transformed reward component should have its own temperature variable
If you choose to transform a reward component, then you must also introduce a temperature parameter inside the transformation function; this parameter must be a named variable in the reward function and it must not be an in- put variable. Each transformed reward component shou...
-
[15]
Make sure the type of each input variable is correctly specified; a float input variable should not be specified as torch.Tensor
-
[16]
Un- der no circumstance can you introduce new input vari- ables
Most importantly, the reward code’s input variables must contain only attributes of the provided environment class definition (namely, variables that have prefixself.). Un- der no circumstance can you introduce new input vari- ables
-
[318]
Dohare, S.; Hernandez-Garcia, J
PMLR. Dohare, S.; Hernandez-Garcia, J. F.; and Rahman, P. 2023. Maintaining Plasticity in Deep Continual Learning. CoRR, abs/2306.13812. Haarnoja, T.; Moran, B.; Lever, G.; Huang, S. H.; Tiru- mala, D.; Humplik, J.; Wulfmeier, M.; Tunyasuvunakool, S.; Siegel, N. Y .; Hafner, R...
2023 arXiv
-
[2008]
initial prompt
Maximum entropy inverse reinforcement learning. In Aaai, volume 8, 1433–1438. Chicago, IL, USA. Appendix Algorithm Description In this section we will introduce the algorithm flow of ROSKA, as illustrated in Alg. 1. Similar Eureka, our ROSKA employs a multi-round iterative pro...
-
[2023]
In Conference on robot learning, 287–
Do as i can, not as i say: Grounding language in robotic affordances. In Conference on robot learning, 287–
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.