{"id":"05c4c7f4-dc53-47d6-829e-3af9846a5bb9","arxiv_id":"2504.12568","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"EPO, a hybrid of neuroevolution and PPO, shows modest reward gains on Atari Pong and sample-efficiency gains on Breakout, but the evidence is weakened by hyperparameter tuning on Breakout and incomplete baselines.","lead":"This paper introduces EPO, which alternates between evolutionary search and PPO fine-tuning to train reinforcement learning policies. It reports better policies on Pong and better sample efficiency on Breakout, but the gains are modest and partly contradicted by the paper's own tables.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Wall-clock-capped comparisons do not measure sample efficiency; EPO's lower sample count at fixed time may reflect lower throughput, so the 26.8%/57.3% gains are not established.","rationale":"I focused on the sample-efficiency comparison protocol because it is the load-bearing quantitative support for half of the central claim. Even if fitness-weighted crossover works perfectly, the experiment as designed still would not demonstrate sample efficiency. Section 5.1 sets wall-clock caps, and the sample counts in Table 4 and Figure 3 vary by roughly 2.3x between methods under identical wall-clock, showing substantial throughput differences. A matched-sample-budget experiment would settle whether the 26.8% and 57.3% improvements are real. The Pong best-reward statistic further weakens the policy-quality evidence, but the sample-efficiency protocol is the cleanest single failure. Therefore the reader's REJECT verdict is unchanged, though my identified concern differs from the reader's weakest assumption about crossover.","tokens_in":11949,"tokens_out":8227,"duration_ms":89464,"concrete_test":"Re-run Breakout with a fixed sample budget matched to EPO's total (about 0.82M interactions) for PPO and pure evolution, or plot reward versus cumulative samples for all methods. If PPO or pure evolution reaches or exceeds EPO's mean reward at the same sample count, the reported sample-efficiency improvement is an artifact of the wall-clock protocol and the central claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that EPO \"improves both policy quality and sample efficiency\" rests on two quantitative pillars: Breakout sample counts (Table 4) and Pong best reward (Table 3). The sample-efficiency pillar is invalid as reported. Section 5.1 fixes training duration in wall-clock seconds (7,200s for Breakout, 10,000s for Pong) for all methods, then Table 4 labels the resulting sample counts as evidence of sample efficiency. That conflates sample efficiency with computational throughput. A method that is slower per environment interaction will consume fewer samples in a fixed wall-clock budget regardless of whether it obtains more reward per sample. Indeed, pure evolution (no PPO fine-tuning) consumes 1.92M samples vs EPO's 0.82M in the same 7,200s, likely because it can run more generations per second; this does not mean EPO is more sample-efficient. To support the claim, the authors would need to compare reward at matched sample budgets or plot reward versus samples. As published, the 26.8% and 57.3% figures are not a valid sample-efficiency result. The Pong pillar is also weak: mean rewards overlap (-6.25±3.42 vs -8.03±5.17), and the \"better policies\" claim rests on a single best-episode reward (11.0 vs 1.0), an extreme-order statistic with no significance test. Thus the central claim is unsupported even before considering whether fitness-weighted crossover is an effective operator.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Evolutionary Policy Optimization (EPO), a hybrid algorithm that combines neuroevolution (elitism, fitness-weighted crossover, mutation) with PPO pre-training and fine-tuning. EPO is evaluated on Atari Pong and Breakout against PPO and a pure evolution baseline, with the central claim that EPO improves both policy quality and sample efficiency. The paper also reports an ablation study and a transfer-learning experiment. The algorithm is clearly described, but the empirical evidence for the central claim is undermined by methodological flaws in the sample-efficiency comparison, in-sample hyperparameter tuning, and weak statistical support.","tokens_in":12299,"tokens_out":5147,"duration_ms":51602,"significance":"If the central claim were established, EPO would be a useful contribution: a simple integration of neuroevolution and policy gradients could improve exploration and exploit local optimization, potentially benefiting sparse-reward and exploration-heavy RL tasks. The paper provides a clear algorithmic description, an ablation study, and experiments on standard Atari benchmarks. However, the reported results do not currently support the claim. The sample-efficiency comparison is confounded with wall-clock time, the Breakout result is in-sample because hyperparameters were tuned on that task, and the policy-quality improvements are within one standard deviation or rely on an extreme-order statistic. These issues affect the paper's main conclusions, so the contribution is not yet validated.","major_comments":[{"comment":"The sample-efficiency comparison is confounded with wall-clock time. All methods are trained for a fixed wall-clock duration (7,200 seconds for Breakout, 10,000 seconds for Pong), and then the total sample count is reported as a measure of sample efficiency. A lower sample count at fixed wall-clock time does not imply better sample efficiency; it may reflect lower throughput. Indeed, pure evolution consumes 1.92e6 samples in the same wall-clock budget as EPO's 0.82e6, likely because it runs more environment interactions per second, not because EPO is more sample-efficient. To support the claims of a 26.8% and 57.3% sample-efficiency improvement, the authors must compare reward at matched sample budgets or plot reward versus samples; the current results do not measure sample efficiency.","section":"Section 5.1, Table 4"},{"comment":"The main Breakout result is measured on the same task used for hyperparameter tuning. Hyperparameters (mutation probability, elite count, population size) were optimized over 75 trials of 240 seconds of training on Breakout, and the Breakout results in Table 4 are then reported for this configuration. This makes the headline sample-efficiency and reward numbers in-sample. The out-of-sample check on Pong is not reported with any numeric detail; the paper only states that 'EPO with the identified hyperparameters consistently performed well' over 20 trials, with no results shown. The generalization claim is therefore unsupported, and the Breakout improvements may reflect tuning to that specific environment.","section":"Section 4.3, Table 4"},{"comment":"The statistical evidence for 'better policy quality' is weak. On Pong, the mean rewards are -6.25 +/- 3.42 for EPO versus -8.03 +/- 5.17 for PPO; these overlap at one standard deviation. The 'best reward' of 11.0 versus 1.0 is an extreme-order statistic with no significance test and is not a reliable measure of policy quality. On Breakout, mean rewards are 3.15 +/- 1.45 versus 3.02 +/- 1.43, again overlapping. No significance tests, confidence intervals for the differences, or effect sizes are provided, so the claim that EPO 'discovers significantly better policies' is not supported by the data.","section":"Tables 3 and 4"},{"comment":"The paper's own Pong results contradict the abstract's claim that EPO improves both policy quality and sample efficiency. Table 3 shows EPO uses 2.36e6 samples versus PPO's 1.58e6 samples at the same wall-clock budget, and the text acknowledges 'EPO sacrifices some sample efficiency but gains a substantial improvement in policy quality.' This direct contradiction means the central claim as stated in the abstract and introduction cannot stand without substantial re-framing.","section":"Section 5.2, Pong results"},{"comment":"The core evolutionary operator—fitness-weighted averaging of neural network weights—is not validated beyond a single sentence. The paper states that 'other crossover techniques, such as random parameter masking, were tested but did not outperform the proposed weighted averaging approach,' but no data or analysis is provided. The assumption that interpolating weights of deep CNNs produces viable offspring that 500 timesteps of PPO fine-tuning can repair is a load-bearing assumption of the algorithm, yet it is not tested in the ablation studies. This leaves a correctness-risk concern about whether the evolutionary search actually explores effectively.","section":"Section 4.1.2, Algorithm 1"}],"minor_comments":[{"comment":"The text contains the placeholder '[NEED TO ADD CITATIONS]' in the Previous Work section, which indicates the manuscript is incomplete and needs a careful revision.","section":"Section 3"},{"comment":"The caption and axis labels use abbreviations 'EPO-TL' and 'EPO-NoPT' without defining them in the figure; the text introduces these names only later in Section 6 or uses different terms, which hurts readability.","section":"Figure 3"},{"comment":"The sentence 'the computational overhead of pre-processing these visual inputs makes Atari a suitable benchmark for evaluating sample complexity' is confusing; preprocessing overhead is unrelated to sample complexity, and the point is not developed.","section":"Section 5.1"},{"comment":"In the appendix table, the phrase 'with with' appears in the title, and it would be helpful to report the standard deviations across runs for the hyperparameter-search rewards, not just the mean.","section":"Section 4.3 and Table 6"},{"comment":"The 'pure evolution' baseline is not fully described: its mutation rate, population size, and other hyperparameters are not given, making it hard to assess the fairness of the comparison.","section":"General"}],"recommendation":"reject","confidential_remarks":"The paper appears to be an early-stage report (it references a GECCO 2025 poster and includes a placeholder for missing citations). The central empirical claims are not supported by the experiments as reported, and the flaws are load-bearing: the sample-efficiency measure is confounded with wall-clock time, the main Breakout result is in-sample due to hyperparameter tuning, and the policy-quality claims lack statistical significance. These issues would require new experiments and a substantial re-framing of the claims, which is beyond a standard minor revision. I recommend rejection, though the authors are encouraged to resubmit after addressing these methodological concerns."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper is a clear, concrete description of a hybrid that alternates PPO pre-training, elitism, fitness-weighted crossover, adaptive mutation, and PPO fine-tuning. That design is worth a look. But the headline claims do not survive contact with the experiments.\n\nThe real strengths: Algorithm 1 is specific enough to reproduce, the hyperparameter table is helpful, and the ablation study (pre-training, fine-tuning, and a transfer-learning attempt) is more than many comparable papers do. The authors also admit in the Pong section that EPO uses more samples than PPO—an honest moment that sits awkwardly next to the abstract's claim of improved sample efficiency.\n\nThe soft spots are not minor. First, Section 5.1 caps all training at wall-clock seconds, and Tables 3 and 4 then report sample counts after that fixed time. That conflates sample efficiency with throughput. In a fixed wall-clock budget, a method that runs fewer environment interactions per second will show a lower sample count even if it gets no more reward per sample. Pure evolution's high sample count (1.92M vs EPO's 0.82M in 7,200s) likely reflects more evaluations per second, not worse sample efficiency. So the 26.8% and 57.3% figures are not evidence for the claimed sample-efficiency gain.\n\nSecond, the main positive Breakout result is in-sample: hyperparameters were tuned on Breakout (Section 4.3) and then the headline numbers come from Breakout. That does not make the result circular in the full sense, but it does mean the reported improvement is partly tuning, not algorithm.\n\nThird, the Pong evidence for \"significantly better policies\" rests on a single best-episode reward (11.0 vs 1.0), while mean rewards overlap well within error bars. That is an extreme-order statistic, not a significance test.\n\nFinally, the related-work section literally contains \"[NEED TO ADD CITATIONS]\" and omits existing hybrids like PDERL and CEM-RL. That is not a minor citation gap; it changes the paper's positioning from novel to incremental.\n\nWho is this for? A reader specifically interested in neuroevolution plus PG hybrids could learn something from the algorithm design and ablations, but should not rely on the quantitative claims. As a submission, this is not ready: the central claim is unsupported, the metric is wrong, and the literature engagement is incomplete.\n\nMy recommendation: send it to peer review, but with the expectation of major revision or rejection in its current form. A competent referee will catch the sample-efficiency issue immediately, and the authors need that feedback. If an editor asks me, I'd say it is a desk-reject candidate only if the venue has stricter completeness standards; otherwise it deserves a review.","headline":"EPO is a clean hybrid of PPO and neuroevolution, but its sample-efficiency claims are confounded by wall-clock-capped comparisons and in-sample tuning.","tokens_in":12816,"tokens_out":4606,"would_cite":false,"duration_ms":47116,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper shows that alternating neuroevolution with PPO improves both sample efficiency and policy quality on Atari benchmarks.","keywords":["evolutionary policy optimization","neuroevolution","proximal policy optimization","exploration vs exploitation","sample efficiency","Atari reinforcement learning","crossover","mutation"],"falsifier":"A direct test would measure the average reward of crossover offspring immediately after crossover and after their 500-step PPO fine-tuning on Breakout. If the fine-tuned crossover offspring are not better, on average, than offspring produced by mutating a single elite, then the crossover operation is not carrying EPO's gains and the algorithm would perform just as well without it. A second check: recompute sample counts including the overhead of fitness evaluations under the same wall-clock budget; if EPO's sample-efficiency advantage over PPO disappears, the claim of 26.8% improvement is an artifact of how samples are counted.","tokens_in":11761,"feed_emoji":"🧬","tokens_out":5789,"duration_ms":52619,"temperature":0.7,"pith_summary":"Evolutionary Policy Optimization (EPO) is a hybrid algorithm that alternates between a policy gradient method (PPO) for fine-grained local optimization and neuroevolution for global exploration. The paper's central claim is that this combination outperforms either approach alone: on Breakout, EPO reduces sample count by 26.8% compared with PPO and by 57.3% compared with pure evolution, and on Pong, EPO's best policy reaches a reward of 11.0 whereas PPO's best reaches 1.0. EPO uses PPO to pre-train a population, then applies elitism, fitness-weighted crossover, adaptive mutation, and a short PPO fine-tuning step to offspring. The practical stakes are that RL systems can get both exploration and exploitation without a strict sample-efficiency penalty, at least on tasks that mix dense and sparse reward structure.","feed_headline":"Neuroevolution plus PPO beats either method alone on Atari","feed_subtitle":"Combining evolution's exploration with PPO's fine-tuning cuts sample cost and discovers stronger policies.","key_machinery":"The central object is the evolutionary loop itself, with three operations: elitism (carrying the top $\\mathrm{E}$ agents forward), fitness-weighted crossover of neural network weight vectors $c = \\alpha p_1 + (1-\\alpha)p_2$ where $\\alpha = f_1/(f_1+f_2+\\epsilon)$, and adaptive Gaussian mutation whose scale grows as parent fitnesses converge. The loop's variation is carried by the crossover and mutation, while PPO acts as the exploitation engine by pre-training the initial clone population and by locally refining offspring; this alternating structure is what lets global search and gradient descent reinforce each other.","core_discovery":"On its own terms, the paper establishes that neuroevolution and policy gradients need not be competing alternatives. By cloning a PPO-pretrained policy into a population, evolving the population through fitness-weighted weight crossover and adaptive Gaussian mutation, and fine-tuning unmutated offspring with PPO for 500 timesteps, EPO discovers policies that plain PPO does not find and uses fewer environment samples to reach comparable reward on Breakout. The paper reports best-reward 11.0 on Pong versus 1.0 for PPO and a 26.8% sample-count reduction on Breakout over PPO. These results are the evidence for the claim that the hybrid is effective for tasks requiring both exploration and local optimization.","pith_inferences":["If fitness-weighted weight averaging is the crucial variation operator, then EPO's success might be fragile when network architectures exhibit permutation symmetry, where interpolating weights can map to functionally meaningless policies; testing crossover in a permutation-invariant representation would separate this effect.","The sample-count comparison on Pong favors PPO, so the sweeping claim of 'sample efficiency' is a per-task phenomenon; a more careful comparison would track the full reward–sample Pareto frontier rather than a single metric.","One testable prediction follows from the mutation-scaling rule: when parent fitnesses are nearly equal, mutation noise increases, so EPO should explore most aggressively when the population is undecided about direction; a diagnostic would record mutation scale over training and correlate it with reward jumps.","The 500-step fine-tuning budget is a free parameter of the hybrid; on harder or higher-dimensional tasks, the required budget likely grows with task complexity, and the paper does not establish a scaling law."],"forward_implications":["On tasks with sparse or deceptive rewards, EPO should reach a given reward level with about a quarter fewer environment interactions than PPO alone, based on the Breakout results.","EPO's best-found policies can be qualitatively better than PPO's best within the same wall-clock budget, as shown by the Pong best-reward gap (11.0 vs. 1.0).","A short fine-tuning budget of 500 PPO timesteps is enough to make crossover offspring useful, so the hybrid's sample overhead per generation is modest.","Transferring a policy from a similar task (Pong to Breakout) does not accelerate EPO, so the gains come from within-task evolution and not from cross-task reuse.","Hyperparameters tuned on Breakout (mutation probability 0.3, population size 8, elite count 3) transferred to Pong without re-tuning, suggesting the method is not highly sensitive to task identity."],"supporting_citations":[{"why":"Supplies the PPO objective used for pre-training and fine-tuning in EPO.","marker":"[18]"},{"why":"Defines the evolution-strategies baseline against which sample efficiency is compared.","marker":"[16]"},{"why":"Provides the deep-neuroevolution baseline that motivates the evolutionary component.","marker":"[20]"},{"why":"Provides the PPO implementation used for all PPO training and fine-tuning.","marker":"[14]"},{"why":"Supplies the Arcade Learning Environment simulator for the Atari benchmarks.","marker":"[2]"},{"why":"Establishes the Atari 2600 domain and the convolutional network approach that EPO's policy network follows.","marker":"[11]"},{"why":"Defines the frame preprocessing and stacking used to convert Atari frames into network inputs.","marker":"[12]"}],"fun_headline_variants":["EPO fuses evolution and PPO to beat both on Atari","Hybrid of evolution and PPO cuts samples and boosts Pong","Neuroevolution plus PPO wins on Atari with fewer samples","Evolutionary Policy Optimization beats PPO and evolution solo","Mixing evolution with PPO improves Atari performance and efficiency"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that interpolating the weights of two trained policies, then running PPO for 500 timesteps, yields offspring worth keeping; if weight averaging routinely destroys the policy structure that PPO cannot repair, the evolutionary loop stops producing useful variation.","fun_headline_variants_meta":{"raw":{"variants":["EPO fuses evolution and PPO to beat both on Atari","Hybrid of evolution and PPO cuts samples and boosts Pong","Neuroevolution plus PPO wins on Atari with fewer samples","Evolutionary Policy Optimization beats PPO and evolution solo","Mixing evolution with PPO improves Atari performance and efficiency"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000532,"raw_usage":{"total_tokens":2501,"prompt_tokens":824,"completion_tokens":1677,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":440,"completion_tokens_details":{"reasoning_tokens":1590}},"tokens_in":440,"tokens_out":1677,"duration_ms":12371,"temperature":1.0,"reasoning_tokens":1590,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:27:56.803681+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct test would measure the average reward of crossover offspring immediately after crossover and after their 500-step PPO fine-tuning on Breakout. If the fine-tuned crossover offspring are not better, on average, than offspring produced by mutating a single elite, then the crossover operation is not carrying EPO's gains and the algorithm would perform just as well without it. A second check: recompute sample counts including the overhead of fitness evaluations under the same wall-clock budget; if EPO's sample-efficiency advantage over PPO disappears, the claim of 26.8% improvement is an artifact of how samples are counted.","supporting_citations":[],"review_version":1}