{"id":"31105370-92b5-4424-85cd-970ac6566af6","arxiv_id":"2505.11135","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Across three semiconductor fab simulation models, CMA-ES based dispatching consistently outperforms PPO, improving tardiness by up to 29% on small benchmarks and up to 4% on a real industrial dataset.","lead":"This paper compares two reinforcement learning approaches, policy gradients (PPO) and evolution strategies (CMA-ES), for dispatching lot movements in semiconductor factories, using two public simulation benchmarks and a real Infineon industrial dataset. It finds that CMA-ES scales better to larger, more realistic fab models, with modest gains in tardiness and throughput.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'scales much better' claim is not yet established because PPO is compared under a restricted action space and a truncated-buffer compute limit, not a matched budget.","rationale":"The reader's formal weakest assumption concerned simulator validity and industry-dataset realism, but the reader's rationale also noted an unbalanced PPO comparison and missing error bars. My concern is more specific: the paper itself excludes batching tools from PPO and truncates PPO episodes on the larger models for memory reasons, while the Limitations section admits that more hardware could make PPO more competitive. This makes the headline comparison between CMA-ES and PPO not yet a clean test of algorithmic scalability. The concrete check is feasible on the open-source SMT2020 model and would directly test whether the gap survives a matched and scope-equivalent setup. Until that check is run, the conditional verdict remains appropriate, so I recommend no change to the reader's CONDITIONAL verdict.","tokens_in":20413,"tokens_out":6702,"duration_ms":66906,"concrete_test":"Re-run the SMT2020 comparison with PPO given the same all-tool control scope as CMA-ES (including a batching-capable action head), a complete-episode training buffer made feasible by streaming or external storage, and a matched budget (same CPU-core-hours, seeds, and number of workers as the CMA-ES runs). If PPO still shows negative tardiness improvement after matched tuning, the scalability claim is supported; if PPO reaches comparable improvement, the claim is refuted as an artifact of the truncated and restricted PPO setup.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central comparison is load-bearing on PPO being a comparable, fairly tuned baseline. Section 3.2 states that batch tools 'are therefore not considered for the PPO experiments', while Section 4.2 reports PPO only on the lithography tool group; complete episodes were used only for Minifab, with truncated episode buffers forced on SMT2020 and industry because full-buffer samples overflow memory. The Limitations section explicitly concedes that 'with more dedicated hardware resources, the PPO potentially would deliver more promising result for the industrial scale scenario.' Since CMA-ES's best results come from controlling all tool types (Fig. 8, Fig. B3) but PPO is never given that control scope, the observed gap conflates algorithmic scalability with action-space restriction, truncated credit assignment, and compute budget. Without seed-level variance or matched CPU budgets, the headline 'scales much better' is an interpretation, not a demonstrated scaling property.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper compares two reinforcement learning approaches, CMA-ES (an evolution-strategy method) and PPO (a policy-gradient method), for lot dispatching in semiconductor frontend fabs. Experiments are run on three simulation models—Minifab, SMT2020, and a proprietary large-scale industry dataset—all implemented in the D-SIMCON commercial simulator. The authors report that CMA-ES achieves double-digit tardiness improvements on Minifab and SMT2020 and up to 4% tardiness and 1% throughput improvement on the industry dataset, while PPO improves tardiness only on Minifab and degrades it on the larger models. They conclude that the evolution-strategy approach 'scales much better' than the policy-gradient approach, and they additionally analyze the influence of the set of controlled tools, computational cost, and generalization to unseen loading scenarios.","tokens_in":20629,"tokens_out":3021,"duration_ms":31759,"significance":"If the main claim were established, this would be a useful contribution to a field where comparisons are often made on incompatible testbeds: the paper uses the same simulator engine for all three scenarios, includes an industry-scale dataset with diverse load mixes and tool dedications, and provides generalization experiments for changing loading scenarios. The work also gives concrete quantitative results on computational cost and shows that controlling more tool groups improves CMA-ES performance. However, the headline comparison between CMA-ES and PPO is currently not a controlled algorithm comparison: the two methods differ in action-space scope, reward function, episode-buffer handling, and tuning effort. Because the paper itself concedes (Section 6) that PPO might perform better with more dedicated hardware, the claim that the proposed method 'scales much better' is overstated as presented. The underlying observations are valuable, but the paper needs a substantial revision to support the central scalability statement.","major_comments":[{"comment":"The comparison between CMA-ES and PPO is not matched in action space. Section 3.2 states that 'Batching tools are therefore not considered for the PPO experiments', and Section 4.2 reports PPO results only for the lithography area. In contrast, CMA-ES's best results are obtained when all tool types are controlled (Fig. 8 for Minifab, Fig. B3 for SMT2020, Fig. B4 for industry). The observed gap therefore confounds algorithmic scalability with the scope of dispatching decisions available to each agent. The paper should either run PPO with the same expanded action space (including batching, which would require an appropriate action representation) or explicitly frame the headline result as a comparison of the two methods under different control scopes.","section":"3.2 and 4.2"},{"comment":"PPO uses truncated episode buffers for SMT2020 and the industry model because full-buffer samples overflow memory, while complete episodes are used only for Minifab. The Limitations section explicitly states that 'with more dedicated hardware resources, the PPO potentially would deliver more promising result for the industrial scale scenario.' This means the reported PPO degradation on larger models may reflect memory and compute constraints rather than algorithmic scaling behavior. To support the 'scales much better' claim, the authors should report matched computational budgets (e.g., CPU-hours and memory usage per method) or, if feasible, run PPO with a larger buffer. At minimum, the conclusion should be rephrased to say that CMA-ES scaled better under the authors' resource constraints, not that it scales better in general.","section":"4.2 and 6"},{"comment":"The two methods are optimized against different objectives. CMA-ES uses the cost functions c_ES or c'_ES, while PPO uses the reward r_PPO,t, which was selected as the best among several candidate rewards tested on the same MiniFab benchmark (Appendix A). Moreover, per-model hyperparameters (rollout fragment length, episode truncation, learning rates) are tuned for PPO, and CMA-ES hyperparameters are also tuned. This asymmetric tuning burden means the comparison reflects one particular configuration of each method, not a general property of the algorithms. The paper should state this limitation prominently and ideally provide a sensitivity analysis for the PPO reward choice on the larger models.","section":"3.3 and Appendix A"},{"comment":"Results are reported as averages over the training progress without confidence intervals or seed-level variance. Because the simulator uses random seeds for stochastic tool failures and load scenarios, the differences between, e.g., CMA-ES LIT (15.1%) and LIT+DIF (15.1%) in Fig. 8 or the negative PPO values in Fig. 5 might be within run-to-run noise. The paper should report multiple seeds and error bars, or at least provide a statistical measure of significance for the central comparison. This is particularly important for the industry dataset, where improvements are small (up to 4% tardiness, 1% throughput).","section":"Figures 5, 7, 8 and Section 4"}],"minor_comments":[{"comment":"The text contains a typo: 'Eearliest Due Date' should be 'Earliest Due Date' in the list of dispatching heuristics in Section 2.1.","section":"2.1"},{"comment":"The heading 'Semiconductor Frontened Manufacturing Testbeds' contains a typo: 'Frontened' should be 'Frontend'.","section":"2.2"},{"comment":"Several entries in Table 3 have formatting problems (e.g., '20243 Yedidsion', 'Zhangg', and superscript markers that are not explained consistently). Please clean up the table formatting and define all footnotes.","section":"Table 3"},{"comment":"Figure 9 reports computation times in hours/days but does not specify the number of CPU cores used for each experiment. Since the paper claims scalability with CPU cores, the core count should be stated explicitly in the caption or text.","section":"4.4"},{"comment":"Reference [21] (Karaboga) is cited with a DOI for the Grey Wolf Optimizer paper [38], which appears incorrect. Please verify all DOIs.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper relies on a commercial simulator (D-SIMCON) whose validation and availability are not fully described; since one of the co-authors is affiliated with D-SIMLAB, a detailed description of the simulation models and their configuration would strengthen reproducibility. The industry dataset is proprietary, but the authors could still release the simulation configuration or anonymized summaries. The central claim of the paper is likely to be accepted only after the comparison between CMA-ES and PPO is made more balanced, either by expanding the PPO action space and resource budget or by clearly limiting the scope of the claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead arXiv:2505.11135. The genuinely new thing is the same-simulator comparison of PPO and CMA-ES across Minifab, SMT2020, and a real Infineon-scale fab, plus the generalization tests across loading scenarios and failure patterns. That is worth having. The paper is also honest about its limitations and does not oversell the absolute gains: 4% tardiness and 1% throughput on the industrial scenario is modest and believable.\n\nThe soft spot is the central scalability claim. The paper concludes CMA-ES \"scales much better\" than a \"comparable\" PPO baseline, but the PPO arm is not comparable. PPO never controls batching tools, uses a different reward, and on SMT2020 and industry it is forced into truncated episodes with a memory-bound buffer. The paper's own Limitations section concedes that with more dedicated hardware PPO might do better on the industrial scale. So the gap between CMA-ES and PPO plausibly reflects action-space restriction, truncated credit assignment, and compute budget as much as algorithmic scalability. That is a real flaw in the argument as stated. The stress-test note lands.\n\nAlso missing: no seed-level variance or confidence intervals, and no code or data release (the industry dataset is commercial, fine, but the open-source models should be reproducible). The PPO reward was selected empirically on the same benchmarks, which is a minor circularity. On the positive side, the comparison of controlled tool subsets is informative, and the finding that more control helps is a useful practical insight.\n\nIs this a serious paper? Yes. The empirical work looks careful, the baselines are sensible, and the generalization experiments are a step beyond what most fab-scheduling RL papers do. A serious referee should see it. The right outcome is major revision: either soften the headline, or match the PPO setup better (complete episodes where feasible, batching control via a modified action space, matched CPU budget, and seed-level reporting). As it stands I would not cite the \"scaling\" claim, but I might cite the three-model same-simulator comparison as a data point.\n\nTake it to peer review, with a referee who will insist on the matched comparison.","headline":"Useful same-simulator comparison of CMA-ES and PPO on real fab data, but the 'scales much better' headline is not yet earned because the PPO baseline is structurally disadvantaged.","tokens_in":21205,"tokens_out":2256,"would_cite":true,"duration_ms":21488,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that an evolution-strategy optimizer (CMA-ES) scales to realistic semiconductor fab dispatching far better than policy-gradient reinforcement learning (PPO), cutting tardiness by double digits on open-source benchmark…","keywords":["Reinforcement Learning","Dispatching","Scheduling","Semiconductor Manufacturing","Scalability","Evolution Strategies","SMT2020","Minifab"],"falsifier":"Run the same CMA-ES and PPO training pipelines on an independently written discrete-event simulator implementation of the Minifab and SMT2020 models with identical due dates and random seeds: if PPO matches or beats CMA-ES there, or if CMA-ES's advantage disappears, the scalability ordering is an artifact of the particular simulator. A second check: deploy the CMA-ES policy on a different real fab with different tool dedications and load mixes and measure tardiness against that fab's own heuristics; the claim fails if the improvement does not reproduce or turns negative.","tokens_in":1727,"feed_emoji":"🏭","tokens_out":1887,"duration_ms":82387,"temperature":0.7,"pith_summary":"The paper compares two reinforcement-learning optimizers, the policy-gradient method PPO and the evolution-strategy method CMA-ES, on the same lot-dispatching task across three simulation environments of increasing realism: the small Minifab model, the large SMT2020 benchmark, and a real frontend-fab dataset with more than a thousand machines. It tries to establish that CMA-ES scales much better than PPO as the simulation grows: CMA-ES improves tardiness in all three settings, while PPO improves only the tiny Minifab and degrades performance on the larger models. It also claims that controlling more bottleneck tool groups, not just lithography, is decisive, and that training on diverse loading scenarios and failure patterns improves generalization. If true, this gives fab owners a practical path: a derivative-free evolution strategy running many parallel simulator instances can beat hand-tuned dispatching heuristics on realistic industrial data, without the credit-assignment and value-estimation problems that hurt PPO in this setting.","feed_headline":"Evolution strategies cut fab tardiness by 29.4%","feed_subtitle":"Policy-gradient learning fails on real factory data, while evolution strategies keep improving as models grow.","key_machinery":"The load-bearing object is the policy network trained by CMA-ES: a small attention-based network (scaled dot-product attention over lot-wise observations, followed by a feed-forward scoring head) that ranks the lots waiting at a tool. CMA-ES treats the network weights as a black-box parameter vector, samples perturbed candidates in parallel, runs a full fab simulation episode for each candidate, and updates the mean and covariance of the search distribution toward the candidates with the best normalized tardiness-and-throughput cost. The other central piece is the experiment design: every model runs on the same commercial discrete-event simulator with the same observation and action interface, so the only variables changed are the optimizer, the set of controlled tool groups, and the training data mix. The attention mechanism is what lets the policy handle queues of arbitrary length, while the episode-level cost function is what lets the evolution strategy sidestep the credit-assignment problem that the paper identifies as PPO's bottleneck.","core_discovery":"On the paper's own terms, the central discovery is that the scalability of reinforcement learning for fab dispatching is determined less by the sophistication of the learning algorithm than by how it handles delayed, noisy rewards inside a large simulator. CMA-ES assigns one cost per full simulation episode, samples perturbations of the policy parameters in parallel, and uses the covariance of successful perturbations to guide the next search; that black-box structure fits the sparse, long-delayed reward signal of a wafer fab. PPO, by contrast, needs frequent state-action-reward samples and an accurate value-function estimate, both of which become unreliable when an episode spans thousands of actions and weeks of simulated time. The measured outcome is that CMA-ES produced a 29.4% average tardiness improvement on Minifab when all tool types were controlled, double-digit tardiness gains on SMT2020, and up to 4% tardiness and 1% throughput improvement on the industrial dataset, while PPO improved tardiness only on Minifab and made it worse on SMT2020 and industry. The paper also finds that the share of dispatching decisions placed under the agent's control matters more than which single tool is optimized: batching tools, for example, only pay off when the agent also controls the upstream tools that feed them.","pith_inferences":["If this scalability ordering holds beyond the particular simulator, derivative-free evolution-strategy policies could transfer to other sparse-reward production scheduling problems, such as back-end testing or reticle logistics, where the credit-assignment problem is similarly severe.","The authors' memory constraint suggests a concrete extension: a PPO variant that compresses or replays episode samples could recover some of the lost performance at industrial scale, since the paper's own data show complete-episode updates are more stable than truncated ones.","A cheap practical recipe implied by the tool-combination results is to run the small Minifab model first to select the bottleneck tool set, then train only on those tools for the expensive industrial model, reducing the training hours documented in the paper.","Training a CMA-ES agent on a deliberately diverse set of loading scenarios rather than a fixed one should reduce the over-optimization of a single training scenario that the paper observes, since its generalization experiments link diversity to robustness."],"forward_implications":["On fab models of realistic size and complexity, evolution-strategy optimization should be the preferred reinforcement-learning choice over policy-gradient methods, because long-delayed and noisy rewards break value-function learning.","The more dispatching decisions the agent controls, especially batching tools together with their upstream feeders, the larger the tardiness improvement, so choosing which bottleneck tool groups to control is part of the optimization itself.","Training on multiple loading scenarios and random seeds produces agents that generalize to unseen scenarios and longer time horizons, though with some loss of improvement.","Training cost is dominated by simulator runtime and grows with the share of controlled tools, but wall-clock time scales with the number of CPU cores used, so the method is embarrassingly parallel.","A policy that beats a fab's current expert-tuned heuristics by 4% tardiness and 1% throughput on industrial data is operationally significant, since fab capacity carries a large weekly cost."],"supporting_citations":[{"why":"Supplies the evolution-strategies-as-scalable-RL formulation that the CMA-ES approach builds on.","marker":"[49]"},{"why":"Prior application of CMA-ES to real frontend-fab dispatching that this paper extends to multiple testbeds.","marker":"[56]"},{"why":"Provides the commercial discrete-event simulator used as the common evaluation platform for all three models.","marker":"[9]"},{"why":"Defines the SMT2020 benchmark, its fab models, and the baseline dispatching rules.","marker":"[25]"},{"why":"Defines the Minifab model and its layout, including the batching tools.","marker":"[55]"},{"why":"Supplies the PPO algorithm used as the policy-gradient baseline.","marker":"[52]"},{"why":"Supplies the covariance-matrix-adaptation update rule used by the ES optimizer.","marker":"[19]"},{"why":"Provides the distributed reinforcement-learning implementation used for the PPO experiments.","marker":"[30]"}],"fun_headline_variants":["Evolution strategies slash fab tardiness by 29%","CMA-ES scales better than PPO for fab dispatch","Black-box optimization wins in semiconductor scheduling","Fab scheduling: evolution strategies beat deep RL","29% better tardiness with evolution strategies in fabs"],"cache_read_input_tokens":23424,"weakest_assumption_plain":"The entire comparison rests on the assumption that the commercial simulator evaluates all three fab models faithfully and that the industry dataset's due dates and loading scenarios are realistic ground truth; if either is distorted, the measured improvements and the claimed ordering of the two methods may not transfer to other fabs.","fun_headline_variants_meta":{"raw":{"variants":["Evolution strategies slash fab tardiness by 29%","CMA-ES scales better than PPO for fab dispatch","Black-box optimization wins in semiconductor scheduling","Fab scheduling: evolution strategies beat deep RL","29% better tardiness with evolution strategies in fabs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000223,"raw_usage":{"total_tokens":1517,"prompt_tokens":1064,"completion_tokens":453,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":680,"completion_tokens_details":{"reasoning_tokens":380}},"tokens_in":680,"tokens_out":453,"duration_ms":4666,"temperature":1.0,"reasoning_tokens":380,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:56:35.526095+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same CMA-ES and PPO training pipelines on an independently written discrete-event simulator implementation of the Minifab and SMT2020 models with identical due dates and random seeds: if PPO matches or beats CMA-ES there, or if CMA-ES's advantage disappears, the scalability ordering is an artifact of the particular simulator. A second check: deploy the CMA-ES policy on a different real fab with different tool dedications and load mixes and measure tardiness against that fab's own heuristics; the claim fails if the improvement does not reproduce or turns negative.","supporting_citations":[{"cited_title":"URL http://www.d-simlab.com/ category/d-simcon/products-d-simcon/forecaster-and-scenario-manager/","cited_arxiv_id":null,"evidence_quote":"Provides the commercial discrete-event simulator used as the common evaluation platform for all three models."},{"cited_title":"IEEE Transactions on Semiconductor Manufacturing 33(4):522–","cited_arxiv_id":null,"evidence_quote":"Defines the SMT2020 benchmark, its fab models, and the baseline dispatching rules."},{"cited_title":"In: ASMC","cited_arxiv_id":null,"evidence_quote":"Defines the Minifab model and its layout, including the batching tools."},{"cited_title":"Evol Comput 9(2):159–195","cited_arxiv_id":null,"evidence_quote":"Supplies the covariance-matrix-adaptation update rule used by the ES optimizer."},{"cited_title":"In: ICML, Proceedings of Machine Learning Research, vol 80","cited_arxiv_id":null,"evidence_quote":"Provides the distributed reinforcement-learning implementation used for the PPO experiments."}],"review_version":1}