REVIEW 3 major objections 4 minor 33 references
An LLM agent can schedule chip design optimization as well as human engineers, with far less manual effort.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-02 18:12 UTC pith:ZT7E7IZQ
load-bearing objection A genuinely new LLM-agent framework for long-horizon ECO scheduling, with sensible ablations and a real tool evaluation — but the headline margins come from unseeded single runs and a hidden knowledge store, so referee with a request for variance and artifacts. the 3 major comments →
Retrieve, Schedule, Reflect: LLM Agents for Chip QoR Optimization
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The paper's central claim is that chip QoR optimization can be treated as an LLM-driven scheduling problem, not a single-shot parameter search. It introduces a Retrieve–Schedule–Reflect loop: a scheduler queries a database of natural-language scheduling paths (the search tree), an executor turns the retrieved advice into legal EDA tool commands, and a reflector distills Pareto-optimal historical traces into scheduling and execution guidance for the next round. The authors report that on 8 ASAP7 open-source designs, this framework achieves post-optimization total slack leaving only 0.23% of the original negative slack unsolved (versus 10.92% for a reinforcement-learning baseline and 0.51% for
What carries the argument
The central object is the scheduling search tree (ST): a set of nodes, each containing an objective (timing/power/area), a natural-language design-state description, a strategy snippet, and tool-manual references. Paths through the ST are extracted into a database D; at runtime the scheduler embeds the current design state and QoR history, retrieves the top-k most cosine-similar paths, and uses them as step-by-step scheduling templates. The second load-bearing mechanism is Pareto-driven reflection: terminal QoR vectors from completed rounds are filtered to the Pareto front, and a reflector agent writes two short natural-language summaries—one about objective ordering (scheduling reflection)
Load-bearing premise
The search tree—the paper's claimed source of 'natural-language expertise'—is asserted to be built from academic papers and engineer input, but its contents are not released or demonstrated; if the tree is thin, ad hoc, or tuned to the benchmark, the RAG advantage evaporates.
What would settle it
Inspect the released search tree and count the number of distinct paths per objective and design-state type; then run the framework on a new set of designs from a different PDK or with a randomly perturbed tree, and check whether the claimed total-slack improvement and the RAG-ablation gap (from -8.44e2 to -1.56e3 in Table 3) still appear. If the gap shrinks to noise or the framework fails on unfamiliar designs, the central claim that expert-grounded RAG is essential would be falsified.
If this is right
- If the framework genuinely matches human-engineer QoR, design teams could reduce manual ECO scheduling effort, reserving engineers for final review rather than iterating optimization runs.
- The reported 4× speedup over RL suggests that knowledge-grounded search can cut the exploration cost of black-box optimization, a pattern that could transfer to other engineering domains with iterative tool usage.
- Natural-language trade-off instructions become a practical way to steer multi-objective optimization, e.g., explicitly trading timing margin for power, without redefining a reward function.
- The Pareto-reflection mechanism provides a general recipe: any iterative optimizer that can summarize its own history in text can improve across restarts, which extends beyond chips to other design flows.
- The open-sourced code and data would let others test whether the framework's advantage depends on the specific search tree and models, or whether it persists across new designs and process nodes.
Where Pith is reading between the lines
- A testable extension is to blind-audit the search tree: if the same tree is applied to designs outside the ICCAD'24 ASAP7 benchmark (different process nodes, macros, or clock frequencies), the RAG ablation gap should persist; if it collapses, the tree is likely overfit to the benchmark.
- The ablation table suggests that reflection contributes more than RAG on timing (removing reflection degrades total slack from -8.44e2 to -1.83e3, while removing RAG degrades to -1.56e3). An editor might infer that the framework's robustness depends more on cross-round feedback than on the specific knowledge base, which has implications for how easily it can be ported.
- The paper reports LLM token costs of $1.25 for a full run on GPT-5-mini; if LLM inference costs continue to fall, the economic barrier to using such agents in production EDA flows would shrink enough to make the approach practical for routine design closure.
- A natural follow-up study would stress-test the language-instruction capability by asking the agent to follow conflicting or unusual trade-offs (e.g., 'minimize leakage only' or 'fix only hold time'), checking whether it still respects hard constraints.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes an agentic LLM framework, Retrieve-Schedule-Reflect, for automating the long-horizon scheduling of iterative chip QoR optimization in commercial EDA tools. Three agents (scheduler, executor, reflector) operate in rounds: the scheduler retrieves natural-language scheduling templates from a search tree via RAG, the executor translates advice into legal EDA commands using tool-manual retrieval, and the reflector summarizes Pareto-optimal traces into scheduling and execution feedback. Experiments on 8 ASAP7 designs from the ICCAD'24 dataset report that the framework outperforms an A3C RL baseline in timing improvement (average total slack ratio 0.23% vs 10.92%), achieves better or comparable power/area, runs in about 3.5h vs 15.4h, and is comparable to a human engineer. Ablations show that removing RAG or reflection degrades timing, and the framework can follow natural-language preferences such as allowing 5% worst-slack degradation to reduce combinational power.
Significance. If the empirical claims are robust, the work would be a useful step toward automating a labor-intensive part of chip design: the schedule-execute-reflect loop currently requiring expert engineers. The design is attractive: human-interpretable search-tree templates, RAG for knowledge grounding, Pareto-driven reflection, and the ability to express custom objectives in natural language are all credible and practical contributions. The authors commit to releasing code and design data, which would strengthen reproducibility. However, the current evidence is not yet sufficient to establish the central quantitative claims: all headline comparisons rest on single runs of a temperature=1 LLM pipeline with no seed control, and the knowledge store that drives the RAG advantage is not released or fully specified. The result is promising but needs additional statistical validation and transparency before the claims can be accepted.
major comments (3)
- [§5.1, §B.1, Tables 1–3] All reported QoR numbers are single-run point estimates from a stochastic pipeline: temperature=1, top_p=1 for every LLM call (B.1), no seed is reported, and the RL baseline itself is stochastic. The central claim of “10% better timing” and the RAG ablation effect (Table 3: Ours total slack -8.44e2 vs w/o-RAG -1.56e3) could be within run-to-run variation. I request repeated runs with multiple seeds (at least 3–5) for the full system, the RAG/reflection ablations, and the RL baseline, with means and standard deviations or confidence intervals. Without this, Contribution 4 is not statistically established.
- [§3.1, Appendix A] The search tree (ST) is the central source of “expertise grounding,” but its construction is asserted rather than demonstrated: the paper states it is “built from academic papers and reports” and “filtered by engineers” (Appendix A), yet only one example path is shown, and the full tree, filtering protocol, and coverage criteria are not released. Since the w/o-RAG ablation roughly doubles the remaining total slack (Table 3), the claimed advantage depends critically on the quality and transferability of this ST. I ask the authors to release the ST (or a substantial excerpt) and to provide concrete coverage criteria, construction details, and some evidence that the ST is not tuned to the specific 8 benchmark designs (e.g., evaluation on a held-out design or a different PDK).
- [§5.1, Table 1 (Human baseline)] The human baseline is described only as “asking an experienced engineer to manually schedule the optimization process for 10 rounds.” No information is given about the engineer’s experience, instructions, or inter-person variability. The human row shows exactly zero total slack on five of eight designs, which is surprising and suggests that different stopping criteria or effort levels may have been used. The comparison “comparable to human engineers” is therefore not yet well supported. Please specify the protocol, the engineer’s background, the exact objective/budget given, and ideally report results from more than one engineer or repeated sessions.
minor comments (4)
- [§2.4 and elsewhere] Typo: “an multi-agent framework” should be “a multi-agent framework.” Also “scheudling” in the introduction should be “scheduling.”
- [References] The reference list contains duplicate entries for Mnih et al. (2016a and 2016b), which are the same paper; the in-text citation in B.3 uses 2016b. Also, §5.4 cites DeepSeek-V3.2 as (Liu et al., 2025b), but the reference list associates DeepSeek-V3.2 with Liu et al. (2025a); the 2025b entry is for MM-Agent. Please correct these.
- [§5.5, Figure 5] The runtime breakdown pie charts are hard to read in the current rendering; consider replacing them with a stacked bar chart or a table with percentages.
- [§5.1] The statement that code segments restricted by the tool license are replaced with “mock equivalents” is helpful, but it would be useful to state explicitly which parts are mocked and how a reader can substitute real commands; otherwise the reproducibility promise is hard to evaluate.
Circularity Check
No significant circularity: the core empirical claims are anchored by external EDA reports (PrimeECO) and independent benchmark designs, with no fitted parameter renamed as a prediction.
full rationale
The paper's contribution is an evaluated agent workflow, not a derivation whose output is forced by its inputs. The QoR numbers in Tables 1-3 are measured by an external golden tool: 'All optimizations and report generation are performed using Synopsys PrimeECO, a widely used golden tool for QoR analysis and ECO optimization.' The search tree used for RAG is constructed externally from 'academic papers and reports with engineer input so that the encoded schedules reflect realistic optimization practice rather than arbitrary sequences' (Sec. 3.1, Appendix A), and the paper does not report fitting any ST node or retrieval parameter to the target QoR values of the eight ICCAD'24 ASAP7 designs. Equations (1)-(12) define representations (node, path, cosine similarity, top-k, trace, Pareto set) and do not contain a parameter fitted to the reported outcomes; the Pareto filter is a selection rule, not a predictor. The only overlapping-author citation is Bai et al. 2024 for the A3C RL baseline: 'We develop and A3C RL algorithm (Mnih et al., 2016b) that is used in EDA tasks (Bai et al., 2024) as baseline.' This is an implementation reference for a comparison method, with hyperparameters given in B.3, and the resulting outputs are externally measured; it is not load-bearing for the paper's central claim. Legitimate concerns remain — the full ST content is not released, all LLM runs use temperature=1/top_p=1 with no seeds (B.1), and single-run point estimates are reported — but these are reproducibility/statistical-validity risks, not circularity under the quoted reduction test.
Axiom & Free-Parameter Ledger
free parameters (6)
- Number of rounds (R_max) =
10
- Iterations per round (I_max) =
10
- RAG top-k =
not reported
- LLM sampling temperature / top-p =
1.0 / 1.0
- RL baseline reward weights and hyperparameters =
TS=10, WS=3, power=3, area=3; violation bonus=10; gamma=0.99, lr=1e-3
- Hand-written prompt templates =
as in Appendix D
axioms (5)
- ad hoc to paper The search tree encodes valid, transferable expert scheduling knowledge.
- domain assumption Slack (WNS/TNS) is an adequate primary measure of timing quality, and negative slack means functional failure.
- domain assumption GPT-5-mini reliably interprets EDA reports and generates legal EDA commands when grounded by retrieved manual snippets.
- standard math Pareto dominance over (timing, power, area) terminal vectors is the correct optimality criterion for trace selection.
- domain assumption The ICCAD'24 ASAP7 benchmark designs are representative of practical ECO workloads.
invented entities (1)
-
Search-tree path database D of natural-language scheduling templates
no independent evidence
read the original abstract
Modern chip design requires multi-objective optimization of timing, power, and area under stringent time-to-market constraints. Although powerful optimization algorithms are integrated into EDA tools, achieving high QoR hinges on effective long-horizon scheduling, which relies heavily on manual expert intervention. To address this issue and automate chip design, we propose an agentic LLM framework that schedules chip optimizations through direct interaction with EDA tools. The agent is grounded in natural language expertise expressed as a search tree through retrieval-augmented generation (RAG). We further improve scheduling quality with Pareto-driven QoR feedback through language reflection. Experimental results show that, compared with black-box search methods such as reinforcement learning, our framework achieves 10% greater timing improvement while consuming less power and area, with more than 4x speedup. The post-optimization QoR is also comparable to that achieved by human experts. Finally, the agent supports customized tasks expressed in natural language, enabling preferential QoR trade-offs. The code and chip design data will be publicly available at https://github.com/YiKangOY/Open-LLM-ECO.
Figures
Reference graph
Works this paper leans on
-
[1]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...
-
[2]
Agnesina, A., Chang, K., and Lim, S. K. Vlsi placement parameter optimization using deep reinforcement learning. In Proceedings of the 39th international conference on computer-aided design, pp.\ 1--9, 2020
2020
-
[3]
and Blaauw, D
Ajayi, T. and Blaauw, D. Openroad: Toward a self-driving, open-source digital layout implementation tool chain. In Proceedings of Government Microcircuit Applications and Critical Technology Conference, 2019
2019
-
[4]
Bai, C., Zhai, J., Ma, Y., Yu, B., and Wong, M. D. Towards automated risc-v microarchitecture design with reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp.\ 12--20, 2024
2024
-
[5]
T., Vashishtha, V., Shifren, L., Gujja, A., Sinha, S., Cline, B., Ramamurthy, C., and Yeric, G
Clark, L. T., Vashishtha, V., Shifren, L., Gujja, A., Sinha, S., Cline, B., Ramamurthy, C., and Yeric, G. Asap7: A 7-nm finfet predictive process design kit. Microelectronics Journal, 53: 0 105--115, 2016
2016
-
[6]
Timing and design closure in physical design flows
Coudert, O. Timing and design closure in physical design flows. In Proceedings International Symposium on Quality Electronic Design, pp.\ 511--516. IEEE, 2002
2002
-
[7]
and Dunlop, A
Fishburn, J. and Dunlop, A. A polynomial programming approach to transistor sizing. In Proc. of ICCAD’85, pp.\ 326--328, 1985
1985
-
[8]
Ptpt: Physical design tool parameter tuning via multi-objective bayesian optimization
Geng, H., Chen, T., Ma, Y., Zhu, B., and Yu, B. Ptpt: Physical design tool parameter tuning via multi-objective bayesian optimization. IEEE transactions on computer-aided design of integrated circuits and systems, 42 0 (1): 0 178--189, 2022
2022
-
[9]
B., Kundu, S., and Wang, Z
Ghose, A., Kahng, A. B., Kundu, S., and Wang, Z. Orfs-agent: Tool-using agents for chip design optimization. In IEEE/ACM International Symposium on Machine Learning for CAD (MLCAD), 2025
2025
-
[10]
Chateda: A large language model powered autonomous agent for eda
He, Z., Wu, H., Zhang, X., Yao, X., Zheng, S., Zheng, H., and Yu, B. Chateda: A large language model powered autonomous agent for eda. arXiv preprint arXiv:2308.10204, 2023
Pith/arXiv arXiv 2023
-
[11]
u ttler, H., Lewis, M., Yih, W.-t., Rockt \
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., K \"u ttler, H., Lewis, M., Yih, W.-t., Rockt \"a schel, T., et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems, 33: 0 9459--9474, 2020
2020
-
[12]
Liu, A., Mei, A., Lin, B., Xue, B., Wang, B., Xu, B., Wu, B., Zhang, B., Lin, C., Dong, C., et al. Deepseek-v3. 2: Pushing the frontier of open large language models. arXiv preprint arXiv:2512.02556, 2025 a
Pith/arXiv arXiv 2025
-
[13]
Mm-agent: Llm as agents for real-world mathematical modeling problem
Liu, F., Yang, Z., Liu, C., Song, T., Gao, X., and Liu, H. Mm-agent: Llm as agents for real-world mathematical modeling problem. arXiv preprint arXiv:2505.14148, 2025 b
Pith/arXiv arXiv 2025
-
[14]
Chipnemo: Domain-adapted llms for chip design
Liu, M., Ene, T.-D., Kirby, R., Cheng, C., Pinckney, N., Liang, R., et al. Chipnemo: Domain-adapted llms for chip design. arXiv preprint arXiv:2311.00176, 2023
Pith/arXiv arXiv 2023
-
[15]
Lu, Y.-C., Nath, S., Khandelwal, V., and Lim, S. K. Rl-sizer: Vlsi gate sizing for timing optimization using deep reinforcement learning. In 2021 58th ACM/IEEE Design Automation Conference (DAC), pp.\ 733--738. IEEE, 2021
2021
-
[16]
Lu, Y.-C., Chan, W.-T., Guo, D., Kundu, S., Khandelwal, V., and Lim, S. K. Rl-ccd: concurrent clock and data optimization using attention-based self-supervised reinforcement learning. In 2023 60th ACM/IEEE Design Automation Conference (DAC), pp.\ 1--6. IEEE, 2023 a
2023
-
[17]
Lu, Y.-C., Nath, S., Pentapati, S., and Lim, S. K. Eco-gnn: Signoff power prediction using graph neural networks with subgraph approximation. ACM Transactions on Design Automation of Electronic Systems, 28 0 (4): 0 1--22, 2023 b
2023
-
[18]
P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., Silver, D., and Kavukcuoglu, K
Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., Silver, D., and Kavukcuoglu, K. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pp.\ 1928--1937. PmLR, 2016 a
1928
-
[19]
P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., Silver, D., and Kavukcuoglu, K
Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., Silver, D., and Kavukcuoglu, K. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pp.\ 1928--1937. PmLR, 2016 b
1928
-
[20]
L., Li, Y., Gaillardon, P.-E., and Yu, C
Neto, W. L., Li, Y., Gaillardon, P.-E., and Yu, C. Flowtune: End-to-end automatic logic optimization exploration via domain-specific multiarmed bandit. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 42 0 (6): 0 1912--1925, 2022
1912
-
[21]
Strongly np-hard discrete gate-sizing problems
Ning, W. Strongly np-hard discrete gate-sizing problems. IEEE transactions on computer-aided design of integrated circuits and systems, 13 0 (8): 0 1045--1051, 1994
1994
-
[22]
Alphasyn: Logic synthesis optimization with efficient monte carlo tree search
Pei, Z., Liu, F., He, Z., Chen, G., Zheng, H., Zhu, K., and Yu, B. Alphasyn: Logic synthesis optimization with efficient monte carlo tree search. In 2023 IEEE/ACM International Conference on Computer Aided Design (ICCAD), pp.\ 1--9. IEEE, 2023
2023
-
[23]
Z., and Yu, B
Pu, Y., Ji, Y., Yu, S., Zhang, Z., Guo, Z., He, Z., Lin, Y., Pan, D. Z., and Yu, B. Gpu acceleration for versatile buffer insertion. In IEEE/ACM International Conference on Computer-Aided Design (ICCAD), 2025
2025
-
[24]
D., Ermon, S., and Finn, C
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 36: 0 53728--53741, 2023
2023
-
[25]
Sharma, T., Kolluru, S., and Stevens, K. S. Learning based timing closure on relative timed design. In IFIP/IEEE International Conference on Very Large Scale Integration-System on a Chip, pp.\ 133--148. Springer, 2020
2020
-
[26]
and Li, Z
Shi, W. and Li, Z. A fast algorithm for optimal buffer insertion. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 24 0 (6): 0 879--891, 2005
2005
-
[27]
Reflexion: Language agents with verbal reinforcement learning
Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K., and Yao, S. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36: 0 8634--8652, 2023
2023
-
[28]
Primeclosure: Ai-driven signoff eco solution
Synopsys . Primeclosure: Ai-driven signoff eco solution. https://www.synopsys.com/implementation-and-signoff/signoff/primeclosure.html, 2022. Accessed: 2026-01-26
2022
-
[29]
Van Ginneken, L. P. Buffer placement in distributed rc-tree networks for minimal elmore delay. In 1990 IEEE International Symposium on Circuits and Systems (ISCAS), pp.\ 865--868. IEEE, 1990
1990
-
[30]
V., Zhou, D., et al
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35: 0 24824--24837, 2022
2022
-
[31]
and Kim, T
Won, D. and Kim, T. Eda tool parameter optimization through fast and reliable machine-learning based rank prediction model. In 2025 ACM/IEEE 7th Symposium on Machine Learning for CAD (MLCAD), pp.\ 1--7. IEEE, 2025
2025
-
[32]
Wu, B.-Y., Liang, R., Pradipta, G., Agnesina, A., Ren, H., and Chhabria, V. A. 2024 iccad cad contest problem c: Scalable logic gate sizing using ml techniques and gpu acceleration. In Proceedings of the 43rd IEEE/ACM International Conference on Computer-Aided Design, pp.\ 1--5, 2024
2024
-
[33]
React: Synergizing reasoning and acting in language models
Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., and Cao, Y. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2023
Pith/arXiv arXiv 2023
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.