PCBWorld: A Benchmark Environment for Engine-Grounded PCB Design Automation

Han-Seul Jeong; HyungSeok Song; Junseok Park; Seohui Bae; Soonyoung Lee; Won-Seok Choi; Youngjoon Park

PCBWorld: Agents Route Real Circuit Boards Through KiCad's Native Engine

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

2026-07-08 20:42 UTC pith:OUMEFHAK

load-bearing objection PCBWorld: Environment is the real contribution; headline transfer claim needs CP@1 to be interpretable the 3 major comments →

arxiv 2607.05915 v2 pith:OUMEFHAK submitted 2026-07-07 cs.AI

PCBWorld: A Benchmark Environment for Engine-Grounded PCB Design Automation

Hyungseok Song , Junseok Park , Won-Seok Choi , Seohui Bae , Han-Seul Jeong , Youngjoon Park , Soonyoung Lee This is my paper

classification cs.AI

keywords PCB routingreinforcement learningKiCaddesign rule checkinteractive agentengine-grounded environmentLLM tool usezero-shot transfer

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper introduces PCBWorld, an open-source environment that wraps the KiCad EDA engine into a Gymnasium-style reinforcement learning interface, exposing 58 Python APIs that mirror the step-level routing operations a human engineer uses. The central claim is that when agents—whether RL policies or tool-using LLMs—route PCBs by issuing segment-level commands through the engine's native API and observing the resulting board state plus Design Rule Check feedback after each action, they substantially outperform agents that either move cell-by-cell on a discretized grid or generate entire board files in a single open-loop pass. The paper demonstrates this with a compact PPO Transformer policy trained from scratch on simple synthetic gridless boards (4–6 nets, 8–21 pads) that transfers zero-shot to the 100 smallest real open-source boards, achieving a Clean Pass rate of 0.86 versus Freerouting's 0.80, while maintaining performance across all grid resolutions where grid-action RL baselines collapse. For LLM agents, interactive routing through the engine yields a Clean Pass of 0.96 on synthetic boards for the largest model, compared to 0.55 for plan-only generation and 0.00 for engine-free file generation. The paper also releases PCBWorld-Bench, comprising three dataset families (synthetic grid-based, synthetic gridless, and 679 real open-source boards in KiCad's native format) scored by eight engine-checked metrics, establishing a shared evaluation protocol under which all routing methods—whether they use PCBWorld or not—are scored identically.

Core claim

The paper's central object is the engine-grounded interactive routing loop: an agent invokes KiCad's native routing operations (start_route, make_line, make_via, finish), observes the engine-updated board state and DRC feedback, and routes one operation at a time. This loop replaces two prior abstractions—grid-cell movement, whose decision horizon grows with grid resolution and breaks credit assignment, and open-loop generation, which cannot guarantee design-rule compliance. By tying the action granularity to routed segments rather than grid cells, the PPO policy's decision horizon stays constant as grids grow finer, which is why it maintains perfect routability from grid 50 through grid 500

What carries the argument

KiCad's Push-and-Shove (PNS) router and Design Rule Check engine, exposed as 58 Python APIs; a Markov Decision Process with a nested-dictionary board state, six action types, and a terminal reward based on a potential function combining design-rule violations, wirelength, and via count; two wrappers (a tokenized RL interface and a serialized tool-call LLM interface) over the same MDP; a log-concave per-net DRC penalty that concentrates the learning signal on closing the last violations.

Load-bearing premise

The zero-shot transfer claim rests on the D3-A evaluation set being representative of real boards, but D3-A contains only the 100 smallest boards from a 679-board corpus (2–13 nets, 6–31 pads), which overlaps heavily with the synthetic training distribution (4–6 nets, 8–21 pads). On medium boards (5–42 nets, 31–100 pads), the PPO policy's Clean Pass drops to 0.45 versus Freerouting's 0.78, and no results are reported for the largest boards. Additionally, the best-of-5 rollout

What would settle it

Train a grid-action RL policy whose action granularity scales with routed segments rather than grid cells, and show it matches PCBWorld's PPO on the D1 grid-scalability benchmark—this would isolate action abstraction rather than engine grounding as the causal factor. Alternatively, show that a plan-only LLM agent with access to a high-fidelity board simulator for mental rehearsal matches the interactive agent's Clean Pass, which would weaken the claim that per-step engine feedback is essential.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

If engine-grounded interactive routing is the right abstraction, then the path to competitive learning-based PCB routing runs through native EDA APIs rather than through grid discretization or end-to-end file generation, redirecting both RL and LLM agent design.
The finding that a policy trained on 4–6 net synthetic boards generalizes to small real boards suggests that synthetic board generators with controllable difficulty could serve as scalable training curricula, though the performance drop on medium boards (CP 0.45 vs Freerouting's 0.78) indicates the curriculum must extend well beyond current parameters.
The sharp gap between interactive and open-loop LLM routing (CP 0.96 vs 0.55 vs 0.00) provides a concrete test bed for the broader hypothesis that iterative tool-use with state feedback is necessary for tasks requiring hard geometric constraints, extending the interactive-agent paradigm into spatial reasoning domains.
The released benchmark, with 679 real boards spanning up to 451 nets and 2,103 pads, establishes difficulty tiers that current methods cannot solve, creating a measurable frontier for tracking progress in automated PCB routing.
The pluggable reward potential allows routing objectives to be swapped (e.g., prioritizing signal integrity or thermal constraints instead of just wirelength and via count), opening the environment to multi-objective optimization once those signals are integrated into the engine.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper introduces PCBWorld, an open-source engine-grounded PCB routing environment built on the KiCad EDA engine. The environment exposes 58 Python APIs wrapping KiCad's native routing and design-rule-check (DRC) operations, formulates PCB routing as an MDP, and provides wrappers for both RL policies and tool-using LLM agents. Alongside the environment, the authors release PCBWorld-Bench, comprising synthetic grid-based (D1) and gridless (D2) boards plus 679 real open-source boards (D3) in KiCad's native format. Evaluation is method-agnostic: any completed .kicad_pcb file is scored by the same eight engine-checked metrics. Experiments show that (i) segment-level KiCad-API actions scale to fine grid resolutions where grid-action RL baselines collapse, (ii) a compact PPO Transformer trained on synthetic D2 boards transfers zero-shot to small real boards (D3-A), approaching or exceeding rule-based routers, and (iii) interactive engine-grounded routing substantially outperforms open-loop LLM generation.

Significance. The paper makes a solid infrastructure contribution: wrapping KiCad's C++ routing engine as a vectorized Gym environment with a well-documented API-to-MDP mapping is non-trivial and useful to the community. The method-agnostic evaluation protocol—scoring any .kicad_pcb file identically regardless of how it was produced—is a genuine strength that enables fair comparison across heterogeneous routing methods. The reward design, including the log-concave DRC penalty with breadth/depth decomposition, is well-motivated. The interactive-vs-open-loop LLM comparison provides a clear, falsifiable result. The code and datasets are promised as open source. The work is positioned as a foundation rather than a final solution, which is appropriate given the limitations the authors themselves acknowledge.

major comments (3)

§4.3 and Table 3: The best-of-5 rollout selection protocol (selecting the rollout with highest potential gain) is applied to all stochastic methods, but its effect is asymmetric across method types. PPO is a stochastic neural policy whose 5 rollouts produce diverse routings, so best-of-5 selection can substantially inflate CP: if single-rollout success probability is p, then CP@5 >= 1-(1-p)^5. For example, CP@1=0.50 yields CP@5>=0.97. Freerouting, while stochastic across seeds, is a deterministic rip-up-and-reroute algorithm whose 5 rollouts within a single seed may be near-identical, giving CP@5 approx CP@1. The paper reports CP@5 in Table 3 but does not report CP@1 for the main comparison, making it impossible to assess whether PPO's 0.86 vs Freerouting's 0.80 on D3-A reflects genuine routing quality or selection advantage. The paper does report single-rollout Rout.@1 for D1 (Figure 6,
§4.2 and Table 2: The zero-shot transfer claim rests on D3-A being representative of 'real boards.' D3-A contains only the 100 smallest boards from the 679-board D3 corpus (2-13 nets, 6-31 pads), while the PPO policy was trained on D2 boards with 4-6 nets and 8-21 pads. The distribution overlap between D2-train and D3-A is substantial, and the transfer is not as out-of-distribution as the 'zero-shot to real boards' framing suggests. On D3-B (5-42 nets, 31-100 pads), PPO's CP drops to 0.45 versus Freerouting's 0.78, and on D3-C no results are reported. The claim that the policy 'approaches rule-based routers' is supported only on the smallest, most D2-like subset of real boards. The authors should temper the zero-shot transfer framing or report results on D3-C to substantiate the generalization claim.
§5.3, Table 3: The D3-A evaluation uses 99 boards (one 73-net board excluded), while D3-B uses only 10 boards. With 4 seeds, D3-B CP for PPO is 0.45 +/- 0.10, meaning the 95% confidence interval is roughly [0.25, 0.65]. The comparison to Freerouting's 0.78 +/- 0.05 on D3-B is based on only 10 boards x 4 seeds = 40 data points. The paper should report the board-level variance (not just seed-level variance) or acknowledge that the D3-B comparison is underpowered.

minor comments (7)

§5.3, Table 3: The paper states 'deterministic methods are run once, as selection does not apply.' However, Freerouting is listed as a stochastic method reporting mean over 4 seeds. The distinction between Freerouting's stochasticity (across seeds) and the within-seed determinism of its 5 rollouts should be clarified in the table caption or methodology.
Table 2: D3-A is described as '100 boards' in the text but Table 3 reports '99 Boards' for D3-A. The discrepancy (one excluded board) is mentioned only in Appendix L. A footnote in the main table would help.
Figure 7: The y-axis label for the DRV subplot in Figure 7a shows values around 0-30 but the axis label is unclear about whether these are per-board averages or totals. Clarifying the aggregation would help interpretation.
§3.2, Eq. (2): The potential function uses lambda_w and lambda_v 'reused from Equation (1),' but Equation (1) presents these as user-specified weights for the constrained optimization. The relationship between the optimization weights and the reward weights could be stated more precisely—are they always identical, or can they differ for training vs. evaluation?
Appendix G, Eq. (5): The DRC penalty hyperparameters (s_agg, s_pn, o) = (3, 1, 2) are stated without justification. A brief sensitivity analysis or rationale for these specific values would strengthen the reproducibility of the reward design.
The paper uses 'GPT-5.4' and 'Qwen3.5-397B' as model names. If these are anonymized or renamed versions of specific models, the actual model identifiers should be provided for reproducibility.
§7 (Discussion): The paragraph on 'explainable, reasoning-based routing' is cut off mid-sentence ('we identify several key observations, which are presented in Appendix N'). While Appendix N does contain the case studies, the main text discussion would benefit from a brief summary of the key findings rather than only deferring to the appendix.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for a careful and constructive review. The three major comments all identify legitimate gaps in our experimental reporting: (1) the best-of-5 selection protocol's asymmetric effect on PPO vs. Freerouting is not controlled for, (2) the zero-shot transfer claim is overstated for D3-A given its distributional overlap with D2-train, and (3) the D3-B comparison is underpowered. We agree with all three points and will revise accordingly—adding CP@1 results, tempering the transfer framing, and reporting board-level variance. No standing objections remain.

read point-by-point responses

Referee: §4.3 and Table 3: The best-of-5 rollout selection protocol is applied to all stochastic methods, but its effect is asymmetric across method types. PPO is a stochastic neural policy whose 5 rollouts produce diverse routings, so best-of-5 selection can substantially inflate CP... Freerouting, while stochastic across seeds, is a deterministic rip-up-and-reroute algorithm whose 5 rollouts within a single seed may be near-identical... The paper reports CP@5 in Table 3 but does not report CP@1 for the main comparison...

Authors: The referee is correct that the best-of-5 selection protocol can asymmetrically inflate CP for PPO relative to Freerouting, and that reporting only CP@5 makes it impossible to assess the magnitude of this effect. This is a fair and important concern. We will add CP@1 (single-rollout, no selection) alongside CP@5 in Table 3 for all stochastic methods on D2, D3-A, and D3-B. This will allow readers to directly assess the selection effect. We note that Figure 6 already reports single-rollout Rout.@1 for D1, but the main D2/D3 comparison table does not include the analogous CP@1, which is an oversight we will correct. We will also add a paragraph in §4.3 explicitly discussing the selection effect, including the referee's observation that Freerouting's within-seed rollouts may be near-identical due to its deterministic rip-up-and-reroute structure, so CP@5 ≈ CP@1 for Freerouting while PPO benefits from selection. If the CP@1 gap between PPO and Freerouting on D3-A narrows substantially, we will adjust the framing of the D3-A comparison accordingly. revision: yes
Referee: §4.2 and Table 2: The zero-shot transfer claim rests on D3-A being representative of 'real boards.' D3-A contains only the 100 smallest boards from the 679-board D3 corpus (2-13 nets, 6-31 pads), while the PPO policy was trained on D2 boards with 4-6 nets and 8-21 pads. The distribution overlap between D2-train and D3-A is substantial, and the transfer is not as out-of-distribution as the 'zero-shot to real boards' framing suggests...

Authors: The referee is correct that D3-A's net and pad counts (2–13 nets, 6–31 pads) substantially overlap with the D2 training distribution (4–6 nets, 8–21 pads), which weakens the 'zero-shot to real boards' framing. We agree that the current language overstates the generalization. We will revise the manuscript in two ways: (1) We will temper the transfer claim throughout the paper, including the abstract, introduction, and §5.3, to explicitly acknowledge that D3-A boards are the smallest real boards and that their complexity distribution overlaps with D2-train. The claim will be reframed as 'zero-shot transfer to small real boards' rather than 'zero-shot transfer to real boards' without qualification. (2) We will add explicit discussion of the D3-A/D2-train distributional overlap in §4.2 and §5.3, noting that D3-A is the subset where transfer is most plausible precisely because the complexity is closest to the training distribution. Regarding D3-C: we did not report D3-C results because PPO and the LLM agents do not complete routing within the step budget on boards with 100+ pads, and Freerouting itself has low completion rates on the largest boards. We will state this explicitly in the revision rather than leaving D3-C results absent without explanation. The D3-B results (CP 0.45 vs. 0.78) already show the limits of generalization, and we will foreground this as the honest boundary of what the current policy achieves. revision: yes
Referee: §5.3, Table 3: The D3-A evaluation uses 99 boards (one 73-net board excluded), while D3-B uses only 10 boards. With 4 seeds, D3-B CP for PPO is 0.45 +/- 0.10, meaning the 95% confidence interval is roughly [0.25, 0.65]. The comparison to Freerouting's 0.78 +/- 0.05 on D3-B is based on only 10 boards x 4 seeds = 40 data points. The paper should report the board-level variance (not just seed-level variance) or acknowledge that the D3-B comparison is underpowered.

Authors: The referee is correct that the current D3-B comparison is underpowered: 10 boards × 4 seeds = 40 data points, and the reported standard deviations are seed-level (across 4 seeds), not board-level. The 95% CI for PPO's D3-B CP is indeed wide enough that the gap to Freerouting may not be as large as the point estimates suggest. We will make two changes: (1) We will report board-level variance in addition to seed-level variance for D3-B (and D3-A), so that the reader can assess the uncertainty at the appropriate granularity. This means reporting per-board CP averaged over seeds, then computing the standard deviation across boards. (2) We will add an explicit acknowledgment in §5.3 that the D3-B comparison is based on only 10 boards and is therefore underpowered, and that the D3-B gap between PPO and Freerouting should be interpreted with caution. We chose 10 D3-B boards because of the high per-board evaluation cost for LLM agents (Appendix L documents that GPT-5.4 takes ~866 seconds per D3-B board), but we should have stated this limitation more prominently. If feasible before the camera-ready deadline, we will expand the D3-B evaluation set to include more boards for the RL and rule-based methods (which are much cheaper to run than LLM agents), though we cannot guarantee this will be complete in time. revision: yes

Circularity Check

0 steps flagged

No significant circularity found; the derivation is self-contained against external benchmarks.

full rationale

The paper's central claims rest on an external engine (KiCad) for both training signal and evaluation, not on a self-defined proxy. The reward potential Φ(s) = -(f_d(n_drv(s)) + λ_w ℓ(s) + λ_v n_via(s)) uses KiCad's DRC API for n_drv, and the evaluation metric CP checks DRV=0 via the same KiCad DRC. While the training reward and evaluation metric both reference the same external checker, this is analogous to using the same physics engine for simulation and scoring — the checker is independent of the authors' method. The best-of-5 selection protocol (selecting the rollout with highest potential gain) is applied uniformly to all methods including external baselines (Freerouting, OrthoRoute, KRT), so it does not asymmetrically define the authors' method into success. The selection criterion (maximize ΔΦ, which heavily weights DRV=0) is approximately aligned with the evaluation criterion (CP requires DRV=0), but this alignment is a property of the evaluation protocol, not a circular derivation: the selection does not guarantee CP=1, and the paper reports CP < 1 for most methods. No self-citation chain is load-bearing for the central claims — the paper cites external tools (KiCad, Freerouting, Jumanji, Sable, PCBench) and evaluates against external baselines under a shared protocol. The one minor concern is that the potential function used for both training and sample selection shares terms with the evaluation metric, but this is standard practice in RL benchmarks (train on shaped reward, evaluate on true objective) and does not constitute circularity.

Axiom & Free-Parameter Ledger

13 free parameters · 5 axioms · 0 invented entities

The paper introduces no new physical entities, particles, forces, or dimensions. The environment wraps an existing EDA engine (KiCad) and uses standard RL/LLM methods. The 'invented' aspects are engineering artifacts (the 58-API wrapper, the tokenization scheme, the dual RL/LLM wrapper architecture) rather than new scientific entities. The free parameters are hyperparameters of the reward function and training setup, all of which are standard for RL. The ad-hoc axioms concern evaluation design choices (D3-A as 'real boards', best-of-5 selection, excluded rip-up actions) rather than theoretical assumptions.

free parameters (13)

lambda_w (wirelength weight) = 0.002
Default training weight for wirelength in the potential function; swept over {0, 0.001, 0.002} in the 3x3 factorial (Sec 5.5).
lambda_v (via weight) = 0.1
Default training weight for via count in the potential function; swept over {0, 0.05, 0.1} in the 3x3 factorial (Sec 5.5).
s_agg (aggregate DRC penalty scale) = 3
Breadth penalty scale in the log-concave DRC penalty f_d (Eq. 5); controls how much the first violation on each new net is penalized.
s_pn (per-net DRC penalty scale) = 1
Depth penalty scale in f_d; controls how much additional violations on the same net are penalized.
o (log-curve knee offset) = 2
Shared offset in f_d that determines where the log penalty curve saturates; a single new violation on a clean net contributes ln(3/2).
gamma (discount) = 0.995
Discount factor for PPO; the per-step reward telescoping equivalence holds exactly only at gamma=1, so this is an approximation.
learning rate = 1e-4
AdamW learning rate with 20-iteration warmup (Table 14).
d_model / layers / heads = 128 / 4 / 8
Transformer architecture dimensions for the RL policy (Table 14).
n_freq (Fourier feature frequencies) = 32
Number of sin/cos Fourier features for coordinate encoding in the RL tokenization (Appendix J.2).
candidate pool size = 64
Maximum number of candidate points the pointer network selects from per step (Appendix J.2).
directional grid offset = 0.5mm
Eight-way directional grid offset around the routing head for candidate pool construction (Appendix J.2).
episode step limit = 256
Maximum steps per episode for D2/D3-A/D3-B (Table 14); also used for D1.
k (rollouts per board) = 5
Number of independent rollouts per board; best-of-5 selected by potential gain (Sec 4.3).

axioms (5)

domain assumption KiCad's PNS router and DRC engine faithfully implement PCB design rules and routing physics
The entire environment depends on KiCad's C++ engine being correct. This is reasonable given KiCad's industry adoption but is an external dependency the paper does not verify (Sec 2.2, Sec 3.1).
ad hoc to paper Segment-level engine operations are a sufficient action space for PCB routing without rip-up-and-reroute
The action space (Table 1) excludes track deletion (delete_track_near is exposed but excluded from the agent's action space, Sec 7). The paper acknowledges this limits the agent's ability to recover from routing deadlocks, but the zero-shot transfer claim is evaluated under this constraint.
domain assumption The 35 stock error-level DRC checks are a sufficient proxy for manufacturability
The evaluation uses KiCad's default error-level DRC checks as the DRV=0 criterion for Clean Pass (Appendix F, Table 9). The paper acknowledges in Sec 7 that industrial practice requires signal integrity, EMI, and thermal checks beyond geometric DRC.
ad hoc to paper D3-A boards (2-13 nets, 6-31 pads) are representative of 'real boards' for the zero-shot transfer claim
The zero-shot transfer claim is evaluated on D3-A, the smallest 100 boards of the 679-board D3 corpus. The paper frames this as 'real boards' but the distribution closely overlaps with D2-train (4-6 nets, 8-21 pads). D3-B and D3-C results show the transfer does not extend to larger boards (Sec 5.3, Table 3).
ad hoc to paper Best-of-5 rollout selection by potential gain is a fair evaluation protocol
All stochastic methods are evaluated with 5 rollouts per board, selecting the one with highest potential gain (Sec 4.3). This inflates CP relative to single-rollout evaluation. The paper reports CP@1 only for the D1 scalability experiment (Fig. 6), not for the main D2/D3 results.

pith-pipeline@v1.1.0-glm · 49938 in / 5006 out tokens · 653907 ms · 2026-07-08T20:42:44.707124+00:00 · methodology

0 comments

read the original abstract

PCB routing is the task of connecting the nets of a board with copper traces under strict design rules, yet learning-based methods still lag behind rule-based routers. We introduce PCBWorld, an open-source engine-grounded PCB routing environment built on KiCad, an electronic design automation (EDA) engine. As a human engineer does, agents in PCBWorld interactively route a board through the engine's native operations, guided by its Design Rule Check (DRC) feedback. The environment supports both RL and tool-using LLM agents. Alongside the environment, PCBWorld-Bench provides three board datasets in the native .kicad_pcb format, two controllable synthetic generators and 679 real open-source boards. It scores any completed board with eight engine-checked evaluation metrics, regardless of the routing method. In our experiments, agents in PCBWorld consistently outperformed grid-action RL policies and open-loop LLM baselines, and an RL policy trained only on synthetic boards transferred zero-shot to real boards, approaching rule-based routers.