{"id":"3b0b8f61-c377-4a7f-96b4-6fc11caf1818","arxiv_id":"2502.03159","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"PICBench evaluates how well LLMs can generate photonic integrated circuit netlists and shows that simulator-driven error feedback greatly improves their syntax and functional correctness.","lead":"PICBench is a new benchmark that asks large language models to write photonic integrated circuit designs as computer-readable netlists, and then automatically checks whether the designs simulate correctly. It tests five commercial AI models on 24 design problems and shows that giving the models feedback about their mistakes sharply improves their success rate.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Functionality evaluation is the load-bearing weak point: Section III-C compares generated and golden frequency responses without stating a tolerance or validating that all correct designs match, so reported functionality scores may not measure design correctness.","rationale":"The reader's weakest_assumption correctly identifies the undefined functionality comparison. I agree. The paper distinguishes syntax (simulator runs successfully) from functionality (simulation results match golden) but never operationalizes the latter: no tolerance, no error metric, no discussion of complex vs. magnitude response, and no set of alternate correct designs used to validate the comparator. The reported functionality Pass@1 values range from 1.67% to 62.50%, and whether these numbers rank actual design capability or merely the ability to reproduce one particular expert netlist is unknown. The restrictions and error-feedback results have an additional independent overfitting concern—restrictions are derived from the same 24 problems used for final evaluation—but the single most load-bearing issue for the central benchmark claim remains the functionality criterion. A conditional accept is appropriate, with the explicit condition that the authors publish a precise matching criterion and demonstrate, using alternate human-validated correct designs, that the criterion accepts functionally correct circuits and rejects incorrect ones. This does not change the reader's conditional verdict, so no verdict adjustment is needed.","tokens_in":8762,"tokens_out":4801,"duration_ms":49536,"concrete_test":"Open the PICBench repository evaluation script and identify the exact comparison used for the functionality check (e.g., np.allclose with rtol/atol, exact equality, or a norm over the wavelength grid). Then run a controlled experiment: for at least three problems (e.g., MZI ps, Crossbar 4x4, Clements 4x4), hand-build 3–5 netlists that an independent PIC designer judges functionally equivalent to the golden solution but that differ in implementation, such as a different but valid component ordering or an alternative decomposition that preserves the specified input/output behavior. Run the evaluator unchanged. If any valid design fails, the functionality criterion is over-constrained and must be redefined before the scores are interpretable; if all pass with the documented tolerance, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that PICBench automatically evaluates functionality by comparing simulation outputs with golden solutions. Section III-C says only \"we simply compare the simulation results between generated code completions and golden reference solutions,\" with no metric, tolerance, wavelength grid, or treatment of complex S-parameters specified. This matters because the same functional specification can be met by multiple netlists: equivalent port relabeling, alternative but functionally identical MZI switching architectures, or phase shifts differing by 2π will change the raw simulated response even though the circuit is correct. If the comparison is exact array equality, the benchmark penalizes correct designs and the low functionality scores (e.g., 6.67% Pass@1 for GPT-4) underestimate true capability. If a lenient threshold is used, incorrect designs could pass. The paper provides no evidence that the golden-response match is necessary and sufficient for correctness, nor any human-validated set of alternate correct designs. Since the paper's headline comparison of LLMs and its feedback/restriction conclusions rest on these functionality numbers, the absence of a defined and validated matching criterion is load-bearing. The artifact may be excellent; the evaluation claim as written is not yet substantiated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces PICBench, a benchmark and evaluation framework for assessing LLMs on photonic integrated circuit (PIC) design. It comprises 24 natural-language design problems spanning optical computing, optical interconnects, optical switches, and fundamental devices. LLM outputs are JSON netlists for the open-source SAX simulator, and the framework automatically evaluates syntax (whether SAX can simulate the netlist) and functionality (whether the simulated frequency response matches a golden solution). The authors evaluate five commercial LLMs, report Pass@1 and Pass@5 scores, and study two prompt-engineering techniques: an error-feedback loop and a set of restrictions derived from common errors. They report substantial improvements from both techniques.","tokens_in":8958,"tokens_out":4800,"duration_ms":45427,"significance":"If the benchmark's evaluation criteria are valid, PICBench fills a clear gap: prior LLM-for-photonics work tested only a few hand-designed cases, while digital-design benchmarks like VerilogEval and RTLLM have no photonic counterpart. The paper ships open-source code and uses the mainstream SAX simulator, making the framework reusable and comparable. The systematic comparison of five LLMs and the demonstration that feedback and restrictions improve scores are useful for the community. However, the significance hinges on the functionality check being a sound and reproducible measure of design correctness, and on the experimental comparisons being fair; both issues are currently questionable.","major_comments":[{"comment":"The functionality evaluation is underspecified to the point of being non-reproducible. The paper states only that \"we simply compare the simulation results between generated code completions and golden reference solutions,\" without defining the metric, the tolerance, the wavelength grid, or the treatment of complex S-parameters. This matters because distinct correct implementations (e.g., port relabeling, alternative but functionally identical MZI topologies, phase shifts differing by 2π) will produce different frequency responses. If the comparison is exact equality, the benchmark penalizes valid designs; if a tolerance is used, it must be reported and justified. The paper also provides no evidence that the golden response is necessary and sufficient for correctness, nor a human-validated set of alternate correct designs for any problem. Since Tables III and IV and the abstract's claims about LLM capability rest on these functionality numbers, the missing criterion is load-bearing and should be supplied.","section":"III-C"},{"comment":"The claim that the restrictions improve LLM performance is circular. The restrictions in Table II are described as \"common error types we collected during our trials\" on the same 24 problems, and the improvement in Table IV is measured on those same problems. This amounts to tuning on the test set and then reporting the test-set accuracy; the measured gains (e.g., Gemini 1.5 Pro syntax Pass@1 from 9.17% to 64.17%) may largely reflect overfitting to the benchmark's specific idiosyncrasies rather than a general improvement in PIC-design ability. To support the claimed impact of restrictions, the authors should evaluate on a held-out set of new problems or use a temporal split in which restrictions are frozen before evaluating on the benchmark.","section":"III-D and Table IV"},{"comment":"The comparison between \"without feedback\" and \"with feedback\" results is not apples-to-apples because the number of LLM calls per problem differs. In the feedback condition, each sample can undergo up to 1 or 3 correction iterations, so Pass@1 with 3 EFs allows up to four sequential attempts per sample, whereas Pass@1 without EF allows only one attempt. The observed improvements (e.g., Claude 3.5 Sonnet functionality Pass@1 from 1.67% to 24.17%) could be due simply to the additional attempts rather than to the feedback. The paper should either control for the total number of LLM calls (e.g., compare Pass@1 with 3 EFs against Pass@4 without feedback) or report the results as a function of the number of attempts. As written, the feedback experiment conflates iteration count with sample count and the conclusion is not supported.","section":"IV-A and IV-B"}],"minor_comments":[{"comment":"The text says \"we do not include any purely device-level design problems in our collection,\" but Table I includes a category called \"Fundamental Devices\" with MZM and MZI ps. The intended distinction (device-level lacking connections) should be clarified to avoid an apparent contradiction.","section":"III-B"},{"comment":"The sentence \"we queried the selected LLMs both without and with feedback for n iterations where we set n = 1 and n = 3\" is ambiguous: it is unclear whether n is the number of feedback iterations per sample or the number of samples per problem, especially because n is already used for the sample count in the Pass@k estimator. Please use distinct notation and specify both values.","section":"IV-A"},{"comment":"The error classification loop is described as involving \"human inspection\" when collecting restrictions, but the feedback loop in Section III-E says the error is \"automatically classified.\" Please clarify whether the classification for feedback is automated or manual, and how.","section":"III-A"},{"comment":"In the JSON format example, the second instance key is written \"<component_name2\" with a missing closing angle bracket; fix the typo.","section":"Figure 3"},{"comment":"The SAX reference is incomplete: it gives only the GitHub URL and an informal citation. Please provide a version, DOI, or formal publication reference, as the evaluation depends on the exact simulator behavior.","section":"Reference [14]"}],"recommendation":"major_revision","confidential_remarks":"The core artifact (a 24-problem PIC netlist benchmark on SAX) is a timely contribution. However, the functionality-comparison criterion must be fully specified and validated, and the experimental comparisons for feedback and restrictions need to control for attempt count and overfitting. These issues are addressable within the manuscript's scope, so I recommend major revision rather than rejection. I would also suggest the authors make the actual problem descriptions and golden netlists available in the supplementary material or clearly via the repository, since the paper itself only lists problem names."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"PICBench is worth taking seriously. It is the first benchmark I know of that adapts LLM code-generation evaluation to photonic netlists, and it ships the two things that matter most: a concrete 24-problem suite with golden solutions, and an automated pipeline that checks both syntax and functionality on top of SAX. That fills a real gap. Prior LLM chip-design benchmarks stop at Verilog and SPICE, and the existing PIC work is either unautomated or limited to seven devices. The domain-specific error taxonomy and the feedback loop are genuinely useful additions, and the paper is honest enough to report low baseline scores rather than cherry-pick results. The tables are consistent, and the finding that feedback plus restrictions sharply improves Pass@1 is plausible.\n\nThe soft spots are real but fixable. The load-bearing one is Section III-C. The paper says functionality is checked by simply comparing the simulated frequency response of the generated netlist against the golden response, but it never states the metric, tolerance, wavelength grid, or how complex S-parameters are handled. That matters because equivalent correct designs can produce different raw responses: port relabeling, alternative MZI architectures, or phase shifts differing by 2π are all legitimate and would not match exactly. Without a defined and validated matching criterion, the functionality scores could mislead. This is not a fatal flaw given the benchmark artifact is what future users will actually run, but the paper’s headline comparisons rest on it.\n\nThe second soft spot is the restrictions experiment. The restriction list in Table II was assembled by watching models fail on the exact 24 problems in Table I, then tested on those same problems. That is a training-on-the-test-set effect, so the magnitude of the gain (e.g., Gemini from 9% to 64% syntax Pass@1) is not an independent measure of what the restrictions would do on unseen problems. The feedback-loop gains are less contaminated and still show a clear effect.\n\nMinor issues: n=5 samples per problem gives coarse Pass@k estimates with no error bars, and the paper does not say whether the repository contains the exact evaluation code and thresholds. Those are easy to address.\n\nBottom line: the benchmark itself is a solid contribution and the right kind of artifact to build in this area. The evaluation methodology needs tightening, not replacement. I would send this to peer review expecting major revision, and I would cite it if I worked on LLM-driven chip or photonic design.","headline":"PICBench is a useful first benchmark for LLM-generated photonic netlists, but its functional pass/fail criterion is underspecified and the restriction gains are measured on the problems that produced the restrictions.","tokens_in":9521,"tokens_out":1002,"would_cite":true,"duration_ms":11930,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces PICBench, a benchmark that asks large language models to write photonic integrated circuit netlists from natural-language descriptions and then checks those netlists by simulation.","keywords":["photonic integrated circuits","PIC design automation","LLM code generation","benchmarking","netlist generation","S-parameter simulation","error feedback","prompt engineering"],"falsifier":"Take any of the 24 problems and run a hand-written valid netlist that implements the same function with a different architecture, such as a 4x4 switching network built from a different but equivalent layout; if the benchmark scores this as a functional failure, the comparison against a single golden response is not actually testing function.","tokens_in":8558,"feed_emoji":"💡","tokens_out":11642,"duration_ms":95691,"temperature":0.7,"pith_summary":"PICBench, introduced in this paper, is a benchmark for photonic integrated circuit (PIC) design by large language models: given a natural-language description of a circuit, the model must output a JSON netlist that names the components and wires them together. The authors built 24 expert-written problems spanning optical computing, interconnects, switches, and fundamental devices, each with a golden netlist and its simulated frequency response. They then measured how often five commercial LLMs produce a netlist that simulates at all (syntax) and one whose simulated response matches the golden response (functionality). Their experiments show that feeding simulator errors back to the model and adding hand-curated format restrictions sharply raises both scores, with best configurations reaching near-total syntax success and Pass@5 functionality above 60 percent.","feed_headline":"LLM photonic-chip netlists hit 95% syntax with feedback","feed_subtitle":"A new benchmark simulates LLM-generated photonic circuits; feedback and restrictions push five models past 90% syntax.","key_machinery":"The engine of the paper is an LLM–simulator feedback loop held together by a structured system prompt. The prompt demands a specific JSON netlist schema and gives an API list of allowed built-in components; a separate restrictions block encodes common failure modes (undefined models, duplicate port connections, dangling ports, extra JSON content) as explicit rules. An open-source frequency-domain circuit simulator takes the generated netlist and either reports a syntax error or returns a frequency response. Errors are classified into a fixed category set and fed back to the model with a request to rewrite the entire netlist; if simulation succeeds, the response is compared with the golden response. This loop is what converts a raw code-generation problem into an iterable, automated design process.","core_discovery":"The paper's central claim is that LLM-based PIC design generation can be benchmarked automatically and improved substantially by prompt-level interventions. Each problem in PICBench pairs a natural-language specification with an expert-written golden netlist; evaluation pipes a model's generated netlist into an open-source frequency-domain simulator. Syntax is judged by whether the simulator can build the circuit and produce a frequency response; functionality is judged by comparing that response with the saved golden response. In experiments, restrictions plus three rounds of error feedback lifted every tested model's syntax Pass@1 above 50 percent, with one configuration reaching 95 percent; functionality Pass@1 reached 43.33 percent at best, and Pass@5 functionality reached 62.50 percent. The authors interpret these gains as evidence that feedback mechanisms and in-context learning are productive directions for automating PIC design.","pith_inferences":["If the evaluation were extended to accept a family of equivalent frequency responses rather than one golden curve, the reported functionality scores would likely rise, because many different component placements can realise the same transfer function.","The restriction list is curated from observed errors; automating its construction from simulator messages would make the loop fully self-improving and could transfer to other structured-code domains such as analog or RF circuit netlists.","PICBench's fast simulation and machine-checkable errors make photonic netlist generation a useful probe for studying how LLMs handle domain-specific formal languages, separate from general coding benchmarks.","The Pass@5 functionality ceiling of 62.5% suggests that sampling more candidates alone will not solve the task; future gains likely require feedback loops that check functional, not merely syntactic, correctness."],"forward_implications":["With restrictions and three error-feedback iterations, all five tested LLMs reach 100% syntax Pass@5 on PICBench, meaning the remaining gap is almost entirely functional correctness.","A single feedback round can be more valuable than five independent samples: several models' Pass@1 syntax with one feedback iteration exceeds their Pass@5 syntax without feedback.","Hand-written restrictions transfer common simulator error messages into the prompt and improve syntax even with no feedback, raising one model's Pass@1 syntax from 9.17% to 64.17%.","Functionality remains the harder test: best Pass@1 with all interventions is 43.33%, so current LLMs can often produce simulatable circuits that do not match the expert design's behavior."],"supporting_citations":[{"why":"The open-source simulator that executes generated netlists and produces the frequency responses used for both syntax and functionality checks.","marker":"[14]"},{"why":"Defines the Pass@k estimator the paper uses to report syntax and functionality scores.","marker":"[15]"},{"why":"A Verilog generation benchmark that supplies the template of natural-language problems plus test-based evaluation that PICBench adapts.","marker":"[8]"},{"why":"A Verilog evaluation dataset whose testing procedure motivates systematic evaluation of generated code.","marker":"[9]"},{"why":"Prior LLM-generated FDTD code for a laser structure, not fully automated; motivates the need for an automated evaluation loop.","marker":"[12]"},{"why":"Prior LLM-assisted PIC layout work tested on only seven simple devices; the gap PICBench fills.","marker":"[13]"}],"fun_headline_variants":["LLM photonic netlists reach 95% syntax with feedback","PICBench: first benchmark for LLM photonic design","Feedback lifts LLM syntax to 95% in photonic circuits","LLMs generate photonic netlists, but functionality lags","Benchmark shows LLMs excel at syntax, not function in photonics"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that comparing an LLM-generated circuit's simulated frequency response with the single expert-written golden response, without an explicit tolerance or a set of equally valid responses, correctly determines whether the circuit implements the requested function.","fun_headline_variants_meta":{"raw":{"variants":["LLM photonic netlists reach 95% syntax with feedback","PICBench: first benchmark for LLM photonic design","Feedback lifts LLM syntax to 95% in photonic circuits","LLMs generate photonic netlists, but functionality lags","Benchmark shows LLMs excel at syntax, not function in photonics"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000163,"raw_usage":{"total_tokens":1238,"prompt_tokens":935,"completion_tokens":303,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":551,"completion_tokens_details":{"reasoning_tokens":213}},"tokens_in":551,"tokens_out":303,"duration_ms":2967,"temperature":1.0,"reasoning_tokens":213,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T05:42:12.380069+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take any of the 24 problems and run a hand-written valid netlist that implements the same function with a different architecture, such as a 4x4 switching network built from a different but equivalent layout; if the benchmark scores this as a functional failure, the comparison against a single golden response is not actually testing function.","supporting_citations":[{"cited_title":"Laporte, “Sax,” 2023","cited_arxiv_id":null,"evidence_quote":"The open-source simulator that executes generated netlists and produces the frequency responses used for both syntax and functionality checks."},{"cited_title":"Rtllm: An open-source benchmark for design rtl generation with large language model,","cited_arxiv_id":null,"evidence_quote":"A Verilog generation benchmark that supplies the template of natural-language problems plus test-based evaluation that PICBench adapts."},{"cited_title":"Verilogeval: Evaluating large language models for verilog code generation,","cited_arxiv_id":null,"evidence_quote":"A Verilog evaluation dataset whose testing procedure motivates systematic evaluation of generated code."},{"cited_title":"From english to pcsel: Llm helps design and optimize photonic crystal surface emitting lasers,","cited_arxiv_id":null,"evidence_quote":"Prior LLM-generated FDTD code for a laser structure, not fully automated; motivates the need for an automated evaluation loop."},{"cited_title":"Towards large-language model assisted layout of silicon photonic integrated circuits,","cited_arxiv_id":null,"evidence_quote":"Prior LLM-assisted PIC layout work tested on only seven simple devices; the gap PICBench fills."}],"review_version":1}