{"id":"c5d86bfb-8d6b-42da-a9df-5b2c19a1e137","arxiv_id":"2412.02410","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"AutoPLC combines retrieval from vendor-specific code libraries, LLM-based planning and API recommendation, and compiler feedback from real PLC IDEs to generate compilable Structured Text code for Siemens SCL and CODESYS ST at over 90% compilation pass rates.","lead":"This paper presents AutoPLC, a system that generates vendor-specific PLC code from natural language descriptions, reaching over 90% compilation success on a 914-task benchmark at about $0.13 per task. It is useful because PLC code is fragmented across proprietary dialects, and a working generator could cut engineering time in industrial automation.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline pass rates may be inflated because Rq2ST is built from the same OSCAT/LGF/Competition code that forms the benchmark, and the substring-name filter does not exclude semantically near-duplicate cases (Section IV-C).","rationale":"The reader identified the same load-bearing weakness: the evaluation permits near-duplicate ground-truth code to be retrieved because the Rq2ST knowledge base and the 914-task benchmark are drawn from the same three source libraries, with only a name-substring exclusion. This is the most serious threat to the central claim because it directly affects the headline >90% compilation pass rates and the comparison against baselines. The paper's own ablation quantifies the mechanism: removing case retrieval reduces pass rates by up to 35 points, so the benefit of retrieval is large enough that even a small contamination rate could explain a substantial share of the reported advantage. The external Agents4PLC benchmark provides some independent support, but its 23 tasks are too few to validate the 914-task claim. The condition is not fatal if a strict disjoint split still yields high pass rates, so the appropriate disposition remains conditional pending the concrete test. I did not find an additional load-bearing concern beyond this one; compilation success as a proxy for correctness is a known limitation that the paper explicitly acknowledges and partially addresses with execution-based evaluation and expert review.","tokens_in":16560,"tokens_out":3077,"duration_ms":34118,"concrete_test":"Re-run AutoPLC on the 914 tasks with a leakage-free Rq2ST: for each task, exclude from Rq2ST all cases from the same source library that share any vendor API call or whose normalized embedding similarity to the target's ground-truth code exceeds 0.8, in addition to the existing substring filter. Regenerate all tasks with the same pipeline and three backbone seeds, then compare LGF and Competition pass rates with Table II. If pass rates fall by more than 10 points, or the gap to Claude-3.5-Sonnet shrinks by more than half, the headline results are materially inflated by knowledge-base overlap.","verdict_should_be":"UNCHANGED","load_bearing_attack":"AutoPLC's central claim—>90% compilation pass rates on the 914-task benchmark—depends on Stage 2 retrieving relevant cases from Rq2ST. But Rq2ST is constructed from exactly the same OSCAT, LGF, and Competition datasets used as evaluation ground truth (Section IV-C). The only exclusion is 'mutual substring containment' of case names, so near-identical tasks with rephrased text, renamed variables, or slightly different APIs remain retrievable. The LLM then receives the ground-truth implementation as a few-shot example, making compilation success substantially easier than for a genuinely held-out test. This is not merely hypothetical: Table III shows removing Case Retrieving drops pass rate by 35.01% on LGF and 19.51% on Competition; while that also reflects the value of RAG, it is exactly the magnitude expected if retrieval supplies near-answer code. The baselines do not have access to Rq2ST, so the comparison in Table II is unfair to them. The external Agents4PLC result (23 tasks) is independent and partially mitigates, but is too small to support the headline. The paper acknowledges semantic overlap in VI-B, but framing it as 'realistic' does not remove the contamination for measuring system capability.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"AutoPLC is a framework for generating vendor-specific Structured Text (ST) code for PLCs from natural-language requirements, targeting Siemens SCL (TIA Portal) and CODESYS ST. The system builds two knowledge bases, APILib (vendor API metadata with three-dimensional indexes) and Rq2ST (requirement-plan-code cases), and uses a four-stage pipeline: NL-based planning with state-machine support for control tasks, case retrieval with LLM reranking, API recommendation with BM25 and type-based augmentation, and iterative code generation with compiler feedback from real vendor IDEs. The evaluation uses a 914-task benchmark curated from OSCAT (718), Siemens LGF (151), and a Siemens competition dataset (45), plus the external 23-task Agents4PLC benchmark. The paper reports compilation pass rates above 90% on all three self-built datasets and 100% on Agents4PLC, with lower average error counts than six general LLM baselines, LLM4PLC, and MapCoder. It also presents an ablation study, an execution-based functional test on a 71-task subset, and a manual expert evaluation of 90 sampled cases. The authors open-source the framework and benchmark.","tokens_in":16812,"tokens_out":4615,"duration_ms":46814,"significance":"If the empirical claims are trustworthy, AutoPLC is a substantial practical contribution: it demonstrates that retrieval-augmented generation combined with vendor-IDE compiler feedback can produce syntactically valid ST code at very low cost ($0.13 per task), and it provides one of the largest open benchmarks for ST generation. The use of real TIA Portal and CODESYS compilation, the ablation study covering all four stages, the external Agents4PLC comparison, and the open-source release are concrete strengths that go beyond what most prior work in this area provides. The manual evaluation and the explicit discussion of limitations show a degree of methodological care. However, the central evaluation is compromised by a data-leakage risk: Rq2ST is built from the same corpora that form the 914-task benchmark, and the leakage filter is only a substring-name match. The functional-correctness check, where AutoPLC passes the full test suite on only about half of the examined tasks, further tempers the headline compilation-success claim. The external benchmark partially mitigates the leakage concern but is too small to fully support the paper's conclusions.","major_comments":[{"comment":"The Rq2ST knowledge base is constructed from the same OSCAT, LGF, and Siemens Competition corpora that define the 914-task benchmark (Section IV-B1), and Stage 2 (Case Retrieving) injects the top-3 retrieved cases into the generation prompt. The only exclusion filter is mutual substring containment of case names; this does not remove near-duplicate implementations with rephrased requirements, renamed variables, or minor API differences. For a typical target task, the ground-truth code (or a close variant) is therefore present in Rq2ST, so the reported 92.72%/91.11%/92.90% pass rates measure performance when the answer is retrievable, not generalization to unseen ST requirements. The ablation in Table III (removing Case Retrieving drops the pass rate by 35.01% on LGF and 19.51% on Competition) is consistent with this leakage. The defense in VI-B that semantic overlap reflects real-world conditions is not a rebuttal: the benchmark is meant to measure system capability, and held-out tasks are required for that. The external Agents4PLC set (23 tasks) is independent and helps, but is too small to support the headline claim.","section":"Section IV-C and VI-B"},{"comment":"The execution-based evaluation covers only 71 tasks, selected as the intersection of compilation-passing cases from AutoPLC and MapCoder. AutoPLC achieves full-suite success on 35 of these 71 tasks (about 49%), and across all 487 test cases the two models pass 76%. The paper reports these numbers but the RQ1 answer and the abstract present only compilation pass rates. Because the 71-task subset is not a random sample of the benchmark, the compilation numbers and the correctness numbers cannot be combined, and the resulting picture is that on the subset where correctness was actually measured, AutoPLC fails roughly half the tasks under the paper's own strict all-test-cases-must-pass criterion. This should be reported as a central limitation and should temper the claim that AutoPLC generates ST code that is practically usable, even though Section VI-C does acknowledge limited functional assessment.","section":"Section V-A and VI-C"},{"comment":"The baseline comparison is not symmetric. AutoPLC uses Rq2ST and APILib, which are built from the same corpora as the benchmark, while the baselines (except Agents4PLC as reported) do not have access to these knowledge bases. The large gaps, e.g., 92.72% vs. 15.23% for Claude-3.5-Sonnet on LGF, may therefore reflect the availability of near-answer retrieval rather than the superiority of the four-stage pipeline. To support the comparative claim, the authors should evaluate a version of AutoPLC without the knowledge bases (which the ablation partially provides) and, ideally, give the baselines access to an equivalent held-out knowledge source, or compare on tasks where the knowledge base is guaranteed not to contain the ground truth.","section":"Section IV-D and Table II"}],"minor_comments":[{"comment":"The text contains multiple 'Fig.??' placeholders, so the figure references in the background section do not render; these should be fixed before publication.","section":"Section II-A"},{"comment":"There are formatting errors in the text such as 'T askand Code' and 'AP Isare'; these should be corrected.","section":"Section III-A"},{"comment":"The filter described in this section is called 'strict,' but it only checks mutual substring containment of names; a more precise term such as 'name-based exclusion' would be less misleading.","section":"Section IV-C"},{"comment":"The reported improvement of 1620.8% over LLM4PLC on Agents4PLC is a relative percentage comparison of pass rates (100.00% vs. 36.36%); the paper should state this as percentage-point difference or otherwise clarify the metric to avoid overstatement.","section":"Section V-A"},{"comment":"The terminology 'Valid. Rate' in Table II is introduced as 'formal verification pass rate' in the introduction; the two terms should be unified for consistency.","section":"Section IV-B2 and Table II"},{"comment":"The phrase 'As shown in Huang et al. (2024) [17]' cites a broad trustworthiness survey without pinpointing the specific finding about external knowledge and feedback; a more precise citation would help the reader verify the claim.","section":"Section V-B"}],"recommendation":"major_revision","confidential_remarks":"The core concern is that the headline compilation pass rates may be substantially inflated by the overlap between Rq2ST and the evaluation benchmark. This is not merely a methodological nuance: the ablation shows that removing case retrieval reduces the pass rate by up to 35 percentage points, which is the magnitude one would expect from retrieving near-answer code. I would require the authors to either (a) re-run the main evaluation with a knowledge base that excludes all cases derived from the test corpora, or (b) provide a quantitative analysis of how often retrieval returns the ground-truth implementation or a near-duplicate, and re-report the pass rates on the subset of tasks where retrieval does not return a near-duplicate. The external Agents4PLC result is helpful but insufficient given its small size. The paper is otherwise well-structured and the engineering effort is real, so I see this as a fixable major-revision, not a rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take. Two things to know up front. First, the artifact is real: a vendor-aware ST generator with two knowledge bases, a 914-task benchmark, and actual integration with TIA Portal and CODESYS. The authors release everything, which is genuinely useful in a field with scarce public resources. Second, the 90%+ compilation pass rate is not a clean measure of capability, because the retrieval base is built from the same OSCAT/LGF/Competition code as the benchmark. The substring filter is too weak to stop near-duplicates, and the ablation numbers are consistent with that: removing case retrieval drops pass rate by 35 points on LGF, exactly what you'd see if retrieval was handing over near-answer code.\n\nThat said, the paper isn't careless. The authors explicitly acknowledge the semantic overlap in VI-B and argue it reflects real-world conditions. They also run an independent check on the Agents4PLC benchmark (23 tasks), getting 100% compilation and 78% validation, which is encouraging but too small to carry the headline. Their error-type analysis and manual evaluation are thorough, and the limitations section is honest.\n\nThe functional correctness story is the other soft spot. On 71 tasks with test suites, AutoPLC and MapCoder both pass full suites on roughly half. Compilation is a weak proxy, and the paper says so. No error bars on stochastic runs, but that feels minor in this subfield.\n\nBottom line: this is a solid engineering paper that deserves a serious referee. The right revision path is to re-run the main evaluation with a clean held-out split or semantic deduplication of the knowledge base, and to spend more space on functional correctness. If the pass rate holds after that, the result is meaningful. If not, the framework and benchmark still stand as a useful contribution. I'd send it to review with major revisions.","headline":"AutoPLC is a real engineering artifact, but the headline 90% pass rate is inflated by benchmark leakage; fix the evaluation split before believing the number.","tokens_in":17377,"tokens_out":2631,"would_cite":true,"duration_ms":26764,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper shows that a general LLM, supplied with vendor API libraries, a requirement-to-code case base, and real vendor compiler feedback, can generate Structured Text PLC code that compiles more than 90% of the time.","keywords":["Structured Text","Programmable Logic Controllers","LLM code generation","vendor-aware generation","retrieval-augmented generation","compiler feedback","IEC 61131-3","industrial automation"],"falsifier":"Hold out entire source library families during retrieval (for example, build the case database only from the open ST library and test on the two SCL sets, then swap), keeping planning, API recommendation, and compiler feedback unchanged; if pass rates stay above 90% the headline result is robust, and if they fall steeply the substring-only filter is doing much of the work.","tokens_in":16345,"feed_emoji":"⚙️","tokens_out":9301,"duration_ms":94449,"temperature":0.7,"pith_summary":"The paper tries to establish that industrial Structured Text (ST) programming, which is fragmented by vendor-specific dialects, can be automated from natural-language requirements without fine-tuning a large language model. It builds two vendor-specific knowledge sources—an API library and a requirement-to-code case library—and routes generation through four stages: natural-language planning, case retrieval with LLM reranking, API recommendation, and an iterative repair loop that feeds compiler errors from the real vendor platform back into the model. On a 914-task benchmark spanning two ST variants, the framework reports compilation pass rates above 90%, with 100% compilation on a 23-task external benchmark and an average cost of about $0.13 per task. Experienced PLC engineers rated the output useful, including code that did not compile. A sympathetic reading is that retrieval plus authentic vendor compiler feedback, rather than larger or specialized models, is the key to making LLM-generated control code usable.","feed_headline":"LLM pipeline compiles PLC code 90% of the time","feed_subtitle":"Vendor API and case retrieval plus real compiler feedback gets past the syntax wall in industrial automation.","key_machinery":"The load-bearing machinery is the paired knowledge infrastructure and a four-stage generation workflow. The API library stores vendor functions with descriptions, parameters, examples, and a generated three-dimensional index (functional summary, usage scenarios, ST keywords), which makes low-level hardware-aware functions retrievable from high-level plans. The case library stores requirement-plan-code-API quadruples, so the model receives proven implementation experience in context. The workflow classifies the task, writes a natural-language plan (state-machine style for process control, linear steps otherwise), retrieves top cases by embedding similarity then LLM reranking, recommends APIs by combining BM25 retrieval, I/O type matching, and similar-case augmentation with LLM filtering, and finally generates code that is repaired against compiler diagnostics from the actual vendor IDE, fixing declaration errors before implementation errors.","core_discovery":"The paper's central claim is that the bottleneck in LLM-based PLC code generation is not reasoning ability but missing vendor knowledge and missing authentic validation. With an API library whose entries are indexed by generated summaries, scenarios, and ST keywords, and a case library of requirement-plan-code-API quadruples, the pipeline lets a general LLM produce code that matches vendor syntax and function signatures. The reported evidence: compilation pass rates of 92.72% on the general-purpose SCL set, 91.11% on the process-control set, and 92.90% on the open ST set, plus 78.30% formal-validation pass rate and 100% compilation on the external 23-task benchmark. The framework outperforms all compared baselines and closes most of the gap between raw LLM performance and the 90%+ target.","pith_inferences":["Beyond the paper's claims, its leakage guard is a substring-name filter between benchmark tasks and cases drawn from the same three source libraries; a stronger test would hold out entire library families to measure how much retrieval of near-duplicate ground-truth code contributes to the reported pass rates.","Beyond the paper's claims, the recipe of explicit state-machine planning, small curated case retrieval, and feedback from the real toolchain is transferable to other stateful, vendor-fragmented languages such as IEC 61499 function blocks or embedded C variants.","Beyond the paper's claims, the expert comments suggest the next concrete extension: teach the repair loop to add input validation and runtime-safety checks, moving the headline metric from 'compiles' toward 'safe to execute'.","Beyond the paper's claims, a community-standard conformance checker that emulates vendor extensions would make future PLC-generation benchmarks reproducible without requiring proprietary IDEs."],"forward_implications":["If correct, the framework shows that per-vendor fine-tuning is unnecessary: a general LLM plus curated knowledge and real compiler feedback reaches above 90% compilation on two different ST dialects.","The ablation data attributes the largest gains to case retrieval and the compiler-feedback repair loop, so subsequent systems should invest in knowledge bases and authentic validation rather than only in stronger models.","At $0.13 per task and about 52 seconds per task, the pipeline is cheap enough to be an interactive engineering aid rather than a batch research tool.","Because the benchmark covers mostly basic functions and small process-control tasks, the established claim is about compilable first drafts; the paper explicitly leaves complex interlocking, recipe management, and large-system integration as future work."],"supporting_citations":[{"why":"Prior LLM-based pipeline for ST generation with iterative verification; serves as a baseline and supplies the 40-task evaluation scale this paper extends.","marker":"[11]"},{"why":"Multi-agent framework with retrieval and formal verification; supplies the external 23-task benchmark and the reported baseline numbers the framework is compared against.","marker":"[34]"},{"why":"Earlier retrieval-augmented control-code generation that motivates injecting function blocks and cases into the prompt.","marker":"[28]"},{"why":"Open-source library that supplies 718 of the 914 benchmark tasks and part of the case database.","marker":"[40]"},{"why":"Vendor-documented library that supplies the 151-task general-purpose SCL subset and part of the case database.","marker":"[45]"},{"why":"Industrial competition dataset that supplies the 45 process-control SCL tasks and part of the case database.","marker":"[4]"},{"why":"Open-source ST compiler used by the adapted prior pipeline baseline; the paper contrasts its unreliable feedback with real vendor IDE compilation.","marker":"[8]"},{"why":"Multi-agent general code-generation baseline adapted to ST/SCL and compared on all benchmarks.","marker":"[20]"},{"why":"BM25 retrieval scoring used in the API recommendation stage to recall candidate vendor functions.","marker":"[42]"}],"fun_headline_variants":["AutoPLC writes vendor-ready PLC code from plain English","90%+ PLC code compiles with AutoPLC's vendor-aware pipeline","From English to Siemens and CODESYS code at 90% compile rate","Open-source AutoPLC turns requirements into compiling PLC code"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark and the case database are built from the same three source libraries, and the only guard against the retriever returning the ground-truth solution is a case-name substring filter.","fun_headline_variants_meta":{"raw":{"variants":["AutoPLC writes vendor-ready PLC code from plain English","90%+ PLC code compiles with AutoPLC's vendor-aware pipeline","From English to Siemens and CODESYS code at 90% compile rate","Open-source AutoPLC turns requirements into compiling PLC code"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000636,"raw_usage":{"total_tokens":2951,"prompt_tokens":982,"completion_tokens":1969,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":598,"completion_tokens_details":{"reasoning_tokens":1897}},"tokens_in":598,"tokens_out":1969,"duration_ms":14967,"temperature":1.0,"reasoning_tokens":1897,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T23:28:56.839653+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Hold out entire source library families during retrieval (for example, build the case database only from the open ST library and test on the two SCL sets, then swap), keeping planning, API recommendation, and compiler feedback unchanged; if pass rates stay above 90% the headline result is robust, and if they fall steeply the substring-only filter is doing much of the work.","supporting_citations":[{"cited_title":"Llm4plc: Harnessing large language models for verifiable pro- gramming of plcs in industrial control systems","cited_arxiv_id":null,"evidence_quote":"Prior LLM-based pipeline for ST generation with iterative verification; serves as a baseline and supplies the 40-task evaluation scale this paper extends."},{"cited_title":"LLM-based and Retrieval-Augmented Control Code Generation","cited_arxiv_id":null,"evidence_quote":"Earlier retrieval-augmented control-code generation that motivates injecting function blocks and cases into the prompt."},{"cited_title":"UDAPDR: Unsupervised domain adaptation via LLM prompting and distillation of rerankers","cited_arxiv_id":null,"evidence_quote":"Vendor-documented library that supplies the 151-task general-purpose SCL subset and part of the case database."},{"cited_title":"Generative AI Application Competition: Code Generation for Industrial Software - Biendata, June 2024","cited_arxiv_id":null,"evidence_quote":"Industrial competition dataset that supplies the 45 process-control SCL tasks and part of the case database."},{"cited_title":"Source code repository, 2024","cited_arxiv_id":null,"evidence_quote":"Open-source ST compiler used by the adapted prior pipeline baseline; the paper contrasts its unreliable feedback with real vendor IDE compilation."}],"review_version":1}