{"id":"011279aa-0e10-4424-9240-c196973be152","arxiv_id":"2506.14627","paper_version":2,"verdict":"UNVERDICTED","confidence":"HIGH","novelty_score":2.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"The paper surveys 94 papers on LLM-assisted formalisation of software requirements, organised around formalisation, traceability, formal methods, and UTP/institutions.","lead":"This preprint is a working-document literature survey of 94 papers on using large language models to turn natural language software requirements into formal specifications. A smart generalist would read it to get a scattered but current map of tools like nl2spec, AssertLLM, SAT-LLM, and Isabelle/UTP without any new experiments.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The selection underlying the 94-paper review is unauditable because Section 2 omits search strings and screening counts, so RQ2's future-directions claims rest on an unverifiable corpus; the appendix's misaligned rows show the data layer itself contains errors.","rationale":"The reader's UNVERDICTED verdict is appropriate: this is a self-described working document with no new empirical result, so the relevant correctness question is whether the survey's corpus is accurate and representative. The weakest point is exactly the reproducibility of the 94-paper selection and the reliability of its summaries. I agree with the reader's weakest_assumption. The missing PRISMA-style accounting is not merely an editorial nicety, because Section 7 derives future research directions from the corpus; if the corpus is biased, those directions are unsupported. The appendix table misalignments strengthen the concern by providing direct, observable evidence that the summary layer contains errors. I do not think this moves the verdict: the paper already declares itself a working document, and a survey with these deficiencies is best marked UNVERDICTED rather than accepted or rejected outright. The practical remedy is to supply the omitted search details, screening counts, and corrected table rows.","tokens_in":21300,"tokens_out":9761,"duration_ms":96664,"concrete_test":"Require the authors to supply the exact search strings and screening counts omitted from Section 2, then independently rerun the five-database search, deduplicate by DOI, and apply the stated inclusion/exclusion criteria at title, abstract, and full-text stages. Compare the regenerated set with the published 94 papers and report precision and recall. If the 94 cannot be reproduced, or if a recognised cluster such as NL-to-LTL translation, assertion generation, or specification-synthesis tools is missing or over-represented, the RQ1/RQ2 conclusions are contingent on an undocumented selection rather than a verifiable literature base.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that it is a structured review of 94 papers on LLM-based formalisation of software requirements, with RQ2 drawing future-direction conclusions from that corpus. For that claim to hold, the selection must be reproducible and the per-paper summaries accurate. Section 2 names databases and the Elicit tool but gives no exact query strings, no deduplication or screening counts, no PRISMA-style flow diagram, and no inter-rater procedure. With raw database counts ranging from 17 (IEEE) to 14,800 (Google Scholar), the step from those results to 94 included papers is unauditable. The appendix gives independent evidence that the summary layer is not fully reliable: Table 1 swaps the descriptions for [4], [5], and [50] (SWOT analysis, JML comparison, and domain-model extraction), and Table 2 repeats [61] Laurel after it already appears in Table 1. These are not accusations of misconduct; the document explicitly labels itself a working draft. But the observable errors and the missing screening protocol mean the survey's characterisation of the field, and the RQ2 conclusions built on it, cannot be checked against the underlying literature.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript is a working-document literature survey that claims to summarise 94 papers on using large language models (LLMs) to formalise software requirements. It addresses RQ1 by reviewing papers that translate natural-language requirements into formal notations and RQ2 by proposing future directions around prompt engineering, chain-of-thought, retrieval-augmented generation, and neuro-symbolic approaches. It also contains separate sections on requirements traceability, formal methods and tools, UTP, and the Theory of Institutions, followed by appendix tables that summarise the reviewed literature. The paper explicitly identifies itself as a draft and refers readers to two related papers by the same authors for the abstract.","tokens_in":21483,"tokens_out":4541,"duration_ms":46855,"significance":"If the corpus selection and per-paper summaries were accurate and reproducible, this survey would be a useful entry point to a fast-moving area, particularly because it collects industrial case studies and connects LLM-based formalisation to verification tools such as Dafny, VeriFast, and SPIN. The paper is honest about its working-document status and clearly states its research questions. However, the value of the survey is entirely dependent on the trustworthiness and auditability of the summary layer, and the current manuscript contains concrete errors in that layer. The paper does not claim a new empirical result or theory, so its contribution is organisational and descriptive.","major_comments":[{"comment":"The selection process for the 94-paper corpus is not reproducible. The section names five databases and the Elicit tool, but it does not provide the exact search strings, the date of each search, the number of records screened, the number excluded at each stage, a PRISMA-style flow diagram, or any inter-rater agreement procedure. With database counts ranging from 17 (IEEE Xplore) to 14,800 (Google Scholar), the step from raw results to the final 94 included papers is unauditable. This matters because the RQ2 future-directions claims in Section 7 are conclusions drawn from that corpus; without a verifiable selection, those conclusions cannot be checked.","section":"Section 2 (Methodology for Literature Review)"},{"comment":"Table 1 misassigns the summaries for references [4], [5], and [50]. The body text at the end of Section 3 attributes the SWOT analysis and initial evaluation to [4], the JML comparison between symbolic NLP and ChatGPT to [50], and the domain-model extraction with industrial case studies to [5]. Table 1 instead assigns the SWOT description to [50], the JML comparison to [5], and the domain-model extractor to [4]. Because the paper's central claim is a structured and accurate summary of the literature, these row-level errors directly undermine the reliability of the survey's data layer and must be corrected and checked across all tables.","section":"Appendix A, Table 1"},{"comment":"Reference [61] (Laurel) appears twice: once as the last row of Table 1 and again as the first row of Table 2, with identical descriptions. The duplication suggests the tables were assembled without a systematic de-duplication check and raises the question whether other papers are duplicated or omitted. The paper claims 94 papers are summarised, but the appendix tables list only a subset of the references and there is no complete enumeration of the included papers anywhere in the manuscript, so the claimed corpus size cannot be independently verified.","section":"Appendix A, Tables 1 and 2"},{"comment":"The abstract is not self-contained: it instructs the reader to 'refer to abstract of [7,8]' instead of providing an abstract, and the phrase 'nighty-four' instead of 'ninety-four' appears in both the abstract and the introduction. The manuscript also states in its title and opening sentence that it is a working document. These features are acceptable for an arXiv draft but are not appropriate for a journal submission; the abstract must stand alone and the provisional-status language should be removed or clearly qualified if the paper is being submitted for formal publication.","section":"Abstract and title"}],"minor_comments":[{"comment":"There are numerous typographical errors, including 'in-sufficient' in Section 2, 'challange' in Section 3, 'compromising' for 'comprising' in the description of [14], 'sematic' for 'semantic' in the same entry, and 'specifes' in Section 3. A careful proofreading pass is needed.","section":"Throughout"},{"comment":"The traceability section presents a reasonable collection of works, but it does not indicate which of these papers were part of the 94-paper LLM corpus and which are general traceability background. This distinction should be made explicit so the reader can see how the section relates to RQ1 and RQ2.","section":"Section 4"},{"comment":"Sections 5 and 6 provide background on formal methods, testing, UTP, and the Theory of Institutions, but their connection to the LLM-focused research questions is not stated. A short paragraph at the start of each section explaining why this background is included and how it supports the survey would improve coherence.","section":"Sections 5 and 6"},{"comment":"The column headers are inconsistent across tables: Tables 1-3 use 'Tool / Framework / Technique', while Tables 4-6 use 'Tool / Framework / Methodology Devised'. In addition, reference [42] appears in two separate rows of Table 5, once for the prompt-engineering review and once for the lost-in-the-middle result; these could be merged or cross-referenced for clarity.","section":"Appendix tables"},{"comment":"A few references lack complete bibliographic detail, such as [6] and [35], which omit page numbers or DOI information. The reader is also directed to two closely related self-citations [7] and [8] in the abstract; the relationship between this manuscript and those papers should be clarified in the text rather than only through the reference list.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The manuscript appears to be one of several near-identically titled documents by the same group (see references [7] and [8]), and this draft even directs readers to those papers for the abstract. Before considering this for publication, the editor should ask the authors to consolidate the documents or clearly differentiate the contribution of each, and to verify that the claimed 94-paper corpus is accurately represented in the appendix tables. I did not find evidence of misconduct, but the overlapping self-releases make the paper's scope and novelty difficult to assess."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nQuick take: this is a self-described working draft that summarises 94 papers on using LLMs to formalise software requirements. There is no new method, no new measurements, no formal results. Its value is as a rough map and a bibliography — but the map is not reliable enough to trust without checking entries, and the selection process is not auditable.\n\nWhat it does well: it gathers a broad reference list spanning the LLM-to-formal-spec core as well as traceability, formal methods tools, UTP, and institutions. For someone entering the area, the reference list alone can save time. The authors are upfront that this is a working document and point to two related conference papers [7, 8], which is honest about its status.\n\nThe soft spots are real and proportionate. Section 2 reports database counts and mentions Elicit, but gives no query strings, no per-stage screening counts, no inter-rater procedure, and no flow diagram. That makes the 94-paper corpus unauditable, and RQ2's future-directions claims rest on that corpus. The appendix confirms the data layer is shaky: [4], [5], and [50] are misdescribed relative to the body (the SWOT analysis, JML comparison, and domain-model extraction descriptions are cycled), and [61] appears in both Table 1 and Table 2. There are also typos like “nighty-four.” None of this is a load-bearing flaw in a scientific claim, because there is no scientific claim — it is a descriptive survey, and the description is currently too error-prone to cite as authoritative.\n\nWho it's for: a newcomer who wants a starting bibliography and is willing to verify every entry. Not for someone who needs a dependable systematic review.\n\nMy recommendation: I would not send this version to referees. I'd desk reject with an invitation to resubmit once the methodology is reported properly, the tables are corrected, and the document is promoted from working draft to actual survey. The topic deserves a serious review; this manuscript is not ready for referee time.","headline":"A self-described working draft that is a useful but unreliable bibliography; the 94-paper corpus is unauditable and the summary tables contain swapped entries, so it should not be cited as a systematic review yet.","tokens_in":22051,"tokens_out":3375,"would_cite":false,"duration_ms":32211,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper presents a structured review of 94 studies on using large language models to formalise natural-language software requirements, and it outlines a research agenda for closing the gap between informal requirements and formally…","keywords":["large language models","software requirements formalisation","formal specifications","requirements traceability","structured literature review","prompt engineering","chain-of-thought reasoning","theorem proving"],"falsifier":"Run the same five-database keyword searches with documented search strings and screening counts, and check whether the 94-paper list is reproduced; if a substantial cluster of relevant 2023–2025 papers on LLM-based formalisation is missing or misclassified, the survey's map of the field would not stand.","tokens_in":21053,"feed_emoji":"📋","tokens_out":9088,"duration_ms":79794,"temperature":0.7,"pith_summary":"The paper is a working-document survey claiming to summarise ninety-four studies of how large language models help write formal software specifications. It organises the field around two questions: which methodologies translate natural-language requirements into formal notations, and what trends and future directions are emerging. Alongside the core survey, it adds background sections on requirements traceability, formal methods and their tools, and the theoretical frameworks UTP and the Theory of Institutions. The authors intend the review to serve as a map of the area and to motivate a research agenda, called VERIFAI, for combining prompt engineering, chain-of-thought reasoning, and neuro-symbolic verification. If the survey is accurate, it gives newcomers and practitioners a single starting point for understanding what LLM-based formalisation can and cannot do today.","feed_headline":"94 papers reviewed: LLMs turning requirements into formal specs","feed_subtitle":"Answers how LLMs translate natural-language requirements into formal notations, plus open challenges.","key_machinery":"The load-bearing object is the corpus of 94 selected papers, assembled through keyword searches in five digital libraries and a manual screening pass using explicit inclusion and exclusion criteria. The review machinery consists of the two research questions, RQ1 and RQ2, which organise the synthesis, and the summary tables that condense each paper's tool, framework, and reported results. This structure lets the authors convert a large, heterogeneous literature into a claim about the field's current methodology and its likely next steps, such as the VERIFAI agenda of prompt refinement, chain-of-thought reasoning, and neuro-symbolic verification.","core_discovery":"On its own terms, the paper's central claim is that the field of LLM-based requirements formalisation is real but fragmented: a growing set of tools translate natural language into temporal logics, assertion languages, and program specifications, often with reported accuracies in the 70–95% range, but each tool is a point solution tied to a particular notation or domain. The survey shows that coupling LLMs with verifiers—SMT solvers, bounded model checkers, theorem provers—is a recurring theme, and that verification of the generated artefacts, not just generation, is the persistent open problem. The paper presents the 94-paper corpus as evidence for this picture and derives future directions from it, including iterative refinement, retrieval-augmented prompting, and hybrid neuro-symbolic pipelines.","pith_inferences":["Because the reported accuracies come from different domains and benchmarks, a quantitative cross-study comparison is not yet possible; a standardised benchmark for NL-to-formal translation would turn this survey into a meaningful leaderboard.","The inclusion of pre-LLM work such as controlled natural language and ARSENAL suggests that the formalisation problem predates LLMs, so LLM tools could be evaluated against those older baselines as a sanity check for genuine progress.","The paper's future-directions section points toward VERIFAI, but the survey itself does not evaluate any single approach; an immediate testable extension would be to run the identified prompting strategies on a fixed set of requirements and compare verified specification rates."],"forward_implications":["The review implies that LLM-based formalisation is best understood as a collection of domain-specific tools rather than a single general-purpose method.","A visible trend across the reviewed papers is coupling LLMs with verifiers such as SMT solvers, model checkers, and theorem provers to check or repair generated specifications.","Prompt engineering techniques, including chain-of-thought and retrieval-augmented generation, are presented as promising levers for improving translation accuracy, though the survey notes zero-shot can sometimes outperform few-shot.","Verification of LLM output, rather than mere generation, is the open problem that recurs across the corpus and drives the proposed future research directions."],"supporting_citations":[{"why":"The AI literature assistant used in the methodology to locate and summarise candidate papers for the review.","marker":"[20]"},{"why":"The nl2spec framework, a canonical example of LLM-based translation of natural language into temporal logics with iterative refinement.","marker":"[17]"},{"why":"AssertLLM, a key example of generating hardware verification assertions from specifications with reported 89% correctness.","marker":"[23]"},{"why":"SAT-LLM, the main example of integrating SMT solvers with LLMs to detect conflicting requirements.","marker":"[24]"},{"why":"SpecGen, an example of LLM-based specification generation with mutation-based repair, evaluated on SV-COMP benchmarks.","marker":"[57]"},{"why":"ESBMC-AI, the example of combining LLMs with bounded model checking to detect and fix software vulnerabilities.","marker":"[81]"},{"why":"Lemur, a representative framework integrating LLMs with automated reasoners for program verification.","marker":"[89]"},{"why":"The 2009 ACM survey on formal specifications and testing, which supplies the classification of formal specification languages used in the formal methods section.","marker":"[41]"},{"why":"The tutorial introduction to UTP and designs, which grounds the paper's section on Unifying Theories of Programming.","marker":"[88]"},{"why":"The institutions paper, which grounds the section on the Theory of Institutions.","marker":"[32]"}],"fun_headline_variants":["94 papers: LLMs turn requirements into formal specs","Survey: LLMs formalise requirements, but verification lags","LLM requirements formalisation: 94 papers, one open gap","Formal specs from plain language: 94-paper survey","LLMs write formal specs, verification still unsolved"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the 94 selected papers are representative and accurate enough to support the survey's characterisation of the field and its future-directions claims.","fun_headline_variants_meta":{"raw":{"variants":["94 papers: LLMs turn requirements into formal specs","Survey: LLMs formalise requirements, but verification lags","LLM requirements formalisation: 94 papers, one open gap","Formal specs from plain language: 94-paper survey","LLMs write formal specs, verification still unsolved"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000166,"raw_usage":{"total_tokens":1232,"prompt_tokens":903,"completion_tokens":329,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":519,"completion_tokens_details":{"reasoning_tokens":246}},"tokens_in":519,"tokens_out":329,"duration_ms":3283,"temperature":1.0,"reasoning_tokens":246,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:48:22.195090+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same five-database keyword searches with documented search strings and screening counts, and check whether the 94-paper list is reproduced; if a substantial cluster of relevant 2023–2025 papers on LLM-based formalisation is missing or misclassified, the survey's map of the field would not stand.","supporting_citations":[],"review_version":2}