{"id":"f306fbae-84b9-45a1-8ede-c09ad6586e5f","arxiv_id":"2504.18050","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"An LLM-based pipeline that extracts formal protocol formats from RFC documents detected 69 parser bugs (36 confirmed) across nine network protocols.","lead":"ParCleanse uses large language models to convert RFC protocol documents into formal message specifications that act as test oracles for parser implementations. In tests across nine protocols in C, Go, and Python it flagged 69 parser bugs, 36 of which developers confirmed.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The traceability claim conflates the RFC document with the LLM's interpretation of it; because the same GPT-4o model both builds the oracle and judges inconsistencies, a shared hallucination can turn a correct parser into a reported 'bug'.","rationale":"The Reader's verdict is CONDITIONAL, and my concern reinforces that conditionality rather than overturning it. The Reader identified 'RFC documents are correct, complete, and unambiguous' as the weakest assumption. I agree that this is one weakness, but the more load-bearing issue is the conflation of the RFC with the LLM's interpretation of the RFC. The paper's own formalization (Section 3.2, Definition 2) acknowledges that the extracted format F' is approximate, yet the abstract and introduction claim traceability to the document. The mechanism that is supposed to bridge the gap is Phase 3, where the same LLM that produced F' is asked to validate the disputed constraint against the same RFC section. This creates a circularity: a systematic misinterpretation in extraction is likely to be confirmed by the same model in diagnosis. The paper's manual evaluation of inconsistency-identification accuracy is useful but not independent; the 36 developer-confirmed bugs are the strongest independent evidence, and they do establish that at least a substantial fraction of the reported bugs are real. However, the 33 unconfirmed bugs and the general traceability promise are not supported to the same degree. My proposed test directly targets the circularity by using an independent judge, either a different model or blinded human experts, on exactly the inconsistencies that produced the bug count. If that test confirms the classifications, the central claim stands; if not, the paper needs to either weaken the traceability claim or add an independent validation stage. The verdict should remain CONDITIONAL: the approach is promising and partially validated by developer confirmations, but the headline bug count and traceability guarantee should not be taken as settled until the independence of the inconsistency-identification step is demonstrated.","tokens_in":24381,"tokens_out":2802,"duration_ms":30695,"concrete_test":"Re-run Phase 3 on all 86 logical inconsistencies from Table 3 using a different LLM family (e.g., Claude or Llama) as the validator, or have two independent human protocol experts read only the cited RFC section and judge whether the parser's behavior violates the document, blinded to ParCleanse's extracted format and classification. If the alternative validator or human experts do not confirm the 72 reported implementation errors at close to the paper's 97% rate, then Phase 3 is not an independent check and the traceability claim is not established.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central promise is that 'any bugs we find in a protocol implementation can be traced back to the document.' This holds only if the extracted format F' actually reflects the RFC, or if every extraction error is corrected before a bug is reported. Section 3.2 (Definition 2) explicitly allows F' to be an approximation, and Table 2 shows dependent-constraint recall of only 44%, so the extracted oracle is not the document itself. The burden then falls entirely on Phase 3 to separate implementation errors from format extraction errors. But Phase 3 feeds the same GPT-4o model the same RFC section that generated the disputed constraint; if the model misinterpreted the section during extraction, it is likely to repeat that misinterpretation during diagnosis. The validation is therefore not independent of the oracle construction. The paper's 97% 'identification accuracy' is the authors' own manual re-check, not a third-party adjudication. The 36 developer-confirmed bugs are genuine evidence for those specific cases, but they do not validate the general traceability claim for the 33 unconfirmed bugs or for future protocols. In fact, Table 3 already reports 11 format extraction errors among the 90 detected inconsistencies, showing that detected inconsistencies cannot always be traced to document-supported properties; a bug report is traceable to a section, but the section may not actually support the extracted constraint.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ParCleanse, a system that uses GPT-4o to extract formal protocol message formats from RFC documents via a divide-and-conquer DocTree strategy, then uses those formats as a quasi-oracle to validate network protocol parsers. Phase 1 extracts per-section formats and merges them bottom-up; Phase 2 transforms the merged format into a Format Graph and generates positive and negative inputs through property-level mutations solved with Z3; Phase 3, when a parser deviates from the extracted format, backtraces the relevant RFC section and asks the LLM to classify the mismatch as either an implementation error or a format-extraction error, refining the format in the latter case. The evaluation covers nine protocols in C, Go, and Python, reports precision/recall of the extracted formats against a manually built ground truth, compares against ChatAFL and ParDiff, and reports 69 detected bugs with 36 confirmed by developers.","tokens_in":24620,"tokens_out":10378,"duration_ms":93802,"significance":"If the central results hold, this is a useful step toward automating oracle construction and traceability for protocol parser validation. The strongest evidence is the 36 developer-confirmed bugs in widely used implementations, together with a public artifact, a syntax-checked extraction pipeline, and a grounded comparison against established baselines. The main caveat is that the quasi-oracle is an approximate LLM interpretation, not the document itself; the traceability claim therefore needs to be qualified. With that qualification, the paper would be a solid contribution to LLM-assisted software validation and protocol testing.","major_comments":[{"comment":"The inconsistency-identification step is not independent of the oracle construction. The same GPT-4o model that extracted the disputed constraint in Phase 1 (Algorithm 1, lines 3–8) is given the same RFC section in Phase 3 and asked whether the constraint or the parser is correct. If the model misread the section during extraction, it is likely to repeat that misreading when diagnosing, so the reported 97% identification accuracy (Table 3, §5.3.2) and the abstract's traceability claim do not establish that each reported bug is traceable to the document rather than to the model's interpretation of the document. I ask for an independent adjudication of at least a random sample of the 90 detected inconsistencies (e.g., by protocol experts not involved in building the ground truth, or by an ablation in which the Phase 3 validator is prompted only with the raw section and no knowledge of the extracted format), with results reported separately for format-extraction errors.","section":"§4.3"},{"comment":"Dependent-constraint extraction is substantially incomplete: the overall recall is 44%, with 0% recall for BFD, DHCP, and TCP and 0% precision/recall for DHCP and TCP. Since Phase 2 (§4.2) generates negative tests by negating extracted constraints, the 56% of ground-truth dependent constraints not extracted are never exercised, which undercuts the 'complete protocol format' claim in §4.1 and the RQ1 conclusion of 'over 90% precision and recall for most elements.' Please quantify how many of the reported bugs concern dependent constraints, report the coverage of ground-truth dependent constraints by the final oracle, and either soften the completeness claims or add experiments on protocols with more dependent constraints.","section":"§5.2.2, Table 2"},{"comment":"The abstract's statement that 'any bugs we find in a protocol implementation can be traced back to the document' is not supported by Definition 2, which explicitly allows the extracted format F' to deviate from the true format F. A report can be traced to an RFC section, but the section may not entail the extracted constraint; Table 3's own 11 format-extraction errors demonstrate this gap. Please revise the traceability claim to 'traced to a document section and to the LLM-extracted constraint, with the constraint being an approximate interpretation that may require confirmation,' and make the residual risk explicit in the contributions and conclusion.","section":"§3.2, Definition 2; Abstract"},{"comment":"The headline count of 69 bugs depends on the Phase 3 classification, which was manually rechecked by the authors at five minutes per inconsistency rather than by independent developers for all cases. The 36 developer confirmations are strong evidence for those specific bugs, but they do not validate the 33 unconfirmed reports. Please report the per-protocol confirmation rate, separate the '69 detected' and '36 confirmed' claims in the abstract and conclusion, and, where possible, provide evidence that the unconfirmed reports were sent to maintainers with sufficient context (e.g., the traceability data) for them to act on.","section":"§5.3.2, Table 3"}],"minor_comments":[{"comment":"The TCP row lists the description as 'Extensible Authentication Protocol,' which is incorrect for RFC 793; this appears to be a copy-paste error from another protocol.","section":"Table 1"},{"comment":"The MergeFormats function (lines 11–16) does not run SyntaxChecker on the merged format, even though node-level formats are syntax-checked in lines 3–8; a syntax error introduced during merging would be caught only later in Phase 2, if at all.","section":"Algorithm 1, §4.1.2"},{"comment":"The table is very dense because many subcolumns are compressed into a single row; separating the inconsistency-identification and bug-detection parts into two tables, or adding clear column headers, would make it substantially easier to verify the totals.","section":"Table 3"},{"comment":"Generating exactly one positive test case per format path may make results sensitive to Z3's chosen assignment; a short sensitivity analysis (e.g., two or three assignments per path for one protocol) would help establish that the 69-bug count is not an artifact of the solver's choices.","section":"§5.3.1"}],"recommendation":"major_revision","confidential_remarks":"In my view the core approach has merit and the confirmed bugs provide genuine evidence, but the paper's biggest risk is the self-consistency of the LLM oracle and validator. If the authors provide an independent adjudication of a sample of inconsistencies and temper the traceability wording, I would be willing to support acceptance. I have no concerns about citation practice."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Let me cut to the chase. This is a solid tool paper with a genuinely useful pipeline: it uses an LLM to parse RFC documents into formal protocol formats via a divide-and-conquer DocTree, generates property-level positive and negative tests with Z3, and when a parser disagrees, backtraces to the exact RFC section and asks the LLM to judge whether the format or the parser is wrong. The evaluation on nine protocols in C, Go, and Python found 69 bugs, 36 confirmed by developers. That is real evidence, not just a demo.\n\nWhat's new: prior LLM-based fuzzing like ChatAFL extracts coarse formats and mostly catches crashes; differential tools like ParDiff miss bugs shared across implementations. ParCleanse's combination of fine-grained mutation and traceable diagnosis is a real step forward. The ablations are convincing: removing divide-and-conquer tanks format quality, removing traceability drops inconsistency identification accuracy from 97% to 84%, and removing fine-grained testing collapses bug detection to mostly crashes. Those numbers support the design decisions.\n\nNow the soft spots, in proportion. The central claim that every bug can be traced back to the document is overstated. Section 3.2 defines the extracted format as an approximation, and dependent-constraint recall is only 44%. If the same LLM that built the oracle also reads the same RFC section during diagnosis, a shared hallucination can turn a correct parser into a false 'bug'. The 97% identification accuracy is the authors' own manual re-check, not independent adjudication. And there are no repeated-run statistics: temperature 0 helps, but GPT-4o is still nondeterministic. So the 36 confirmed bugs are solid for those specific cases, but they don't license the general traceability promise, especially for the 33 unconfirmed bugs.\n\nThat said, the paper is honest about the approximation and explicitly assumes RFCs are ground truth in Section 6. The limitations section names TCP header padding and dependent constraints as known weak spots. This is not a paper that hides its weaknesses.\n\nWho should read it: anyone working on protocol testing, LLM-based oracles, or specification mining from natural language. It deserves serious peer review. My recommendation: send it to review, with a request for the full artifact with a commit hash, a clearer separation of confirmed and unconfirmed bugs in the headline, and ideally a sample of bug reports adjudicated by an independent party.","headline":"A solid, well-engineered tool paper that turns RFCs into executable test oracles via LLMs and finds real developer-confirmed bugs; the traceability claim is stronger than the evidence supports, but it deserves serious peer review.","tokens_in":25162,"tokens_out":2752,"would_cite":true,"duration_ms":25737,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ParCleanse shows that an LLM can distill RFC documents into a formal message format used as a quasi-oracle for validating protocol parsers, finding 69 bugs (36 confirmed) across nine protocols.","keywords":["network protocol parsers","large language models","RFC documents","formal specification extraction","traceability","quasi-oracle","divide-and-conquer","parser validation"],"falsifier":"Run ParCleanse on an RFC that carries a known erratum or a deliberately planted error in one field constraint, and observe whether a correct parser is reported as buggy; if it is, the oracle has inherited the document's error, showing the load-bearing assumption is wrong.","tokens_in":24145,"feed_emoji":"🐛","tokens_out":13642,"duration_ms":110580,"temperature":0.7,"pith_summary":"ParCleanse claims that a network protocol parser can be checked against a quasi-oracle — an approximate formal format, extracted automatically from the protocol's own RFC document — instead of relying on hand-written models or other implementations. The paper's pipeline splits an RFC into a tree of sections, uses a large language model to translate each section into a formal message format in a small DSL, merges the sub-formats bottom-up, and then generates one valid packet per format path plus one mutated packet per violated property. Any disagreement between parser and format is traced back to the exact RFC section and triaged by the LLM as either an implementation bug or a format-extraction error, with 97% accuracy on the nine protocols studied. The result is 69 unique bugs (36 confirmed) in C, Go, and Python parsers, mostly silent input-validation faults that crash-free fuzzing and differential testing miss. If the approach carries over, it would let standards documents themselves act as the test oracle across the network protocol ecosystem.","feed_headline":"RFC-reading LLM builds parser oracle, finds 69 bugs","feed_subtitle":"ParCleanse distills RFCs into machine-checkable formats, probes nine protocol parsers, and confirms 36 bugs.","key_machinery":"The load-bearing object is the DocTree, a hierarchy of RFC sections in which each node stores the section's text, an LLM-written summary, and the protocol sub-format extracted from that section; edges record dependencies that the table of contents does not show, such as a TLV section being a child of the packet-format section. The DocTree enables three things at once: divide-and-conquer extraction that keeps LLM context small, bottom-up merging of sub-formats into one complete format, and traceability, since every generated constraint records the section that authored it. The second mechanism is the Format Graph, a DAG of fields and subgraphs built from the merged format; each path through it is encoded as an SMT formula, and the solver produces a positive packet for the path and a negative packet for each individual constraint negation. The combination means every test input and every later diagnosis can cite the exact RFC section that justifies it, which is what lets the pipeline separate parser bugs from its own extraction errors.","core_discovery":"The central discovery is operational: a large language model, guided by a document-structure tree, can convert the natural-language format descriptions of RFCs such as those for Babel, BFD, BGP-4, IPv4/IPv6, ICMPv4/v6, DHCP, and TCP into a formal message grammar whose constraints are precise enough to drive test generation. The paper demonstrates that this grammar, used as a quasi-oracle, catches parsers that accept a packet violating a documented constraint (for example, a Babel Router-Id TLV whose RouterId field is all zeroes, where the RFC states it must satisfy $RouterId \\neq 0$) or reject a packet that satisfies every constraint. Because each constraint retains a pointer to the RFC section it came from, suspected violations are re-adjudicated against that section, and the paper reports that this triage is correct for 87 of 90 detected inconsistencies (97%), separating 79 implementation errors from 11 format-extraction errors. The paper further claims that this pipeline outperforms the state of the art: 100% precision/recall on message types versus 89%/55% for the LLM-based fuzzing baseline, and 33 unique new bugs on the C protocols versus 4 for differential analysis, with 68 of 69 total bugs new and 36 confirmed.","pith_inferences":["The oracle inherits the standard's own ambiguities, so the same pipeline could be pointed at RFCs with known errata to make the tool double as a standards-quality checker rather than only a parser validator.","Where extraction is weakest — dependent constraints at 44% recall, with TCP's implicit padding alignment as the documented failure mode — the property-level mutation strategy could be extended to cross-field mutations, not just single-constraint negations, to probe interactions between dependent fields.","The black-box accept/reject/crash signal could be enriched with partial-parse or warning outputs from parsers, potentially turning silent state corruptions into earlier, observable inconsistencies.","Because the DSL is language-independent and the tool needs only an executable, the same DocTree pipeline is a natural candidate for protocols with bit-level layouts (for example TLS or QUIC), where implicit alignment rules are likely to stress the extraction step further."],"forward_implications":["A parser can be validated directly against its standard document, so bugs shared by every implementation of a protocol — which differential testing cannot see — become detectable.","Because every reported inconsistency is traceable to the RFC section that defines the violated property, the paper's ablation shows that without this backtrace inconsistency-classification accuracy drops from 97% to 84% and format-error detection to 0%.","Property-level mutation is what finds silent logical bugs: whole-format testing yields mostly crashes (211 of 233 inconsistencies), while fine-grained testing yields 75 logical inconsistencies and 69 unique bugs.","Divide-and-conquer via DocTree is essential for long documents: feeding the whole RFC to the LLM drops field-name recall from 95% to 54% and independent-constraint recall from 82% to 26%.","The reported 97% inconsistency-classification accuracy and 100% precision/recall on message types indicate that the extracted format can serve as an oracle for these nine protocols."],"supporting_citations":[{"why":"Supplies the Go parsers for IPv4, ICMPv4, and ICMPv6 used in the evaluation; without it the multi-language claim is untested.","marker":"[2]"},{"why":"The online RFC repository from which the nine standards documents were downloaded; it is the sole input to the extraction pipeline.","marker":"[3]"},{"why":"Supplies the Python parsers for IPv6, DHCP, and TCP in the evaluation.","marker":"[4]"},{"why":"The C protocol suite whose Babel, BFD, and BGP-4 parsers are test subjects, including the motivating RouterId bug.","marker":"[12]"},{"why":"The SMT solver that turns each format path into satisfiable assignments for positive inputs and single-property violations for negative inputs.","marker":"[13]"},{"why":"The LLM-based protocol fuzzing baseline whose format-extraction results ParCleanse reports outperforming on every format element.","marker":"[31]"},{"why":"The syntax checker that validates each LLM-generated sub-format and returns errors for regeneration, grounding Phase 1.","marker":"[39]"},{"why":"The differential parser-analysis baseline whose bug counts on the C protocols ParCleanse reports exceeding (33 unique new compared with 4).","marker":"[58]"}],"fun_headline_variants":["LLM turns RFCs into parser oracle, catches 69 bugs","Natural-language specs become bug detectors for parsers","Parse bugs traced to RFC sections via LLM oracle","LLM-powered oracle validates protocol parsers, finds 69 bugs","From RFC text to formal grammar: 69 parser bugs caught"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The pipeline treats the RFC text as the complete, correct, and unambiguous ground truth, so any error, omission, or ambiguity in the standard is inherited by the oracle and can turn a correct parser into a reported bug.","fun_headline_variants_meta":{"raw":{"variants":["LLM turns RFCs into parser oracle, catches 69 bugs","Natural-language specs become bug detectors for parsers","Parse bugs traced to RFC sections via LLM oracle","LLM-powered oracle validates protocol parsers, finds 69 bugs","From RFC text to formal grammar: 69 parser bugs caught"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000224,"raw_usage":{"total_tokens":1527,"prompt_tokens":1078,"completion_tokens":449,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":694,"completion_tokens_details":{"reasoning_tokens":366}},"tokens_in":694,"tokens_out":449,"duration_ms":3943,"temperature":1.0,"reasoning_tokens":366,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:25:40.774526+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run ParCleanse on an RFC that carries a known erratum or a deliberately planted error in one field constraint, and observe whether a correct parser is reported as buggy; if it is, the oracle has inherited the document's error, showing the load-bearing assumption is wrong.","supporting_citations":[{"cited_title":"Go Networking","cited_arxiv_id":null,"evidence_quote":"Supplies the Go parsers for IPv4, ICMPv4, and ICMPv6 used in the evaluation; without it the multi-language claim is untested."},{"cited_title":"IETF DataTracker","cited_arxiv_id":null,"evidence_quote":"The online RFC repository from which the nine standards documents were downloaded; it is the sole input to the extraction pipeline."},{"cited_title":"Impacket","cited_arxiv_id":null,"evidence_quote":"Supplies the Python parsers for IPv6, DHCP, and TCP in the evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The C protocol suite whose Babel, BFD, and BGP-4 parsers are test subjects, including the motivating RouterId bug."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The syntax checker that validates each LLM-generated sub-format and returns errors for regeneration, grounding Phase 1."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The differential parser-analysis baseline whose bug counts on the C protocols ParCleanse reports exceeding (33 unique new compared with 4)."}],"review_version":1}