{"id":"5ee750d2-2cc8-470f-8480-18ac3ee4ad3e","arxiv_id":"2509.21074","paper_version":4,"verdict":"REJECT","confidence":"LOW","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"The headline claim, that RepLLM reproduces 95% of benchmarks in two hours with 10% token savings, is absent from the body, which instead reports a different, semi-automated system with no baseline comparison.","lead":"The abstract promises RepLLM, an automated multi-agent system that reproduces 95% of networking benchmarks in about two hours, while the full text describes RePro, a semi-automated prompt-engineering pipeline averaging 38.6 hours. The two parts disagree on the framework, the numbers, and the evaluation, so the headline result is not supported by the submitted manuscript.","discovery_kind":"unclear","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The abstract's headline claim describes RepLLM, but the full text describes a different framework (RePro) and reports no 2-hour, 95%-benchmark, or token-saving results; this unsupported central claim is the load-bearing failure.","rationale":"The stress-test pass agrees with the reader that the paper should be rejected, but the decisive issue is not the evaluation's memorization risk—it is that the artifact's central claim is about a system that the submitted text does not describe. The abstract promises RepLLM's four agents, Shared Memory, sandboxed static-dynamic debugging, 95% benchmark reproduction in roughly two hours, and 10% token savings over baselines. The body, from its own title and abstract onward, presents RePro, a semi-automated prompt-engineering method, with average reproduction time of 38.6 hours and 32.2 human prompts. The only occurrence of 'RepLLM' in the body is Figure 8's caption, which labels a diagram of RePro's pipeline. No measurement of benchmark-reproduction rate, end-to-end wall-clock time, or token consumption appears anywhere in the evaluation. This is not a disagreement with current consensus; it is an internally unsupported central claim. The reader's weakest assumption about LLM memorization and participant familiarity (Sections 7.1 and 5.3) is a genuine secondary concern that would apply even to a coherent RePro evaluation, but it presupposes that the evaluated system is the claimed system; here even that precondition fails. Therefore the verdict should remain REJECT, unchanged from the reader's recommendation: the abstract's quantitative claims cannot be checked against the body, and the concrete source-search test would settle whether this is a versioning error or a substantive mismatch.","tokens_in":28893,"tokens_out":4116,"duration_ms":31679,"concrete_test":"Search the submitted LaTeX/PDF source for 'RepLLM', 'shared memory', 'Content Parsing', 'Architecture Design', 'Audit&Repair', '95%', 'two hours', and 'token consumption'. Confirm whether the body ever defines RepLLM or reports those results. If these tokens appear only in the abstract and Figure 8's caption, the mismatch is confirmed and the abstract's claims should be considered unverified. If a separate RepLLM section or appendix exists in the source but was omitted from the PDF, that versioning error still invalidates the submitted artifact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim, stated in the abstract, is that RepLLM, an end-to-end multi-agent framework with four specialized agents and Shared Memory, reproduces 95% of benchmarks in about two hours with up to 10% token reduction over state-of-the-art baselines. The submitted body is a different paper: its title, abstract, and all sections describe RePro, a semi-automated prompt-engineering framework with a three-stage pipeline (system description extraction, framework-level code generation, function-level code generation, plus code optimization), which reports an average reproduction time of 38.6 hours and 32.2 human prompts (§1, §7.3). No section defines RepLLM's agents, Shared Memory, sandbox-isolated static-dynamic debugging, or a 95% benchmark-reproduction rate. Figure 8's caption says 'RepLLM Architecture', but the surrounding text and the diagram itself describe RePro's stages, not four specialized agents. Sections 7.2–7.5 report per-system latency ratios, error counts, and prompt counts, but never a 95% benchmark-reproduction figure, never a two-hour end-to-end time, and never a token-consumption comparison with baselines. Thus the abstract's headline numbers cannot be checked or reproduced from the manuscript. If the arXiv record is the version of record, the paper's central claim is unsupported by its own evidence.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript as submitted presents RepLLM in its abstract, described as an end-to-end multi-agent framework with four specialized agents, a Shared Memory mechanism, and sandbox-isolated static-dynamic debugging, and it claims reproduction of 95% of original benchmarks within about two hours and up to 10% token reduction over state-of-the-art baselines. The full text, however, defines and evaluates a different system, RePro, a semi-automated prompt-engineering framework with a three-stage pipeline (system description extraction, framework-level and function-level code generation, and code optimization). The body's evaluation covers six open-source networking systems reproduced with five LLMs and reports an average reproduction time of 38.6 hours, an average of 32.2 human-written prompts per reproduction, and detailed syntactic/semantic error-repair analyses. No section of the body defines RepLLM's agents, Shared Memory, or sandbox debugging, and no measurement of token consumption or of a 95% benchmark-reproduction rate appears anywhere in the evaluation.","tokens_in":29203,"tokens_out":6873,"duration_ms":56622,"significance":"If the body's claims for RePro were rigorously established, this would be a useful empirical study of LLM-assisted reproduction of networking systems: the prompt templates in Figures 2–7 are concrete and reusable, and the breakdown of syntactic versus semantic error repair times and human prompt involvement (Results 4–7) is informative for practitioners building similar tools. However, the manuscript's headline contribution, as stated in the abstract, is entirely unsupported by the body: the framework name, architecture, and quantitative claims are different. The evaluation also has confounds that preclude attributing the observed behavior to the proposed framework rather than to LLM memorization of widely available open-source code plus substantial human repair. The paper does provide detailed prompt designs and a transparent error analysis, which I credit as useful material, but the central claims as written are not verifiable from the submitted text.","major_comments":[{"comment":"The abstract's central claim describes RepLLM and reports \"reproduce 95% of the original benchmarks within approximately two hours\" and \"reducing token consumption by up to 10% compared with state-of-the-art baselines,\" but the full text defines and evaluates RePro, a different framework. §7.3, Result 2 reports an average reproduction time of 38.6 hours, no section reports token consumption, and RepLLM's four specialized agents, Shared Memory mechanism, and sandbox-isolated static-dynamic debugging methodology are never defined in the body; the only occurrence of the name \"RepLLM\" in the body is the caption of Figure 8, which actually depicts RePro's pipeline. The abstract's headline numbers therefore cannot be checked or reproduced from the manuscript.","section":"Abstract; §6.1; §7.3"},{"comment":"Table 3 omits DCTCP and HPCC despite §7.2 claiming that \"For NS-3 simulations, DCTCP and HPCC results align closely with the originals\" and despite Figure 9 showing qualitative comparisons for both systems. The \"Avg. Error (%)\" metric in Table 3 is never defined, no number of runs or error bars is provided, and NCFlow's normalized latency of 13.031 is inconsistent with the text's statement that \"most systems have normalized latency near 1.0.\" This undermines the accuracy claim embodied in Result 1 and makes the fidelity comparison for two of the six evaluated systems non-quantitative.","section":"§7.2; Table 3"},{"comment":"The evaluation design does not separate the framework's contribution from LLM memorization and human repair. Participants were asked to reproduce papers with which they were \"already familiar\" (§7.1), all six selected systems have publicly available open-source implementations (Table 2), and semantic error correction explicitly uses a Human-in-the-Loop procedure in which humans identify the error location and design targeted prompts (§5.3). Given that the evaluated LLMs were likely trained on these widely available codebases, the reported reproductions may reflect retrieval of memorized code combined with iterative human guidance rather than generation from the paper text. A concrete remedying test would be to evaluate on papers without public code, with participants unfamiliar with the target paper, or with an additional condition that ablates the human repair component.","section":"§7.1; §5.3"},{"comment":"The evaluation contains no comparison to any state-of-the-art baseline framework, despite the abstract's claim of token reduction \"compared with state-of-the-art baselines,\" and token consumption is never measured. The paper compares reproduced systems only against the original open-source implementations; it does not compare RePro with prior LLM-based reproduction approaches discussed in §3.3, such as the 294.5-prompt pipeline of [35] or LASER [33]. Without such a baseline, the reported 38.6 hours and 32.2 manual prompts cannot substantiate the claimed efficiency advantage, and the abstract's quantitative claim is unverifiable.","section":"§7.1–§7.5; Abstract"},{"comment":"Results 4–7 report precise quantitative values—for example, 100 versus 37 minutes per semantic/syntactic repair, 2.68 versus 1.57 prompts per repair, and 62.1% versus 26.3% human prompt proportions—but none of these numbers appear in any table and they cannot be derived from the figures, which use glyph-only labels and do not include axis legends. The regression in Figure 15 is described as having \"error bands,\" yet no regression coefficients, confidence intervals, R² values, or sample sizes are reported. These omissions leave the paper's quantitative error-analysis conclusions unsupported.","section":"§7.4; Results 4–7"}],"minor_comments":[{"comment":"The full-text title is \"RePro: Leveraging Large Language Models for Semi-Automated Reproduction of Networking Research Results,\" while the abstract describes RepLLM; the manuscript's own front matter is internally inconsistent about which system is being presented.","section":"Title; Abstract"},{"comment":"Figure 8's caption reads \"RepLLM Architecture,\" but the diagram and the surrounding text describe RePro's multi-stage pipeline; the caption should be reconciled with the content.","section":"§6.1; Figure 8"},{"comment":"GRooT's \"Absolute Latency\" values are negative (-0.029 and -0.028), which is impossible for a latency; clarify whether this is a speedup, a different metric, or a labeling error.","section":"§7.2; Table 3"},{"comment":"Several evaluation figures (Figures 10–14) use encoded glyph labels in place of readable textual axis labels and legends, making the plots very difficult to interpret; plain-text labels and legends are needed.","section":"§7.3; Figure 10; Figure 11; Figure 12; Figure 14"},{"comment":"The contribution statement \"First Semi-Automated Framework for Network Research Reproduction\" appears to conflict with §3.3 and Appendix B, which describe LASER [33] as a semi-automated LLM-assisted reproduction framework, albeit for LEO satellite networks; the claim should be qualified to acknowledge this prior work.","section":"§1; §3.3; Appendix B"},{"comment":"The sentence \"For DCTCP and HPCC, we use Cursor [2] to facilitate error correction\" is inconsistent with the same section's statement that participants were \"explicitly prohibited from manual coding\" and with the advertised automation; the role of Cursor should be clarified.","section":"§7.1"},{"comment":"The conclusion claims that RePro \"substantially reduces reproduction time,\" but the evaluation does not include a manual-reproduction baseline; a direct comparison with manual effort would be needed to support this claim.","section":"§8; §7.3"}],"recommendation":"reject","confidential_remarks":"The divergence between the abstract (RepLLM) and the body (RePro) is so large that this may be a submission error: the body does not contain the system described in the abstract. If the authors intended to submit the RePro paper, they need to correct the title and abstract and substantially strengthen the evaluation; if they intended to submit the RepLLM paper, the core content is missing. Additionally, the relationship to the authors' prior HotNets '23 paper [35] should be clarified, since RePro appears to be an extension of that line of work, and no code or supplementary material is provided to support reproducibility claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this arXiv record has two different papers stapled together. The abstract advertises RepLLM, an end-to-end multi-agent framework with four specialized agents, Shared Memory, and a two-hour, 95%-benchmark, 10%-token-saving result. The full text describes RePro, a semi-automated prompt-engineering pipeline whose own evaluation reports 38.6 hours average, 32.2 human prompts, and never measures tokens. Figure 8's caption says 'RepLLM Architecture' but the diagram and surrounding text are RePro's stages. The headline numbers in the abstract cannot be checked against the body.\n\nNow the fair part. The body does real work. RePro's four-step pipeline—system description extraction, framework-level code generation with SCoT, function-level generation with SeCoT, and iterative debugging—is a legitimate extension of the authors' HotNets'23 study and of published SCoT/SeCoT prompting. The evaluation covers six named network systems across traffic engineering, graph mining, transport protocols, and verification. The prompt counts (32.2 human-written, about 95 total) are substantially lower than the 294.5 in the prior work, and the error analysis distinguishing syntactic vs semantic repair is a useful datapoint for anyone building on this. That is a genuine empirical contribution, albeit a modest one.\n\nThe soft spots are serious. First, the abstract/body mismatch is not a typo—it is a load-bearing failure. If the arXiv record is the version of record, the paper claims results it does not present. Second, the evaluation has no baselines: no comparison against plain LLM prompting, no comparison against LASER or the HotNets'23 approach. Third, the systems are all open-source and well-known (DCTCP, HPCC, etc.), so LLM memorization and participant familiarity (explicitly admitted in Section 7.1) make it ambiguous whether the pipeline generates code from the paper or retrieves memorized code plus human repair. Fourth, Table 3 omits DCTCP and HPCC rows despite textual claims, and there are no error bars or statistical tests anywhere. No artifacts are released. These are not minor polish issues.\n\nWho is this for? A reader interested in prompt engineering for systems code might skim the body for the template designs. But I would not cite it in its current form, and I would not bring it to reading group. The underlying idea has merit, but the submission is not honest as written. My recommendation: desk reject, with an invitation to resubmit a corrected manuscript whose abstract matches the body, adds baselines, releases artifacts, and addresses the memorization concern.","headline":"The abstract advertises a multi-agent framework the body never describes; the body's RePro evaluation is interesting but too contaminated to support the paper's claims.","tokens_in":29802,"tokens_out":2618,"would_cite":false,"duration_ms":22441,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims an LLM pipeline can turn networking papers into code that reproduces most benchmarks in hours.","keywords":["large language models","code generation","research reproduction","computer networking","prompt engineering","chain-of-thought reasoning","multi-agent framework","network systems"],"falsifier":"Run the Section 7 protocol on the same six systems with participants who have never seen the papers and no access to the public repositories while recording end-to-end wall-clock time. If the mean comes out near 38.6 hours rather than the abstract's roughly two hours, or errors exceed 5%, the headline claim is contradicted by the paper's own evaluation setup.","tokens_in":28688,"feed_emoji":"🤖","tokens_out":10293,"duration_ms":84832,"temperature":0.7,"pith_summary":"The paper aims to show that a structured, LLM-driven pipeline can take a published computer-networking paper and turn it into runnable code that reproduces the paper's own measured results. The abstract frames this as RepLLM, a four-agent end-to-end framework with shared memory that recovers 95% of original benchmarks in about two hours; the body, however, describes a semi-automated framework named RePro with a three-stage prompt-engineering pipeline and reports an average reproduction time of 38.6 hours across six systems. Read as a whole, the manuscript's central thesis is that decomposing a paper into functional modules, generating code framework-first with structured chain-of-thought reasoning, then debugging module by module with semantic chain-of-thought, cuts the human effort of reproduction from weeks of manual work to about thirty human-written prompts per system. That matters because only about 40% of top networking papers release their code, so a cheaper paper-to-code path would let researchers verify and build on results that would otherwise sit behind missing implementations.","feed_headline":"LLM pipeline claims 95% benchmark reproduction in two hours","feed_subtitle":"A paper-reading agent framework aims to turn networking publications into working code without manual reimplementation.","key_machinery":"The load-bearing machinery is the prompt-engineering pipeline. Structured Chain-of-Thought (SCoT) generates a pseudo-code skeleton constrained to the three fundamental control structures—sequence, branch, and loop—before any code is written, forcing the model to decide module inputs, outputs, and logical flow first. Semantic Chain-of-Thought (SeCoT) then guides function-level implementation by making the model state data flow and control flow explicitly before emitting code. The third piece is the paper-content mapping: every generated function is annotated with a verbatim excerpt of the source paper as its requirement, so the LLM's later reasoning is anchored to the original text rather than to a paraphrase. These pieces are orchestrated in a four-stage workflow—system description extraction, framework-level generation, function-level generation, and code optimization—and evaluated against five LLMs with different context windows and output limits.","core_discovery":"On the body's own terms, the paper's discovery is that a semi-automated prompt-engineering workflow can faithfully re-implement heterogeneous network systems from their paper text. The workflow extracts the system's sub-field, problem, inputs, outputs, and architecture; divides the reproduction into independent functional modules; generates a framework-level scaffold using Structured Chain-of-Thought and few-shot in-context learning; maps verbatim paper content onto each function; fills in functions using Semantic Chain-of-Thought that tracks data and control flow; and then repairs errors by feeding compiler and test feedback back to the LLM, with humans supplying targeted prompts only for stubborn semantic bugs. Across six open-source systems—a traffic-engineering solver, a graph-pattern miner, a DNS verifier, a control-plane verifier, a datacenter transport protocol, and a congestion-control algorithm—the reproduced implementations come within 0–3.1% of original performance on average, with normalized latencies near 1.0 in most cases. The abstract separately claims that a four-agent end-to-end variant, RepLLM, reproduces 95% of original benchmarks in roughly two hours with up to 10% lower token consumption; the body does not evaluate that variant, and its own reported average is 38.6 hours.","pith_inferences":["A decisive test the paper leaves implicit: run the pipeline on papers whose code is not public and with participants unfamiliar with the work; without that, reported reproduction could be retrieval plus human repair.","The modular paper-to-code recipe likely transfers to other system-heavy fields with design-centric papers, such as operating systems, distributed databases, or programmable hardware.","Because semantic repair dominates cost, one natural extension is to close the loop on benchmark numbers: parse the paper's result tables and feed mismatches between reproduced and published values back into the audit stage automatically.","The large gap between the abstract's two-hour claim and the body's 38.6-hour average suggests the authors credit the multi-agent architecture, not prompt structure alone, for most of the speedup; a controlled ablation would show which."],"forward_implications":["If the body's results hold, an LLM prompt pipeline can reproduce six heterogeneous network systems with 0–3.1% average error, making papers without public code viable reproduction targets.","Reproduction drops to an average of 38.6 hours and 32.2 human-written prompts—against the 294.5-prompt baseline for unstructured prompting—making error repair, not paper reading or code writing, the main bottleneck.","Semantic errors are the expensive class, taking about 100 minutes and 2.68 prompts per fix versus 37 minutes and 1.57 for syntactic errors, so future gains depend on automated semantic validation.","If the abstract's RepLLM variant delivers its claimed two-hour, 95% reproduction, the multi-agent coordination and shared-memory design would mark an order-of-magnitude improvement over the body's own 38.6-hour pipeline."],"supporting_citations":[{"why":"Documents the unstructured-prompt baseline requiring an average of 294.5 handcrafted prompts, which RePro aims to beat.","marker":"[35]"},{"why":"Supplies Structured Chain-of-Thought, the method used to generate framework-level code skeletons.","marker":"[23]"},{"why":"Supplies Semantic Chain-of-Thought, the method used for data-flow- and control-flow-aware function generation.","marker":"[27]"},{"why":"Defines chain-of-thought prompting, the underlying reasoning technique the paper adapts.","marker":"[34]"},{"why":"NCFlow is one of the six reproduced systems, providing the traffic-engineering evaluation target.","marker":"[8]"},{"why":"Arya is one of the six reproduced systems, providing the graph-pattern-mining evaluation target.","marker":"[44]"},{"why":"GRooT is one of the six reproduced systems, providing the DNS-verification evaluation target.","marker":"[21]"},{"why":"Tiramisu is one of the six reproduced systems, providing the control-plane-verification evaluation target.","marker":"[7]"},{"why":"DCTCP is one of the six reproduced systems, providing the NS-3 transport-protocol evaluation target.","marker":"[9]"},{"why":"HPCC is one of the six reproduced systems, providing the NS-3 congestion-control evaluation target.","marker":"[24]"}],"fun_headline_variants":["LLM agents reproduce network systems from paper text","RepLLM: multi-agent framework auto-reimplements networking papers","Paper claims 95% in 2h, but tests show 38.6h average","Collaborative LLM agents rebuild network research code","From publication to working code: RepLLM's automated pipeline"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that reproducing six systems that already have public open-source code, with participants who are already familiar with the papers and may guide repairs by hand, measures the framework's ability to generate code from paper text rather than the models' memory of existing code plus human repair.","fun_headline_variants_meta":{"raw":{"variants":["LLM agents reproduce network systems from paper text","RepLLM: multi-agent framework auto-reimplements networking papers","Paper claims 95% in 2h, but tests show 38.6h average","Collaborative LLM agents rebuild network research code","From publication to working code: RepLLM's automated pipeline"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000872,"raw_usage":{"total_tokens":3811,"prompt_tokens":1019,"completion_tokens":2792,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":635,"completion_tokens_details":{"reasoning_tokens":2704}},"tokens_in":635,"tokens_out":2792,"duration_ms":18427,"temperature":1.0,"reasoning_tokens":2704,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:47:04.822821+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the Section 7 protocol on the same six systems with participants who have never seen the papers and no access to the public repositories while recording end-to-end wall-clock time. If the mean comes out near 38.6 hours rather than the abstract's roughly two hours, or errors exceed 5%, the headline claim is contradicted by the paper's own evaluation setup.","supporting_citations":[{"cited_title":"Xiang, Y","cited_arxiv_id":null,"evidence_quote":"Documents the unstructured-prompt baseline requiring an average of 294.5 handcrafted prompts, which RePro aims to beat."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies Structured Chain-of-Thought, the method used to generate framework-level code skeletons."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines chain-of-thought prompting, the underlying reasoning technique the paper adapts."},{"cited_title":"Abuzaid, S","cited_arxiv_id":null,"evidence_quote":"NCFlow is one of the six reproduced systems, providing the traffic-engineering evaluation target."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Arya is one of the six reproduced systems, providing the graph-pattern-mining evaluation target."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"GRooT is one of the six reproduced systems, providing the DNS-verification evaluation target."},{"cited_title":"Abhashkumar, A","cited_arxiv_id":null,"evidence_quote":"Tiramisu is one of the six reproduced systems, providing the control-plane-verification evaluation target."},{"cited_title":"Alizadeh, A","cited_arxiv_id":null,"evidence_quote":"DCTCP is one of the six reproduced systems, providing the NS-3 transport-protocol evaluation target."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"HPCC is one of the six reproduced systems, providing the NS-3 congestion-control evaluation target."}],"review_version":2}