{"id":"ab127c2f-6609-47a7-b0a2-89007fe673ed","arxiv_id":"2608.11924","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"An end-to-end research paper generation pipeline implemented as composable skills in an existing coding assistant reports high citation validity and figure editability while bounding failed research directions.","lead":"Spark-to-Paper is a system that writes complete research papers using thirteen composable skills inside an AI coding assistant, without a separate agent platform. It plans experiments before writing results, runs integrity and review checks, and reports 99.5% citation validity, 96.4% figure editability, and 92% detection of injected unsupported claims, at $8.1 and 3.2 hours per paper.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Claim-admission labels in Appendix C are assigned by the same model that wrote the manuscript, with no ground-truth audit; the 36-probe ablation measures recall for seeded unsupported claims, not label accuracy, so the evidence-centrality claim is not yet supported.","rationale":"The reader's verdict already identifies the same load-bearing assumption, and my read does not move it. The architectural half of the central claim, thirteen composable skills inside a coding assistant with no standing infrastructure, is directly demonstrated by the run instrumentation and is not in dispute. The evidence-centrality half, however, has no direct quantitative support: the only reported number tied to semantic claim assessment is the fabrication-detection ablation, and that number measures the system's ability to remove seeded unsupported claims, not the correctness of the five-way labels that drive claim revision. A biased labeler could still pass the full stack while systematically converting unsupported claims into supported ones, because Appendix A's deterministic gates explicitly exclude semantic judgment. That is precisely the failure that would falsify the paper's strongest contribution. I therefore agree with the reader's weakest_assumption and recommend no change to the CONDITIONAL verdict; the condition should be that the authors release the probe corpus and provide an independent held-out audit of the claim-admission labels.","tokens_in":16941,"tokens_out":5881,"duration_ms":62580,"concrete_test":"Construct a held-out evaluation set of at least 100 manuscript claims with known ground-truth support labels, drawn from the eight generated papers' own experiment artifacts plus the 36 seeded probes; have three independent human raters label each claim as supported, partially-supported, unsupported, contradicted, or needs-confirmation; run Spark-to-Paper's claim-admission protocol on the same claims; and report exact-match accuracy and a confusion matrix against the majority human label, including the unsupported-labeled-as-supported error rate. Also release the 36-probe corpus so the original ablation can be re-run. If exact-match accuracy is below roughly 85%, or if unsupported-labeled-as-supported errors exceed a small threshold, the evidence-centrality claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that experimental evidence governs claim acceptance, revision, or abandonment rests on the claim-admission protocol in Appendix C, where each claim is labeled supported, partially-supported, unsupported, contradicted, or needs-confirmation and the assigned label triggers the revision action. These labels are produced by the same Claude-family model that wrote the manuscript; the only quantitative evidence offered is the 36-probe ablation in Section 7.3. That ablation has three gaps. First, the probe corpus is not released, so its representativeness cannot be checked; a fixed set of 36 probes cannot establish the error rate of a five-way semantic classification. Second, the measured quantity is recall for seeded unsupported claims (14% to 92%), not the accuracy of the five-way labels; it does not measure the false-negative rate for claims that are unsupported but labeled supported, nor does it measure over-weakening of supported claims. Third, there is no independent audit of the label assignments from the eight full runs; the case studies in Figure 6 are qualitative and potentially cherry-picked. The deterministic gates in Appendix A explicitly disclaim semantic judgment, so if the model systematically overrates the support its own text provides, the gates will pass and the system will terminate with a confident manuscript whose claims are not actually evidence-backed. This is exactly the failure mode that would falsify the evidence-centrality half of the central claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents Spark-to-Paper, a system that generates complete research papers inside an existing coding assistant using thirteen composable skills, without a separate orchestration service. The system separates model-based judgment from deterministic checks, separates experiment planning from reporting so that required evidence is specified before results are observed, and uses the resulting evidence to retain, weaken, remove, or contradict manuscript claims. It also introduces a bounded recovery mechanism for a failure mode called the Self-Refutation Loop and generates editable vector figures via deterministic plotting for measured results and code-based reconstruction for method diagrams. The evaluation reports 99.5% citation validity and 96.4% figure editability over eight controlled topics, a fabrication-detection improvement from 14% to 92% when the full integrity and review stack is added, 74% review precision, and average costs of 11.9M tokens, $8.1, and 3.2 hours per manuscript.","tokens_in":17103,"tokens_out":5484,"duration_ms":60847,"significance":"If the empirical claims hold, the paper makes a genuinely useful architectural contribution: it shows that a lightweight skill-based pipeline inside a coding assistant can match or exceed the artifact-level reliability of standalone autonomous research systems while remaining infrastructure-free. The separation of deterministic integrity gates from model-based semantic judgment is a principled design that makes many properties independently checkable, and the paper is commendably transparent about the provenance of each number: controlled measurements, retrospective audits, and self-reported showcase counts are labeled as such. The fixed-probe ablation and blinded human review of raised issues are appropriate evaluation instruments. However, the central claim that experimental evidence governs claim acceptance, revision, or abandonment currently rests on a model-internal claim-admission protocol that has not been independently audited, and the main quantitative evaluation is small and partly retrospective. These are fixable with additional reporting and analysis, and the underlying system design appears sound.","major_comments":[{"comment":"The central evidence-centrality claim is currently supported only by a model-internal claim-admission protocol with no independent audit. The reported 14%→92% ablation measures detection of 36 externally seeded unsupported claims, not the accuracy of the five-way labels (supported, partially-supported, unsupported, contradicted, needs-confirmation) that actually trigger the revision actions in Table 5. Since the deterministic gates in Appendix A explicitly disclaim semantic judgment, a systematic tendency of the model to label insufficiently supported claims as supported would pass every gate and terminate with an over-claimed manuscript. Please release the probe corpus, report label-level confusion across all five categories, and add an independent audit of claim-admission labels from the full runs, or narrow the evidence-centrality claim accordingly.","section":"§7.3 and Appendix C"},{"comment":"The main quantitative comparison rests on only eight controlled topics and on retrospective audits of prior systems' self-released artifacts. The paper states that the protocol was registered with an external timestamp and that all outcomes are reported, but no protocol identifier, topic list, or raw outputs are provided, so the pre-registration claim cannot be independently verified. In addition, the reported quality metrics are limited to citation validity and figure editability; these do not directly measure scientific soundness or the correctness of the evidence-to-claim mapping, which is the mechanism the paper identifies as its main contribution. A controlled evaluation of claim-label accuracy on the eight full runs would substantially strengthen the central claim.","section":"§7.2, Table 3"},{"comment":"The efficiency and quality averages may exclude abandoned trajectories. Section 5.3 states that only trajectories whose evidence supports claims are consolidated into a final manuscript and that failed trajectories are preserved as failure reports, but the eight-topic evaluation reports no count of such failures or their cost. If the 11.9M-token, $8.1, and 3.2-hour averages exclude self-refutation aborts and restarts, they understate the true cost of producing a successful manuscript. Please report the trajectory-level breakdown, including the number of aborted trajectories and their resource consumption, so that the cost and reliability claims are fully measurable.","section":"§5.3 and §7.2"},{"comment":"Figure editability is computed over approximately 1,900 elements 'excluding figures that are intentionally rasterized by design,' but the paper does not define the criterion for 'intentionally rasterized' nor state how raster fallbacks from failed HTML reconstruction are classified. Because Section 6.1 explicitly allows fallback to raster when reconstruction is unreliable, the 96.4% editability figure could be inflated by excluding reconstruction failures. Please define the exclusion rule and report editability with fallback-to-raster counted as non-editable, or provide evidence that no reconstruction failures were excluded from the denominator.","section":"§6 and Table 3"}],"minor_comments":[{"comment":"The header '∆tokens (M)∆USD' is missing a separator and should be two distinct column headers, e.g., '∆tokens (M)' and '∆USD', for readability.","section":"Table 4"},{"comment":"The human-written preprint reference point is described as 'eight sampled papers' but no sampling criteria are given; please specify how these papers were selected so that the 97.8% citation-validity interval can be interpreted as a fair reference.","section":"§7.2"},{"comment":"The phrase 'incorrect expectations are highlighted by Bold and Underline text' should read 'highlighted in bold and underlined text'; the current phrasing is ambiguous about which visual style is applied.","section":"Figure 6 caption"},{"comment":"Review-driven revision and experiment-driven claim admission are described separately, but their interaction is unclear when a review issue and an evidence-based claim revision target the same passage; please clarify the ordering or arbitration between these two correction loops.","section":"§3.1, Stage 5 and §4"}],"recommendation":"major_revision","confidential_remarks":"The paper is transparent and the system design is appealing, but the evidence-centrality claim depends on a model-internal label assignment that has not been independently validated, and the main evaluation is small and partly retrospective. The requested additions—an independent audit of claim-admission labels, release of the probe corpus and topic list, trajectory-level cost reporting, and a clearer figure-editability denominator—are all within the scope of a revision and would make the paper's central claims directly checkable. I do not see grounds for rejection if these can be supplied."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this if you work on LLM agents or automated science; it is worth your time. What is genuinely new: the paper implements the full research-to-paper loop inside Claude Code as thirteen composable skills, with no separate orchestration service, and adds three mechanisms that address real failure modes—pre-committed result tables that stay empty until experiments run, a capped self-refutation loop that writes failure reports instead of forcing success, and reconstruction of method figures as editable HTML/PDF vectors. The evaluation is mostly honest: it separates controlled measurements on eight topics from retrospective audits of prior systems, gives bootstrap intervals, validates citations externally, and uses blinded raters for review precision. It even labels its own showcase counts as existence evidence, not benchmarks. That is the right posture.\n\nThe soft spots are real but not disqualifying. The headline comparison against prior systems mixes topics, backbones, and pricing environments, so the claimed gap in citation validity is weaker than it looks. Eight topics is a small sample. The 36-probe fabrication corpus is fixed and unreleased. And the stress-test note lands: the claim-admission labels—supported, partially-supported, unsupported, contradicted, needs-confirmation—are assigned by the same Claude-family model that wrote the manuscript, and the ablation measures recall on seeded unsupported claims, not the accuracy of the five-way classification. The deterministic gates deliberately disclaim semantic judgment, so a model that overrates its own evidence would sail through. The paper is upfront that claim-level diagnosis is model-based, and the design is plausible, but the central claim that experimental evidence governs claim acceptance is not yet measured. That needs an independent audit of labels on a sample of real runs. This is a gap in evidence, not evidence of failure.\n\nCitation pattern is fine. They engage the relevant agent, self-critique, and citation-integrity literature, and the self-citations are to directly relevant prior work. No commit hash pins the code, which is minor but worth flagging for reproducibility.\n\nWho is this for: people building research agents and people designing evaluations for them. It deserves a serious referee. I would send it out, asking for the probe corpus, a commit hash, and an external audit of claim-admission labels before acceptance. Not desk-reject material.","headline":"A solid, honest systems paper that gets the architecture right and is upfront about its measurements, but its central evidence-centrality claim hinges on model-assigned labels that nobody has audited.","tokens_in":17724,"tokens_out":2216,"would_cite":true,"duration_ms":22976,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that end-to-end research paper generation can be implemented as a lightweight, composable skill workflow inside an existing coding assistant, with experimental evidence kept central to how claims are accepted, revised, or…","keywords":["autonomous research agents","research paper generation","composable skills","coding assistants","evidence-grounded claims","citation integrity","figure editability","self-refutation loop"],"falsifier":"An external audit would take a random sample of claims that Spark-to-Paper labeled supported or contradicted and check each against the raw experiment logs, metrics, and source data; if a substantial fraction of supported labels turn out not to be implied by the measurements, the evidence-centrality claim fails. A cheaper check would rerun the 36-probe fabrication corpus while also measuring claim-label accuracy on probes whose evidence status is known.","tokens_in":16658,"feed_emoji":"📄","tokens_out":8432,"duration_ms":78920,"temperature":0.7,"pith_summary":"This paper tries to show that the whole arc of producing a research paper—planning, literature search, writing, running experiments, revising claims, making figures, and assembling a LaTeX project—can be decomposed into thirteen composable skills that run inside an ordinary coding assistant, with no separate agent platform or orchestration service. The reason to care is cost and control: if this works, a researcher can get a complete manuscript with verified citations, executed experiments, and editable figures at roughly $8 and 3.2 hours, with claims revised or abandoned according to measured evidence rather than forced into a successful narrative. The design separates model judgment from deterministic, checkable operations, and separates experiment planning from reporting so that required evidence is committed before results are seen.","feed_headline":"Coding-assistant skills write papers with 99.5% valid citations","feed_subtitle":"The full pipeline runs experiments, checks claims, and makes editable figures without a separate research platform.","key_machinery":"The central mechanism is the skill-pipeline architecture: thirteen skills that share a common project directory and are orchestrated only by a high-level ordering, with each skill deciding how to carry out its task from the current project state. Within each skill, the system separates model-based judgment from deterministic operations that can be executed and checked, such as citation resolution, LaTeX compilation, and plotting from measured data. Two mechanisms do the paper's heavy lifting: the evidence-grounded claim admission protocol, which labels claims as supported, partially supported, unsupported, contradicted, or needs-confirmation and maps each label to a revision action, and the bounded Self-Refutation Loop, which caps experiment–critique–revision cycles at seven and converts an unsupported trajectory into a failure report instead of forcing an apparent success. The figure pipeline also matters: quantitative plots are generated deterministically from measurements, while explanatory figures are produced by an image model as a visual target and then reconstructed in HTML as editable vector PDFs.","core_discovery":"The central claim is that end-to-end research paper generation can be implemented as a lightweight, composable skill workflow inside an existing coding assistant, and that this workflow can keep experimental evidence central to how claims are accepted, revised, or abandoned. On the paper's own terms, Spark-to-Paper establishes this by using thirteen skills that communicate through persistent project artifacts, with deterministic gates for machine-checkable properties and model-based critique for semantic ones. The evaluation reports 99.5% citation validity across 384 references, 96.4% figure editability across roughly 1,900 figure elements, fabrication detection rising from 14% for a single-pass draft to 92% with the full integrity stack, and adversarial review precision of 74%. The full system is reported to use 11.9M tokens, cost $8.1, and take 3.2 hours per manuscript on average.","pith_inferences":["If the claim-admission classifications are as reliable as the 36-probe ablation suggests, the same protocol could be applied to human-authored drafts as a pre-submission integrity audit, not just to generated papers; the paper does not test this application.","The skill-based decomposition implies a modular upgrade path: a better citation resolver, a different plotting library, or a stronger review model could be swapped into one skill without rebuilding the orchestration; the paper does not directly demonstrate this modularity.","The reported cost and runtime assume current model pricing and a particular coding assistant; scaling to many parallel papers or switching to a weaker model would likely change the token and dollar figures, and the paper's single-backbone evaluation does not address cross-model transfer.","A natural extension would be to run the same pipeline on a topic with a known published answer and compare the generated claims against ground-truth literature, which would measure whether evidence-grounded revision produces scientifically sound conclusions beyond citation integrity."],"forward_implications":["A complete paper with verified references, executed experiments, and editable figures can be produced for roughly $8 and 3 hours inside a coding assistant, making the cost comparable to a long interactive session rather than a dedicated research platform.","Because experiment planning is committed before results are observed, the generated paper cannot silently swap in an easier evaluation after the fact; any change to the protocol becomes a visible revision or an abandoned trajectory.","The full integrity and review stack detects 92% of seeded unsupported claims, compared with 14% for a single-pass draft, so most fabrication that survives ordinary drafting would be caught.","Failure is treated as a first-class outcome: when experiments keep contradicting the hypothesis, the trajectory is written up as a failure report and a new idea is tried, so the output corpus can contain negative results rather than only polished successes.","Editable vector figures are the default for both explanatory and quantitative content, which means the generated artifacts can be modified by a human rather than only viewed as embedded bitmaps."],"supporting_citations":[{"why":"Prior end-to-end autonomous research system; supplies the released-paper baseline for citation validity and cost.","marker":"[13]"},{"why":"Successor end-to-end system; provides a second baseline for citation validity and runtime.","marker":"[28]"},{"why":"Prior agent-based research assistant; gives a released-artifact baseline for citation validity and cost.","marker":"[20]"},{"why":"Closest existing skill suite for research, writing, and review inside a coding assistant; shows the gap this paper fills.","marker":"[26]"},{"why":"Earlier skill-based pipeline that stops before experiments and editable figures; defines the partial-automation baseline.","marker":"[27]"},{"why":"Pairs an automated researcher with an automated reviewer; closest analogue to the adversarial review stage.","marker":"[25]"},{"why":"Supplies the self-critique mechanism the long-horizon correction loop is built on.","marker":"[21]"},{"why":"Provides the iterative refine-with-self-feedback pattern used in review-and-revise.","marker":"[14]"},{"why":"Documents fabricated bibliographic citations by large language models, motivating the citation gate.","marker":"[23]"}],"fun_headline_variants":["Skill-based paper generator hits 99.5% citation validity","13 composable skills write verified research papers","Paper pipeline: 99.5% valid citations, 96.4% editable figures","From experiment to paper: skills keep claims evidence-based","Coding-assistant skills cut paper fabrication to 8%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the model-based protocol for deciding whether a measured result actually supports a claim gets that judgment right; if it misclassifies claims, the deterministic gates catch only structural violations and the system stops being evidence-based and becomes a loop counter.","fun_headline_variants_meta":{"raw":{"variants":["Skill-based paper generator hits 99.5% citation validity","13 composable skills write verified research papers","Paper pipeline: 99.5% valid citations, 96.4% editable figures","From experiment to paper: skills keep claims evidence-based","Coding-assistant skills cut paper fabrication to 8%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000234,"raw_usage":{"total_tokens":1540,"prompt_tokens":1029,"completion_tokens":511,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":645,"completion_tokens_details":{"reasoning_tokens":425}},"tokens_in":645,"tokens_out":511,"duration_ms":4650,"temperature":1.0,"reasoning_tokens":425,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:22:12.049128+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"An external audit would take a random sample of claims that Spark-to-Paper labeled supported or contradicted and check each against the raw experiment logs, metrics, and source data; if a substantial fraction of supported labels turn out not to be implied by the measurements, the evidence-centrality claim fails. A cheaper check would rerun the 36-probe fabrication corpus while also measuring claim-label accuracy on probes whose evidence status is known.","supporting_citations":[{"cited_title":"academic-research-skills: A claude code skill suite for research, writing, and review","cited_arxiv_id":null,"evidence_quote":"Closest existing skill suite for research, writing, and review inside a coding assistant; shows the gap this paper fills."},{"cited_title":"Walters and Esther Isabelle Wilder","cited_arxiv_id":null,"evidence_quote":"Documents fabricated bibliographic citations by large language models, motivating the citation gate."}],"review_version":1}