Pith. sign in

REVIEW 3 major objections 2 minor 1 cited by

DeNovoSWE: Scaling Long-Horizon Environments for Generating Entire Repositories from Scratch

T0 review · 3 major / 2 minor · reviewed 2026-06-27 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read DeNovoSWE supplies 4,818 automatically generated instances for training models to build entire repositories from documentation.

desk verdict DeNovoSWE reports a big lift on repo-scale generation after fine-tuning, but the dataset's correctness rests on an unverified agentic pipeline. read the letter →

arxiv 2606.10728 v2 pith:2ZTO7RSV submitted 2026-06-09 cs.SE

Jiale Zhao , Guoxin Chen , Fanzhe Meng , Wayne Xin Zhao , Ruihua Song , Ji-Rong Wen , Kai Jia This is my paper · ORCID
classification cs.SE
keywords whole-repositorygenerationLLMagentsdatasetconstructionlong-horizontaskssoftwareengineeringagenticworkflowfine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper addresses the lack of training data for AI agents that must create complete software repositories from high-level specs rather than perform small fixes. It presents an automated sandboxed workflow that decomposes the task, generates code, and repairs issues through critic loops to produce thousands of full-repository examples without any human review. A difficulty-aware filter then selects trajectories to keep both quality and variety. Fine-tuning a 30B model on the resulting dataset raises its success rate on a demanding benchmark from 5.8 percent to 47.2 percent. This demonstrates that scalable, annotation-free data construction can extend agent capabilities to long-horizon repository-level work.

What carries the argument

The sandboxed agentic workflow that uses divide-and-conquer decomposition combined with critic-repair loops and difficulty-aware filtering to generate verifiable whole-repository instances from documentation without human annotation.

What would settle it

Fine-tuning a model on DeNovoSWE and observing no gain over the base model on the BeyondSWE-Doc2Repo benchmark, or a manual review finding that a large fraction of the generated repositories fail basic functionality tests.

Watch

Extended reading notes

Core claim

DeNovoSWE comprises 4,818 high-quality instances for generating complete repositories from documentation, built through a sandboxed agentic workflow with divide-and-conquer and critic-repair, and fine-tuning on it improves long-horizon SWE performance from 5.8% to 47.2% on BeyondSWE-Doc2Repo.

Load-bearing premise

The sandboxed agentic workflow using divide-and-conquer and critic-repair produces high-quality, verifiable whole-repository instances suitable for training without human annotation or verification.

Editorial extensions

If this is right

  • Training data for whole-repository generation can be produced at scale without manual effort or verification.
  • Fine-tuned models achieve substantially higher success rates on benchmarks requiring full repository creation from documentation.
  • Difficulty-aware trajectory filtering maintains both data quality and diversity during automated curation.
  • The same workflow extends agent performance from localized code changes to complete project implementation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Similar automated decomposition and repair loops could generate training data for other multi-step creative tasks such as hardware design or scientific workflow construction.
  • Performance gains on synthetic benchmarks may translate to real repositories only if the sandbox environment closely matches production constraints.
  • Mixing DeNovoSWE-style data with existing bug-fix datasets could produce agents that both create new projects and maintain them.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 2 minor

Summary. The paper introduces DeNovoSWE, a dataset of 4,818 instances for generating complete software repositories from documentation. Instances are produced automatically via a sandboxed agentic workflow employing divide-and-conquer and critic-repair, with difficulty-aware trajectory filtering and no human annotation. Fine-tuning Qwen3-30B-A3B on DeNovoSWE is reported to raise performance on the BeyondSWE-Doc2Repo benchmark from 5.8% to 47.2%.

Significance. If the generated repositories are verifiably correct and complete at scale, the automated construction method would represent a meaningful advance in creating training data for long-horizon SWE agents without manual effort. The reported performance lift would indicate that such data can meaningfully improve model capabilities on repository-level tasks.

major comments (3)
  1. Abstract: the assertion that the 4,818 instances are 'high-quality' and 'verifiable' is not accompanied by any reported metrics on functional correctness, test passage rates, specification adherence, or error rates after the critic-repair step; this directly underpins the claim that the data is suitable for training without human verification.
  2. Dataset construction (workflow description): no procedure, equation, or table is supplied showing how the critic enforces repository-scale correctness (e.g., end-to-end test execution results or coverage statistics), leaving the 'verifiable' property as an assertion rather than a demonstrated property.
  3. Evaluation section: the headline improvement (5.8% o 47.2%) on BeyondSWE-Doc2Repo lacks any reported controls for contamination or overlap between DeNovoSWE generation sources and the benchmark, which is load-bearing for interpreting the lift as genuine generalization.
minor comments (2)
  1. The abstract and workflow description would benefit from an explicit diagram or pseudocode outlining the divide-and-conquer and critic-repair loop.
  2. Minor notation inconsistency: the model name 'Qwen3-30B-A3B' appears without a reference or version clarification in the abstract.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive comments. We address each major point below with clarifications on our methodology and commit to revisions where they strengthen the presentation of DeNovoSWE's construction and evaluation.

read point-by-point responses

  1. Referee: Abstract: the assertion that the 4,818 instances are 'high-quality' and 'verifiable' is not accompanied by any reported metrics on functional correctness, test passage rates, specification adherence, or error rates after the critic-repair step; this directly underpins the claim that the data is suitable for training without human verification.

    Authors: The abstract condenses the core claims, but we agree it lacks explicit quantitative support for the 'high-quality' and 'verifiable' descriptors. These properties derive from the sandboxed critic-repair loop that retains only trajectories passing automated verification against documentation and execution checks. In the revision we will update the abstract to reference the supporting metrics and add a dedicated table in Section 3 reporting post-repair test passage rates, specification adherence, and error statistics. revision: yes

  2. Referee: Dataset construction (workflow description): no procedure, equation, or table is supplied showing how the critic enforces repository-scale correctness (e.g., end-to-end test execution results or coverage statistics), leaving the 'verifiable' property as an assertion rather than a demonstrated property.

    Authors: Section 3 outlines the critic-repair process within the divide-and-conquer workflow, where the critic agent performs sandboxed execution and documentation alignment checks before accepting a trajectory. We acknowledge that aggregate statistics and explicit decision criteria are not tabulated. The revision will include a new table and pseudocode block detailing the critic's verification steps together with summary statistics on end-to-end test results and coverage. revision: yes

  3. Referee: Evaluation section: the headline improvement (5.8% to 47.2%) on BeyondSWE-Doc2Repo lacks any reported controls for contamination or overlap between DeNovoSWE generation sources and the benchmark, which is load-bearing for interpreting the lift as genuine generalization.

    Authors: The source repositories and documentation for DeNovoSWE were drawn from a distinct collection pipeline that does not intersect with BeyondSWE-Doc2Repo projects. We did not, however, report an explicit overlap analysis in the original manuscript. The revision will add a contamination-control subsection that quantifies any lexical or structural overlap and confirms the reported lift reflects generalization. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; derivation is self-contained

full rationale

The abstract describes DeNovoSWE as generated via a sandboxed divide-and-conquer + critic-repair workflow with difficulty-aware filtering, then reports an empirical fine-tuning gain on BeyondSWE-Doc2Repo (5.8% to 47.2%). No equations, self-citations, or load-bearing steps are quoted that reduce the performance claim to a definitional fit, renamed input, or self-referential construction. The dataset curation and benchmark evaluation are presented as distinct processes; the reported improvement is an external measurement rather than a quantity forced by the generation pipeline itself. Absent any exhibited reduction of the central result to its own inputs, the chain does not exhibit circularity.

Assumptions & free parameters 0 free parameters · 0 assumptions · 0 invented entities

Abstract supplies no information on free parameters, axioms, or invented entities; full text required for assessment.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DeNovoSWE: Scaling Long-Horizon Environments for Generating Entire Repositories from Scratch." pith.science (2026). https://pith.science/paper/2ZTO7RSV

@misc{pith2026260610728,
  author       = {Pith},
  title        = {Pith review of: DeNovoSWE: Scaling Long-Horizon Environments for Generating Entire Repositories from Scratch},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2ZTO7RSV}},
  note         = {Machine review of arXiv:2606.10728}
}
read the original abstract

As the capabilities of LLM-based code agents continue to advance, their expected role is expanding beyond localized bug fixing in existing codebases toward architecting and implementing complete software repositories from high-level specifications. However, training agents for such long-horizon software engineering tasks remains difficult due to the scarcity of large-scale, verifiable whole-repository generation data. In this paper, we introduce \textbf{DeNovoSWE}, a large-scale dataset for whole-repository generation. DeNovoSWE comprises 4,818 high-quality instances, where each instance requires generating a complete repository from documentation. Our dataset is automatically constructed through a carefully designed sandboxed agentic workflow, enabling scalable curation without human annotation. DeNovoSWE is constructed with "divide and conquer" and critic-repair philosophy. To balance data quality and diversity, we further introduce a difficulty-aware trajectory filtering strategy. Fine-tuning Qwen3-30B-A3B on DeNovoSWE substantially improves long-horizon SWE performance, raising its score on the challenging BeyondSWE-Doc2Repo benchmark from 5.8% to 47.2%.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. CalibForge: Adversarial Solver Calibration for Scaling Learnable Terminal Tasks

    cs.LG 2026-08 conditional novelty 6.0 of 10

    CalibForge generates terminal-agent training tasks by revising candidates until a strong solver passes and a weak solver fails, and students trained on the resulting 5,431 tasks gain up to 30 points on held-out benchmarks.

Reference graph

Works this paper leans on

27 extracted references · cited by 1 Pith paper

  1. [1]

    Read the README and repository tree carefully

  2. [2]

    Review the capability outline and identify the highest-level concepts it suggests

  3. [3]

    Inspect package metadata and entry points such as`pyproject.toml`,`setup.py`, `package.json`, main`__init__`files, CLI modules, or service entry modules

  4. [4]

    Open the main source packages under the likely source roots and confirm the repository's public workflows and core abstractions

  5. [5]

    model" structure (prediction) against a

    Draft an overview that feels like the beginning of real project documentation. 18 DeNovoSWE: Scaling Long-Horizon Environments for Generating Entire Repositories from Scratch Few-shot example: ```markdown ## 1. Overview The`target_repo`library is a comprehensive toolkit for evaluating the quality of biomolecular structure prediction models. It supports pr...

  6. [6]

    Check whether the overview explains the repository's overall purpose rather than just enumerating modules

  7. [7]

    Verify that any mentioned interfaces, workflows, or domains are supported by repository evidence

  8. [8]

    Make sure the section reads like documentation opening prose, not analysis notes

Show all 27 references
  1. [10]

    capabilities

    Submit the final overview section with the`finish`tool. 19 DeNovoSWE: Scaling Long-Horizon Environments for Generating Entire Repositories from Scratch Repository ability prompt. You are a documentation architect for software repositories. Your job is to derive an ordered repo...

  2. [11]

    VerifiableCredential

    **`build(...)`** **Signature:** ```python def build( self, issuer_did: str, subject_did: str, credential_id: Optional[str] = None, types: Optional[List[str]] = None, contexts: Optional[Union[List, str]] = None, issuance_date: Optional[datetime] = None, expiration_date: Optiona...

  3. [12]

    assertionMethod

    **`sign(...)`** **Signature:** ```python 23 DeNovoSWE: Scaling Long-Horizon Environments for Generating Entire Repositories from Scratch def sign( self, issuer_signing_key: nacl.signing.SigningKey, verification_method_id: str, proof_purpose: Optional[str] = "assertionMethod" )...

  4. [13]

    Canonicalizes the document using **URDNA2015**

  5. [14]

    Signs the hash using the`issuer_signing_key`(Ed25519)

  6. [15]

    Encodes the signature in **base58btc**

  7. [16]

    Appends a`proof`object (type`Ed25519Signature2020`) to the VC

  8. [17]

    ``` **Description:** Verifies the cryptographic integrity of the VC

    **`verify_signature(...)`** **Signature:** ```python def verify_signature( self, issuer_public_key: nacl.signing.VerifyKey, expected_issuer_did: Optional[str] = None, expected_subject_did: Optional[str] = None ) -> bool: ... ``` **Description:** Verifies the cryptographic inte...

  9. [18]

    Removes the`proofValue`from the proof block

  10. [19]

    Canonicalizes the document (URDNA2015)

  11. [20]

    Verifies the signature against the canonical doc using`issuer_public_key`

  12. [21]

    Optionally checks if`issuer`and`credentialSubject.id`match expected values

  13. [22]

    VerifiableCredential

    **Serialization & Factories** `to_dict() -> Dict`: Exports the VC to a Python dictionary. `to_json() -> str`: Exports the VC to a JSON string. `@classmethod from_dict(data: Dict) -> "VerifiableCredential"`: Creates a VC instance from a dictionary. `@classmethod from_json(json_...

  14. [23]

    Inspect the local source files for the highlighted APIs and any ambiguous symbol names

  15. [24]

    Inspect related unit tests, call sites, and fixtures for the current ability when they exist

  16. [25]

    Verify that exact names, signatures, defaults, ownership relations, raised errors, and return structures are correct

  17. [26]

    Make sure every required symbol name appears in the section and the focus stays on the current ability only

  18. [27]

    {pre_finish_guidance}

  19. [28]

    Submit the final section with the`finish`tool using the current ability name and the markdown body. 26

Pith tools

Reviewed June 27, 2026 · model on record in the stance chip above.