Recursive Harness Self-Improvement

Donghyun Lee; Hyunin Lee; Jeffrey Seely; Jinglue Xu; Matei Zaharia; Yujin Tang

REVIEW 4 major objections 5 minor 1 cited by

A few self-comparison rounds of a prompt-level harness outrank maximum-reasoning agents at lower inference cost.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 23:04 UTC pith:4ASF3VZC

load-bearing objection A genuinely new lightweight harness-optimization idea with well-controlled experiments, but the headline cost claim omits the optimization loop and the evaluation rests entirely on LLM-judge preferences. the 4 major comments →

arxiv 2607.15524 v1 pith:4ASF3VZC submitted 2026-07-17 cs.LG cs.AI

Recursive Harness Self-Improvement

Hyunin Lee , Jinglue Xu , Jeffrey Seely , Donghyun Lee , Matei Zaharia , Yujin Tang This is my paper

classification cs.LG cs.AI

keywords recursive self-improvementharness optimizationprompt-level agent loopmulti-agent workflowstest-time scalingLLM-as-a-judgecontext managementinformation-theoretic objective

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

RHI asks whether the invisible 'harness' — the multi-agent workflow, roles, instructions, and communication contracts wrapped around a fixed model — can itself be optimized cheaply. The paper's central claim is yes: representing the harness as a prompt-level specification and revising it through pairwise feedback against its own previous output, a few iterations suffice to push low-reasoning-effort agents above their same-family maximum-reasoning baselines on 30 synthetic ML research tasks, while cutting inference cost by up to 60%. The gains are attributed to task-specific context management (better contracts and workflow hops) rather than longer reasoning traces. If right, RHI is a cheap alternative to test-time compute scaling and a practical way to raise execution-trace quality for future model training.

Core claim

Across 30 synthetic ML research tasks spanning quantitative finance, robotics, and pharmacy, applying RHI to high-reasoning Sonnet-4.6, Opus-4.7, and Opus-4.8 agents produces harnesses that win more pairwise LLM-judge comparisons than the same models run at xhigh, max, or ultracode reasoning effort: Sonnet-4.6-high+H[2] beats Sonnet-4.6-max, Opus-4.7-high+H[1] beats Opus-4.7-xhigh/max, and Opus-4.8-high+H[2] beats Opus-4.8-xhigh/ultracode/max, with normalized cost reduced by up to 60% relative to ultracode. Output-token usage stays nearly constant while performance improves; the paper therefore argues the mechanism is more effective inter-agent information flow, formalized as an implicit obj

What carries the argument

The harness is represented as a prompt-level object: agent design (roles and instructions) plus an agent workflow made of 'contracts' (what information is passed between agents) and 'hops' (the orchestrator–subagent control flow). RHI's update rule is trajectory-local: each iteration generates one new repository, compares it pairwise against the previous output via an LLM judge, appends the preference to a history, and asks an LLM harness optimizer to rewrite the harness. The paper formalizes the implicit objective as increasing mutual information between workflow components and the task while decreasing task-conditional total correlation (redundancy) among components.

Load-bearing premise

The whole comparison rests on a single proxy: LLM-as-a-judge pairwise preferences between repositories are treated as the true signal of repository quality, and even the benchmark tasks are LLM-transformed job postings, with no human or external anchor; if judges reward surface coherence rather than real task quality, 'beats max reasoning' could be an artifact of optimizing for the judge.

What would settle it

Conduct the same 30-task comparison with an independent, non-LLM scoring procedure — e.g., human expert ratings of research reports or executable checks such as whether code runs, metrics.json is valid, and numerical claims reproduce — and see whether the RHI harnesses still beat max-reasoning baselines. If the LLM judge's preferences diverge from those scores, the headline claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

A few lightweight harness updates can substitute for scaling reasoning effort: RHI beats max-reasoning baselines in pairwise comparisons on the benchmark.
Prompt-level, task-specific harness optimization can outperform a provider-built system-level dynamic workflow (ultracode) on the benchmark.
Cost reductions come mainly from reduced cache read/write usage, not shorter outputs, so the method is relevant whenever prompts and context dominate inference cost.
RHI improves how a fixed model is used but does not close the gap to stronger base models, so it is complementary to, not a replacement for, train-time scaling.
The improved harnesses produce higher-quality execution traces, the first half of a model–harness co-evolution loop the paper argues should be a focus of future work.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The paper leaves untested whether RHI's advantage survives a non-LLM signal: replacing the LLM judge with human ratings or executable metrics would show whether the gains reflect genuine task quality or optimization for the judge.
Because the harness is just text, RHI could be wrapped around any agent that accepts a prompt, which may allow the learned contracts to transfer across models.
If the implicit info-theoretic objective is real, explicit optimization of a mutual-information-minus-redundancy objective over harness text might outperform the implicit version.
The trajectory-local comparison with cached outputs suggests a natural Bayesian or thresholded stopping rule around pairwise win probabilities, making the 'momentum' interpretation quantitative.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

A genuinely new lightweight harness-optimization idea with well-controlled experiments, but the headline cost claim omits the optimization loop and the evaluation rests entirely on LLM-judge preferences.

read the letter

Two things to know. First, the core algorithm is real: RHI's trajectory-local relaxation—one new execution and one pairwise comparison against the immediate predecessor per iteration—is a genuine alternative to population-based harness search, and the controlled comparisons across three base models, two judge families, and three seeds are thoughtfully done. Second, the 'cheap alternative to test-time scaling' claim does not survive accounting for the optimization loop. The stress-test arithmetic checks out: for opus-4.8, the three RHI runs cost 1.35+1.85+1.69=4.89 normalized, already above ultracode at 4.15, before counting judge calls. The paper cites Wang et al. (2026) for the principle that harness evolution must be compared under fixed inference budgets, then omits exactly those search costs from Claim 3. So the honest statement is that RHI reduces per-output inference cost after an unpaid optimization phase; the cheap-alternative framing is unsupported as written.

What is genuinely new: Eq. (3)'s trajectory-local self-comparison objective is not in the cited prior work (Meta-Harness, AutoHarness, TTHE, GEPA, OPRO all use populations or scalar/verifier signals). The paper is also unusually candid about confounds—it explicitly flags the Opus-4.7 output-token confound and the correlational nature of the embedding analyses. The information-theoretic hypothesis (Eq. 6) is clearly labeled as a post-hoc account, not a prediction, which is fine.

The soft spots are load-bearing. The benchmark is LLM-generated, the update signal and the final evaluation are the same pairwise LLM-preference protocol, and no code or data are shipped. If the judges reward surface coherence rather than repository quality, the 'beats max reasoning' result could be an artifact of optimizing for the judge. The f_ext/f_int evidence is partly built into the optimizer prompt, so it is weak confirmation. These are not fatal—the algorithm is coherent and falsifiable—but they cap how much should be concluded until external validation exists.

Who this is for: anyone working on harness optimization or LLM-as-judge evaluation methodology. It deserves a serious referee, with the cost accounting and judge validity as the primary revision targets. I'd bring it to a reading group mostly to argue about evaluation design.

Referee Report

4 major / 5 minor

Summary. The paper introduces Recursive Harness Self-Improvement (RHI), an iterative procedure that represents a coding agent's harness as a textual prompt and updates it using pairwise LLM-judge comparisons between the outputs of consecutive harness versions. On 30 synthetic ML research tasks derived from three industry job postings, the authors report that a few RHI iterations applied to high-reasoning Claude agents let those agents win more pairwise LLM-judge comparisons than stronger test-time scaling settings (xhigh/max/ultracode), while sometimes lowering per-run inference cost. The paper also presents ablations suggesting the gains come from task-specific contracts/hops rather than longer outputs, and proposes an information-theoretic hypothesis for the update trajectory.

Significance. If validated, the central result would be practically significant: task-specific, prompt-level harness optimization could improve agent performance below the cost of increasing reasoning effort, and could improve execution-trace quality for future model training. The study has useful internal controls: three base models, multiple LLM judge configurations, three seeds, and explicit measurement of cost, output tokens, and cache read/write usage. The authors are also appropriately cautious in several places: they label the information-theoretic story as a hypothesis, call the embedding evidence correlational (Sections 6.2 and 6.3), and admit that the opus-4.7 output-token evidence is inconclusive (Section 5.4). However, the two headline claims rest on unvalidated assumptions: that LLM pairwise preferences measure repository quality, and that the cost comparison can ignore the cost of the optimization loop. Both assumptions are load-bearing, so the paper's abstract and Claims 1 and 3 are currently stronger than the evidence.

major comments (4)

[§5.5, Figure 7e; Algorithm 1] Claim 3's 'up to 60% cost reduction' compares only the final RHI agent run: normalized cost 1.69 for opus-4.8-high+H[2] versus 4.15 for ultracode. Producing H[2] requires executing H[0], H[1], and H[2], whose normalized agent costs are 1.35, 1.85, and 1.69; the sum is 4.89, already larger than the ultracode baseline (4.15) and more than double the max baseline (2.19). Each iteration also adds at least one Leval pairwise comparison and one Lharness update per task, which are not captured by the agent cost function used in Section 5.2. Because harnesses are task-specific, this search cost is not amortized across tasks. The paper itself cites Wang et al. (2026) in Section 2.2 for the principle that harness evolution must be compared under fixed inference budgets including search cost; Claim 3 violates that principle. The cost claim should be recomputed to include the full optimization and j
[§3.2, Algorithm 1; §4.2; §5.1–5.5] The evaluation loop is partly closed. The harness is updated using pairwise preferences from an LLM evaluator Leval (Eq. (5), Algorithm 1), and the headline claim that H[2] outperforms max/ultracode is measured by LLM pairwise preferences under the same protocol (Section 4.2). The benchmark tasks are LLM-transformed job postings (Section 4.1) and no human or external quality anchor is used. Consequently, 'winning 20/30' or 'outperforming ultracode' may reflect the degree to which RHI learns to satisfy the same kind of LLM judge rather than to produce higher-quality research artifacts. This is the weakest assumption in the paper. A concrete fix: evaluate a subset with human domain experts or against execution-based/functional checks and report judge–human agreement; alternatively, optimize with one judge and evaluate with a different judge plus external metrics to show transfer. Without s
[§4.1] The 30-task suite is generated by transforming three industry job postings into research prompts with standardized deliverables (research_report.md, plots, metrics.json, index.json). No evidence is provided that this distribution is representative of quantitative finance, robotics, or pharmacy ML research, or that the deliverable rubric captures what makes a research repository good. The benchmark may be well matched to LLM evaluability rather than to research quality. This is not by itself disqualifying, but combined with the closed evaluation loop it makes the external validity of the central comparison unknown. Please provide either human validation of the task/deliverable interface or a comparison of RHI gains on an existing non-synthetic benchmark.
[§6.3.2, Eq. (6); Tables 2–3] The information-theoretic 'implicit objective' is presented as a formalization in the abstract, but it is a post-hoc hypothesis with free parameter beta and a chosen Cext. The supporting evidence is correlational and estimator-dependent: mutual information and total correlation are computed from Gaussian CCA/whitened embeddings with two embedding models and permutation debiasing. The monotonic trends do not establish that Lharness optimizes J(g_i), and no predictive test is given. The authors do label the claim as a hypothesis and note the correlational nature, which is appropriate, but the abstract-level 'formalize' language overstates the contribution. Please either move this to a clearly labeled interpretation or add a predictive test, for example comparing RHI's updates to directly optimizing J(g_i) on a held-out set.

minor comments (5)

[§6.2.2, near Fig. 11] The text 'Together with the low-dimensional visualizations in Figure 11' appears to refer to Figure 10; Figure 11 contains cosine-similarity curves, not t-SNE/UMAP projections.
[§5.3] 'Winning 20 of 30 pairwise comparisons' is a mean over judges and seeds. Reporting per-judge and per-seed counts, or a confidence interval, would make the strength of the claim clearer.
[§2.2, Table 1] Table 1's per-iteration cost assumes the previous output is cached, so N_trace=1 and N_pair=1. State this caching assumption explicitly in the table caption or text.
[Throughout] Minor typos and notation issues: 'thses' in §6.2.1, 'addtionally' in §6.3, and 'uni00A0' artifacts in figure labels. Also, H[i] and H(i) are used interchangeably; define once.
[Reproducibility] No code or data release is mentioned. The appendices are extensive, but for an empirical paper of this type, releasing the task suite, evolved harnesses, and evaluator prompts would be important for reproducibility.

Circularity Check

0 steps flagged

No significant circularity: the headline wins are empirical, though the LLM-judge loop and cost accounting warrant external-validation caveats.

full rationale

I find no step in which a claimed prediction is equivalent to its input by construction. The RHI update (Algorithm 1; Eq. 3) uses pairwise LLM preferences over consecutive harness outputs as a local ascent signal, and the headline comparisons (Section 4.2; Figures 5-7) measure pairwise LLM preferences of the final harnesses against same-family test-time-scaling baselines. Because the baselines are not optimized against Leval, the reported wins are not forced by the optimization; they are empirical outcomes reported over multiple judges and seeds. The judge-loop is a construct-validity and external-anchor concern (LLM-as-a-judge is both the optimization reward and the evaluation metric), not a circular reduction. Likewise, the information-theoretic hypothesis in Section 6.3 is explicitly labeled correlational and post-hoc, so it is not used to derive the main results. The cost-efficiency claim (Claim 3, §5.5) excludes the RHI optimization loop and is inconsistent with the paper's own citation of Wang et al. (2026) in §2.2 that search cost must be included; this is an accounting/correctness gap, not circularity. Self-citations (e.g., Lee et al. 2026 Meta-Harness) are used for positioning only and are not load-bearing. Overall, the central derivation is self-contained and the main claims have independent empirical content, though they remain relative to the LLM-judge protocol rather than an external human-verified benchmark.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 1 invented entities

The central empirical result requires only a few background premises, but they are consequential: the LLM judge must be a valid quality oracle, the synthetic tasks must represent real ML research, and the embedding-based information diagnostics must be meaningful. The hypothesized objective J adds a free parameter beta and a postulated functional form with no predictive validation.

free parameters (3)

Stopping threshold epsilon = not reported
Algorithm 1 breaks when the improvement rate s_i < epsilon; the value of epsilon is not given, and it controls how many RHI iterations are used for the reported harnesses.
Initial domain-specific harness H(0) = hand-written prompt per domain (Appendix B shows pharmacy example)
All performance improvements are measured relative to this human-designed initialization; no sensitivity analysis over initial harnesses is provided.
beta in hypothesized objective J(g_i) = no value given (beta > 0)
Equation (6) defines the redundancy-penalty weight beta, but it is never estimated or used in the algorithm; it is a free parameter of the post-hoc hypothesis.

axioms (5)

domain assumption Pairwise LLM preferences reflect the true multi-criteria quality ordering of code repositories.
Used in Eq. (1), Eq. (3), and the evaluation protocol (Section 4.2); no human or non-LLM validation is provided.
domain assumption There exists a task utility u_x and link sigma such that Pr(H > H') = sigma(u_x(H) - u_x(H')).
Section 3.1 uses this Bradley-Terry-style model to argue the trajectory-local objective preserves the same latent utility ordering as the population objective.
ad hoc to paper Synthetic LLM-generated tasks from three job postings are representative of quantitative finance, robotics, and pharmacy ML research.
Section 4.1 constructs the benchmark by transforming three job postings with an LLM; there is no external curation or human validation of task difficulty.
domain assumption The coding agent is a black box whose behavior is determined by prompt-level harness injection; the harness text is the only optimized object.
The entire method and evaluation depend on the claim that editing the textual harness changes agent behavior without code-level or weight-level access (Sections 2.1 and 3.3).
domain assumption Embedding models and Gaussian approximations are adequate for estimating mutual information and total correlation of harness components.
Sections 6.3.3–6.3.4 use Gaussian CCA and Gaussian total correlation on text embeddings; the authors explicitly call these diagnostic statistics rather than exact information-theoretic quantities.

invented entities (1)

Implicit information-theoretic objective J(g_i) = sum I(z_hc;X) - beta TC(z_hc|X) no independent evidence
purpose: Post-hoc account of RHI's update trajectory and functional specialization guidance
Equation (6) is introduced after observing the empirical trends in Tables 2 and 3; no out-of-sample prediction or independent test is offered. The authors themselves label it a hypothesis, not a demonstrated objective.

pith-pipeline@v1.3.0-alltime-deepseek · 50719 in / 13273 out tokens · 145201 ms · 2026-08-01T23:04:15.096574+00:00 · methodology

0 comments

read the original abstract

Under model--harness co-evolution, harnesses are not merely inference-time scaffolds but data-generating components whose execution traces can shape future foundation models. This motivates harness-in-the-loop learning: optimizing harnesses for both immediate agent performance and the quality of traces used for future model training. However, continually updating provider-built scaffolds is costly and labor-intensive. We therefore investigate whether optimizing user-constructed harnesses in a task-specific manner can improve execution-trace quality while remaining computationally lightweight and requiring only a few update iterations. To this end, we introduce Recursive Harness Self-Improvement (RHI), which represents the harness as a prompt-level specification of the agent loop and iteratively refines it using pairwise feedback over its own revision history. Across 30 synthetic machine-learning research tasks spanning quantitative finance, robotics, and pharmacy, a few RHI iterations suffice to substantially raise the performance ceiling of low-reasoning-effort agents, exceeding the corresponding maximum-reasoning-effort setting while reducing inference cost by up to 60%. We show that these gains arise primarily from improved task-specific context management through more effective inter-agent information flow rather than longer reasoning traces. Finally, we formalize this behavior as an information-theoretic hypothesis for RHI's implicit optimization objective, suggesting RHI as a practical algorithm for continual learning within the paradigm of model--harness co-evolution.

discussion (0)

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

Agent Harness Distillation: Inference-Time Harness Extraction and Exploitation in Autonomous Multi-Agent Systems
cs.CR 2026-07 conditional novelty 6.0

Agent Harness Distillation extracts inference-time multi-agent orchestration from black-box AMAS and transfers it to weaker backbones, exposing IP leakage that a deception defense can blunt.

Reference graph

Works this paper leans on

60 extracted references · cited by 1 Pith paper

[1]

research_report.md as a conference-style paper (7-9 pages equivalent) with equivariant architecture details and profiling discussion
[3]

deliverables/plots/: - deliverables/plots/angular_error_histogram.png - deliverables/plots/error_by_residue_type.png - deliverables/plots/training_curves.png - deliverables/plots/runtime_memory_profile.png - deliverables/plots/ablation_equivariant_vs_invariant.png
[4]

Reproducible code (Python): src/extract_sidechain_dataset.py, src/model_se3.py, src/train.py, src/evaluate.py, src/profile.py; requirements.txt
[5]

results/metrics.json with MAE degrees, median error, per-residue breakdown, runtime
[6]

agent_id

results/ablation_results.json for invariant baseline vs equivariant variants. 33 Recursive Harness Self-Improvement Initial Harness (H[0]) This is an initial harness design. This is domain-specific. Harness(0) Create an agent team with following agent candidates to solve this problem: [ { "agent_id": "agent_orchestrator", "role": "Lead scientist-orchestra...
[7]

agent_structural_data_rcsb and agent_torsion_geometry must jointly produce DatasetRecord v1 before model, training, or eval implementation
[8]

38 Recursive Harness Self-Improvement

agent_geometric_ml and agent_baseline_ablation must jointly produce ModelIO and AblationGrid v1 before training begins. 38 Recursive Harness Self-Improvement
[9]

agent_eval_benchmarks must publish ResultsContract v1 before scicomm writes quantitative claims or index.json summaries
[10]

agent_profile_systems must publish ProfilingContract v1 before the report discusses speed or memory
[11]

Minimum orchestrator-subagent hops: Hop A initial delegation with scope and acceptance criteria

agent_artifact_validator and agent_quality_safety review frozen artifacts independently and can open recall tickets. Minimum orchestrator-subagent hops: Hop A initial delegation with scope and acceptance criteria. Hop B contract acknowledgement after InterfaceContract v1, where each consuming agent returns ack_or_change_request. Hop C evidence review afte...
[12]

agent_interface_contract_steward publishes RequiredPathManifest v1 before any implementation; every producing and validating agent must return ContractAck or ChangeRequest
[13]

44 Recursive Harness Self-Improvement

agent_structural_data_rcsb and agent_torsion_geometry jointly produce DatasetRecord v2 and DataProvenanceManifest v1 before model, training, or eval implementation consumes data. 44 Recursive Harness Self-Improvement
[14]

agent_structural_data_rcsb and agent_eval_benchmarks jointly produce SplitLeakageReview v1 before training/evaluation claims are frozen
[15]

agent_geometric_ml and agent_baseline_ablation jointly produce ModelRegistry v1, ModelIOContract v2, and AblationGrid v2 before training begins; agent_training_reproducibility must ack importability, checkpoint keys, and model_ids
[16]

agent_eval_benchmarks publishes ResultsContract v2 before scicomm writes quantitative accuracy claims or index.json summaries
[17]

agent_profile_systems publishes ProfilingContract v2 and validates exact src/profile.py before scicomm writes speed or memory claims
[18]

agent_scicomm_publication sends ClaimsEvidenceMatrix to agent_claims_consistency_referee; all referee blockers are routed to the owning specialist or scicomm for patch
[19]

Minimum orchestrator-subagent hops: Hop A EvidenceRubric fan-out: orchestrator sends task deliverables, prior evidence defects, and acceptance criteria to all agents

agent_artifact_validator and agent_quality_safety review frozen artifacts independently and can open recall tickets; their final checks occur after any patch round, not only before patches. Minimum orchestrator-subagent hops: Hop A EvidenceRubric fan-out: orchestrator sends task deliverables, prior evidence defects, and acceptance criteria to all agents. ...
[20]

agent_interface_contract_steward publishes RequiredPathManifest v2 and InterfaceContract v3 before implementation; every producing and validating agent must return ContractAckV3 or ChangeRequestV3 with field-level comments
[21]

agent_structural_data_rcsb and agent_torsion_geometry jointly produce DatasetRecord v3 and DataProvenanceManifest v2 before model, training, eval, profile, or equivariance code consumes data
[22]

agent_structural_data_rcsb and agent_eval_benchmarks jointly produce SplitLeakageReview v2 before training and evaluation claims are frozen
[23]

agent_geometric_ml and agent_baseline_ablation jointly produce ModelRegistry v2, ModelIOContract v3, and AblationGrid v3 before training begins; agent_training_reproducibility, agent_eval_benchmarks, agent_profile_systems, agent_equivariance_referee, and agent_metrics_artifact_integrator must ack importability, checkpoint keys, model_ids, and factory signatures
[24]

53 Recursive Harness Self-Improvement

agent_equivariance_referee publishes EquivarianceAuditContract v1 and results/equivariance_tests.json before scicomm writes equivariance claims or final acceptance; failures trigger recall to model or training. 53 Recursive Harness Self-Improvement
[25]

agent_eval_benchmarks publishes ResultsContract v3 before metrics integration and scicomm accuracy claims
[26]

agent_profile_systems publishes ProfilingContract v3 and validates exact src/profile.py before metrics integration and speed or memory claims
[27]

agent_profile_sanity_referee audits ProfilingContract before scicomm profiling text; blockers route to profile or metrics integrator
[28]

agent_metrics_artifact_integrator freezes MetricsIntegrationContract only after eval, profile, ablation, training, data, and equivariance inputs are consumed and results/metrics.json contains runtime, profiling, memory, and equivariance summaries or explicit waivers
[29]

agent_scicomm_publication sends ClaimsEvidenceMatrix to agent_claims_consistency_referee; all referee blockers route to the owning specialist or scicomm for patch
[30]

agent_artifact_validator, agent_quality_safety, agent_profile_sanity_referee, and agent_reproducibility_log_auditor review frozen artifacts independently and can open recall tickets; their final checks occur after any patch round, not only before patches. Minimum orchestrator-subagent hops: Hop 0 EvidenceRubric fan-out: orchestrator sends task deliverable...
[31]

Interface steward publishes RequiredPathManifest v4, InterfaceContract v4, and MetricsPointerMatrix v2 before implementation; every producing and validating agent returns field-level ContractAckV4 or ChangeRequestV4
[32]

Data agent publishes DataProvenanceManifest v4, RCSB query attempts, and filter verification; RCSB protocol referee must pass or conditionally pass before training and before any report data claim
[33]

Data and geometry jointly produce DatasetRecord v4 and no-sidechain-input leakage evidence before model, training, eval, profile, or equivariance code consumes data
[34]

Data and eval jointly produce SplitLeakageReview v3 before training and evaluation claims are frozen
[35]

Model and ablation jointly produce ModelRegistry v4, ModelIOContract v4, and AblationGrid v4; training, eval, profile, equivariance, and metrics agents must ack importability, factory signatures, model_ids, and checkpoint keys
[36]

Training produces TrainingHistory and CheckpointMetadata; eval and metrics must ack epoch-selection fields before metrics freeze
[37]

64 Recursive Harness Self-Improvement

Entrypoint CI audits main.py before final; stale hello-world is blocking. 64 Recursive Harness Self-Improvement
[38]

Equivariance referee publishes EquivarianceAuditContract v2 and results/equivariance_tests.json with command evidence before scicomm writes equivariance claims
[39]

Eval publishes ResultsContract v4 and metrics_lineage before metrics integration and report accuracy claims
[40]

Profile publishes ProfilingContract v4 and validates exact src/profile.py before metrics integration and speed or memory claims
[41]

Profile sanity referee audits ProfilingContract and metrics runtime-memory merge before scicomm profiling text
[42]

Metrics integrator freezes MetricsIntegrationContract only after eval, training, profile, ablation, data, RCSB, and equivariance inputs are consumed and results/metrics.json contains all required pointers
[43]

JSON consistency referee audits frozen metrics before plots, index summaries, and report are considered final
[44]

Scicomm sends ClaimsEvidenceMatrix to claims referee; all claims blockers route to owning specialist or scicomm for patch
[45]

Minimum orchestrator-subagent hops: Hop 0 EvidenceRubric fan-out: send task deliverables, pairwise history, and current v3 file evidence to all agents

Validator, quality, RCSB referee, profile sanity, JSON consistency referee, claims referee, entrypoint CI, and log auditor review after every patch round, not only before patches. Minimum orchestrator-subagent hops: Hop 0 EvidenceRubric fan-out: send task deliverables, pairwise history, and current v3 file evidence to all agents. Hop 1 Parallel defect pre...
[46]

write your history log: write your plan, execution (or tool-execution), reflection during the reasoning logs in a logs.txt file

Following query solved with claude coding ——————————– # Task (task description) # Current multi agent design (v0) (multi agent design description) # Save rules Use uv for all Python workflows--run code with uv run, install dependencies with uv add, use uvx for tools. write your history log: write your plan, execution (or tool-execution), reflection during...
[47]

Current submission code repo from Claude Code (evidence only) # Workspace for current iteration (multi-agent-design-v0) Path:`...` ## Directory tree (representative) (tree description) ## Programmatic preflight (heuristic evidence only) Derived from the **task text** (expected paths) vs this workspace; not a substitute for reading the Deliverables block. ...
[48]

- Repo B:

Pairwise history summary - Comparison: multi-agent-design-v0 vs multi-agent-design-v1 - Key: v0 vs v1 - Winner: tie - Judge model: gpt-5.5 - Judged at (UTC): 2026-05-24T10:00:05.337848+00:00 - Repo A: ... - Repo B: ... - Rationale (verbatim excerpt): The two submissions are indistinguishable in the provided evidence:↩→ both point to the same workspace pat...

2026
[49]

Output to orchestrator

Instructions Improve the multi-agent design from v1 to v2 to enhance the quality of the query’s deliverables by creating genuine multi-agent advantages over single-agent execution. Preserve the original intent and required deliverables, but address weaknesses revealed by evidence. • Explicitly strengthen "Output to orchestrator" contracts so downstream ag...
[50]

Reproducible code including dataset preparation, fine-tuning, evaluation, paraphrase genera- tion (public paraphrase model or rule-based templates), andrequirements.txt. 5.results/metrics.jsonincluding: •Top-1 accuracy •Macro F1 •Expected calibration error (ECE) •Paraphrase robustness drop •Per-class metrics 6.results/model_comparison.json comparing froze...
[51]

Reproducible Python code for data collection, feature engineering, modeling, and evaluation. 5.results/metrics.jsonincluding: •IC / RankIC •Predictive𝑅 2 •Strategy Sharpe ratio •Beta •Maximum drawdown •Turnover •Transaction costs •Placebo-test metrics 6.results/feature_mapping.json documenting page-to-ticker mappings and confi- dence scores, andresults/mo...
[52]

Reproducible Python code for data processing, representation learning, multimodal fusion, training, and evaluation. 5.results/metrics.jsonincluding: •Macro and micro F1 •Hierarchical accuracy across EC levels 1–4 •Coverage statistics 6.results/ablation_results.json comparing fusion variants (concatenation, attention, gating) with configurations and evalua...
[53]

research_report.mdasaconference-stylepaper(6–9pagesequivalent)centeredoncalibration and decision-making
[54]

deliverables/index.json
[55]

deliverables/plots/: reliability_diagram.png, ece_vs_temperature.png, roc_curve.png, deci- sion_utility_curve.png, calibration_by_difficulty_bin.png
[56]

Reproducible code: data generation; feature extraction; model training; calibration methods (temperature/isotonic); evaluation; requirements.txt
[57]

results/metrics.json: auroc, average_precision, ece, brier_score, expected_utility_gain
[58]

project":

results/model_comparison.json comparing uncalibrated vs calibrated variants with per-bin metrics. # Labels (for orientation only) •SubmissionAcorresponds to run type: ‘base‘ •SubmissionBcorresponds to run type: ‘defaultTeam‘ # Workspace for submission **A** (evidence only) Path: ‘...‘ ## Directory tree (representative) query318/ |-- data/ |`-- episodes.pa...
[59]

Deliverable coverage— map the task’sDeliverables/Deliverable:/ inline deliverables list (and any other explicit output requirements in ‘# Task‘) to evidence; mark gaps or placeholders
[60]

3.Reproducibility— dependencies, entry points, seeds, documented data or generation

Numerical/empirical rigor— appropriate methodology, baselines, honest limitations; con- sistency between report and metrics when applicable. 3.Reproducibility— dependencies, entry points, seeds, documented data or generation. 4.Presentation— report structure, clarity, figure integration (infer from paths and excerpts). 5.Engineering— layout, modularity, r...
[61]

winner":

Task alignment— penalize solving the wrong problem or drifting from the stated objective. ## Output JSON schema (exact keys) { "winner": "A" | "B" | "tie", "rationale": "<string, cite concrete evidence from both workspaces>" } Return JSON only. F. Distribution of normalized cost, output tokens, and cache read/write 87 Recursive Harness Self-Improvement hi...

[1] [1]

research_report.md as a conference-style paper (7-9 pages equivalent) with equivariant architecture details and profiling discussion

[2] [3]

deliverables/plots/: - deliverables/plots/angular_error_histogram.png - deliverables/plots/error_by_residue_type.png - deliverables/plots/training_curves.png - deliverables/plots/runtime_memory_profile.png - deliverables/plots/ablation_equivariant_vs_invariant.png

[3] [4]

Reproducible code (Python): src/extract_sidechain_dataset.py, src/model_se3.py, src/train.py, src/evaluate.py, src/profile.py; requirements.txt

[4] [5]

results/metrics.json with MAE degrees, median error, per-residue breakdown, runtime

[5] [6]

agent_id

results/ablation_results.json for invariant baseline vs equivariant variants. 33 Recursive Harness Self-Improvement Initial Harness (H[0]) This is an initial harness design. This is domain-specific. Harness(0) Create an agent team with following agent candidates to solve this problem: [ { "agent_id": "agent_orchestrator", "role": "Lead scientist-orchestra...

[6] [7]

agent_structural_data_rcsb and agent_torsion_geometry must jointly produce DatasetRecord v1 before model, training, or eval implementation

[7] [8]

38 Recursive Harness Self-Improvement

agent_geometric_ml and agent_baseline_ablation must jointly produce ModelIO and AblationGrid v1 before training begins. 38 Recursive Harness Self-Improvement

[8] [9]

agent_eval_benchmarks must publish ResultsContract v1 before scicomm writes quantitative claims or index.json summaries

[9] [10]

agent_profile_systems must publish ProfilingContract v1 before the report discusses speed or memory

[10] [11]

Minimum orchestrator-subagent hops: Hop A initial delegation with scope and acceptance criteria

agent_artifact_validator and agent_quality_safety review frozen artifacts independently and can open recall tickets. Minimum orchestrator-subagent hops: Hop A initial delegation with scope and acceptance criteria. Hop B contract acknowledgement after InterfaceContract v1, where each consuming agent returns ack_or_change_request. Hop C evidence review afte...

[11] [12]

agent_interface_contract_steward publishes RequiredPathManifest v1 before any implementation; every producing and validating agent must return ContractAck or ChangeRequest

[12] [13]

44 Recursive Harness Self-Improvement

agent_structural_data_rcsb and agent_torsion_geometry jointly produce DatasetRecord v2 and DataProvenanceManifest v1 before model, training, or eval implementation consumes data. 44 Recursive Harness Self-Improvement

[13] [14]

agent_structural_data_rcsb and agent_eval_benchmarks jointly produce SplitLeakageReview v1 before training/evaluation claims are frozen

[14] [15]

agent_geometric_ml and agent_baseline_ablation jointly produce ModelRegistry v1, ModelIOContract v2, and AblationGrid v2 before training begins; agent_training_reproducibility must ack importability, checkpoint keys, and model_ids

[15] [16]

agent_eval_benchmarks publishes ResultsContract v2 before scicomm writes quantitative accuracy claims or index.json summaries

[16] [17]

agent_profile_systems publishes ProfilingContract v2 and validates exact src/profile.py before scicomm writes speed or memory claims

[17] [18]

agent_scicomm_publication sends ClaimsEvidenceMatrix to agent_claims_consistency_referee; all referee blockers are routed to the owning specialist or scicomm for patch

[18] [19]

Minimum orchestrator-subagent hops: Hop A EvidenceRubric fan-out: orchestrator sends task deliverables, prior evidence defects, and acceptance criteria to all agents

agent_artifact_validator and agent_quality_safety review frozen artifacts independently and can open recall tickets; their final checks occur after any patch round, not only before patches. Minimum orchestrator-subagent hops: Hop A EvidenceRubric fan-out: orchestrator sends task deliverables, prior evidence defects, and acceptance criteria to all agents. ...

[19] [20]

agent_interface_contract_steward publishes RequiredPathManifest v2 and InterfaceContract v3 before implementation; every producing and validating agent must return ContractAckV3 or ChangeRequestV3 with field-level comments

[20] [21]

agent_structural_data_rcsb and agent_torsion_geometry jointly produce DatasetRecord v3 and DataProvenanceManifest v2 before model, training, eval, profile, or equivariance code consumes data

[21] [22]

agent_structural_data_rcsb and agent_eval_benchmarks jointly produce SplitLeakageReview v2 before training and evaluation claims are frozen

[22] [23]

agent_geometric_ml and agent_baseline_ablation jointly produce ModelRegistry v2, ModelIOContract v3, and AblationGrid v3 before training begins; agent_training_reproducibility, agent_eval_benchmarks, agent_profile_systems, agent_equivariance_referee, and agent_metrics_artifact_integrator must ack importability, checkpoint keys, model_ids, and factory signatures

[23] [24]

53 Recursive Harness Self-Improvement

agent_equivariance_referee publishes EquivarianceAuditContract v1 and results/equivariance_tests.json before scicomm writes equivariance claims or final acceptance; failures trigger recall to model or training. 53 Recursive Harness Self-Improvement

[24] [25]

agent_eval_benchmarks publishes ResultsContract v3 before metrics integration and scicomm accuracy claims

[25] [26]

agent_profile_systems publishes ProfilingContract v3 and validates exact src/profile.py before metrics integration and speed or memory claims

[26] [27]

agent_profile_sanity_referee audits ProfilingContract before scicomm profiling text; blockers route to profile or metrics integrator

[27] [28]

agent_metrics_artifact_integrator freezes MetricsIntegrationContract only after eval, profile, ablation, training, data, and equivariance inputs are consumed and results/metrics.json contains runtime, profiling, memory, and equivariance summaries or explicit waivers

[28] [29]

agent_scicomm_publication sends ClaimsEvidenceMatrix to agent_claims_consistency_referee; all referee blockers route to the owning specialist or scicomm for patch

[29] [30]

agent_artifact_validator, agent_quality_safety, agent_profile_sanity_referee, and agent_reproducibility_log_auditor review frozen artifacts independently and can open recall tickets; their final checks occur after any patch round, not only before patches. Minimum orchestrator-subagent hops: Hop 0 EvidenceRubric fan-out: orchestrator sends task deliverable...

[30] [31]

Interface steward publishes RequiredPathManifest v4, InterfaceContract v4, and MetricsPointerMatrix v2 before implementation; every producing and validating agent returns field-level ContractAckV4 or ChangeRequestV4

[31] [32]

Data agent publishes DataProvenanceManifest v4, RCSB query attempts, and filter verification; RCSB protocol referee must pass or conditionally pass before training and before any report data claim

[32] [33]

Data and geometry jointly produce DatasetRecord v4 and no-sidechain-input leakage evidence before model, training, eval, profile, or equivariance code consumes data

[33] [34]

Data and eval jointly produce SplitLeakageReview v3 before training and evaluation claims are frozen

[34] [35]

Model and ablation jointly produce ModelRegistry v4, ModelIOContract v4, and AblationGrid v4; training, eval, profile, equivariance, and metrics agents must ack importability, factory signatures, model_ids, and checkpoint keys

[35] [36]

Training produces TrainingHistory and CheckpointMetadata; eval and metrics must ack epoch-selection fields before metrics freeze

[36] [37]

64 Recursive Harness Self-Improvement

Entrypoint CI audits main.py before final; stale hello-world is blocking. 64 Recursive Harness Self-Improvement

[37] [38]

Equivariance referee publishes EquivarianceAuditContract v2 and results/equivariance_tests.json with command evidence before scicomm writes equivariance claims

[38] [39]

Eval publishes ResultsContract v4 and metrics_lineage before metrics integration and report accuracy claims

[39] [40]

Profile publishes ProfilingContract v4 and validates exact src/profile.py before metrics integration and speed or memory claims

[40] [41]

Profile sanity referee audits ProfilingContract and metrics runtime-memory merge before scicomm profiling text

[41] [42]

Metrics integrator freezes MetricsIntegrationContract only after eval, training, profile, ablation, data, RCSB, and equivariance inputs are consumed and results/metrics.json contains all required pointers

[42] [43]

JSON consistency referee audits frozen metrics before plots, index summaries, and report are considered final

[43] [44]

Scicomm sends ClaimsEvidenceMatrix to claims referee; all claims blockers route to owning specialist or scicomm for patch

[44] [45]

Minimum orchestrator-subagent hops: Hop 0 EvidenceRubric fan-out: send task deliverables, pairwise history, and current v3 file evidence to all agents

Validator, quality, RCSB referee, profile sanity, JSON consistency referee, claims referee, entrypoint CI, and log auditor review after every patch round, not only before patches. Minimum orchestrator-subagent hops: Hop 0 EvidenceRubric fan-out: send task deliverables, pairwise history, and current v3 file evidence to all agents. Hop 1 Parallel defect pre...

[45] [46]

write your history log: write your plan, execution (or tool-execution), reflection during the reasoning logs in a logs.txt file

Following query solved with claude coding ——————————– # Task (task description) # Current multi agent design (v0) (multi agent design description) # Save rules Use uv for all Python workflows--run code with uv run, install dependencies with uv add, use uvx for tools. write your history log: write your plan, execution (or tool-execution), reflection during...

[46] [47]

Current submission code repo from Claude Code (evidence only) # Workspace for current iteration (multi-agent-design-v0) Path:`...` ## Directory tree (representative) (tree description) ## Programmatic preflight (heuristic evidence only) Derived from the **task text** (expected paths) vs this workspace; not a substitute for reading the Deliverables block. ...

[47] [48]

- Repo B:

Pairwise history summary - Comparison: multi-agent-design-v0 vs multi-agent-design-v1 - Key: v0 vs v1 - Winner: tie - Judge model: gpt-5.5 - Judged at (UTC): 2026-05-24T10:00:05.337848+00:00 - Repo A: ... - Repo B: ... - Rationale (verbatim excerpt): The two submissions are indistinguishable in the provided evidence:↩→ both point to the same workspace pat...

2026

[48] [49]

Output to orchestrator

Instructions Improve the multi-agent design from v1 to v2 to enhance the quality of the query’s deliverables by creating genuine multi-agent advantages over single-agent execution. Preserve the original intent and required deliverables, but address weaknesses revealed by evidence. • Explicitly strengthen "Output to orchestrator" contracts so downstream ag...

[49] [50]

Reproducible code including dataset preparation, fine-tuning, evaluation, paraphrase genera- tion (public paraphrase model or rule-based templates), andrequirements.txt. 5.results/metrics.jsonincluding: •Top-1 accuracy •Macro F1 •Expected calibration error (ECE) •Paraphrase robustness drop •Per-class metrics 6.results/model_comparison.json comparing froze...

[50] [51]

Reproducible Python code for data collection, feature engineering, modeling, and evaluation. 5.results/metrics.jsonincluding: •IC / RankIC •Predictive𝑅 2 •Strategy Sharpe ratio •Beta •Maximum drawdown •Turnover •Transaction costs •Placebo-test metrics 6.results/feature_mapping.json documenting page-to-ticker mappings and confi- dence scores, andresults/mo...

[51] [52]

Reproducible Python code for data processing, representation learning, multimodal fusion, training, and evaluation. 5.results/metrics.jsonincluding: •Macro and micro F1 •Hierarchical accuracy across EC levels 1–4 •Coverage statistics 6.results/ablation_results.json comparing fusion variants (concatenation, attention, gating) with configurations and evalua...

[52] [53]

research_report.mdasaconference-stylepaper(6–9pagesequivalent)centeredoncalibration and decision-making

[53] [54]

deliverables/index.json

[54] [55]

deliverables/plots/: reliability_diagram.png, ece_vs_temperature.png, roc_curve.png, deci- sion_utility_curve.png, calibration_by_difficulty_bin.png

[55] [56]

Reproducible code: data generation; feature extraction; model training; calibration methods (temperature/isotonic); evaluation; requirements.txt

[56] [57]

results/metrics.json: auroc, average_precision, ece, brier_score, expected_utility_gain

[57] [58]

project":

results/model_comparison.json comparing uncalibrated vs calibrated variants with per-bin metrics. # Labels (for orientation only) •SubmissionAcorresponds to run type: ‘base‘ •SubmissionBcorresponds to run type: ‘defaultTeam‘ # Workspace for submission **A** (evidence only) Path: ‘...‘ ## Directory tree (representative) query318/ |-- data/ |`-- episodes.pa...

[58] [59]

Deliverable coverage— map the task’sDeliverables/Deliverable:/ inline deliverables list (and any other explicit output requirements in ‘# Task‘) to evidence; mark gaps or placeholders

[59] [60]

3.Reproducibility— dependencies, entry points, seeds, documented data or generation

Numerical/empirical rigor— appropriate methodology, baselines, honest limitations; con- sistency between report and metrics when applicable. 3.Reproducibility— dependencies, entry points, seeds, documented data or generation. 4.Presentation— report structure, clarity, figure integration (infer from paths and excerpts). 5.Engineering— layout, modularity, r...

[60] [61]

winner":

Task alignment— penalize solving the wrong problem or drifting from the stated objective. ## Output JSON schema (exact keys) { "winner": "A" | "B" | "tie", "rationale": "<string, cite concrete evidence from both workspaces>" } Return JSON only. F. Distribution of normalized cost, output tokens, and cache read/write 87 Recursive Harness Self-Improvement hi...