{"id":"100d49c0-e8cd-4078-90ed-954b0c7b96cf","arxiv_id":"2608.13560","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"AutoDesign recursively improves a design harness, and the resulting DesignHarness raises PosterBench scores by 5.0 to 19.6 points across seven model configurations.","lead":"This paper introduces AutoDesign, a system that rewrites its own poster-design software based on rollout feedback, and a new 100-paper benchmark, PosterBench, for paper-to-poster generation. The evolved system beats the commercial Claude Design by 7.45 points and wins a system-blind human preference study.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 100-paper Main Track score is not established as held-out: Algorithm 1's Dtrain and Ddev are never specified as disjoint from PosterBench Main Track or mini, so the 78.32 headline could be an optimization-set result.","rationale":"The reader's weakest assumption identifies exactly the load-bearing gap: the paper never states that the PosterBench Main Track and PosterBench-mini are disjoint from Algorithm 1's Dtrain and Ddev. I reviewed the full text including the appendix for any explicit held-out statement and found none. Section A.1's comparison matrix fixes source papers across a row but does not address train/dev overlap; Section 3.2's 'independent development set' language is about preventing overfitting to Dtrain, not about protecting the benchmark. This matters because the two headline numbers, 78.32 on the 100-paper Main Track and +12.4 across seven configurations on PosterBench-mini, are only meaningful as generalization measurements if the evaluated papers were never used to update or gate the harness. If the overlap exists, the reported gains could reflect memorization of evaluation-set patterns rather than a reusable design harness; if it does not, the results are clean. The paper deserves credit for releasing records and checkpoints, which makes the proposed intersection test feasible, and the appendix's detailed protocol suggests good-faith evaluation. Because the flaw is an unstated data-split condition rather than a demonstrated contamination, the correct disposition is conditional acceptance pending the held-out check, which is the reader's existing verdict. My stress-test therefore does not move the verdict, but it sharpens the required clarification.","tokens_in":26221,"tokens_out":5025,"duration_ms":52473,"concrete_test":"Cross-reference the released per-case records (Section A.5) and the optimization record L from Algorithm 1 to compute Dtrain ∩ MainTrack, Ddev ∩ MainTrack, and the same intersections with PosterBench-mini. If any intersection is non-empty, recompute the 78.32 headline and the Table 4 gains restricted to the disjoint complement of Dtrain ∪ Ddev; if removing all overlapping cases changes either by more than the reported margins, the headline is not a held-out result. If the intersections are empty, report the construction of Dtrain and Ddev explicitly, including sizes and how they were sampled from the full paper pool.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that AutoDesign generalizes: 'On the PosterBench Main Track, AutoDesign scores 78.32' (Section 5.1) and that the learned DesignHarness raises the average PosterBench Score from 54.99 to 67.39 across seven configurations (Section 5.2.1, Table 4). This claim is only valid if the evaluation sets are disjoint from the optimization sets. Algorithm 1 optimizes against Dtrain and gates acceptance with Ddev, but the paper never states that either set is held out from PosterBench or PosterBench-mini. Section 3.2 says Ddev is 'independent' only relative to Dtrain ('guards against overfitting the harness to the training tasks'); it does not say Ddev is separate from the benchmark. Section A.1 and Table 5 list 'source paper' as a fixed factor across compared systems, but they do not exclude papers used in Dtrain or Ddev. PosterBench-mini is described as 'the original 10-paper subset used for the controlled ablations' (Section 5.1), and Table 4 reports the harness gain on exactly that subset. If Dtrain or Ddev overlaps those 10 papers, the +12.4 improvement is in-sample, not a controlled generalization result; if either set overlaps the 100 Main Track papers, the 78.32 headline is contaminated. The distinction drawn in Sections 3.2 and A.4 between Rmeta and the frozen PosterBench protocol separates evaluator functions, not data instances, so it does not resolve the concern. Without an explicit disjoint split, Tables 1–4 are all vulnerable to evaluation-set overfitting.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript introduces AutoDesign, a two-loop meta-harness optimization framework that iteratively improves a design harness (prompts, tools, orchestration, validation, and evaluation feedback) around a fixed model, and applies it to the academic paper-to-poster task. It also introduces PosterBench, a 100-paper Main Track plus a 10-paper PosterBench-mini subset, with a seven-dimension rubric spanning faithfulness, coverage, density, visual evidence, layout, readability, and aesthetics. The paper reports that AutoDesign with DesignHarness scores 78.32 on the Main Track, 7.45 points above the commercial Claude Design system; that attaching DesignHarness improves all seven tested model/coding-agent configurations by between 5.0 and 19.6 points, raising the average PosterBench Score from 54.99 to 67.39; and that a system-blind human study gives AutoDesign the highest Bradley-Terry preference estimate at 64.0%. The central evaluation claim is that these numbers reflect generalization of the learned harness, which requires the optimization sets Dtrain and Ddev from Algorithm 1 to be disjoint from the PosterBench evaluation sets. The manuscript does not state that this disjointness holds.","tokens_in":28123,"tokens_out":4396,"duration_ms":44318,"significance":"If the reported results are genuinely held-out, the paper makes a useful contribution: it demonstrates a concrete instantiation of harness-level optimization with persistent updates, provides a multi-dimension benchmark with a documented protocol, and ships released evaluation records that support auditing. The controlled ablation across seven configurations showing consistent gains, and the cost-performance analysis, are valuable even apart from the headline number. The main uncertainties are data separation and evaluator alignment: the absence of an explicit disjointness statement for Dtrain/Ddev versus PosterBench, and the fact that the optimization-time evaluator Rmeta and the final PosterBench metric share the same seven-dimension vocabulary and were both authored by the same team. Until those points are resolved, the headline 78.32 and the +12.4-point average gain are not established as unbiased generalization results.","major_comments":[{"comment":"Algorithm 1 optimizes the harness against Dtrain and gates updates with Ddev, but the manuscript never states that either set is disjoint from the 100-paper PosterBench Main Track or the 10-paper PosterBench-mini. The 'independent development set' in Section 3.2 is described only as guarding against overfitting to Dtrain, not as being held out from PosterBench; Section A.1 and Table 5 list the source paper as a fixed factor but do not exclude optimization-set papers. Consequently, the headline Main Track score 78.32 (Section 5.1) and the average gain from 54.99 to 67.39 (Section 5.2.1, Table 4) are unbiased only under an unstated assumption. The authors must state the split explicitly; if any overlap exists, the evaluations must be re-run on disjoint papers or the affected numbers must be reported as in-sample results.","section":"3.2 / Algorithm 1 / 5.1 / A.1"},{"comment":"The paper separates Rmeta from PosterBench by role ('Rmeta supplies feedback... PosterBench evaluates completed systems'), but both evaluators are authored by the same team and use the same seven-dimension vocabulary with the same weight emphasis on density, layout, and readability. Optimizing against Rmeta therefore implicitly optimizes toward PosterBench, so consistent gains across seven configurations do not by themselves establish that the harness improves human-judged quality. Please provide a quantitative check of the coupling, such as the correlation between Rmeta and PosterBench scores on a held-out sample of artifacts, or an ablation using an independently specified evaluator, so that readers can assess the degree of metric alignment.","section":"3.2 / A.4 / Table 6"},{"comment":"The abstract and Section 1 claim that AutoDesign 'reaches average conference-poster quality in human evaluation,' but the human study in Section 5.3 is a system-blind pairwise preference comparison among four systems, not a rating of posters against real conference posters. The reported benchmark-human agreement (r = 0.34) and the nominal Krippendorff alpha of 0.101 in Appendix A.5 are modest. The pairwise-preference claim is supported by the data, but the 'conference-poster quality' claim should be removed or replaced with a direct human rating against real conference posters.","section":"5.3 / A.5 / 1"}],"minor_comments":[{"comment":"The 78.32 Main Track score is reported without any uncertainty estimate; given N=100 and the visible per-dimension variance, a bootstrap confidence interval would clarify the comparison with the 70.87 Claude Design baseline.","section":"5.1"},{"comment":"The gain is reported as '+12.4%' in the abstract and as '+12.40 points' in Section 1; please use consistent units.","section":"Abstract / 1"},{"comment":"The trajectory values (49.00, 80.88, 88.39) are not linked to a specific source paper or configuration; the caption or appendix should give the case id and the model/coding-agent details.","section":"Figure 1(a)"},{"comment":"The 64.0% Bradley-Terry estimate should be accompanied in the main text by its 95% interval (55.2–77.8%), since that interval overlaps the 51.7% point estimate for Claude Code and readers may otherwise over-interpret the gap.","section":"5.3"},{"comment":"The note that Krippendorff's alpha is a nominal agreement diagnostic rather than the ranking estimator is useful but appears only in the appendix; one sentence in Section 5.3 would prevent misinterpretation.","section":"A.5"}],"recommendation":"major_revision","confidential_remarks":"The load-bearing issue is the missing explicit disjointness of Dtrain/Ddev from the PosterBench evaluation sets. If the authors cannot document that split, the headline result is not an evaluation of generalization. I would also flag that the benchmark and the optimized metric come from the same team; the paper does disclose this in A.4, but the practical effect on the headline number deserves scrutiny during revision. The paper otherwise has solid experimental breadth and the released records are a strength."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, this is the most complete attempt I've seen at turning the harness-optimization idea into a concrete artifact for a real design task, and it ships code, data records, and a benchmark. Second, the headline number is not yet trustworthy, because the paper never states that the 100-paper Main Track and the 10-paper mini subset are disjoint from the training and development sets used in Algorithm 1.\n\nWhat is actually new: the five-component decomposition of a design harness is a useful organizational device; the bounded one-component update with a train/dev acceptance gate is a clean recipe; and PosterBench's seven-dimension protocol with protected gates is a serious attempt at a hard-to-evaluate task. Seven configurations all improve with the learned harness. Combined with the released repository and per-case record schema, I believe the qualitative claim that the harness helps. That deserves real credit.\n\nThe soft spots, in proportion. The big one is data provenance. Section 3.2 says Ddev is \"independent,\" but only relative to Dtrain. Algorithm 1 never mentions PosterBench. Appendix A.1's comparison matrix lists fixed factors, but not whether the optimization sets were drawn from the same papers. So the 78.32 Main Track could be an optimization-set score, and the +5 to +19.6 gains on mini could be in-sample. That is the difference between a generalization result and tuning to the test set. It is not a minor wording issue. Also, the same team designed Rmeta and PosterBench with the same seven dimension names. The paper says the functions are separate, but the shared vocabulary and authorship make the split question more pressing, not less. Second, the headline 78.32 has no uncertainty estimate; the human study's bootstrap interval is wide and the nominal Krippendorff coefficient is 0.101. The human evidence is weak, though the paper is honest about it.\n\nNone of this kills the core idea. The recipe is plausible, the artifacts are concrete, and the benchmark could be widely used. But the authors need to state explicitly which papers are in Dtrain and Ddev, whether either set overlaps PosterBench or PosterBench-mini, and release the assignment. If they do, this is a useful contribution; if they cannot, the numbers are not interpretable.\n\nWho is this for: people building self-improving agent harnesses and anyone evaluating multimodal generation systems. PosterBench alone may be worth the paper. I would send it to peer review, and a serious referee should demand the data-split clarification before acceptance.","headline":"A genuinely reusable harness-optimization recipe and a new benchmark, but the headline results are not yet interpretable because the paper never states that the train/dev sets are disjoint from the evaluation sets.","tokens_in":27147,"tokens_out":2709,"would_cite":true,"duration_ms":28188,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AutoDesign claims the harness around a fixed model—not the model itself—is the right optimization target, and demonstrates it with a 78.32 score on the 100-paper PosterBench Main Track, 7.45 points above Claude Design.","keywords":["meta-harness optimization","design harness","paper-to-poster generation","PosterBench","long-horizon agentic design","recursive self-improvement","multimodal design","acceptance gate"],"falsifier":"Check the released per-case records: if any of the 100 Main Track case IDs (for example '2017-attention-is-all-you-need') also appears in the training set $D_{\\text{train}}$ or development set $D_{\\text{dev}}$ of Algorithm 1, the reported 78.32 is not an unbiased held-out score. A complementary check is to rerun the meta-harness loop with the acceptance gate disabled; if the ungated harness scores about the same on papers outside the optimization sets, the development-set criterion is not what prevents overfitting.","tokens_in":26052,"feed_emoji":"🎨","tokens_out":15238,"duration_ms":124940,"temperature":0.7,"pith_summary":"AutoDesign is trying to establish that a multimodal design system can improve itself by optimizing the 'harness'—the prompts, tools, validation rules, and control flow surrounding a fixed model—rather than the model's weights. The paper proposes two nested loops: an inner loop in which the harness converts a source paper into an editable poster through repeated critique-and-repair, and an outer loop in which a meta-harness studies recorded rollouts and scores, proposes a change to exactly one of five harness components, and accepts the change only if training performance rises and development performance does not fall. To test the idea, the paper builds PosterBench, a 100-paper benchmark with a seven-dimension rubric, and reports that the evolved DesignHarness raises the average PosterBench Score across seven model/coding-agent configurations from 54.99 to 67.39, reaches 78.32 on the Main Track, and wins a system-blind human preference study. A sympathetic reader would care because the result suggests that durable, transferable design skill can be accumulated in software around a model at low cost, without retraining the model.","feed_headline":"Beats Claude Design by 7.45 points in paper-to-poster generation","feed_subtitle":"AutoDesign's recursive harness optimization lifted scores on all seven model setups — and cut cost to under $3 per poster.","key_machinery":"The load-bearing mechanism is the design harness $H$ with its five-component decomposition, together with the two-loop meta-harness cycle built on it. The inner loop instantiates the harness as a designer–critic pair, $y_k = M^{\\mathrm{design}}(y_{k-1}, f_{k-1}; x, c)$ and $f_k = M^{\\mathrm{critic}}(y_k; x, c)$, so the artifact is refined locally as editable code while rule-based validation and vision-language-model critique supply repair feedback. The outer loop aggregates trajectories $\\tau_t$ and scores $s_t$ across training tasks, and a coding agent acting as the meta-harness optimizer $P$ proposes one component-restricted update from this evidence plus an optimization record $L$; the acceptance gate then promotes the candidate only when $J_{\\text{train}}(H') > J_{\\text{train}}(H)$ and $J_{\\text{dev}}(H') \\geq J_{\\text{dev}}(H)$, with the development set kept hidden from the optimizer so that it guards against overfitting the harness to training papers. What carries the empirical argument is PosterBench, a frozen, externally specified evaluator producing a weighted seven-dimension rubric score (faithfulness, coverage, density, visual evidence, layout, readability, aesthetics) with record-level ceilings, applied after optimization so that the optimized harness and its competitors are measured under the same protocol.","core_discovery":"The paper's central claim is that a static production procedure for multimodal output can be turned into a persistent learning system by making the design harness itself the object of optimization. Formally, the harness is $H$ with $y \\sim H(\\pi_\\theta, x, c)$ where the model parameters $\\theta$ stay fixed and the optimization acts on the surrounding system, decomposed into five functional components: context and memory, tools and specifications, execution runtime, orchestration, and evaluation and feedback. The meta-harness objective is $H^\\star = \\arg\\max_H J(H)$ with $J(H)$ the expected quality of produced artifacts under a human-aligned evaluator; each outer-loop iteration restricts the update to one component and gates promotion on $J_{\\text{train}}(H') > J_{\\text{train}}(H)$ and $J_{\\text{dev}}(H') \\geq J_{\\text{dev}}(H)$. Iterating this loop produces DesignHarness, an executable paper-to-poster system whose designer module authors editable HTML, validates it with rule-based blocking checks, critiques rendered previews with a vision-language model, and finalizes the best valid candidate. On the newly introduced PosterBench protocol, the claim is confirmed across seven configurations: attaching DesignHarness improves every one of them by 5.0 to 19.6 points, and the full system scores 78.32, surpassing the commercial Claude Design by 7.45 points and receiving the highest Bradley–Terry preference (64.0%) in a system-blind human study.","pith_inferences":["The gain pattern in the seven configurations—largest on the weakest models (+19.6 for DeepSeek V4 Pro, +17.8 for Seed 2.1 Pro, versus +5.0 and +5.6 for Claude 4.8 and GPT-5.5)—suggests harness optimization acts partly as a skill-transfer mechanism, a pattern worth probing on other design tasks.","A direct ablation would settle the acceptance gate's role: rerun the outer loop with the development-set criterion disabled and compare final scores; if the ungated harness matches the gated one, the gate is not what prevents overfitting.","The paper treats the optimization record $L$ as the system's memory; wiping $L$ and rerunning optimization from the initial harness would quantify how much of the final gain is accumulated experience rather than the initial scaffold."],"forward_implications":["Harness capability transfers across models and coding agents: the same DesignHarness improves all seven tested configurations, so the learned system-level skill is not tied to a single model.","High-quality posters become cheap: with the LongCat 2.0 configuration the harness reaches 55.13 at roughly $0.27 per poster, making conference-poster quality accessible without frontier-model prices.","PosterBench scores are most informative when the gap is large: blind human agreement with the benchmark-preferred poster rises from 51.9% for 0–3 point gaps to 74.4% for gaps of at least 20 points.","The same meta-harness loop is claimed to extend to slides, webpages, and videos, provided each medium gets its own evaluator, rendering checks, and promotion gate; these outputs are pilots in this paper.","Because posters remain editable HTML with provenance links to the source paper, a user can apply localized revisions after generation instead of regenerating the whole artifact."],"supporting_citations":[{"why":"Defines the model harness as an optimization target distinct from model weights, supplying the framing AutoDesign builds on for meta-harness optimization.","marker":"(Lee et al., 2026b)"},{"why":"Provides the model-versus-scaffold distinction in self-improving-agent taxonomies that justifies optimizing the system around a fixed model.","marker":"(Ren et al., 2026)"},{"why":"Supplies the held-out development-split acceptance gate that AutoDesign uses to decide whether a harness update is promoted.","marker":"(Nguyen et al., 2026)"},{"why":"Establishes the response-level critique-and-revise pattern that the inner design loop instantiates with rule-based and vision-language critics.","marker":"(Madaan et al., 2023)"},{"why":"Introduces the prior paper-to-poster generation setting and benchmark against which AutoDesign's task framing and comparisons are anchored.","marker":"(Pang et al., 2025)"},{"why":"Provides an earlier automated poster-pipeline system, one of the baselines establishing the task of extracting and laying out paper content.","marker":"(Jaisankar et al., 2025)"},{"why":"Supplies the paired-comparison model used to estimate system preference from the system-blind human study.","marker":"(Bradley and Terry, 1952)"},{"why":"The closest prior approach searching over agentic-system designs; AutoDesign defines its harness-level unit of optimization in contrast to it.","marker":"(Hu et al., 2025)"}],"fun_headline_variants":["AutoDesign: Self-improving harness beats Claude by 7.45 points","Meta-harness optimizer boosts all seven models by 5–19.6 points","Paper-to-poster: AutoDesign's $3 harness beats Claude by 7.45","Recursive harness loop: DesignHarness lifts every setup to 78.32"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the 100 papers scored in the PosterBench Main Track are disjoint from the training and development papers used to optimize and gate the harness, so the headline 78.32 is a held-out measurement; the paper states the training/development split in Algorithm 1 but never explicitly says the Main Track excludes those papers.","fun_headline_variants_meta":{"raw":{"variants":["AutoDesign: Self-improving harness beats Claude by 7.45 points","Meta-harness optimizer boosts all seven models by 5–19.6 points","Paper-to-poster: AutoDesign's $3 harness beats Claude by 7.45","Recursive harness loop: DesignHarness lifts every setup to 78.32"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00052,"raw_usage":{"total_tokens":2610,"prompt_tokens":1132,"completion_tokens":1478,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":748,"completion_tokens_details":{"reasoning_tokens":1389}},"tokens_in":748,"tokens_out":1478,"duration_ms":13084,"temperature":1.0,"reasoning_tokens":1389,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:09:00.068448+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Check the released per-case records: if any of the 100 Main Track case IDs (for example '2017-attention-is-all-you-need') also appears in the training set $D_{\\text{train}}$ or development set $D_{\\text{dev}}$ of Algorithm 1, the reported 78.32 is not an unbiased held-out score. A complementary check is to rerun the meta-harness loop with the acceptance gate disabled; if the ungated harness scores about the same on papers outside the optimization sets, the development-set criterion is not what prevents overfitting.","supporting_citations":[],"review_version":1}