{"id":"ed478948-418c-4150-afae-30a0d87251af","arxiv_id":"2602.18548","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A new e-commerce-sourced benchmark evaluates design-to-code LLMs by requiring executable React generation and iterative revision from a noisy design IR plus a reference image.","lead":"1D-Bench is a new benchmark that tests design-to-code AI models by having them build React pages from a reference screenshot plus an imperfect exported design file, then revise the code over several rounds using rendering feedback. Results show iterative editing usually helps today's multimodal models, while a small reinforcement-learning training experiment produced little reliable gain.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The composite visual-similarity metric is validated only on synthetic perturbations, not on real model outputs; rankings and multi-round similarity gains could shift under an independently validated scorer.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: all conclusions about visual fidelity, model rankings, and multi-round similarity improvements flow through the hand-set composite S. I independently checked the central claims and found the RSR improvements are objective and robust, but the similarity score is only calibrated on 50 synthetic perturbation pairs. This is a real, testable gap, not an internal inconsistency or an ad hominem issue. The paper's fixed toolchain, WriteTool interface, multi-round protocol, and honest negative RL pilot are all genuine strengths, and the benchmark is plausibly useful, so the conditional verdict remains appropriate. No other concern—such as absence of an IR-to-render baseline or ambiguity about data release—is as directly load-bearing as metric validity, because every headline result depends on S. If the proposed human-preference test on real outputs passes and rankings are stable under alternative weights, the concern would be resolved; if not, the headline conclusions would need re-reporting.","tokens_in":20803,"tokens_out":6041,"duration_ms":58018,"concrete_test":"Take ~30 instances and, for each, two real outputs from Table 2 (e.g., Claude single- vs multi-round; GPT-5.2 single- vs multi-round; Claude vs Gemini multi-round). Run the same 20-annotator pairwise protocol as Figure 3A, comparing the authors' metric winner against the human majority via agreement and rank correlation. Also recompute S and FinalScore under alternative specifications (S_img only; equal weights; no OCR/template component) and check whether the Claude/Gemini/GPT ordering and multi-round similarity deltas survive. If human-metric agreement on real outputs is comparable to the synthetic validation and rankings are stable, the concern is resolved; otherwise Table 2 and the 'generally improves' conclusion need to be re-derived with a re-validated scorer.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Every headline quantity in Table 2—Claude's multi-round FinalScore 80.4, the GPT-5.2 similarity gain 78.3→84.9, the ranking of Gemini/Claude/GPT—is computed with the hand-set composite S=0.5·S_img+0.3·S_comp+0.2·S_layout, whose image-level weights are LPIPS 0.8 / SSIM 0.1 / pixel 0.1 (§3.3, Appendix B). The only validation of S against humans is the 50-instance synthetic perturbation study in Figure 3A, generated by controlled DOM/style edits (sibling swaps, node moves, numeric CSS drifts). That is a different distribution from actual model outputs, which fail with missing files, wrong image assets, layout collapse, and font fallback. Nothing in the paper shows that S ranks such real outputs the way human designers would, and no sensitivity analysis or alternative metric is reported. If LPIPS or the OCR/template-matching components are over-weighting artifacts of the synthetic perturbations, the headline scores and the claim that multi-round editing 'often improves visual similarity' could change. The objective RSR gains (e.g., GPT-5.2 63.6%→93.2%) are robust and support the executability benefit of multi-round, but the similarity-based component of the central claim rests on a scorer not validated on real model outputs.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces 1D-Bench, a benchmark for design-to-code generation where each instance provides a reference rendering and a potentially noisy exported intermediate representation (IR). Models must generate an executable React codebase under a fixed toolchain; the benchmark defines single-round and multi-round settings, with the latter iteratively feeding back similarity scores, diff heatmaps, and failure logs. The authors collect 984 instances from internal e-commerce workflows, construct a stratified 204-instance evaluation set, and evaluate five commercial and open-weight multimodal LLMs. Their central claim is that multi-round generation generally improves rendering success rate and often visual similarity, leading to higher FinalScore (e.g., Claude Sonnet 4.5 reaches 80.4). They also present a pilot post-training study using synthetic repair trajectories and GRPO, reporting limited and unstable gains.","tokens_in":21162,"tokens_out":8177,"duration_ms":71905,"significance":"The paper addresses a real gap in design-to-code benchmarking: most prior benchmarks target isolated HTML, use synthetic or crawled data, and lack an iterative editing protocol. 1D-Bench's grounding in real e-commerce workflows, its fixed executable React toolchain, and its multi-round evaluation with execution feedback are notable strengths. The data pipeline is detailed, including automatic filters, stratification, and manual review, and the authors release the dataset and scripts (anonymized). The honest reporting of the negative pilot RL results is also a credit. If the metric validation is strengthened, this benchmark could become a useful community resource for comparing models on realistic design-to-code tasks.","major_comments":[{"comment":"The central claim that multi-round editing improves visual similarity rests on the hand-set metric S = 0.5·S_img + 0.3·S_comp + 0.2·S_layout with sub-weights w_lpips=0.8, w_ssim=0.1, w_pix=0.1. The only human validation (Figure 3A) uses 50 synthetic perturbation pairs generated by controlled DOM/style edits (sibling swaps, node moves, numeric CSS drifts); real model outputs fail in different ways (missing assets, layout collapse, font fallback). The text states that human preference correlates with the metric score difference but never reports the agreement accuracy or correlation coefficient. This is load-bearing: if the components are tuned to the synthetic perturbation distribution, the multi-round similarity gains and model rankings in Table 2 could shift under an independently calibrated metric. I recommend validating S on a sample of actual model outputs with pairwise human judgmen","section":"§3.3, Appendix B (composite metric)"},{"comment":"The headline results are single-run means without confidence intervals or significance tests. With 204 instances, differences such as Claude 80.4 vs Gemini 79.5 vs GPT-5.2 79.1 in multi-round FinalScore are likely within noise, and the multi-round vs single-round gain for Gemini (79.6→79.5) is not distinguishable from zero. To support the central claim that 'iterative editing generally improves final performance,' the authors should report per-instance bootstrap confidence intervals for S and RSR, or paired tests, for each model.","section":"Table 2, §4.2"},{"comment":"The multi-round protocol is underspecified: the maximum number of rounds, the early-stopping threshold, and the average number of rounds used per model are not stated. This matters because a model that exhausts all rounds has many more chances to fix build errors; GPT-5.2's RSR jump from 63.6% to 93.2% could be largely an effect of additional attempts. The authors should report the round limit, the stopping criterion, and the distribution of rounds per model, and ideally show the per-round progression of RSR and S.","section":"§3.2, Appendix C"}],"minor_comments":[{"comment":"The diagram labels the benchmark 'D2C-Bench' while the paper title and abstract use '1D-Bench'. Please make the naming consistent.","section":"Figure 1A"},{"comment":"The percent sign is missing in '63.6% to 93.2' for the GPT-5.2 RSR increase.","section":"§4.2, Table 2"},{"comment":"The figure lacks axis labels and does not state the number of annotators per pair or the consensus rule. Please add these details; the current presentation makes the calibration result hard to interpret.","section":"Figure 3A"},{"comment":"The reference list contains formatting inconsistencies (e.g., an entry starting with '5 Team'; 'Qwen Team' lacks a year). Please proofread.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"This is a solid benchmark paper, but the metric validation is the main weakness. If the authors add a human preference study on real model outputs and a sensitivity analysis for the composite metric, the paper could become acceptable. The missing statistical uncertainty and multi-round protocol details should also be addressed. The scope fits cs.SE well."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: bring this to a design-to-code reading group and cite it if you work on UI code generation benchmarks. The combination is new — noisy exported IR plus reference image as input, fixed-toolchain executable React output, and a multi-round execution-feedback protocol on real industrial e-commerce data. That fills a real gap in Table 1, where most benchmarks are single-round HTML from screenshots.\n\nWhat it does well: the data pipeline is careful (rule filtering, dedup, VLM screens, manual review, stratified sampling), the execution harness is described in enough detail to reproduce, and the prompts are included verbatim. The RSR results are robust and genuinely interesting — GPT-5.2 going from 63.6% to 93.2% rendering success with iterative feedback is a concrete, objective gain. The RL pilot is honestly reported as a negative result, and the segmented-GRPO setup is a reasonable attempt.\n\nThe soft spot is exactly what the stress-test note flags. The composite similarity score S = 0.5·S_img + 0.3·S_comp + 0.2·S_layout, with LPIPS 0.8/SSIM 0.1/pixel 0.1, is validated only on 50 synthetic perturbation pairs, not on real model outputs. That validation shows the metric tracks human preference for sibling swaps and numeric CSS drifts, but real model failures — missing assets, layout collapse, font fallback — are a different distribution. No sensitivity analysis and no alternative metric are reported. So the final scores and the Claude-vs-GPT-vs-Gemini ranking rest on a hand-set scorer that could shift under independent validation. That is a moderate weakness, not a fatal one: the main multi-round claim (iteration improves executability) is carried by RSR, and the similarity gains are secondary. Still, it needs to be fixed.\n\nMinor points: there is no baseline that ignores the IR, so we cannot see how much the noisy structure actually helps; error bars are absent from Table 2; and the paper doesn't explicitly state whether the full 204-instance set is released beyond the anonymous repo. All are addressable.\n\nOn the stress-test note: I think it is fair. The concern is real, but it doesn't undermine the benchmark's core value — it just means the fine-grained scores should be read with caution.\n\nWho this is for: anyone building or evaluating design-to-code systems. It deserves a serious referee. My recommendation: send it to review, and require the authors to (a) validate the metric on real model outputs or at least report a sensitivity analysis over the weights, and (b) run an IR-ablation baseline.","headline":"A solid, well-documented benchmark with a genuinely useful multi-round execution protocol; the hand-set similarity metric is the soft spot and is validated only on synthetic perturbations, so the fine-grained model ranking is less certain than the RSR trends.","tokens_in":21578,"tokens_out":2480,"would_cite":true,"duration_ms":24468,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"1D-Bench claims that multi-round editing with visual feedback improves UI code generation over one-pass generation, with render success rising from 63.6% to 93.2% for one model.","keywords":["design-to-code","UI code generation","benchmark","multi-round editing","visual feedback","rendering success","visual similarity","agentic reinforcement learning"],"falsifier":"Take the real renderings from the five models, have human annotators rank pairs, and compare with the composite metric; if the metric's winner matches the human majority on real outputs no better than chance (unlike on the 50 synthetic pairs), the hand-set weights do not transfer and the rankings collapse.","tokens_in":20732,"feed_emoji":"🖥️","tokens_out":5451,"duration_ms":45121,"temperature":0.7,"pith_summary":"This paper introduces 1D-Bench, a benchmark that measures whether multimodal models can turn a design image plus a possibly noisy exported intermediate representation (IR) into an executable React project. Its central claim is that iterative, multi-round editing with visual feedback — screenshots, similarity scores, diff heatmaps, and failure logs — generally improves final performance over one-pass generation, mainly by converting rendering failures into working pages. To make results comparable, the benchmark fixes the toolchain, the scaffold, and the scoring protocol, and reports a single FinalScore that multiplies average visual similarity by the rendering success rate. The paper also reports a pilot study showing that supervised fine-tuning and reinforcement learning on synthetic repair trajectories produce limited, unstable gains, which the authors attribute to sparse terminal rewards and high-variance full-file edits.","feed_headline":"Multi-round editing lifts UI code render success to 93%","feed_subtitle":"New benchmark 1D-Bench shows visual feedback beats one-pass generation, though RL training gains stay limited.","key_machinery":"The evaluation engine: a deterministic build-and-render harness that screenshots each generated React app and compares it to the reference through a composite similarity score S = 0.5·S_img + 0.3·S_comp + 0.2·S_layout, where S_img is mostly LPIPS (weight 0.8), S_comp is OCR/edge-based element completeness, and S_layout is center-distance-based. The FinalScore is mean similarity over successful renders times the rendering success rate, and this same pipeline supplies the feedback (score, diff heatmap, failure logs) that drives multi-round edits.","core_discovery":"Under a fixed toolchain and a WriteTool-only action space, the paper reports that every model evaluated improves its rendering success rate in the multi-round setting, with the largest gain moving from 63.6% to 93.2%. Mean visual similarity also rises for most models, and the best multi-round FinalScore is 80.4. At the same time, a pilot post-training recipe — supervised fine-tuning on synthetic repair traces followed by policy-gradient reinforcement learning with terminal-only rewards — fails to produce consistent improvement, suggesting that the bottleneck is not model inference but the credit-assignment signal.","pith_inferences":["The paper validates its score only on synthetic perturbations; a natural extension is to test metric–human agreement on the real model outputs, where errors are correlated rather than isolated.","Because IR quality is a controllable variable, 1D-Bench could also measure how model performance degrades as IR noise increases, turning robustness from a qualitative claim into a measurable curve.","The paper's negative RL result suggests a concrete next step — replacing full-file overwrites with line-level diff edits to densify the reward signal — which follows from the authors' own diagnosis of high-variance actions."],"forward_implications":["If the central claim holds, single-round scores understate model ability; benchmarks should report multi-round behavior as the more realistic measure.","The finding that most render failures are fixable with logs means compiler/runtime feedback is a low-cost, high-value signal for design-to-code models.","The RL pilot's failure implies that naive terminal-reward optimization over full-file overwrites is unlikely to improve iterative UI generation; better-shaped or denser rewards are needed.","The FinalScore product makes the benchmark robust to the common failure mode of generating pretty but non-executable code."],"fun_headline_variants":["Visual feedback lifts UI code render success to 93%","1D-Bench: Iterative edits beat one-pass UI generation","Multi-round UI coding hits 93% render, RL gains stay weak","1D-Bench shows feedback wins, but RL training lags"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The entire benchmark rests on the hand-set composite similarity score S, which was calibrated on 50 synthetic perturbation pairs; if that score misorders real renderings, the model rankings and the multi-round gains would change.","fun_headline_variants_meta":{"raw":{"variants":["Visual feedback lifts UI code render success to 93%","1D-Bench: Iterative edits beat one-pass UI generation","Multi-round UI coding hits 93% render, RL gains stay weak","1D-Bench shows feedback wins, but RL training lags"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000181,"raw_usage":{"total_tokens":1148,"prompt_tokens":751,"completion_tokens":397,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":495,"completion_tokens_details":{"reasoning_tokens":332}},"tokens_in":495,"tokens_out":397,"duration_ms":4427,"temperature":1.0,"reasoning_tokens":332,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T21:55:04.134786+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the real renderings from the five models, have human annotators rank pairs, and compare with the composite metric; if the metric's winner matches the human majority on real outputs no better than chance (unlike on the 50 synthetic pairs), the hand-set weights do not transfer and the rankings collapse.","supporting_citations":[],"review_version":1}