{"id":"97ecb528-a36f-4296-a9b5-9c81355193c4","arxiv_id":"2504.20294","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"mrCAD is a large dataset of multimodal human instructions for generating and refining CAD designs, and a benchmark showing that VLMs struggle with refinement instructions while humans excel at them.","lead":"Researchers collected a dataset of 6,082 human communication games where one player draws and writes instructions for another to build and refine CAD designs. They show that current vision-language models follow the initial build instructions far better than they follow later refinement instructions.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Models are evaluated one-shot while humans refine interactively; the negative refinement PI may be an artifact of the protocol, not a capability gap.","rationale":"The central empirical assertion is that VLMs are worse at refinement than generation, supported by negative proportional improvement on refinement rounds. For this conclusion to hold, the comparison must isolate instruction-following ability from the decision procedure used to act. The current design does not: human makers in the dataset act interactively with live visual feedback, while models are evaluated by generating a single batch of tool calls with no intermediate feedback. This is a textbook confound. It specifically depresses refinement performance because refinement requires precise, incremental edits; one-shot planning errors are irreversible, whereas humans can correct them within the same round. The reader's concern about the chamfer metric is valid but less decisive: even if the metric is noisy, it would affect generation and refinement similarly, and a no-op baseline (PI=0) still shows models actively harming designs. The protocol confound, in contrast, directly explains why the human/model gap is large for refinement but not for generation. The dataset contribution is unaffected, and the generation-better-than-refinement finding may survive, but the stronger claim of a 'severe gap in refinement ability' compared to humans is not established by the current evaluation. The verdict should remain conditional, with the new condition that the protocol confound be addressed (e.g., via an interactive evaluation or a human one-shot control).","tokens_in":13718,"tokens_out":13641,"duration_ms":146286,"concrete_test":"Use the released mrCAD gym to re-run the evaluation of GPT-4o and Qwen-7B-FT on the refinement rounds with an interactive loop: after each tool call, render the updated design and feed it back to the model, allowing additional corrective tool calls until the model signals completion (or up to the median number of human actions per round). Compute PI on the final state. If the mean refinement PI for these models rises from negative to near zero or positive, the original negative PI is an artifact of the one-shot protocol.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2.2 defines the Maker as generating a sequence of actions from P_maker(A|m,D_i,R_{1:i-1}), and in the game the Maker works in a live CAD interface for 120 seconds, able to see the design after each click and correct errors. In the model evaluation (Section 6.2, Figure 9), the VLM is prompted to output a complete list of tool calls in one response; the environment then applies all actions and returns the final design. There is no opportunity for the model to observe the intermediate state and issue corrective actions. This asymmetry is harmless in generation rounds, where an approximate shape from an empty canvas yields positive PI, but it is severe in refinement rounds, where a single mis-specified coordinate can move the wrong curve or overshoot a small correction. Humans can undo and retry within the round; models cannot. The human refinement PI of +0.119 thus comes from a different, more forgiving decision procedure than the models' negative PI. The paper's headline claim that VLMs 'often make changes that actually increase the distance to the target' is measured with a one-shot policy, not an interactive agent. This confound is not mentioned in Section 6 or the limitations.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces mrCAD, a large-scale dataset of human-human communication games in a 2D CAD environment. In each game, a Designer sees a target CAD and issues instructions to a Maker using text, drawings, or both; the Maker manipulates the CAD over multiple rounds. The corpus contains 6,082 games and 15,163 instruction-execution rounds. The authors analyze human behavior, finding that refinement improves reconstruction accuracy, that generation instructions are drawing-heavy while refinement instructions are more text-heavy, and that refinement language is more imperative. They define a chamfer-based distance metric and a proportional-improvement (PI) metric, and use mrCAD as a benchmark to evaluate GPT-4o, GPT-4o-mini, Claude-3.7, Qwen2.5-VL-7B, and a LoRA-finetuned Qwen model. Table 1 reports that all evaluated models achieve positive PI on generation rounds but negative PI on refinement rounds, while humans achieve positive PI on both, leading to the central claim that state-of-the-art VLMs are substantially worse at following refinement instructions than generation instructions.","tokens_in":13939,"tokens_out":5731,"duration_ms":58422,"significance":"If the headline empirical claim holds, mrCAD is a valuable and significant resource for the study of multimodal instruction following, human-AI collaboration, and interactive refinement. Its strengths include the large scale of human-human interaction data, a grounded programmatic CAD environment with an objective distance metric, a released benchmark and gym environment, and human baseline comparisons. The human-behavior analyses, especially the shift from drawing-heavy generation to language-heavy refinement, are interesting in their own right. However, the central model-vs-human comparison currently rests on a protocol asymmetry and on point estimates without uncertainty quantification, and the distance metric that anchors the PI measure is not validated. These issues need to be addressed before the claim that models are worse at refinement can be taken as established.","major_comments":[{"comment":"The model evaluation protocol is one-shot, while the human Maker is interactive. Section 2.2 gives the human Maker 120 seconds in a live CAD interface, with the ability to see the design after each click and to undo or retry. In contrast, the VLM evaluation shown in Figure 9 prompts the model to output a complete list of tool calls in one response, and the environment then applies all actions and returns the final design, with no intermediate observation or opportunity for corrective actions. This asymmetry is particularly severe for refinement rounds, where a single mis-specified coordinate can move the wrong curve or overshoot a small correction. The paper's central claim that models 'often make changes that actually increase the distance to the target' (Sec. 6.3) is therefore measured under a different, less forgiving decision procedure than the human refinement PI of +0.119. I ask the authors to either evaluate models in an interactive loop with environment feedback after each action, or explicitly restrict the claim to one-shot instruction following and provide a human baseline collected under the same one-shot protocol.","section":"Sec. 6.2 / Fig. 9 / Sec. 2.2 / Table 1"},{"comment":"No uncertainty quantification is reported for the model PI values. Table 1 reports point estimates for each model (e.g., GPT-4o refinement PI = -0.119, Qwen-7B FT refinement PI = -0.017), but the benchmark contains 5,751 rounds, and it is not shown whether these negative values are statistically distinguishable from zero or from the positive generation PI. Report bootstrap confidence intervals or per-round standard errors for each model, and ideally significance tests for the generation-vs-refinement and model-vs-human contrasts.","section":"Sec. 6.3 / Table 1"},{"comment":"The chamfer distance metric relies on an ad hoc default penalty of 1/4 for unmatched points when computing point-to-design distances, and the paper provides no validation that this metric agrees with human judgments of geometric similarity. Because the PI metric is built entirely on this distance, the 'destructive change' conclusion could be an artifact of the metric if, for example, small translations of many curves are penalized disproportionately. Please validate the distance metric against human similarity ratings or alternative geometric distances, and report the sensitivity of Table 1 to the threshold theta and to the default penalty.","section":"Sec. 2.1 / Appendix B.2 / Fig. 2"},{"comment":"The appendix discloses a 'small error' in the JavaScript implementation of the accuracy function that caused lower-performing trials to receive larger distances than they should have, and states that some participants may have performed extra rounds to meet the target threshold. Because the dynamic submission threshold and the 'life' system depend on this function, the bug may have influenced which rounds were collected and how many refinement rounds dyads performed, potentially inflating the human refinement PI. Please quantify the impact on the dataset, including the number of affected trials and the direction of bias, and explain why the reported human PI and the modality analyses are not systematically distorted.","section":"Appendix B.2"}],"minor_comments":[{"comment":"Section 3.1 states that each round gives the Designer 30 seconds and the Maker 120 seconds, while Table 2 reports Designer turn time as 60s (180s) and Maker turn time as 120s (360s). Please reconcile these numbers.","section":"Sec. 3.1 vs Appendix Table 2"},{"comment":"The text reports 6,082 rollouts, 15,163 rounds, 6,078 generation rounds, and 9,085 refinement rounds. Since 6,078 + 9,085 = 15,163, the generation-round count differs from the rollout count by 4; please clarify whether some rollouts lack a first round or whether the counts refer to different subsets.","section":"Sec. 5.1"},{"comment":"The text refers to 'Table 2' when reporting the PI results, but the table is labeled 'Table 1'. Please correct the cross-reference.","section":"Sec. 6.3"},{"comment":"The Figure 3 caption is garbled in places (e.g., 'A B C distance text drawing' and 'with re/f_inement'), and Section 5.2 contains an extra brace in 'Fig. 6} B'. Please clean up these formatting artifacts.","section":"Fig. 3 / Sec. 5.2"},{"comment":"The distance-metric paragraph contains a malformed expression 'd(D,E) /uni221D Σ'; please use proper mathematical notation or a code listing.","section":"Sec. 2.1"}],"recommendation":"major_revision","confidential_remarks":"The JavaScript accuracy bug disclosed in Appendix B.2 is a notable data-quality issue that should be investigated carefully before publication. I would also encourage the editor to verify that the paper's protocol comparison (interactive humans vs one-shot models) is addressed head-on in the revision, since it is the main threat to the headline claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Marcus,\n\nThe dataset is the real contribution, and it's a good one. Just over 6,000 communication games, 15,000 instruction-execution rounds, with text and drawing from human dyads reconstructing CADs. The collection design is thoughtful: dynamic submission threshold, ability-based pay, careful exclusion criteria. The analysis of human communication is the strongest part. Generation instructions are drawing-heavy; refinement shifts toward text and imperatives, and refinement drawings are sparser. That is a genuine empirical finding.\n\nThe benchmark part is the soft spot. The headline claim — VLMs are worse at refinement, often increasing distance to target — is measured one-shot: the model outputs a full action list in a single response and the environment applies everything. Humans, by contrast, had 120 seconds in a live editor with undo and visual feedback after each click. So the human +0.119 refinement PI comes from a different, far more forgiving decision procedure than the models' negative PI. If you let a model see the intermediate state after each action and correct its own mistakes, that negative result might shrink or disappear. The paper never mentions this asymmetry in the limitations. That doesn't undermine the dataset or the human analyses, but it does mean the \"severe gap\" claim is not proven as stated.\n\nSmaller issues: no error bars or multiple seeds on model PIs, and the generation/refinement split is round 1 vs rounds 2+, which the authors admit is imprecise. The chamfer metric with a default penalty of 1/4 for unrelated points is coarse but defensible as a proxy, though some validation against human judgment would help. The appendix also reports a JavaScript bug in the accuracy function that affected participants' displayed scores; post-hoc recalculation means the released data is fine, but it is worth flagging for readers.\n\nWhere does this land? The dataset deserves a serious referee and will be a useful resource for multimodal instruction following and HCI work. The model-evaluation section needs either an interactive evaluation or a carefully hedged conclusion. I'd bring this to reading group and would cite it for the dataset and the human-modality findings, but not for the claim that SOTA VLMs cannot refine.\n\nRecommendation: send to peer review, with a request to address the one-shot vs interactive asymmetry, add error bars or seeds, and soften the refinement-gap claim as appropriate.","headline":"A solid, large-scale dataset of multimodal CAD instructions; the human-modality analysis is the real contribution, while the VLM refinement-gap claim is confounded by a one-shot vs interactive evaluation.","tokens_in":14462,"tokens_out":2753,"would_cite":true,"duration_ms":32004,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that state-of-the-art vision-language models follow generation instructions well but routinely make a design worse when asked to refine it, and introduces a large multimodal CAD dataset that exposes the gap.","keywords":["multimodal instruction following","computer-aided design","refinement","vision-language models","human-AI collaboration","benchmark dataset","chamfer distance","communication game"],"falsifier":"Re-score a sample of mrCAD refinement rounds using an independent similarity measure—such as human pairwise judgments of whether the edit moved the design toward the target, or a differently constructed geometric metric without the fixed 1/4 penalty—and recompute the average proportional improvement for each evaluated model; if the sign stops being negative or humans no longer beat the models, the paper's central claim is an artifact of its distance metric.","tokens_in":13546,"feed_emoji":"🎨","tokens_out":12619,"duration_ms":117666,"temperature":0.7,"pith_summary":"The paper tries to establish that current vision-language models have a specific blind spot: they can create a design from an instruction but often cannot act on a follow-up instruction to change it. To study this, the authors built mrCAD, a dataset of 6,082 two-player communication games in which a Designer who can see a target computer-aided design (CAD) tells a Maker how to build or modify it using text, drawings, or both, over 15,163 instruction-execution rounds. On this benchmark, every model they evaluated improved the CAD on first-round 'generation' instructions but made it worse on later 'refinement' instructions: measured by proportional improvement in geometric distance to the target, refinements produced negative scores for all models while human Makers averaged +0.119. The authors argue this matters because iterative repair is a core mechanism of human collaboration, and it is precisely the kind of interactive editing data that is largely missing from the corpora used to train today's models.","feed_headline":"AI models make designs worse when told to refine them","feed_subtitle":"A 15,163-round human dataset shows models push CADs away from the target exactly where humans improve","key_machinery":"The load-bearing object is the mrCAD benchmark: a two-player communication game in a 2D CAD environment. The Designer sees only the target and sends a message that can combine text with freehand strokes; the Maker applies discrete actions (make, remove, or move curves and control points). To score each round, the paper uses a symmetric chamfer distance between CADs—sample points on each curve, take nearest-point distances, average both directions—then normalizes a round's progress as proportional improvement, $PI = (\\Delta(D_i,D^*) - \\Delta(A(D_i),D^*))/\\Delta(D_i,D^*)$, where $D^*$ is the target. A negative $PI$ is exactly the paper's operational definition of 'made the design worse,' and it is what separates the flagging refinement results from the positive generation results. Vision-language models are prompted with rendered images of the history and instruction and must emit tool calls.","core_discovery":"On its own terms, the central discovery is that refinement instruction-following does not emerge from generation ability. In the mrCAD benchmark, a vision-language model placed in the Maker role sees the rendered current design, the history, and the Designer's multimodal message, then emits editing actions. Across refinement rounds, every evaluated model—including a supervised fine-tuned variant trained on mrCAD itself—had negative mean proportional improvement, meaning its edits on average increased the distance between the current CAD and the hidden target. The same models made positive progress on generation rounds, and humans showed +0.119 proportional improvement on refinement. The paper reads this as a sign that models have learned to generate but not to repair, and ties the gap to a training-data asymmetry: finished artifacts and their descriptions are abundant online, while the process of interactively modifying an artifact is rarely recorded.","pith_inferences":["The paper's round-based split (round 1 = generation, later rounds = refinement) is acknowledged to be imprecise; because some later instructions still request new content, average refinement scores likely understate how poorly models handle genuine repair, suggesting the real gap is larger than the reported negative numbers.","A clean test of the paper's data-scarcity explanation would be to collect the same two-player refinement game in other iterative domains (code, documents, 3D scenes); if the same generation/refinement asymmetry appears, the limitation is general rather than CAD-specific.","The metric's treatment of 'unrelated' points (a fixed 1/4 penalty) means designs that are geometrically far apart can be scored as equally bad; checking model edits against human similarity judgments would reveal whether the measured 'worsening' matches what a user experiences.","The paper's results imply that reinforcement learning or other interactive training signals, rather than supervised fine-tuning on static demonstrations, are the most promising route to giving models refinement ability."],"forward_implications":["Supervised fine-tuning on mrCAD data improves a model's performance on generation rounds but leaves refinement performance near zero or negative, so closing the refinement gap will need more than imitation of human demonstrations.","Ablating either text or drawing from multimodal instructions hurts model performance, with the biggest drop on generation rounds, confirming that both modalities carry instruction content the models rely on.","Because online data skews toward finished artifacts, the generation–refinement gap should be expected to persist in models trained on internet-scale data, independent of CAD.","The programmatic distance metric lets mrCAD serve as a grounded benchmark where instruction-following is measured directly in geometry, without a learned similarity model."],"supporting_citations":[{"why":"Supplies the two-player designer–maker communication game procedure that mrCAD scales up.","marker":"(McCarthy et al., 2024)"},{"why":"Source of the naturalistic 2D CAD designs used as target tasks.","marker":"(Seff et al., 2020)"},{"why":"Basis of the symmetric chamfer distance metric used to score reconstruction accuracy.","marker":"(Butt and Maragos, 1998)"},{"why":"Provides the GPT-4o model evaluated as Maker in the benchmark.","marker":"(OpenAI, 2024)"},{"why":"Provides the Qwen2.5-VL base model used for open-weights and fine-tuned evaluations.","marker":"(Bai et al., 2025)"},{"why":"Provides the Claude 3.7 Sonnet model evaluated as Maker.","marker":"(Anthropic, 2025)"},{"why":"Provides the LoRA fine-tuning method used for the supervised models.","marker":"(Hu et al., 2022)"}],"fun_headline_variants":["AI models worsen CADs when asked to refine","New benchmark: VLMs fail at design refinement","Refinement gap: AI edits push CADs off-target","Humans refine designs, AI models make them worse","mrCAD dataset shows AI can't follow refinement commands"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's conclusion depends on its chamfer-distance metric faithfully ranking how close two CAD designs are; if that metric misranks visually similar designs, the measured 'worsening' on refinement rounds could be an artifact of the ruler rather than a real failure of the models.","fun_headline_variants_meta":{"raw":{"variants":["AI models worsen CADs when asked to refine","New benchmark: VLMs fail at design refinement","Refinement gap: AI edits push CADs off-target","Humans refine designs, AI models make them worse","mrCAD dataset shows AI can't follow refinement commands"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000225,"raw_usage":{"total_tokens":1456,"prompt_tokens":926,"completion_tokens":530,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":542,"completion_tokens_details":{"reasoning_tokens":455}},"tokens_in":542,"tokens_out":530,"duration_ms":5857,"temperature":1.0,"reasoning_tokens":455,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:32:13.504913+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-score a sample of mrCAD refinement rounds using an independent similarity measure—such as human pairwise judgments of whether the edit moved the design toward the target, or a differently constructed geometric metric without the fixed 1/4 penalty—and recompute the average proportional improvement for each evaluated model; if the sign stops being negative or humans no longer beat the models, the paper's central claim is an artifact of its distance metric.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the two-player designer–maker communication game procedure that mrCAD scales up."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Basis of the symmetric chamfer distance metric used to score reconstruction accuracy."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Claude 3.7 Sonnet model evaluated as Maker."}],"review_version":1}