REVIEW 4 major objections 5 minor 32 references
mrCAD: Multimodal Refinement of Computer-aided Designs
T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read This paper claims that state-of-the-art vision-language models follow generation instructions well but routinely make a design worse when asked to refine it, and introduces a large multimodal CAD dataset that exposes the gap.
desk verdict A solid, large-scale dataset of multimodal CAD instructions; the human-modality analysis is the real contribution, while the VLM refinement-gap claim is confounded by a one-shot vs interactive evaluation. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the mrCAD benchmark: a two-player communication game in a 2D CAD environment. The Designer sees only the target and sends a message that can combine text with freehand strokes; the Maker applies discrete actions (make, remove, or move curves and control points). To score each round, the paper uses a symmetric chamfer distance between CADs—sample points on each curve, take nearest-point distances, average both directions—then normalizes a round's progress as proportional improvement, $PI = (\Delta(D_i,D^*) - \Delta(A(D_i),D^*))/\Delta(D_i,D^*)$, where $D^*$ is the target. A negative $PI$ is exactly the paper's operational definition of 'made the design worse,' and it is what separates the flagging refinement results from the positive generation results. Vision-language models are prompted with rendered images of the history and instruction and must emit tool calls.
What would settle it
Re-score a sample of mrCAD refinement rounds using an independent similarity measure—such as human pairwise judgments of whether the edit moved the design toward the target, or a differently constructed geometric metric without the fixed 1/4 penalty—and recompute the average proportional improvement for each evaluated model; if the sign stops being negative or humans no longer beat the models, the paper's central claim is an artifact of its distance metric.
Extended reading notes
Core claim
On its own terms, the central discovery is that refinement instruction-following does not emerge from generation ability. In the mrCAD benchmark, a vision-language model placed in the Maker role sees the rendered current design, the history, and the Designer's multimodal message, then emits editing actions. Across refinement rounds, every evaluated model—including a supervised fine-tuned variant trained on mrCAD itself—had negative mean proportional improvement, meaning its edits on average increased the distance between the current CAD and the hidden target. The same models made positive progress on generation rounds, and humans showed +0.119 proportional improvement on refinement. The paper reads this as a sign that models have learned to generate but not to repair, and ties the gap to a training-data asymmetry: finished artifacts and their descriptions are abundant online, while the process of interactively modifying an artifact is rarely recorded.
Load-bearing premise
The paper's conclusion depends on its chamfer-distance metric faithfully ranking how close two CAD designs are; if that metric misranks visually similar designs, the measured 'worsening' on refinement rounds could be an artifact of the ruler rather than a real failure of the models.
Editorial extensions
If this is right
- Supervised fine-tuning on mrCAD data improves a model's performance on generation rounds but leaves refinement performance near zero or negative, so closing the refinement gap will need more than imitation of human demonstrations.
- Ablating either text or drawing from multimodal instructions hurts model performance, with the biggest drop on generation rounds, confirming that both modalities carry instruction content the models rely on.
- Because online data skews toward finished artifacts, the generation–refinement gap should be expected to persist in models trained on internet-scale data, independent of CAD.
- The programmatic distance metric lets mrCAD serve as a grounded benchmark where instruction-following is measured directly in geometry, without a learned similarity model.
Reading between the lines
- The paper's round-based split (round 1 = generation, later rounds = refinement) is acknowledged to be imprecise; because some later instructions still request new content, average refinement scores likely understate how poorly models handle genuine repair, suggesting the real gap is larger than the reported negative numbers.
- A clean test of the paper's data-scarcity explanation would be to collect the same two-player refinement game in other iterative domains (code, documents, 3D scenes); if the same generation/refinement asymmetry appears, the limitation is general rather than CAD-specific.
- The metric's treatment of 'unrelated' points (a fixed 1/4 penalty) means designs that are geometrically far apart can be scored as equally bad; checking model edits against human similarity judgments would reveal whether the measured 'worsening' matches what a user experiences.
- The paper's results imply that reinforcement learning or other interactive training signals, rather than supervised fine-tuning on static demonstrations, are the most promising route to giving models refinement ability.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper introduces mrCAD, a large-scale dataset of human-human communication games in a 2D CAD environment. In each game, a Designer sees a target CAD and issues instructions to a Maker using text, drawings, or both; the Maker manipulates the CAD over multiple rounds. The corpus contains 6,082 games and 15,163 instruction-execution rounds. The authors analyze human behavior, finding that refinement improves reconstruction accuracy, that generation instructions are drawing-heavy while refinement instructions are more text-heavy, and that refinement language is more imperative. They define a chamfer-based distance metric and a proportional-improvement (PI) metric, and use mrCAD as a benchmark to evaluate GPT-4o, GPT-4o-mini, Claude-3.7, Qwen2.5-VL-7B, and a LoRA-finetuned Qwen model. Table 1 reports that all evaluated models achieve positive PI on generation rounds but negative PI on refinement rounds, while humans achieve positive PI on both, leading to the central claim that state-of-the-art VLMs are substantially worse at following refinement instructions than generation instructions.
Significance. If the headline empirical claim holds, mrCAD is a valuable and significant resource for the study of multimodal instruction following, human-AI collaboration, and interactive refinement. Its strengths include the large scale of human-human interaction data, a grounded programmatic CAD environment with an objective distance metric, a released benchmark and gym environment, and human baseline comparisons. The human-behavior analyses, especially the shift from drawing-heavy generation to language-heavy refinement, are interesting in their own right. However, the central model-vs-human comparison currently rests on a protocol asymmetry and on point estimates without uncertainty quantification, and the distance metric that anchors the PI measure is not validated. These issues need to be addressed before the claim that models are worse at refinement can be taken as established.
major comments (4)
- [Sec. 6.2 / Fig. 9 / Sec. 2.2 / Table 1] The model evaluation protocol is one-shot, while the human Maker is interactive. Section 2.2 gives the human Maker 120 seconds in a live CAD interface, with the ability to see the design after each click and to undo or retry. In contrast, the VLM evaluation shown in Figure 9 prompts the model to output a complete list of tool calls in one response, and the environment then applies all actions and returns the final design, with no intermediate observation or opportunity for corrective actions. This asymmetry is particularly severe for refinement rounds, where a single mis-specified coordinate can move the wrong curve or overshoot a small correction. The paper's central claim that models 'often make changes that actually increase the distance to the target' (Sec. 6.3) is therefore measured under a different, less forgiving decision procedure than the human refinement PI of +0.119. I ask the authors to either evaluate models in an interactive loop with environment feedback after each action, or explicitly restrict the claim to one-shot instruction following and provide a human baseline collected under the same one-shot protocol.
- [Sec. 6.3 / Table 1] No uncertainty quantification is reported for the model PI values. Table 1 reports point estimates for each model (e.g., GPT-4o refinement PI = -0.119, Qwen-7B FT refinement PI = -0.017), but the benchmark contains 5,751 rounds, and it is not shown whether these negative values are statistically distinguishable from zero or from the positive generation PI. Report bootstrap confidence intervals or per-round standard errors for each model, and ideally significance tests for the generation-vs-refinement and model-vs-human contrasts.
- [Sec. 2.1 / Appendix B.2 / Fig. 2] The chamfer distance metric relies on an ad hoc default penalty of 1/4 for unmatched points when computing point-to-design distances, and the paper provides no validation that this metric agrees with human judgments of geometric similarity. Because the PI metric is built entirely on this distance, the 'destructive change' conclusion could be an artifact of the metric if, for example, small translations of many curves are penalized disproportionately. Please validate the distance metric against human similarity ratings or alternative geometric distances, and report the sensitivity of Table 1 to the threshold theta and to the default penalty.
- [Appendix B.2] The appendix discloses a 'small error' in the JavaScript implementation of the accuracy function that caused lower-performing trials to receive larger distances than they should have, and states that some participants may have performed extra rounds to meet the target threshold. Because the dynamic submission threshold and the 'life' system depend on this function, the bug may have influenced which rounds were collected and how many refinement rounds dyads performed, potentially inflating the human refinement PI. Please quantify the impact on the dataset, including the number of affected trials and the direction of bias, and explain why the reported human PI and the modality analyses are not systematically distorted.
minor comments (5)
- [Sec. 3.1 vs Appendix Table 2] Section 3.1 states that each round gives the Designer 30 seconds and the Maker 120 seconds, while Table 2 reports Designer turn time as 60s (180s) and Maker turn time as 120s (360s). Please reconcile these numbers.
- [Sec. 5.1] The text reports 6,082 rollouts, 15,163 rounds, 6,078 generation rounds, and 9,085 refinement rounds. Since 6,078 + 9,085 = 15,163, the generation-round count differs from the rollout count by 4; please clarify whether some rollouts lack a first round or whether the counts refer to different subsets.
- [Sec. 6.3] The text refers to 'Table 2' when reporting the PI results, but the table is labeled 'Table 1'. Please correct the cross-reference.
- [Fig. 3 / Sec. 5.2] The Figure 3 caption is garbled in places (e.g., 'A B C distance text drawing' and 'with re/f_inement'), and Section 5.2 contains an extra brace in 'Fig. 6} B'. Please clean up these formatting artifacts.
- [Sec. 2.1] The distance-metric paragraph contains a malformed expression 'd(D,E) /uni221D Σ'; please use proper mathematical notation or a code listing.
Circularity Check
No significant circularity; the central benchmark claims are empirical, evaluated on held-out rounds with external or fine-tuned VLMs, and the only self-citation is a framing reference that does not carry the argument.
full rationale
mrCAD is a dataset-and-benchmark paper whose central claims are empirical: (1) human generation and refinement instructions differ in modality composition, and (2) VLMs follow generation instructions better than refinement instructions. Neither claim is derived from a fitted parameter that is later relabeled as a prediction, nor from a self-citation chain. The chamfer distance metric (Section 2.1, Appendix B.2) and the proportional-improvement metric (Section 6.1) are defined independently of the conclusions and are applied symmetrically to human and model actions. Models are evaluated on held-out dense-set rounds, with off-the-shelf API models and a Qwen model fine-tuned on a separate coverage split; the measured PI values are not inputs to the metric or to the model definitions. The only self-citation, 'Following McCarthy et al. (2024)' in Section 2, frames the two-player communication game but is not load-bearing for the dataset analyses or the VLM evaluation, which are newly collected and run here. The one-shot model protocol versus the interactive human Maker protocol is a genuine validity/fairness concern that could affect the generation-vs-refinement comparison, but it is not circularity: the model outputs are not defined in terms of the measured PI, nor is the measured PI an input that is later reported as a finding. Thus no step in the paper's derivation chain reduces to its own inputs.
Assumptions & free parameters
free parameters (3)
- Chamfer distance default penalty =
0.25 (1/4 of quadrant)
- Benchmark inclusion threshold theta =
0.2
- Analysis inclusion threshold =
shown in Fig. 4A (not stated numerically)
assumptions (4)
- domain assumption SketchGraphs CADs are naturalistic design tasks
- domain assumption Chamfer distance with 1/4 default is a valid measure of design similarity
- domain assumption Proportional improvement is a meaningful normalization for instruction-following
- domain assumption Round 1 can be treated as generation and rounds 2+ as refinement
Cite this review
Pith. "Pith review of mrCAD: Multimodal Refinement of Computer-aided Designs." pith.science (2026). https://pith.science/paper/AR7URJO7
@misc{pith2026250420294,
author = {Pith},
title = {Pith review of: mrCAD: Multimodal Refinement of Computer-aided Designs},
year = {2026},
howpublished = {\url{https://pith.science/paper/AR7URJO7}},
note = {Machine review of arXiv:2504.20294}
}
read the original abstract
A key feature of human collaboration is the ability to iteratively refine the concepts we have communicated. In contrast, while generative AI excels at the \textit{generation} of content, it often struggles to make specific language-guided \textit{modifications} of its prior outputs. To bridge the gap between how humans and machines perform edits, we present mrCAD, a dataset of multimodal instructions in a communication game. In each game, players created computer aided designs (CADs) and refined them over several rounds to match specific target designs. Only one player, the Designer, could see the target, and they must instruct the other player, the Maker, using text, drawing, or a combination of modalities. mrCAD consists of 6,082 communication games, 15,163 instruction-execution rounds, played between 1,092 pairs of human players. We analyze the dataset and find that generation and refinement instructions differ in their composition of drawing and text. Using the mrCAD task as a benchmark, we find that state-of-the-art VLMs are better at following generation instructions than refinement instructions. These results lay a foundation for analyzing and modeling a multimodal language of refinement that is not represented in previous datasets.
Figures
Figures from the paper (6 more)
Reference graph
Works this paper leans on
-
[1]
URL: " 'urlintro :=
ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Anthropic . 2025. https://assets.anthropic.com/m/785e231869ea8b3b/original/claude-3-7-sonnet-system-card.pdf Claude 3.7 sonnet system card . Technical report, Anthropic. Accessed: 2025-03-01
work page 2025
-
[4]
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. 2025. http://arxiv.org/abs/250...
arXiv 2025
-
[5]
M Akmal Butt and Petros Maragos. 1998. Optimum design of chamfer distance transforms. IEEE Transactions on Image Processing, 7(10):1477--1484
work page 1998
-
[6]
Federico Cassano, Luisa Li, Akul Sethi, Noah Shinn, Abby Brennan-Jones, Jacob Ginesin, Edward Berman, George Chakhnashvili, Anton Lozhkov, Carolyn Jane Anderson, et al. 2023. Can it edit? evaluating the ability of large language models to follow code editing instructions. arXiv preprint arXiv:2312.12450
arXiv 2023
-
[7]
Hansheng Chen, Ruoxi Shi, Yulin Liu, Bokui Shen, Jiayuan Gu, Gordon Wetzstein, Hao Su, and Leonidas Guibas. 2024. Generic 3d diffusion adapter using controlled multi-view editing. arXiv preprint arXiv:2403.12032
arXiv 2024
-
[8]
Mark Dingemanse and Nick J Enfield. 2024. Interactive repair and the foundations of language. Trends in Cognitive Sciences, 28(1):30--42
work page 2024
Show all 32 references
-
[9]
Judith E Fan, Wilma A Bainbridge, Rebecca Chamberlain, and Jeffrey D Wammes. 2023. Drawing as a versatile cognitive tool. Nature Reviews Psychology, 2(9):556--568
2023
-
[10]
Tsu-Jui Fu, Wenze Hu, Xianzhi Du, William Yang Wang, Yinfei Yang, and Zhe Gan. 2024. Guiding instruction-based image editing via multimodal large language models. In International Conference on Learning Representations
2024
-
[11]
Jiaxin Ge, Zora Zhiruo Wang, Xuhui Zhou, Yi-Hao Peng, Sanjay Subramanian, Qinyue Tan, Maarten Sap, Alane Suhr, Daniel Fried, Graham Neubig, et al. 2025. Autopresent: Designing structured visuals from scratch. arXiv preprint arXiv:2501.00912
2025 arXiv
-
[12]
Nelson Goodman. 1976. Languages of art: An approach to a theory of symbols. Hackett publishing
1976
-
[13]
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. https://openreview.net/forum?id=nZeVKeeFYf9 Lo RA : Low-rank adaptation of large language models . In International Conference on Learning Representations
2022
-
[14]
Holly Huey, Caren M Walker, and Judith E Fan. 2021. How do the semantic properties of visual explanations guide causal inference? In Proceedings of the annual meeting of the cognitive science society, volume 43
2021
-
[15]
Anya Ji, Noriyuki Kojima, Noah Rush, Alane Suhr, Wai Keen Vong, Robert D Hawkins, and Yoav Artzi. 2022. Abstract visual reasoning with tangram shapes. arXiv preprint arXiv:2211.16492
2022 arXiv
-
[16]
Jin-Hwa Kim, Nikita Kitaev, Xinlei Chen, Marcus Rohrbach, Byoung-Tak Zhang, Yuandong Tian, Dhruv Batra, and Devi Parikh. 2019. https://doi.org/10.18653/v1/P19-1651 C o D raw: Collaborative drawing as a testbed for grounded goal-driven communication . In Proceedings of the 57th...
2019 doi
-
[17]
Alexander Ku, Peter Anderson, Roma Patel, Eugene Ie, and Jason Baldridge. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.356 Room-across-room: Multilingual vision-and-language navigation with dense spatiotemporal grounding . In Proceedings of the 2020 Conference on Empirica...
2020 doi
-
[18]
Royi Lachmy, Valentina Pyatkin, Avshalom Manevich, and Reut Tsarfaty. 2022. Draw me a flower: Processing and grounding abstraction in natural language. Transactions of the Association for Computational Linguistics, 10:1341--1356
2022
-
[19]
Bryan Lawson. 2006. How designers think. Routledge
2006
-
[20]
William P McCarthy, Justin Matejka, Karl DD Willis, Judith E Fan, and Yewen Pu. 2024. Communicating design intent using drawing and text. In Proceedings of the 16th Conference on Creativity & Cognition, pages 512--519
2024
-
[21]
OpenAI. 2024. http://arxiv.org/abs/2410.21276 Gpt-4o system card
2024 arXiv
-
[22]
OpenAI . 2025. https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/ Gpt-4o mini: Advancing cost-efficient intelligence . Accessed: 2025-03-01
2025
-
[23]
Tomislav Pejsa, Julian Kantor, Hrvoje Benko, Eyal Ofek, and Andrew Wilson. 2016. Room2room: Enabling life-size telepresence in a projected augmented reality environment. In Proceedings of the 19th ACM conference on computer-supported cooperative work & social computing, pages ...
2016
-
[24]
Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. https://nlp.stanford.edu/pubs/qi2020stanza.pdf Stanza: A Python natural language processing toolkit for many human languages . In Proceedings of the 58th Annual Meeting of the Association for Co...
2020
-
[25]
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learni...
2021
-
[26]
Aditya Sanghi, Hang Chu, Joseph G Lambourne, Ye Wang, Chin-Yi Cheng, Marco Fumero, and Kamal Rahimi Malekshan. 2022. Clip-forge: Towards zero-shot text-to-shape generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18603--18613
2022
-
[27]
Ari Seff, Yaniv Ovadia, Wenda Zhou, and Ryan P Adams. 2020. Sketchgraphs: A large-scale dataset for modeling relational geometry in computer-aided design. arXiv preprint arXiv:2007.08506
2020 arXiv
-
[28]
Ari Seff, Wenda Zhou, Nick Richardson, and Ryan P Adams. 2021. Vitruvion: A generative model of parametric cad sketches. arXiv preprint arXiv:2109.14124
2021 arXiv
-
[29]
Chenglei Si, Yanzhe Zhang, Zhengyuan Yang, Ruibo Liu, and Diyi Yang. 2024. http://arxiv.org/abs/2403.03163 Design2code: How far are we from automating front-end engineering?
2024 arXiv
-
[30]
Anthony Williams and Robert Cowdroy. 2002. How designers communicate ideas to each other in design meetings. In DS 30: Proceedings of DESIGN 2002, the 7th International Design Conference, Dubrovnik
2002
-
[31]
Yifei Zhou, Song Jiang, Yuandong Tian, Jason Weston, Sergey Levine, Sainbayar Sukhbaatar, and Xian Li. 2025. http://arxiv.org/abs/2503.15478 Sweet-rl: Training multi-turn llm agents on collaborative reasoning tasks
2025 arXiv
-
[32]
C Lawrence Zitnick and Devi Parikh. 2013. Bringing semantics into focus using visual abstraction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3009--3016
2013
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.