{"id":"063ba3d5-cc62-410e-8dd8-fb4d99b52d7e","arxiv_id":"2412.02368","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A human-scored benchmark shows that even GPT-4o averages below 4/5 correctness and all tested models struggle with scientific diagram prompts that combine spatial, numeric, and attribute requirements.","lead":"ScImage is a new benchmark of 404 text prompts for generating scientific diagrams such as binary trees, matrices, and bar charts, scored by 11 human scientists. It finds that even the best model, GPT-4o, averages below 4 out of 5 on correctness, and all models fail most prompts that combine spatial, numeric, and attribute requirements.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Small per-template samples (n=4) and template imbalance make the numeric/spatial/attribute decomposition's conclusions non-robust; the central claim survives, but fine-grained claims do not.","rationale":"The reader's CONDITIONAL verdict is appropriate. The headline finding, that current MLLMs including GPT-4o score below 4/5 on correctness on ScImage, is credible and supported by Table 3's averages and the reported human evaluation procedure (calibration, pairwise annotation, agreement scores). I do not see an internal inconsistency that would overturn this central claim. The most load-bearing soft spot is the representativeness of the benchmark's internal 'dimension' construct: with 101 templates and four samples each, the per-dimension and per-object-category results in Tables 4 and 5 have tiny effective sample sizes at the template level, and the paper makes general claims (e.g., 'spatial understanding appears to be the most challenging', 'image generation models struggle most with numeric comprehension') from those aggregates. The paper itself flags small sample sizes in Appendix H, which keeps this from being an unacknowledged flaw, but the per-dimension analysis is used in the abstract and conclusion as a core contribution. The concrete test - template-level bootstrapping or a stratified re-evaluation - would determine whether those per-dimension conclusions are properties of the dimension or artifacts of particular templates. If the test fails, the verdict would need adjustment from CONDITIONAL to REJECT or major revision; if it passes, the benchmark's contribution is materially strengthened. I therefore agree partially with the reader's weakest-assumption identification: both point at representativeness, but I localize the risk more specifically in the interaction between template-level sampling and the per-dimension claims, rather than in the overall 404-prompt coverage. The reader's additional concerns about missing error bars, internal count inconsistencies, and undisclosed annotator independence are valid but secondary; the headline finding does not hinge on them. My verdict remains CONDITIONAL because the central claim is credible but the benchmark's fine-grained findings need the template-level robustness check, plus the release of exact prompts and scores, before the benchmark should be adopted as a standard.","tokens_in":27082,"tokens_out":1935,"duration_ms":18423,"concrete_test":"Re-run the human evaluation on a stratified sample that holds template coverage constant: for each of the seven dimension combinations in Table 4, sample at least two independent templates per object category (or bootstrap over templates, treating template as the unit of resampling) and recompute the category means and their bootstrap confidence intervals. If the observed ordering (e.g., spatial lowest for code models, numeric lowest for image models) persists with non-overlapping intervals at the template level, the decomposition claim is robust; if the intervals overlap or the ordering flips, Section 5's per-dimension findings should be labeled template-specific rather than general.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The benchmark's core construct is the three-dimensional (attribute/numeric/spatial) decomposition of scientific image understanding, and the paper's Section 5 claims (e.g., spatial is hardest for code models, numeric hardest for image models) rest on Table 4's category scores. The dataset construction in Section 3.2 generates only four queries per template, and templates are highly unevenly distributed across object categories (Table 5 sample sizes: tables n=4, annotations n=9, graph theory n=20, matrices n=8). A category score can therefore be driven by a single template. Concretely, the 'spatial hardest' conclusion for code models is based on spatial-category prompts drawn from few templates; if those templates happen to be unusually hard (e.g., 'The English text (the name of the object) is {preposition} the {object}'), spatial difficulty may reflect template difficulty rather than a general deficit in spatial understanding. The paper's own Limitations section (Appendix H) admits sample sizes for some object categories are 'comparatively small,' but the central Section 4 claim that GPT-4o scores below 4 on correctness does not depend on this decomposition and is supported by the means in Table 3. The load-bearing concern is therefore not the headline finding but the generalizability of the per-dimension and per-category analysis, which the abstract and Section 6 also state as findings. This concern is compounded by the absence of error bars or significance tests; scores like 3.50 vs. 3.51 (GPT-4o TikZ vs. Python) are reported without any uncertainty quantification.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ScImage, a benchmark for evaluating multimodal large language models on scientific text-to-image generation. The benchmark consists of 101 prompt templates and 404 English generation queries constructed from a dictionary of scientific objects, attributes, spatial relations, and numeric requests, organized around three understanding dimensions (attribute, numeric, spatial) and their combinations. The authors evaluate seven model-output configurations (GPT-4o, Llama 3.1 8B, AutomaTikZ, DALL·E, Stable Diffusion, with code-based and direct-image modes) using 11 human annotators who score correctness, relevance, and scientific style on 1--5 scales, plus a multilingual extension into German, Chinese, and Farsi. The central finding is that GPT-4o is the best model but still scores below 4 on all three criteria, indicating systematic mistakes, and that all models struggle with combined understanding dimensions. The paper also reports per-dimension and per-object-category analyses, and shows that existing automatic metrics correlate only weakly with human judgments.","tokens_in":27322,"tokens_out":3945,"duration_ms":47244,"significance":"If the benchmark is validated, this is a useful contribution: it is one of the first structured evaluations of scientific text-to-image generation, with a carefully designed human protocol that includes calibration, double annotation, detailed guidelines, and inter-annotator agreement reporting. The authors' decision to release the benchmark and the human scores is a concrete strength, as is the demonstration that standard automatic metrics achieve at most Kendall correlation 0.26 with human judgments. The headline result -- that current MLLMs cannot reliably produce accurate scientific diagrams, with GPT-4o remaining below 4 on correctness -- is plausible and is supported by the reported means and the large gaps between GPT-4o and other models. However, the paper's more fine-grained claims about which understanding dimension is hardest for which model type, and about performance by object category, rest on small and uneven samples with no uncertainty quantification; these claims need strengthening before the benchmark can serve as a reusable reference.","major_comments":[{"comment":"The per-dimension conclusions, such as \"spatial understanding appears to be the most challenging across all textual models\" and the corresponding claims in Section 6, are based on point averages over 40--80 prompts per category with no confidence intervals, error bars, or significance tests. Since Section 3.2 generates only four queries per template, a single hard template can dominate a category mean. For example, the \"Annotation\" template \"The English text (the name of the object) is {preposition} the {object}\" is labeled Spatial and may be unusually difficult for code-based models. Please report per-template clustered means, bootstrap confidence intervals, and pairwise significance tests before asserting dimension-level rankings as findings.","section":"Section 5, Table 4"},{"comment":"The object-category analysis has severely imbalanced and small sample sizes: table n=4, annotations n=9, matrices n=8, and graph theory n=20. Statements such as \"graph theory representation poses great challenges for models\" and the discussion of Llama's near-zero table-generation score are therefore fragile; a category mean can be driven by one or two templates. Appendix H acknowledges that sample sizes are \"comparatively small,\" but the main text does not apply this caution consistently. Please either report per-template scores and confidence intervals for these categories, collapse small categories, or explicitly mark these comparisons as exploratory in the abstract and conclusion.","section":"Section 3.2, Table 5"},{"comment":"The multilingual evaluation compares 20 translated prompts per language across seven model configurations, and claims such as \"English does not always lead to best results\" and \"Farsi is worst on average\" are reported without any inferential statistics. Given the small number of prompts and the known sensitivity of LLM outputs to prompt wording, these cross-language ranking differences could easily be noise. Please add significance tests or confidence intervals, and consider mixed-effects models that treat prompts and annotators as random factors.","section":"Section 3.2, Table 6"},{"comment":"The paper presents ScImage as a \"comprehensive benchmark\" and generalizes findings beyond the specific 404 prompts, but it provides no external validation that the 101 templates and the four sampled dictionary entries per template are representative of the distribution of real scientific figure-generation requests. The construction is internally motivated by a survey of existing benchmarks and DaTikZ, yet there is no coverage analysis against held-out scientific captions or other external sources. Please add a coverage or representativeness analysis, or temper the \"comprehensive\" claim in the abstract and Section 6.","section":"Section 3.2"}],"minor_comments":[{"comment":"The first sentence of Section 3.1 reads \"he ScImage evaluates\" and should be \"The ScImage evaluates.\"","section":"Section 3.1"},{"comment":"The heading \"Relevence\" should be \"Relevance,\" and the paper alternates between \"scientificness\" and \"scientific style\" for the same criterion; please harmonize the terminology.","section":"Appendix E.2"},{"comment":"The star ratings in the failure and gold-standard example tables are difficult to read, especially when averaged across annotators; consider showing the numeric scores alongside or instead of stars.","section":"Tables 10--12"},{"comment":"The assignment of 0 for compile errors is applied to all three criteria, but this is only stated explicitly in the correctness guideline (Table 13) and in the context of correctness in Section 4; please clarify in Section 3.3 that the 0 penalty applies uniformly to relevance and scientific style as well.","section":"Section 3.3"},{"comment":"The table caption and the surrounding text should note that some categories contain very few instances; at minimum, add the sample sizes to the caption and refer readers to the limitations discussion in Appendix H.","section":"Table 5"}],"recommendation":"major_revision","confidential_remarks":"The manuscript has substantial overlap between the authors of ScImage and the DaTikZ/AutomaTikZ resources: the object vocabulary is extracted from DaTikZ, and the authors' own AutomaTikZ model is evaluated as a baseline. This is not a correctness problem per se, but the editors may want the authors to be explicit about the provenance and about any dependencies between the benchmark construction and the evaluated baseline. The main revision needed is statistical: the benchmark's per-dimension and per-category findings currently lack uncertainty quantification, and the tiny sample sizes in several categories should be handled or down-weighted. The central \"GPT-4o below 4\" finding is much more robust than the fine-grained claims and should survive the revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things worth knowing. First, the paper is a serious, reusable benchmark for a real gap: generating accurate scientific figures from text. Second, the headline result holds: the best model, GPT-4o in TikZ or Python mode, averages below 4.0 on correctness, and all other models are far worse. The fine-grained claims about which dimension is hardest per model type are weaker and should not be taken at face value.\n\nWhat is actually new is the package. ScImage gives the community 404 templated prompts across nine object categories, spanning attribute, numeric, and spatial understanding and their combinations, then tests seven generation modes (TikZ, Python, direct raster) and four input languages. The human evaluation is a cut above usual practice: eleven PhD-level and above annotators, a calibration session, double annotation, and detailed rubrics with example images in the appendix. Inter-annotator agreement is acceptable (weighted kappa mostly above 0.5). They also show standard automatic metrics barely correlate with human judgment (max Kendall correlation 0.26), which justifies spending money on human ratings. The code, prompts, and scores are released. That is reproducible evidence.\n\nThe soft spots are real but manageable. The per-category and per-dimension analyses sit on very few templates: four queries per template, and object categories as small as n=4 (tables) or n=8 (matrices). A single awkward template can move a category mean, so the 'spatial is hardest for code models' conclusion may be template-specific. The paper acknowledges small sample sizes in Appendix H, but the abstract and conclusion still state those findings without hedging. There are also no error bars or significance tests anywhere, and the reported annotation totals are internally inconsistent (Section 3.3 gives numbers that do not sum cleanly). Annotator independence is not described, and exact model versions are not pinned down. These are fixable but as published make the fine-grained rankings descriptive rather than definitive.\n\nWho is this for? People building or evaluating scientific figure generation systems. It is measurement infrastructure, not a new capability or theory. It deserves a serious referee: the core result is solid, the artifacts are valuable, and the weak spots are exactly what peer review can push the authors to fix. I would engage with it and ask for error bars, a cleaner data release, and more careful claims about the decomposition.","headline":"A solid, reusable human-scored benchmark for scientific text-to-image generation; the headline result (even GPT-4o below 4/5 correctness) holds, but the per-dimension claims need error bars and more data.","tokens_in":27958,"tokens_out":3175,"would_cite":true,"duration_ms":33692,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Current multimodal LLMs cannot reliably draw accurate scientific figures from text alone, even the best one tested.","keywords":["ScImage","scientific text-to-image generation","multimodal large language models","benchmark","human evaluation","spatial understanding","numeric understanding","attribute binding"],"falsifier":"Take a fresh sample of about 200 prompts drawn from real scientific figure captions (for example, from the DaTikZ corpus), score them with the same rubric, and check whether GPT-4o's mean correctness stays below 4 and whether the model ranking by object category is reproduced. If the mean rises above 4.5 or the rankings flip, the benchmark's conclusions are specific to ScImage rather than to scientific text-to-image generation as a whole.","tokens_in":26836,"feed_emoji":"📐","tokens_out":4005,"duration_ms":39862,"temperature":0.7,"pith_summary":"This paper introduces ScImage, a benchmark for testing how well multimodal large language models generate scientific figures from text. It argues that current models, even the best tested (GPT-4o), cannot reliably produce accurate scientific diagrams: GPT-4o averages below 4 out of 5 on correctness, relevance, and scientific style, and every other model scores far lower. The benchmark decomposes scientific image understanding into spatial, numeric, and attribute comprehension, and finds different failure modes for code-based versus direct image models. The authors provide the prompts, the human-annotated scores for roughly 3,000 generated images, and an analysis across object categories and input languages, positioning ScImage as a reusable reference for future work. If the finding holds, automated scientific figure production from text alone is not yet trustworthy enough for publication without human verification.","feed_headline":"Best AI image models still score below 4/5 on science figures","feed_subtitle":"New ScImage benchmark of 404 prompts finds GPT-4o leads but all tested models fall short on scientific figures.","key_machinery":"ScImage is a template-and-dictionary benchmark: 101 query templates are instantiated with elements sampled from a hand-built dictionary of scientific objects, attributes, spatial relations, and numeric values, yielding 404 prompts that each explicitly require one or more of three comprehension dimensions (attribute binding, numeric understanding, spatial understanding). The evaluation machinery is a panel of 11 scientists scoring generated images on correctness, relevance, and scientific style (1 to 5, with 0 for compile errors), with pairwise annotation and a calibration session; it is this human scoring that carries the claims, since automatic metrics correlate at most 0.26 (Kendall) with it.","core_discovery":"The paper's central claim is that on ScImage, a new benchmark of 404 text prompts covering spatial, numeric, and attribute comprehension of scientific objects, current multimodal LLMs cannot reliably generate accurate scientific figures. In both text-code-image mode (Python or TikZ) and direct image mode, even the best model, GPT-4o, scores below 4 out of 5 on correctness, relevance, and scientific style; other models score mostly between 1.5 and 2.2. The benchmark and its roughly 3,000 human-annotated images are offered as a reusable yardstick for measuring progress in scientific text-to-image generation.","pith_inferences":["If the below-4 finding holds, automated scientific-figure pipelines (for example, LLM-to-code-to-image) need a verification stage, such as a separate visual checker, before outputs are used in papers.","The complementary failure modes (spatial hardest for code models, numeric hardest for image models) suggest that an ensemble or a router combining code-based layout with direct image generation could outperform any single approach.","A testable extension is to run newer models through the same 404 prompts and track whether the gap closes; the 20-prompt multilingual subset could also be expanded to test whether the observed language effects are stable.","The three comprehension dimensions could serve as a diagnostic for targeted fine-tuning, for example training code models on spatial relations and image models on counting."],"forward_implications":["GPT-4o in text-code-image mode achieves the best scores but still averages below 4 on correctness, relevance, and scientific style, meaning every output has at least some mistakes on average.","Code-based generation yields more scientifically styled images than direct image generation; direct models such as DALL·E and Stable Diffusion tend to add irrelevant details and score worst on scientificness.","Spatial understanding is the weakest dimension for code-generating models, while numeric understanding is the weakest for direct image generators.","Graph theory representations (nodes and edges) are the hardest object category across models, with average correctness below 1.7.","Automatic metrics (best PickScore Kendall correlation 0.26) correlate only weakly with human judgments, so the human-annotated scores remain the reference for this task."],"supporting_citations":[{"why":"Supplies the DaTikZ caption-code dataset from which ScImage's object entities were extracted, and the AutomaTikZ model that is one of the evaluated systems.","marker":"Belouadi et al. (2024a)"},{"why":"T2I-CompBench, the non-scientific compositional text-to-image benchmark that ScImage explicitly adapts and extends to the scientific domain.","marker":"Huang et al. (2023)"},{"why":"MMVP, cited to argue that automated multimodal metrics are unreliable for precise spatial judgments, motivating the paper's human evaluation design.","marker":"Tong et al. (2024)"},{"why":"VGbench, a prior evaluation of vector graphics generation that ScImage contrasts with its broader multi-format and multi-language setup.","marker":"Zou et al. (2024)"},{"why":"ChartMimic, a chart-to-code evaluation that ScImage contrasts with its coverage of multiple object types and output formats.","marker":"Shi et al. (2024)"}],"fun_headline_variants":["No AI image model clears 4/5 on science figures","ScImage: All LLMs fail at scientific image accuracy","GPT-4o tops but still under 4/5 on science drawing","AI can't render science: best score below 4","Multimodal LLMs stumble on scientific figures"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The 404 prompts assembled from 101 templates, and the three-dimensional decomposition they encode, are representative of scientific image generation; if not, the model rankings and per-category findings do not generalize beyond the test set.","fun_headline_variants_meta":{"raw":{"variants":["No AI image model clears 4/5 on science figures","ScImage: All LLMs fail at scientific image accuracy","GPT-4o tops but still under 4/5 on science drawing","AI can't render science: best score below 4","Multimodal LLMs stumble on scientific figures"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000282,"raw_usage":{"total_tokens":1648,"prompt_tokens":904,"completion_tokens":744,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":520,"completion_tokens_details":{"reasoning_tokens":660}},"tokens_in":520,"tokens_out":744,"duration_ms":8966,"temperature":1.0,"reasoning_tokens":660,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T23:33:32.218767+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a fresh sample of about 200 prompts drawn from real scientific figure captions (for example, from the DaTikZ corpus), score them with the same rubric, and check whether GPT-4o's mean correctness stays below 4 and whether the model ranking by object category is reproduced. If the mean rises above 4.5 or the rankings flip, the benchmark's conclusions are specific to ScImage rather than to scientific text-to-image generation as a whole.","supporting_citations":[],"review_version":1}