{"id":"f9807c3f-6dea-4658-afb9-e2c00490a8e5","arxiv_id":"2411.14062","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A fully automatic benchmark pipeline grades LMMs by regenerating images from their captions and measuring embedding similarity to the originals, ranking over 50 models.","lead":"This paper offers a fully automated way to grade how well image-understanding AI models describe what they see: let the model caption an image, feed that caption to a text-to-image generator, and compare the original with the regenerated picture. Across more than 50 models, most top-scoring systems produced descriptions too vague or incomplete to be rebuilt faithfully, which is the paper's main claim and its main methodological bet.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Without a ceiling calibration, the claim that even top LMMs score below 0.600 SIM is not yet interpretable as an LMM deficiency, because the reconstruction ceiling of the text-to-image models on this dataset is unknown.","rationale":"The reader's weakest_assumption is the measurement neutrality of the text-to-image stage. My reading of the manuscript finds this is indeed the most load-bearing assumption. The paper's own Appendix E admits a class of failures (table images) where the pipeline breaks, but more importantly, the entire quantitative claim rests on the untested assumption that a perfect description would yield a high SIM-Score and that per-pattern differences are attributable to the LMM rather than the generator. The paper provides no calibration, no human-caption baseline, no error bars, and no released code or data. I agree with the reader's verdict of CONDITIONAL and with the suggested remedy of ceiling calibration. The qualitative evidence in Sec. 4.3 and Figs. 17-21 is genuinely convincing that many models produce vague or instruction-violating captions, so a REJECT verdict would be too strong. The lack of variance reporting and the demotion of FID-Score to secondary status also weaken the quantitative instrument, but the ceiling calibration is the one test that would settle the central claim. I did not find a more load-bearing concern: the dataset construction's use of GPT-4o for pattern annotation is transparent and human-checked, and the 13 patterns are a reasonable taxonomy. The compression argument in Sec. 1 is a design choice, not an internal inconsistency. The concern is thus about interpretability of the headline numbers, not about the overall framework's novelty or usefulness.","tokens_in":31591,"tokens_out":2025,"duration_ms":18213,"concrete_test":"Feed human-written reference captions (e.g., JourneyDB original text prompts, or 2-3 expert-written captions per image) through the same pipeline: same four generators, same Unicom encoder, same SIM-Score formula, on the same 1,284 images. Compute the mean and per-pattern SIM-Score ceiling. If the human-caption ceiling is substantially above 0.6 (e.g., 0.75+), the headline claim stands; if the ceiling is near 0.6, then GPT-4o and InternVL2 are near-optimal and the claim that these models fail basic description is unsupported. Additionally, repeat the LMM evaluation with 3-5 seeds per model and report standard deviations to confirm that the 0.01-0.02 SIM differences in Table 1 exceed generator noise.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that SIM-Score measures LMM understanding via reconstruction fidelity (Sec. 1, 'compression' argument). This requires the text-to-image stage to be a faithful and roughly difficulty-uniform channel: a good caption should yield a reconstructed image close to the input, and a bad caption should yield a farther one. The paper never tests that assumption. No ground-truth or human-written captions are ever fed through the four generators on MMGenBench-Test to establish what SIM-Score a perfect caption achieves. The paper's four-generator comparison (Fig. 6) only shows that the four generators rank LMMs similarly; it does not anchor the scale. A score of 0.599 for InternVL2-76B could mean any of: (a) LMMs are genuinely poor describers; (b) the text-to-image model cannot reconstruct JourneyDB images even from perfect captions, capping all scores near 0.6; or (c) the Unicom encoder measures coarse semantic similarity rather than fine-grained fidelity, so a short, vague caption that names the right coarse content scores as high as a detailed accurate one. The pattern-level conclusions in Sec. 4.2 / Fig. 7 are where this matters most: 'Contextual', 'Orientation', 'Count', and 'Motion' are exactly the dimensions that text-to-image models are known to render poorly, and the paper never separates generator difficulty from LMM description failure. Appendix E concedes the pipeline 'may not achieve complete accuracy for tasks... such as table images', which is an admitted class of failure but not an anchor for the remaining patterns. The stochasticity of Eq. 2 is also unreported: no seeds or variance are given, so it is unclear whether the 0.01-0.02 differences between models on Table 1 are above noise. The claim that 'numerous LMMs excelling in existing benchmarks fail' may survive this test, but the absolute claim that 'the most advanced LMMs score below 0.600' and the per-pattern ranking are not yet interpretable without a ceiling.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces MMGenBench-Pipeline, a fully automatic evaluation protocol for large multimodal models (LMMs) based on text-to-image reconstruction. For each input image, the LMM under test generates a caption-prompt; one of four text-to-image models (FLUX.1-dev, Stable Diffusion 3.5, Kolors, Lumina) synthesizes an auxiliary image; and the Unicom image encoder is used to compute cosine similarity (SIM-Score) and FID between the original and reconstructed images. The authors construct MMGenBench-Test (1,284 JourneyDB images annotated with 13 image patterns) and MMGenBench-Domain (10,000 images) and evaluate over 50 LMMs. They report that the best models score below 0.600 SIM-Score, that model rankings are consistent across the four generators, and that LMMs are particularly weak on Contextual, Orientation, Count, and Motion patterns, concluding that existing benchmarks overstate basic image understanding and description abilities.","tokens_in":31687,"tokens_out":5609,"duration_ms":55847,"significance":"The pipeline is conceptually attractive as a low-cost, scalable, and domain-agnostic evaluation tool, and the paper has several concrete strengths: the four-generator cross-validation in Fig. 6, the broad model coverage (over 50 LMMs), the planned release of code and data, and the human-verified 13-pattern test set. The central claim—that reconstruction fidelity measures understanding via a compression argument (Sec. 1)—is plausible but is only as strong as the assumption that the text-to-image stage is a faithful and difficulty-uniform channel. The paper does not yet supply the calibration experiment needed to support that assumption, so the absolute and pattern-level conclusions are premature in their current form. With the addition of the calibration and sensitivity analyses described below, the pipeline could become a useful complement to existing benchmarks.","major_comments":[{"comment":"The interpretation of absolute SIM-Score as a measure of LMM understanding is unanchored because no ceiling calibration is performed. To claim that a score below 0.600 indicates that even top LMMs are inadequate describers, one must know what SIM-Score a perfect caption achieves on this dataset. I request an experiment that feeds human-written or ground-truth captions—for example, the original JourneyDB prompts—through the same four generators and the Unicom encoder on the same 1,284 MMGenBench-Test images, reporting overall and per-pattern SIM-Score. If the reconstruction ceiling is near 0.6 for these images, the observed scores reflect generator and encoder limits rather than LMM deficiencies. This experiment is the load-bearing test for the paper's central claim in Sec. 1 and Sec. 4.2.","section":"Sec. 2.1, Eq. (4), Sec. 4.2"},{"comment":"The pattern-level conclusion that LMMs are weak on 'Contextual', 'Orientation', 'Count', and 'Motion' is confounded by known weaknesses of current text-to-image models on exactly these dimensions. The four-generator comparison in Fig. 6 shows only that the generators rank LMMs similarly; it does not establish that the generator renders each pattern with roughly equal difficulty. Without a per-pattern ceiling calibration, the observed pattern differences can be fully explained by generator difficulty, and the claim that LMMs are proficient at coarse-grained patterns but weak at fine-grained relationships is unsupported. The per-pattern ceiling experiment from the first major comment should be the basis for any pattern-level claim.","section":"Sec. 4.2, Fig. 7, Table 2"},{"comment":"The SIM-Score relies on a single image encoder (Unicom) whose sensitivity to the fine-grained attributes the benchmark claims to measure is unvalidated. Even a faithful text-to-image generator would not yield a meaningful score if the encoder cannot distinguish a detailed, accurate caption from a short, vague one that names only coarse content. I request a controlled sensitivity experiment: take a set of captions, systematically degrade specific attributes (e.g., remove object counts, alter spatial relations, change orientation), generate images from each version, and measure whether SIM-Score drops accordingly. This would directly address the concern, acknowledged in Appendix E for table images, that the pipeline may be inaccurate for content that text-to-image and image representation models handle poorly.","section":"Eq. (4), Sec. 2.1, Appendix E"}],"minor_comments":[{"comment":"The label 'MCQY/NCaption' appears garbled; it should be split or clarified to identify the different task types shown.","section":"Fig. 2(a)"},{"comment":"The model name 'LLaV A-OV' contains an internal space in the text and tables; the official name 'LLaVA-OneVision' should be used for consistency.","section":"Throughout"},{"comment":"The row 'XComposer2 [10]' cites reference [10], which is InternVL; the citation should be corrected to the appropriate XComposer reference.","section":"Table 3"},{"comment":"The caption should state explicitly that the percentages are relative to the total number of pattern annotations (4,297), not to the 1,284 images, since images can have multiple patterns.","section":"Fig. 4"},{"comment":"The paper should state whether the full raw LMM output—including prefixes, suffixes, and safety refusals shown in Fig. 9—was passed to the text-to-image model without parsing, since this affects the interpretation of instruction-following failures and their impact on SIM-Score.","section":"Sec. 4.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a computer vision benchmarking venue, and the core idea is worth pursuing. The missing ceiling calibration is the main risk to the paper's central claim, but it is addressable in a revision with a well-defined experiment. I do not see any ethical or attribution concerns beyond the citation error noted in the minor comments."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The good news first: this is one of the few recent LMM evaluation papers that actually gives you something new. The round-trip loop — image to caption to generated image to similarity — as a fully automatic benchmark is not in JourneyDB, AutoBench-V, MMStar, or VLMEvalKit. The 13-pattern taxonomy and the 50+ model sweep are real work, and Fig. 6 shows the four text-to-image generators rank models similarly, which is a good internal consistency check. No parameters are fitted, and GPT-4o's pattern annotations never enter the SIM computation, so there is no load-bearing circularity. The qualitative examples (safety over-refusal, instruction-prefix pollution, vague captions from 72B models) make the headline observation credible: many strong leaderboard models describe images poorly. The related work is broad and the citation pattern looks fair.\n\nThe problem is that the measurement instrument is uncalibrated. The paper never feeds known-good captions through the same generators. So the claim that top LMMs score below 0.600 SIM is not interpretable: the ceiling might be 0.62 for any caption, or the Unicom encoder might reward vague coarse descriptions. Per-pattern conclusions in Fig. 7 hurt most here — Contextual, Orientation, Count, and Motion are exactly the dimensions text-to-image models render poorly. You cannot separate generator difficulty from LMM description failure without a human-caption anchor. The stochasticity in Eq. 2 is also unreported: no seeds, no error bars, and 0.01–0.02 differences in Table 1 are treated as signal. FID-Score is demoted after behaving inconsistently across the 1,284- and 10,000-image sets, which is a bit of a red flag. And the 'fully automated' label is overstated: Sec. 3.3 involves human double-checking, and Appendix E admits table images break the pipeline. Code and data are still 'will be released.'\n\nThese are addressable, not fatal. The central argument — that reconstruction fidelity measures something about understanding — is reasonable for a first-order instrument. The qualitative evidence stands on its own. The paper is for people building LMM evaluation tools and for anyone interpreting leaderboard rankings; the ranking is probably roughly right, the absolute values need anchoring.\n\nRecommendation: send it to peer review. A serious referee should ask for human-caption ceiling calibration, variance across seeds, and released artifacts. With those, this becomes a solid benchmark paper. Without them, it stays a promising but unproven instrument.","headline":"A useful new evaluation loop, but uncalibrated: the ranking is credible, the absolute scores and pattern-level claims are not yet.","tokens_in":32581,"tokens_out":3256,"would_cite":false,"duration_ms":31574,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Reconstruction fidelity is a practical measure of an LMM's image understanding.","keywords":["automated evaluation pipeline","large multimodal models","text-to-image generation","image understanding","image description","reconstruction fidelity","benchmark","cosine similarity"],"falsifier":"Take a set of human-written captions for the same images (for example, the annotations used to build the test set), feed them through the same four text-to-image models and the same encoder, and compare the resulting similarity scores with the LMMs' scores. If human captions do not score clearly higher, or if the same weak patterns—context, orientation, count, motion—stay weak even with human captions, then the pipeline is measuring generator limitations rather than LMM understanding.","tokens_in":31181,"feed_emoji":"🖼️","tokens_out":9358,"duration_ms":84157,"temperature":0.7,"pith_summary":"The paper argues that an LMM can be said to understand an image only if it can compress that image into a caption from which a text-to-image model can rebuild the same scene. It operationalizes this idea as MMGenBench-Pipeline: the LMM writes a detailed caption, a diffusion model generates a fresh image from it, and an image encoder measures how close the original and regenerated images are. On a newly constructed test set of 1,284 images across 13 visual patterns, the strongest evaluated models reach only moderate reconstruction similarity, with the best scores below 0.6 on a cosine scale, and several models that top existing benchmarks score substantially lower on this test. The paper interprets these results as evidence that fine-grained understanding—counting, orientation, motion, and contextual relations—remains a weak point for current LMMs, and that reconstruction-based evaluation exposes failures that short-answer benchmarks hide.","feed_headline":"Rebuild test: best image models score under 0.6 on understanding","feed_subtitle":"A new benchmark regenerates images from captions and exposes the gap.","key_machinery":"The load-bearing mechanism is the reconstruction loop, a three-stage composition: the LMM maps the input image to a caption $P_{\\mathrm{gen}} = \\mathrm{LMM}(I_{\\mathrm{inp}}, \\mathrm{prompt})$; a text-to-image model maps that caption to a fresh image $I_{\\mathrm{gen}} = G(\\epsilon; P_{\\mathrm{gen}}, \\theta)$; and an image encoder maps both images to embeddings whose cosine similarity defines the SIM-Score. The loop is treated as a lossy compression channel: a caption is good exactly when regenerating from it lands near the original in representation space. Four state-of-the-art text-to-image generators are used to keep the measurement from depending on any one generator, and their rankings are shown to be consistent.","core_discovery":"On the paper's own terms, the discovery is a measurement identity: a caption that preserves the essential information of an image will, after regeneration, produce an image whose representation is close to the original, so reconstruction similarity is a direct readout of the LMM's image understanding. The paper reports that no evaluated model crosses 0.6 on this score; the best open model reaches 0.599 and the strongest closed API model 0.566. Per-pattern scores split cleanly, with coarse attributes like artistic style, surreal content, symbols, and color scoring higher, and fine-grained attributes like contextual relations, orientation, counting, and motion scoring lower. The paper attributes this gap to LMMs' training on short captions and to their tendency to overfit benchmark-style tasks, and it points to short descriptions, instruction-format failures, and safety over-refusal as the main observable failure modes.","pith_inferences":["A calibration run the paper does not report—human-written reference captions through the same generators and encoder—would establish the absolute ceiling; without it, the 0.6-level scores may encode generator difficulty as much as LMM quality.","The pattern-level weakness on count, orientation, and motion may be partly inherited from text-to-image models, which are known to struggle with those same attributes, so cross-pattern gaps are best read as a joint system diagnostic.","The same loop could be inverted into a training signal: captions that reconstruct well could be up-weighted in instruction tuning, converting the benchmark into a data-curation filter.","Because the score depends on the chosen generator and encoder, absolute numbers are only comparable when those components are frozen; upgrading either one silently rescales the leaderboard."],"forward_implications":["Standard leaderboards can be complemented with a reconstruction-based score that directly penalizes vague or incomplete descriptions.","Training LMMs on longer, more detailed image-description pairs should improve their reconstruction scores, since the paper's failure analysis points to short training captions as a bottleneck.","Because the pipeline needs only images, it can be pointed at any new domain without manual annotation or task-specific question writing.","Per-pattern scores offer a diagnostic checklist—count, orientation, motion, and contextual relations—for comparing models on fine-grained description ability.","Instruction-following quality becomes measurable independently of content knowledge through observable formatting failures in generated captions."],"supporting_citations":[{"why":"Supplies the source image pool from which MMGenBench-Test and MMGenBench-Domain are built.","marker":"[60]"},{"why":"The default text-to-image model used to turn generated captions into auxiliary images.","marker":"[34]"},{"why":"A second text-to-image model used to cross-check that evaluation scores are consistent across generators.","marker":"[16]"},{"why":"The image representation model whose embeddings are compared to compute SIM-Score and FID-Score.","marker":"[5]"},{"why":"The strongest closed API model evaluated, and the model used to extract and re-annotate the 13 image patterns.","marker":"[53]"},{"why":"The inference toolkit used to run the set of evaluated LMMs under a standardized prompt.","marker":"[15]"}],"fun_headline_variants":["No LMM tops 0.6 on new image-understanding test","Best image models score 0.599 on understanding benchmark","Image understanding gap: top scores below 0.6","New benchmark: image models fail to describe images","Even top LMMs score under 0.6 on image understanding"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The text-to-image models render a correct, detailed caption faithfully enough, and with roughly equal difficulty across all 13 image patterns, so that differences in reconstruction similarity reflect the LMM's description quality rather than the generator's own strengths and weaknesses.","fun_headline_variants_meta":{"raw":{"variants":["No LMM tops 0.6 on new image-understanding test","Best image models score 0.599 on understanding benchmark","Image understanding gap: top scores below 0.6","New benchmark: image models fail to describe images","Even top LMMs score under 0.6 on image understanding"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000291,"raw_usage":{"total_tokens":1706,"prompt_tokens":955,"completion_tokens":751,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":571,"completion_tokens_details":{"reasoning_tokens":666}},"tokens_in":571,"tokens_out":751,"duration_ms":7330,"temperature":1.0,"reasoning_tokens":666,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T15:35:42.065214+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a set of human-written captions for the same images (for example, the annotations used to build the test set), feed them through the same four text-to-image models and the same encoder, and compare the resulting similarity scores with the LMMs' scores. If human captions do not score clearly higher, or if the same weak patterns—context, orientation, count, motion—stay weak even with human captions, then the pipeline is measuring generator limitations rather than LMM understanding.","supporting_citations":[{"cited_title":"Journeydb: A benchmark for generative im- age understanding","cited_arxiv_id":null,"evidence_quote":"Supplies the source image pool from which MMGenBench-Test and MMGenBench-Domain are built."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The default text-to-image model used to turn generated captions into auxiliary images."},{"cited_title":"Gpt-4o system card, 2024","cited_arxiv_id":null,"evidence_quote":"The strongest closed API model evaluated, and the model used to extract and re-annotate the 13 image patterns."}],"review_version":1}