{"id":"bf08c488-b465-4f9a-8f8e-a115cc939508","arxiv_id":"2412.05939","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Multi-grained concept annotations, when integrated into interleaved image-text documents, improve MLLM comprehension and generation beyond caption-only pretraining at much smaller data scale.","lead":"The paper builds MMGiC, a dataset that pairs images with captions, object labels, descriptions, and cropped object regions, and trains a 7B multimodal model on these interleaved documents. Models trained on this smaller multi-grained dataset matched or beat models trained on 13 times more image-caption data, and combining the two sources gave the best scores on 12 benchmarks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Object-region gains in Table 1 may reflect added text (location/labels) or low-fidelity visual tokens rather than region grounding; an ablation removing region images is needed.","rationale":"The paper presents a substantial new dataset and a mostly well-controlled study: the within-image recipe ablation in Table 1 is the cleanest evidence for the core claim, since it avoids the image-source confound between MMGiC and IC. The strongest soft spot, correctly identified by the reader, is the reliance on the frozen LaVIT visual tokenizer for object-region grounding. I extend this concern by noting that the R component in Table 1 is confounded with additional text (location and region-level object labels), so the observed gains cannot be uniquely attributed to visual grounding in the regions unless the visual tokens are ablated away. This is not an accusation of invalidity: the multi-grained data clearly helps, and the overall pattern of results across 12 benchmarks is credible. But the specific mechanism of 'locating and learning concepts in object regions' is underdetermined by the current experiments. Other issues—single runs without error bars, best-of-8 template selection, and tuning mask_prob on evaluation benchmarks—are real but secondary; they affect the precision of the headline numbers rather than the qualitative ordering of recipes. Since the reader already assigned CONDITIONAL, my concern reinforces that verdict without moving it; the proposed ablation is the concrete next step. No ad hominem, no overstatement: the paper is a solid empirical exploration that needs one isolating ablation and a tokenizer-fidelity check to support its mechanistic claim.","tokens_in":53533,"tokens_out":4919,"duration_ms":53114,"concrete_test":"Retrain the CLDR recipe (Table 1, row 3) with the region image tokens replaced by a fixed placeholder sequence (e.g., the same number of [IMG] tokens using a constant embedding), while keeping the Location and Objects text identical. Compare against CLDR and CLD on COCO CIDEr and SEED-Bench-IMG. If performance remains at the CLDR level, the region-image tokens contribute nothing and the gains are due to the text accompanying regions; if performance drops to near CLD, the visual regions carry the signal. To further address the tokenizer-fidelity concern, compute CLIP similarity between original and LaVIT-reconstructed cropped regions for a random sample of 1000 regions; if the similarity is low or uncorrelated with per-sample caption gains, the 'visual grounding' mechanism is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that MMGiC helps MLLMs locate and learn concepts at multiple granularities rests heavily on the incremental benefit of object regions (R) seen in Table 1 (row 3 vs row 2) and on the qualitative grounding examples in Figures 3 and 4. However, the R component is not purely visual: adding R simultaneously introduces cropped-region visual tokens, location text (e.g., 'Bottom Right'), and region-level object labels. Thus the +4.66 CIDEr on COCO and the corrected 'laying' vs 'sitting' prediction in Figure 3 could be driven by the additional textual annotations that come with the regions, rather than by the model grounding concepts in the region images. The frozen LaVIT visual tokenizer (Section 3.1, Appendix B.3) is acknowledged to lose detail—text-rich and chart images are poorly reconstructed in Figure 6—and no evidence is given that cropped object regions, which are often small and may be heavily compressed after resizing, are faithfully encoded into the discrete tokens used by the LLM. If the region visual tokens are too lossy, they may contribute little beyond a second global view of the image, and the observed gains would be an artifact of the extra text in the template. The paper does not report an ablation that removes the region image tokens while keeping the region text, nor does it quantify reconstruction fidelity for the actual cropped regions used in training. This underdetermines the mechanistic interpretation of 'grounding concepts to regions' and, importantly, would change the practical recommendation about which components of MMGiC matter.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces MMGiC, a new multimodal dataset of 3.5M images with multi-grained concept annotations (captions, category labels, label descriptions, and object regions), built from four object detection datasets, and a general autoregressive discrete MLLM framework. It reports controlled data-recipe experiments (Table 1), a comparison and collaboration study with large-scale image-caption data (Table 2), supervised fine-tuning (SFT) evaluation on 12 comprehension and generation benchmarks (Tables 3-4), and a per-dimension analysis on SEED-Bench (Section 4.4). The paper claims that multi-grained annotations help MLLMs better locate and learn concepts, that MMGiC outperforms image-caption data even at much smaller scale, and that combining MMGiC with captions yields further gains, e.g., 3.95% and 2.34% absolute improvements on POPE and SEED-Bench over captions alone.","tokens_in":53792,"tokens_out":7965,"duration_ms":79680,"significance":"The dataset is a substantial new public resource, and the paper provides three controlled baselines that differ only in pre-training data, with a common SFT protocol and evaluation on a broad set of benchmarks. The authors are transparent about several limitations, especially the frozen visual tokenizer's poor reconstruction of text-rich and chart images (Appendix B.3) and the difficulty of fair cross-dataset comparison (Appendix C.1). If the core claims hold, the work would be a useful demonstration that multi-grained annotations can improve both comprehension and generation in MLLMs. However, the central mechanistic claim about object regions grounding concepts is not yet supported by the current ablations, and the evaluation protocol involves selection on the test benchmarks, so the headline numbers are optimistically biased. The paper's strengths are its scale, its controlled within-dataset recipe comparisons, and its honest discussion of known limitations.","major_comments":[{"comment":"The comparison that motivates the paper's flagship claim about object regions bundles several changes at once: adding R simultaneously introduces cropped-region visual tokens, location text (e.g., 'Bottom Right'), and region-specific object labels. The observed gains (e.g., +4.66 CIDEr on COCO, +3.4 on NoCaps) could therefore be driven by the additional textual annotations rather than by the model grounding concepts in the region images. Please add ablations over the row-2 recipe: (a) row-2 plus region text without the region image, and (b) row-2 plus the region image without the new region text. In addition, because Appendix B.3 documents that the frozen LaVIT tokenizer loses detail on text-rich and chart images, please quantify the tokenizer's fidelity on the actual cropped object regions used in training (e.g., reconstruction similarity or a region classification probe) so the grounding interpretation can be evaluated.","section":"Section 4.1, Table 1 (rows 2 vs. 3)"},{"comment":"The evaluation protocol selects the best of 'at least 8 task instruction templates' per benchmark, and the hyperparameter mask_prob is tuned on the same image captioning and generation benchmarks used in Tables 1 and 2 (swept from 0 to 1 in steps of 0.1, with 0.9 chosen). This constitutes selection on the test set, and no variance or seed information is reported. The headline improvements, including the 3.95% POPE and 2.34% SEED-Bench gains in Table 3, may partly reflect this selection rather than a robust effect of the dataset. Please report the mean and standard deviation (or range) over templates, fix mask_prob before evaluation using a held-out validation split, and provide results across at least 3 seeds, or explicitly justify why seed and template variation is negligible.","section":"Appendix E.2 and Appendix E.1"},{"comment":"The baseline MLLM-IC is not actually trained with image-caption data alone: the shared SFT recipe includes 1M MMGiC 'playback' samples for all three baselines. Therefore the abstract's claim that the combination achieves gains 'over image-caption data alone' is not supported by Table 3, because the image-caption baseline also received multi-grained MMGiC annotations during SFT. Please either fine-tune an IC-only variant without MMGiC playback, or restrict the 'image-caption data alone' phrasing to the pre-training comparison in Table 2 and describe Table 3 as comparing pre-training strategies under a shared mixed SFT protocol.","section":"Section 3.3 and Table 3"},{"comment":"The cross-dataset comparison between MMGiC and IC does not isolate annotation granularity, because the two data pools differ in image source and content: MMGiC is built from human-annotated object detection datasets, while IC is web-caption data. Appendix C.1 itself attributes part of MMGiC(C)'s advantage over IC-PART to the 'higher quality' of the images (more noun chunks per image). The phrase 'fair comparison' should therefore be qualified: the controlled evidence for the multi-grained hypothesis is the within-MMGiC recipe ablation in Table 1, whereas Table 2 varies data source and image complexity alongside annotation type.","section":"Section 4.2, Table 2 and Appendix C.1"}],"minor_comments":[{"comment":"The term 'zero-shot' is misleading after SFT; consider using 'direct transfer' or 'unseen benchmark' to describe evaluation on tasks not seen during SFT.","section":"Section 4.3 and Tables 3-4"},{"comment":"The dataset name is written inconsistently as 'MMG IC', 'MMGiC', and 'MMGIC'; please standardize on one spelling. There are also typos such as 'th' in 'th \"bagel\"' (Section 4.1) and 'laying' where 'lying' is intended in Figure 3.","section":"Throughout"},{"comment":"The loss-scale hyperparameter α=0.1 is introduced without any ablation or sensitivity analysis; since the paper already tunes mask_prob, α is another free parameter that should be fixed before evaluation or shown to be robust.","section":"Appendix E.1"},{"comment":"The notation 'MMGiC → IC' and 'IC → MMGiC' is used without definition in the caption; please define the arrow as sequential curriculum training in the table caption or in the main text near the table.","section":"Table 2 caption"},{"comment":"The selection of 9 evaluation dimensions from SEED-Bench and the exclusion of 'Text Recognition' is reported only in a footnote; this is a substantive design choice that should be stated in the main text, along with its rationale.","section":"Section 4.4"}],"recommendation":"major_revision","confidential_remarks":"This is a substantial empirical study with a potentially useful dataset, and the authors are transparent about several limitations. The verdict hinges on the missing R-component ablation and the template/mask_prob selection on evaluation benchmarks; both issues affect the central claim and the headline numbers. The SFT protocol confound for MLLM-IC also needs to be addressed. I would be willing to review a revised version that adds the ablations and reports variance or fixed hyperparameters before evaluation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a real dataset contribution and a mostly clean empirical study. The new thing is MMGiC, a 3.5M-image interleaved document set that puts captions, object labels, GPT-4 label descriptions, and cropped object regions into one template, trained with plain next-token prediction and no extra losses. That format is new, and the paper gives it a fair shot: the three main baselines differ only in pretraining data, and the 12-benchmark eval covers comprehension and generation.\n\nWhat it does well: the data construction is careful (manual checks on descriptions, WordNet disambiguation, IoU-based region merging), and the appendices are candid about the frozen LaVIT tokenizer's weak reconstruction of text/charts and about the MMLU drop after pretraining. The comparison with 52M-image caption data shows MMGiC holds its own and that IC-first-then-MMGiC ordering helps; that is a useful practical result.\n\nSoft spots, in proportion: the stress-test concern about the R component is real. In Table 1, adding R also adds location text and region labels, so you cannot attribute the +4.66 CIDEr or the 'laying vs sitting' fix to the region image tokens themselves. The paper does not ablate 'region text without region image' or quantify reconstruction fidelity for the actual crops. This weakens the 'grounding' story but not the whole paper: the multi-grained recipe still beats caption-only data in several places, and C+LD already helps before R enters.\n\nThe other issues are standard for this area but worth naming: single runs, no error bars, mask_prob tuned on the eval benchmarks, and best-of-8 instruction templates. Those make the precise deltas (e.g., 3.95 on POPE) look firmer than they are. Also, the SFT composition is ambiguous: Section 3.3 says all baselines playback 1M MMGiC, but Appendix C.5 describes MLLM-IC's default SFT without it. That needs a fix. And code/data are not out yet, which matters for a dataset paper.\n\nWho it's for: anyone building MLLM pretraining corpora. The dataset and the curriculum result are worth engaging with. It deserves a serious referee — conditional accept with required ablations and release.","headline":"A genuinely new multi-grained pretraining dataset with a mostly clean comparison; the object-region ablation is confounded with added text, and the evaluation needs error bars, but the dataset and curriculum findings are worth taking seriously.","tokens_in":54408,"tokens_out":3014,"would_cite":true,"duration_ms":29798,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adding object labels, descriptions, and cropped regions to image captions helps multimodal LLMs locate and generate concepts, with absolute gains of 3.95% and 2.34% on POPE and SEED-Bench when the two data types are combined.","keywords":["multimodal large language models","multi-grained concept annotations","vision-language alignment","object region grounding","label descriptions","image-text interleaved documents","multimodal comprehension and generation","concept learning"],"falsifier":"Retrain the same MLLM on a variant of MMGiC where each object-region crop is replaced by a random square crop of the same size and location statistics, keeping all text annotations identical; if the observed gains on POPE, SEED-Bench, and image generation persist, the benefit is not from region-level visual grounding.","tokens_in":53295,"feed_emoji":"🖼️","tokens_out":6591,"duration_ms":57788,"temperature":0.7,"pith_summary":"Multimodal large language models are usually pre-trained on image captions alone, which describe a scene coarsely. This paper asks whether adding fine-grained concept annotations — object labels, label descriptions, and cropped object regions — makes the model understand and generate concepts better. It introduces MMGiC, a dataset of over 3.5 million images with such multi-grained annotations, restructured into image–text interleaved documents, and trains a general autoregressive MLLM on it. The paper's experiments on 12 benchmarks support the hypothesis: multi-grained annotations improve comprehension and generation over captions alone, and combining them with caption data yields further gains, for example 3.95% and 2.34% absolute improvements on POPE and SEED-Bench. If the claim holds, fine-grained concept annotation is a broadly applicable data upgrade for MLLMs rather than a special-purpose trick.","feed_headline":"Fine-grained annotations boost multimodal LLMs by up to 3.95%","feed_subtitle":"A new dataset pairs captions with object labels, descriptions, and cropped regions to ground concepts in images.","key_machinery":"The load-bearing mechanism is the structured template that converts each image into an interleaved document containing the whole image, its caption, fine-grained labels with descriptions, and cropped object regions with location descriptions. This document is tokenized into a discrete sequence of visual and textual tokens and trained with an autoregressive next-token objective, with no extra loss functions or grounding components. The template is what lets coarse and fine annotations complement each other within one context, and it is also what lets the dataset drop into existing MLLM frameworks.","core_discovery":"MMGiC provides multimodal multi-grained concept annotations — coarse image captions, fine-grained object/attribute/relationship labels with descriptions, and visual object-region crops — organized by a structured template into image–text interleaved documents. Under a standard autoregressive discrete MLLM trained only with next-token prediction, these annotations integrate and complement each other: label descriptions anchor labels to visual detail, and object regions ground text to specific image locations. The paper claims this improves both comprehension and generation, and that MMGiC and conventional caption data have complementary strengths in depth versus breadth of concept representation, so appropriately combining them improves performance further.","pith_inferences":["Because the paper's gains concentrate on dimensions like instance identity, spatial relation, and instance interaction, a testable prediction is that models trained on MMGiC should show larger improvements on spatially demanding VQA than on holistic scene questions.","The template mechanism suggests that automatically synthesizing multi-grained annotations for web-scale caption data could scale these gains, though the paper's own caution about annotation noise implies quality filtering will be critical.","The frozen tokenizer's poor reconstruction of text-rich and chart images predicts that switching to a higher-fidelity visual tokenizer should extend MMGiC's benefits to OCR-heavy and table/chart benchmarks, which the current model underperforms on.","The method is dataset-agnostic in principle, so one could construct MMGiC-style documents for other concept types, such as events or abstract attributes, and test whether the same integration effect appears."],"forward_implications":["Training MLLMs with MMGiC instead of captions alone improves zero-shot image captioning and text-to-image generation even with roughly an order of magnitude less data.","Adding caption data (IC) to MMGiC further improves downstream comprehension, including gains of 3.95% on POPE and 2.34% on SEED-Bench over captions alone.","Fine-grained labels without descriptions or regions can hurt; descriptions and object regions are needed to anchor those labels.","Training order matters: caption data first, then MMGiC, outperforms joint training on several tasks, while joint training first and MMGiC last gives the best average performance.","MMGiC also works as instruction fine-tuning data, improving an MLLM pre-trained on captions only across all evaluated benchmarks."],"supporting_citations":[{"why":"Supplies the frozen visual encoder, discrete tokenizer, decoder, and diffusion model that the framework inherits.","marker":"(Jin et al., 2023)"},{"why":"Defines the autoregressive discrete next-token framework and extended vision-language vocabulary used for training.","marker":"(Ge et al., 2023b)"},{"why":"One of the four object-detection sources; provides object, attribute, and relationship annotations for MMGiC.","marker":"(Kuznetsova et al., 2020)"},{"why":"Provides large-scale object detection annotations with category labels and bounding boxes that become fine-grained labels and regions.","marker":"(Shao et al., 2019)"},{"why":"Adds vast-vocabulary object detection annotations to MMGiC.","marker":"(Wang et al., 2023a)"},{"why":"Contributes dense object, attribute, and relationship annotations with WordNet synset mapping.","marker":"(Krishna et al., 2017)"},{"why":"Synthesizes the coarse image captions used in MMGiC when source datasets lack them.","marker":"(Li et al., 2023b)"},{"why":"Generates the label descriptions for object, attribute, and relationship categories.","marker":"(Achiam et al., 2023)"},{"why":"Provides the LLaMA-2-7B language model that is extended with visual tokens and partially fine-tuned.","marker":"(Touvron et al., 2023b)"},{"why":"Ranks and filters the synthesized captions by image-text similarity.","marker":"(Radford et al., 2021)"}],"fun_headline_variants":["Multi-grained annotations boost multimodal LLMs","New dataset pairs captions with object labels and regions","Fine-grained concepts ground multimodal LLMs better","MMGiC: annotations that align vision and language at multiple scales","Combining coarse and fine annotations lifts LLM benchmarks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The frozen visual tokenizer must preserve enough visual detail in the discrete tokens of small object-region crops for the model to learn from them; the paper itself notes that text-rich and chart images are poorly reconstructed, so if region detail is lost, gains attributed to object regions could instead come from label descriptions or other components.","fun_headline_variants_meta":{"raw":{"variants":["Multi-grained annotations boost multimodal LLMs","New dataset pairs captions with object labels and regions","Fine-grained concepts ground multimodal LLMs better","MMGiC: annotations that align vision and language at multiple scales","Combining coarse and fine annotations lifts LLM benchmarks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000123,"raw_usage":{"total_tokens":1080,"prompt_tokens":907,"completion_tokens":173,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":523,"completion_tokens_details":{"reasoning_tokens":99}},"tokens_in":523,"tokens_out":173,"duration_ms":2442,"temperature":1.0,"reasoning_tokens":99,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T20:10:33.245187+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the same MLLM on a variant of MMGiC where each object-region crop is replaced by a random square crop of the same size and location statistics, keeping all text annotations identical; if the observed gains on POPE, SEED-Bench, and image generation persist, the benefit is not from region-level visual grounding.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Ranks and filters the synthesized captions by image-text similarity."}],"review_version":1}