{"id":"cdaf5343-c5e7-43a2-9288-d845cf540b8e","arxiv_id":"2507.20411","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"CONCAP combines retrieved captions with retrieved concepts to improve multilingual image captioning, reaching 34.2 average CIDEr on XM3600 against 31.8 for Pangea and 25.9 for mBLIP while training on 566K pairs.","lead":"This paper introduces CONCAP, a multilingual image captioning model that retrieves both similar captions and image-specific concepts to guide caption generation. It reports higher average captioning scores on the XM3600 benchmark than larger multilingual models while training on a fraction of the data.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Evaluation-time caption retrieval datastore is undisclosed; if it indexes XM3600 reference captions, the 2.4-point L36 gain over Pangea is an artifact of test-set retrieval rather than of target-language grounding.","rationale":"Good-faith reading: CONCAP's core mechanism, concatenating retrieved captions and concepts into the decoder prompt, is well supported by internal ablations: NoRAG (26.9) to ConRAG (30.3), CapRAG (31.4), and CONCAP (34.2); the Text-only ablation drops to 13.4 L4, showing the model is not merely copying retrieved text. The oracle concept experiment and retriever comparison also lend qualitative support. I therefore do not see an internal inconsistency in the main L36 result itself. The load-bearing vulnerability is external validity: the evaluation-time retrieval datastore is undisclosed. Since CONCAP is a retrieval-augmented model, the exact memory it queries at test time is as important as its trainable parameters; without that datastore being named, the comparison against Pangea and mBLIP cannot be fully assessed. The paper's own leakage-avoidance steps for the XM100 concept-lexicon experiments (Section 5, Figure 4) show the authors recognize this risk, making the silence in Section 4.1 more significant. A secondary data-quality issue noted by the reader, the Table 2 vs Table 10 inconsistency for CapRAGM, affects an ablation rather than the headline L36 number; it should be corrected but is not the primary reason for conditional status. The concrete test above would settle whether the central comparison is fair.","tokens_in":19384,"tokens_out":11064,"duration_ms":93834,"concrete_test":"Re-run the XM3600 evaluation with the caption-retrieval datastore fixed to COCO-35L target-language captions only, with all XM3600 captions removed, and separately compute the overlap between the top-4 retrieved captions and the XM3600 reference captions for each test image. If the L36 CIDEr drops below Pangea's 31.8, or if any retrieved caption comes from the test image's own reference set, the reported 2.4-point advantage is attributable to test-set retrieval and the central claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"CONCAP's central claim is that retrieved captions and concepts in the target language, not test-set information, drive the 34.2 vs 31.8 L36 CIDEr advantage over Pangea. Section 4.1 specifies the training datastore (English COCO pivot) but for evaluation only says retrieval is \"performed using the mSigLIP model directly in the target language\" without naming the indexed caption corpus. Two readings matter. (i) If the datastore is the XM3600 caption set, then at inference the model sees human reference captions for the very images being captioned, so the comparison to retrieval-free baselines is not a fair test of the method. (ii) If the datastore is COCO-35L target-language captions, then retrieval is still English-translated, Western-centric data, which undercuts the paper's \"seeing beyond English\" framing but does not by itself invalidate the headline numbers. The distinction changes the interpretation of every main-result and ablation table. The authors' separate care to exclude XM100 words and images from the enriched concept lexicons (Section 5, Figure 4) shows leakage is a recognized risk, yet no analogous statement is made for caption retrieval at evaluation. The missing datastore specification is therefore not cosmetic; it determines whether the central comparison is valid.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript presents CONCAP, a retrieval-augmented multilingual image captioning model built on mBLIP. The model augments the decoder prompt with (i) top-n captions retrieved from a caption datastore and (ii) top-m image-specific concepts retrieved from language-specific wordlists. Training uses a 566K-pair subsample of COCO-35L, with frozen vision encoder and language decoder plus trainable Q-Former, projection, and LoRA layers. Evaluation on XM3600 reports L36 CIDEr 34.2, above Pangea (31.8) and mBLIP (25.9) despite far fewer parameters and training examples. Ablations show NoRAG 26.9, CapRAG 31.4, ConRAG 30.3, and CONCAP 34.2, which the authors interpret as additive gains; a Text-only control in Appendix A.8 supports the visual-grounding interpretation. Additional analyses cover retriever choice, number of concepts, concept-list enrichment, and per-language performance.","tokens_in":19656,"tokens_out":6934,"duration_ms":62512,"significance":"If the reported results are reproducible, CONCAP is a valuable data-efficient alternative to scaling in multilingual captioning. The evaluation is on an external human-annotated benchmark, hyperparameters are tuned on a dev split, and the appendix provides per-language tables and a Text-only control. The oracle experiment on JEEM and the enrichment analysis are honest probes of the method's limitations. The main caveats are the unspecified evaluation caption datastore and internal inconsistencies in the ablation tables, which need to be resolved before the headline comparison can be fully trusted.","major_comments":[{"comment":"The evaluation-time caption retrieval datastore is not identified. The text says that retrieval is performed \"using the mSigLIP model directly in the target language\" and motivates this by the geographic diversity of XM3600, but it never states which caption corpus is indexed at inference. If the datastore is the XM3600 reference captions and the query image's own captions are not excluded, then the reported L36 advantage over Pangea could be an artifact of test-set leakage rather than of target-language retrieval. Please specify the datastore and the exclusion policy for the query image, and note that no analogous anti-leakage statement is made for caption retrieval even though Section 5 excludes XM100 words and images from enriched concept lexicons.","section":"Section 4.1 (Caption Retrieval)"},{"comment":"The CapRAG and CapRAGM rows are inconsistent. In Table 2, CapRAG lists en/es/zh/hi = 66.2/53.3/23.9/20.2, which are exactly the CapRAGM values in Table 10, while CapRAGM in Table 2 lists 38.3/30.4/16.4/13.9, values that do not appear in Table 10. Because Section 4.4's conclusion that target-language caption retrieval \"actively hurts performance\" rests on CapRAGM, the correct per-language values must be confirmed and the table corrected.","section":"Table 2 vs Table 10"},{"comment":"The CapRAGM comparison changes two variables at once: the retriever (English CLIP vs mSigLIP) and the retrieval language (English pivot vs target language). The conclusion that \"the quality of the retriever is of utmost importance\" is therefore not uniquely identified; the observed drop could also reflect noisier target-language captions or the interaction of both factors. Please disentangle these variables or soften the causal claim.","section":"Section 4.4"}],"minor_comments":[{"comment":"The text cites \"Table 9\" for concept list sizes, but Table 9 in the appendix is the prompt-template table; there are two items numbered Table 9. Renumber the appendix tables.","section":"Appendix A.4"},{"comment":"The Text-only control replaces the image with a solid-color mean-RGB image, but it is not stated whether retrieval for this condition uses the original image or the solid image; this determines exactly what the control isolates.","section":"Appendix A.8"},{"comment":"The oracle result is reported after one epoch of training and on 200 images, and the claim that gains fade with continued finetuning is presented without supporting learning curves. Adding the epoch progression would make this claim substantiated.","section":"Section 5"}],"recommendation":"major_revision","confidential_remarks":"The evaluation datastore question is the key risk. I recommend asking the authors to state the datastore and exclusion rule before final acceptance. If the datastore is XM3600 without excluding the query image's own captions, the headline comparison is invalid; if it is COCO-35L or another non-test corpus, the issue can be resolved in revision. The Table 2/Table 10 inconsistency should also be resolved, but it appears to be a presentation error rather than a fundamental flaw."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nYou should know that CONCAP is a genuinely useful recipe: a retrieval-augmented mBLIP variant that combines retrieved captions with retrieved image concepts and gets a 34.2 average CIDEr on XM3600 from 566K training pairs and 111M trainable parameters, beating Pangea's 31.8. The combination is new in multilingual captioning, and the ablations show the two signals are additive. The Text-only control in Appendix A.8 is a nice touch; it shows the model isn't just parroting retrieved text.\n\nThe paper also earns credit for honesty: concept enrichment from external lexicons hurts, and the oracle experiment shows the benefit fades with training. Those are useful negative results.\n\nThe soft spots are real, and one is load-bearing. Section 4.1 says evaluation-time caption retrieval is done 'directly in the target language' but never names the indexed corpus. If it's the XM3600 reference captions, the 2.4-point gain over Pangea is partly test-set leakage. If it's COCO-35L translations, the 'seeing beyond English' framing is overstated. Figure 4 shows the authors carefully exclude XM100 words/images for concept enrichment, so leakage is a known risk, yet no analogous statement exists for caption retrieval. This ambiguity sits under every main-result and ablation table. It needs to be resolved, not patched.\n\nSecond, the tables disagree. Table 2 lists CapRAGM at 38.3 CIDEr for English; Table 10 lists 66.2 for the same model/setting. Several other columns differ too. That inconsistency makes the appendices hard to trust even though the main CONCAP-vs-baseline numbers are internally consistent.\n\nThird, no code or data are released, which for an empirical method means the numbers can't be independently checked.\n\nNone of these are fatal to the core idea — the mechanism story is coherent and the main comparison holds up internally — but they are exactly the issues a careful referee would catch. As it stands, I would not cite the headline numbers in my own work until the datastore is disclosed and the tables reconciled. Anyone working on multilingual captioning or RAG for vision-language models will get value from this paper, though.\n\nMy recommendation: send it to peer review; it deserves referee time. After the datastore question is answered and the tables fixed, it is a solid conference paper.","headline":"Data-efficient multilingual captioning with genuinely additive RAG signals, but the evaluation datastore is undisclosed and the tables disagree.","tokens_in":20217,"tokens_out":3946,"would_cite":false,"duration_ms":36342,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CONCAP shows that retrieved captions plus image-specific concepts can lift multilingual captioning past models many times its size.","keywords":["multilingual image captioning","retrieval-augmented generation","concept retrieval","low-resource languages","XM3600 benchmark","CIDEr","vision-language models","data-efficient training"],"falsifier":"Decisive check: replace the mSigLIP retrieval with random caption and concept sampling and measure CIDEr on XM3600. If CONCAP retains most of its 34.2 score, retrieval is not the cause; if it falls toward the 26.9 no-retrieval level, retrieval is load-bearing. A second check is to identify the evaluation-time caption corpus: if it is the same translated COCO-35L data used for training, compare with a native-language datastore.","tokens_in":19149,"feed_emoji":"🌍","tokens_out":6406,"duration_ms":60251,"temperature":0.7,"pith_summary":"The paper sets out to show that a multilingual image captioning model can reach strong results without massive multilingual training by conditioning generation on two retrieved signals: full captions of similar images and short image-specific concepts in the target language. CONCAP, built on the lightweight mBLIP architecture, is trained on 566K image-caption pairs with 111M trainable parameters, yet scores 34.2 on the CIDEr caption-quality metric averaged over 36 languages of the XM3600 benchmark, ahead of Pangea (31.8 with 7B parameters and 6M examples) and mBLIP (25.9 with 2.71M examples). The result matters because it points to a cheaper route toward closing the English-centric gap: retrieval supplies target-language phrasing and broad visual coverage that would otherwise require expensive scaled-up multilingual training.","feed_headline":"Concepts retrieval lifts multilingual captions past a 7B model","feed_subtitle":"Trained on 566K pairs, CONCAP scores 34.2 CIDEr on XM3600, ahead of 7B Pangea and mBLIP.","key_machinery":"The load-bearing mechanism is a retrieval-augmented prompt assembled from two datastores indexed by a multilingual vision-language retriever (mSigLIP). For captions, the image encoder retrieves the top-4 nearest captions from a per-language datastore; for concepts, single tokens extracted from the training captions are wrapped in language-specific templates such as \"a photo of a dog\", the top-10 are retrieved per image, and both sets are inserted into a fixed English prompt before the decoder: \"Similar images show: ..., This image might contain: ..., Caption in {language}:\". The decoder is the mBLIP stack, with frozen vision encoder and language model and trainable Q-Former, projection, and LoRA layers. Concept retrieval is what makes the method robust: it does not depend on translated English caption pivots, and it supplies terms that retrieved full captions miss.","core_discovery":"The central discovery is that caption retrieval and concept retrieval are complementary and additively beneficial in multilingual captioning. In an otherwise identical setup, the no-retrieval model scores 26.9 CIDEr; caption-only retrieval gives 31.4; concept-only retrieval gives 30.3; combining both gives 34.2. The paper further finds that concepts alone are a surprisingly effective retrieval signal, that concept retrieval remains useful even when caption retrieval fails (direct target-language caption retrieval at training time drops to 20.4), and that enriching the concept list with external cultural and web-sourced terms hurts rather than helps, whereas oracle cultural concepts improve an Arabic cross-cultural test at early training but the benefit fades with continued finetuning.","pith_inferences":["The paper never names the caption datastore used during XM3600 evaluation. If it is the same translated, Western-centric COCO-35L data used for training, then CONCAP's margin partly reflects retrieving near neighbors of its own training distribution, and the claim of 'seeing beyond English' would be overstated for target-language grounding.","A direct two-way test follows: index a datastore of native, human-written captions per XM3600 language and re-run evaluation; if CIDEr does not rise or hold, the retrieval memory is doing the work, not the target-language concepts.","The oracle result on the 200 Arabic cultural images suggests a training dynamic worth probing: early finetuning can absorb new lexical items, but later epochs lock in the decoder's vocabulary; curriculum schedules that introduce cultural concepts late could extend the oracle gain beyond one epoch."],"forward_implications":["If CONCAP's results hold, a 111M-parameter model trained on 566K pairs can beat a 7B model trained on 6M examples on average across 36 languages, shifting the cost-benefit balance of multilingual captioning toward retrieval memory rather than scale.","The additive gain of captions and concepts means future retrieval-augmented captioning systems should treat sentence-level style guides and lexical content cues as separate channels rather than as redundant evidence.","Because concept-only augmentation survives a weak caption retriever, target-language concept retrieval offers a fallback for languages where translated caption parallels are unavailable.","The finding that broader concept lexicons hurt performance implies that datastore quality, not size, governs concept retrieval, so curation efforts should focus on relevance filtering rather than raw coverage."],"supporting_citations":[{"why":"Supplies the mBLIP architecture and initialization that CONCAP builds on, including the frozen vision encoder and decoder with LoRA.","marker":"Geigle et al., 2024"},{"why":"Provides the XM3600 evaluation benchmark, the COCO-35L training data, and the language groupings and CIDEr-based evaluation protocol.","marker":"Thapliyal et al., 2022"},{"why":"Establishes retrieved-caption augmentation for lightweight captioning and the English-pivot retrieval setup that CONCAP adopts for training.","marker":"Ramos et al., 2023b"},{"why":"PAELLA is the direct multilingual retrieval-augmented baseline and the source of the 566K subsampling strategy; it is also the prior work showing that caption retrieval helps multilingual captioning.","marker":"Ramos et al., 2024"},{"why":"Provides Pangea, the 7B model and PangeaIns dataset that CONCAP compares against and draws cultural-concept enrichment terms from.","marker":"Yue et al., 2025"},{"why":"Supplies mSigLIP, the multilingual retriever that indexes and queries both caption and concept datastores in the target language.","marker":"Zhai et al., 2023"},{"why":"EVCap demonstrates concept-based retrieval augmentation for captioning, the idea CONCAP extends to the multilingual setting.","marker":"Li et al., 2024a"},{"why":"Defines CIDEr, the metric on which all reported comparisons rest.","marker":"Vedantam et al., 2015"}],"fun_headline_variants":["Captions + concepts retrieval beats 7B models on low-resource languages","Concept-aware RAG closes multilingual captioning gap — no 7B needed","Retrieval that sees concepts: CONCAP hits 34.2 CIDEr on XM3600","Why caption-only retrieval falls short: concepts add 2.8 CIDEr","Concept retrieval rescues captions when target-language pairs run dry"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim stands on the assumption that retrieving single-token concepts from noisy, translation-derived wordlists with a multilingual image-text retriever yields visual grounding that stays useful across languages; if the retriever or the wordlists are too noisy in a target language, concept augmentation adds little and the reported gains would not transfer.","fun_headline_variants_meta":{"raw":{"variants":["Captions + concepts retrieval beats 7B models on low-resource languages","Concept-aware RAG closes multilingual captioning gap — no 7B needed","Retrieval that sees concepts: CONCAP hits 34.2 CIDEr on XM3600","Why caption-only retrieval falls short: concepts add 2.8 CIDEr","Concept retrieval rescues captions when target-language pairs run dry"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000457,"raw_usage":{"total_tokens":2255,"prompt_tokens":866,"completion_tokens":1389,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":482,"completion_tokens_details":{"reasoning_tokens":1285}},"tokens_in":482,"tokens_out":1389,"duration_ms":11236,"temperature":1.0,"reasoning_tokens":1285,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:45:08.289671+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Decisive check: replace the mSigLIP retrieval with random caption and concept sampling and measure CIDEr on XM3600. If CONCAP retains most of its 34.2 score, retrieval is not the cause; if it falls toward the 26.9 no-retrieval level, retrieval is load-bearing. A second check is to identify the evaluation-time caption corpus: if it is the same translated COCO-35L data used for training, compare with a native-language datastore.","supporting_citations":[],"review_version":2}