{"id":"5c60adb0-7f2f-48c6-941f-eb776c05b03c","arxiv_id":"2501.15046","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"CAOS is a framework that uses word-embedding similarities and an ensemble of vision-language models to detect and explain object hallucinations in image captioning.","lead":"This paper introduces CAOS, a set of scores that measure how similar a vision-language model's hallucinated objects are to the real objects in an image, to objects mentioned earlier in the caption, and to commonly seen training objects. It is a tool for comparing and diagnosing why different AI image captioning models invent objects.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Oracle self-inclusion: the four LVLMs that vote on out-of-domain object presence are also the evaluated models, so the hallucination labels and CAOS scores for 4 of 5 models may be biased; a leave-one-out oracle test is required.","rationale":"The reader identified the oracle self-inclusion as the weakest assumption; after reading the full text and Algorithm 1, I agree this is the single most load-bearing concern. The paper's central contribution is a diagnostic framework whose quantitative conclusions (e.g., MiniGPT-4 hallucinates fewer and more contextually related objects, InstructBLIP has high CAOS_K, trade-off between CAOS_T and CAOS_K) all derive from the set H of hallucinated objects. For out-of-domain objects, H is constructed by an oracle that includes the model under evaluation. The manual validation of the oracle uses MultimodalGPT, which is not in the ensemble and thus does not control for the circularity. A 93.43% ensemble accuracy on MultimodalGPT implies a nontrivial error rate; for the four in-ensemble models, the error may be systematically correlated with the model's own caption-generation behavior. Since the vote is by simple majority with ties broken toward 'Absent', one model's vote can be decisive in 2-2 splits, so the label for an object can depend on the very model whose hallucination is being judged. This does not invalidate CAOS as a descriptive metric, but it makes the specific numerical scores and rankings in Table 1 and Figures 3-5 provisional. The paper has independent support: the LLM-augmented object detection is manually evaluated (97% precision, perfect recall), the oracle is manually evaluated on a held-out model, and the prompt-consistency check shows stability across 14 instructions. These strengthen the framework's descriptive value but do not address the self-inclusion. Missing code and error bars are secondary; the oracle test is the check that would settle the main threat. If the leave-one-out results show negligible label flips, the CONDITIONAL verdict could be upgraded; if flips are large, the rankings should be revised. Thus I keep the reader's CONDITIONAL verdict.","tokens_in":13133,"tokens_out":7247,"duration_ms":66829,"concrete_test":"Leave-one-out oracle ablation. For each of InstructBLIP, LLaVA-7B, mPLUG-Owl, and MiniGPT-4, rerun Algorithm 1 on the same 2000-image subset using an oracle that excludes that model (3 members, same tie-breaking), and compare the resulting out-of-domain hallucination labels and all six CAOS scores against Table 1. Report the fraction of label flips and the change in each CAOS score. If any CAOS score shifts by more than 0.02 on average, or the relative ordering of the five models on CAOSavg and the ratio scores changes, the self-inclusion is material; if flips are <1% of out-of-domain objects and scores change negligibly, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim—that CAOS reveals which factors drive hallucination and yields rankings of LVLMs—depends on correctly labeling out-of-domain hallucinated objects. Algorithm 1 (lines 5 and 7) uses an oracle ensemble composed of InstructBLIP, LLaVA-7B, mPLUG-Owl, and MiniGPT-4. These are exactly four of the five models evaluated in Table 1. For each of these models, its own vote participates in deciding whether an object it generated is present in the image. The paper validates the oracle only on MultimodalGPT, which is not in the ensemble, so the 93.43% accuracy figure does not measure the error rate for the in-ensemble models. Because ties are broken in favor of 'Absent', a single extra vote from the evaluated model can flip a 2-1 majority of the other three members into a 2-2 tie resolved as 'Absent', labeling a genuinely present object as hallucinated (or, when the other three are split 2-1 in favor of Present, the model's own 'Present' vote can confirm its own hallucination as genuine). Either way, the hallucination set H is perturbed, and since CAOS_T, CAOS_X, CAOS_K are averages over H, the reported scores and the model comparisons in Figures 3-5 are not independent of the models being scored. The conclusion's limitation statement acknowledges oracle errors 'may occasionally lead to inaccuracies' but does not address the systematic dependence introduced by self-inclusion.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Context-Aware Object Similarities (CAOS), a set of metrics for evaluating object hallucination in large vision-language models (LVLMs). CAOS detects both in-domain and out-of-domain objects in generated captions, labels out-of-domain objects as genuine or hallucinated using an ensemble of LVLMs, and computes the maximum cosine similarity between each hallucinated object and (a) ground-truth objects in the image (CAOS_T), (b) preceding objects in the caption (CAOS_X), and (c) top-k frequent objects in the training set (CAOS_K), together with ratio and average scores. Experiments on 2000 MSCOCO validation images with five LVLMs and two embedding models report these scores, a saturation analysis for k, and a prompt-robustness study. The central claim is that CAOS provides a more nuanced, context-aware characterization of why and when LVLMs hallucinate than existing metrics such as CHAIR and POPE.","tokens_in":13433,"tokens_out":4392,"duration_ms":41237,"significance":"If the method were fully validated, CAOS would be a useful addition to the object-hallucination evaluation toolbox: it explicitly targets out-of-domain hallucinations, which CHAIR and POPE largely miss; it connects hallucination to interpretable semantic factors (image content, caption context, and training-set statistics); and it is evaluated across multiple prompts and embedding models. The paper also reports a manual oracle evaluation on MultimodalGPT and prompt-stability experiments, which are useful checks. However, the current evidence is conditional: the oracle self-inclusion issue, the post hoc choice of k=3, and the absence of uncertainty quantification and baseline comparisons leave the reported scores and model rankings not yet established. The framework is promising but needs methodological repair before its claims can be accepted.","major_comments":[{"comment":"The oracle in Algorithm 1 (lines 5 and 7) is an ensemble of InstructBLIP, LLaVA-7B, mPLUG-Owl, and MiniGPT-4, which are four of the five models whose hallucination scores are reported in Table 1. For each of those four models, the model's own vote participates in deciding whether an out-of-domain object it generated is labeled genuine or hallucinated; with ties broken in favor of 'Absent', the evaluated model's vote can flip a 2-1 majority of the other three members, and in other configurations it can confirm its own hallucination as genuine. The oracle validation reported in the 'Oracle using an ensemble of LVLMs' subsection uses only MultimodalGPT, which is not in the ensemble, so the reported 93.43% accuracy does not estimate the error rate for the in-ensemble models. This systematic dependence perturbs the hallucination set H and therefore all averaged CAOS scores in Table 1 and Figures 3-5. The limitation paragraph in the conclusion mentioning 'rare misdetections' does not address the self-inclusion problem; a leave-one-out oracle or an oracle that excludes the evaluated model is required.","section":"Algorithm 1 / Oracle using an ensemble of LVLMs"},{"comment":"The choice k=3 for CAOS_K is selected post hoc from the same evaluation data, based on the observed 'saturation' in Figure 4. Because CAOS_K and CAOS_avg in Tables 1 and 2 are reported with this fixed k, the fitted parameter is part of the reported outcome rather than an independent design choice. The manuscript should either justify k=3 with a held-out criterion, report the main results across the full range of k shown in Figure 4, or provide a sensitivity analysis demonstrating that the model comparisons in Table 1 are robust to k.","section":"Effect of varying k on CAOSK"},{"comment":"All model comparisons in Table 1 and Figure 3 are single point estimates over 2000 images, with no error bars, confidence intervals, or significance tests. Several differences that support the paper's narrative are small (e.g., CAOS_avg-GloVe is 0.43 for LLaVA, mPLUG-Owl, and MultimodalGPT), and the claims in the Results section about a trade-off between CAOS_T/CAOS_X and CAOS_K need statistical support, such as bootstrap resampling over images or paired tests across captions. The prompt-consistency analysis reports only means across 14 instructions; Figure 6 shows variability but no formal test that the ordering is stable across prompts.","section":"Results / Table 1 and Figure 3"},{"comment":"The CAOS scores are defined as maximum cosine similarities to ground-truth objects, preceding objects, and top-k frequent objects. Because a maximum over a set of embeddings tends to increase with the size of the candidate set, the absolute values and cross-model differences may partly reflect the number of candidates rather than semantic grounding. A baseline such as random embeddings, chance-level similarities, or similarities of non-hallucinated objects is needed to interpret the magnitudes in Table 1; without such a baseline, the claim that a high CAOS_avg indicates hallucinations are 'accounted for by the mentioned factors' is not yet supported.","section":"CAOS: Context-Aware Object Similarities / Algorithm 1"}],"minor_comments":[{"comment":"The prose refers to a metric 'CAOS_T/K', but Algorithm 1 (lines 21-22) computes and names it 'CAOS_X/K'; the notation should be made consistent throughout.","section":"CAOS definitions"},{"comment":"The sentence 'a relatively greater fraction of it's hallucinations are related to the ground-truth objects' contains a typo: 'it's' should be 'its'.","section":"Results section"},{"comment":"The en-dash spacing in score names such as 'CAOS_T–GloVe' is visually inconsistent with the table body and could be misread as a minus; unify the formatting.","section":"Table 1 caption"},{"comment":"The caption states that some scores are 'normalized using multiplication by 2' but does not define the normalization for precision, recall, or the number of objects; please specify the transformation and the radar axes.","section":"Figure 3 caption"},{"comment":"The instructions in Table 3 include trailing spaces and inconsistent punctuation (e.g., items 1 and 2), which can matter for prompt-sensitive LVLMs; the exact prompts used should be reported verbatim without formatting artifacts.","section":"Consistency across different prompt styles"}],"recommendation":"major_revision","confidential_remarks":"The core idea is worth pursuing, but the manuscript currently reports results whose validity is undercut by the oracle self-inclusion and the post hoc choice of k. These are fixable within the scope of the paper, but they require re-running at least part of the evaluation. I would also encourage the authors to release code and data, since a metrics paper benefits from reproducibility; the absence of any release is a limitation in the appendix that should be stated explicitly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the CAOS metric family is a real addition to the LVLM hallucination-evaluation toolkit, and the paper is worth a serious referee. The specific rankings in Table 1 should not be taken at face value until the oracle is decoupled from the evaluated models.\n\nWhat's new and what works: CAOS computes max cosine similarity between a hallucinated object and three reference sets—ground-truth objects (T), preceding objects in the caption (X), and top-k frequent training objects (K). That combination, plus the LLM-augmented detection of out-of-domain objects, is not in CHAIR, POPE, H-POPE, or NOPE. The paper also gives a sensible manual evaluation of the object-detection augmentation: 100 random images, rule-based recall 59%, LLM-augmented recall 100% with 97% precision. That is a concrete, reproducible check. The sequential-analysis idea is clearly motivated, and the CAOS_T/X and CAOS_X/K ratios give a useful way to talk about why hallucinations happen, not just how often.\n\nWhere it gets soft: the oracle that labels out-of-domain objects as Present/Absent is an ensemble of InstructBLIP, LLaVA-7B, mPLUG-Owl, and MiniGPT-4—exactly four of the five evaluated models. For those models, their own vote participates in deciding whether their own generated objects are real. That is not a minor detail. The 93.43% oracle accuracy was measured only on MultimodalGPT, which is not in the ensemble, so it does not cover the models being scored. With ties broken toward Absent, the evaluated model's vote can flip a 2-1 majority into a 2-2 tie that becomes 'Absent', labeling a genuinely present object as hallucinated, or can help confirm its own hallucination as genuine. Either way, the hallucination set H is perturbed and CAOS averages over H—so the scores are partially self-confirming. The paper's limitation paragraph admits oracle errors 'may occasionally lead to inaccuracies' but does not address this systematic dependence.\n\nTwo smaller issues: k=3 for CAOS_K was chosen after seeing the saturation in Figure 4 on the same data, which is post hoc selection. And there are no error bars or significance tests over captions/images, so we cannot tell whether the model differences in Table 1 are meaningful. A random-embedding baseline would also help show that CAOS_K measures what it claims rather than generic embedding proximity.\n\nBottom line: the framework is defensible as a descriptive diagnostic, and the paper is clearly written and honest in its stated limitations. But the reported rankings and the claim that CAOS 'uniquely integrates' statistics with semantics are stronger than the current evidence. This deserves peer review—with a requested leave-one-out oracle test, a random baseline, and significance testing.","headline":"CAOS is a useful new diagnostic for object hallucination, but the oracle's self-inclusion and post hoc k=3 choice make the reported model rankings provisional.","tokens_in":14006,"tokens_out":2094,"would_cite":true,"duration_ms":17813,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Similarity scores trace why vision-language models hallucinate objects.","keywords":["object hallucination","large vision-language models","context-aware object similarities","evaluation metrics","cosine similarity","LLM-augmented object detection","oracle ensemble","caption generation"],"falsifier":"Take a fresh sample of, say, 300 MSCOCO validation images, have human annotators label the presence or absence of every out-of-domain object that the LLM-extraction step proposes, and compare those labels with the ensemble oracle's votes. If the oracle's accuracy falls well below the reported 93.43% on this larger sample, or if its 'Present' votes systematically align with hallucinations produced by models that are members of the ensemble, then CAOS's out-of-domain detection and any conclusions drawn from it would not stand. Excluding each ensemble member from the oracle and re-computing scores for that member would also settle whether oracle self-membership biases the results.","tokens_in":12890,"feed_emoji":"🖼️","tokens_out":7408,"duration_ms":55519,"temperature":0.7,"pith_summary":"The paper proposes Context-Aware Object Similarities (CAOS), a set of evaluation scores for object hallucination in large vision-language models. For every hallucinated object in a generated caption, CAOS measures the maximum cosine similarity of its word embedding to three reference sets: ground-truth objects in the image ($\\mathrm{CAOS}_T$), objects that appeared earlier in the caption plus ground-truth objects ($\\mathrm{CAOS}_X$), and the most frequent objects in the training data ($\\mathrm{CAOS}_K$). Two ratios and an average of these scores summarize whether a model's hallucinations are driven by image content, by its own generated context, or by dataset priors. The paper also uses an LLM to extract out-of-domain objects and an ensemble of LVLMs to verify their presence, extending hallucination evaluation beyond a fixed in-domain vocabulary. If the framework is sound, CAOS supplies a diagnostic that existing metrics like CHAIR and POPE lack: not just how often a model hallucinates, but why.","feed_headline":"Similarity scores trace why vision-language models hallucinate","feed_subtitle":"Three cosine-similarity axes separate hallucinations driven by image content, prior context, and training-data frequency.","key_machinery":"The central machinery is the CAOS score family. For a hallucinated object $h$ with embedding $E(h)$, the three primary scores are $\\mathrm{CAOS}_T = \\max_{o \\in T} \\cos(E(h), E(o))$ over ground-truth objects $T$, $\\mathrm{CAOS}_X = \\max_{o \\in X} \\cos(E(h), E(o))$ over earlier caption objects plus ground-truth objects, and $\\mathrm{CAOS}_K = \\max_{o \\in K} \\cos(E(h), E(o))$ over the top-$k$ frequent training objects. The derived scores $\\mathrm{CAOS}_{T/X} = \\mathrm{CAOS}_T / \\mathrm{CAOS}_X$, $\\mathrm{CAOS}_{X/K} = \\mathrm{CAOS}_X / \\mathrm{CAOS}_K$, and the three-way average $\\mathrm{CAOS}_{\\mathrm{avg}}$ are then computed over all hallucinated objects in a caption. Two embedding models, GloVe and MiniLM-L6, provide the vector representations. The pipeline also rests on LLaMA-2-7B with five-shot in-context learning to identify objects the rule-based parser would miss, and on an ensemble of four LVLMs (InstructBLIP, LLaVA, mPLUG-Owl, MiniGPT-4) that votes 'Present' or 'Absent' to label out-of-domain objects.","core_discovery":"On the paper's own terms, the central discovery is that object hallucinations can be decomposed into three context-aware similarity axes, and that doing so separates the models studied into two groups. InstructBLIP, LLaVA, and mPLUG-Owl show higher $\\mathrm{CAOS}_K$ than $\\mathrm{CAOS}_T$ or $\\mathrm{CAOS}_X$, indicating their hallucinations disproportionately reproduce the most frequent training objects; MiniGPT-4 and Multimodal-GPT show relatively higher $\\mathrm{CAOS}_T$ and $\\mathrm{CAOS}_X$, meaning their errors are more often semantically anchored to objects actually in the image or already mentioned in the caption. The paper further reports that LLM-augmented object extraction reaches perfect recall and 97% precision on a 100-caption manual check, compared with 59% recall for rule-based parsing, and that its ensemble oracle labels out-of-domain object presence correctly in 93.43% of 259 cases. These results are offered as evidence that CAOS can detect hallucinations outside the training vocabulary and rank models by the cause of their errors rather than only by error rate.","pith_inferences":["A direct extension would calibrate $\\mathrm{CAOS}_{T/X}$ and $\\mathrm{CAOS}_{X/K}$ against human severity ratings, since the paper treats high ratios as 'more tolerable' without a human-judgment validation.","Because the oracle ensemble includes four of the five models under evaluation, a clean follow-up would hold out each ensemble member and check whether CAOS scores for that model change materially; the paper does not report this check.","The same machinery could be adapted to attribute, spatial, and relational hallucinations by replacing the reference sets of object nouns with attribute terms or relation tuples.","The saturation of $\\mathrm{CAOS}_K$ at $k=3$ suggests a testable causal claim: a de-biasing intervention targeting top-3 training objects should reduce hallucination rates across all models if frequent-object similarity is a genuine driver."],"forward_implications":["Under CAOS, all five evaluated LVLMs have higher $\\mathrm{CAOS}_K$ than $\\mathrm{CAOS}_T$ and $\\mathrm{CAOS}_X$, so frequent training objects are a dominant hallucination driver across models.","There is an inverse relationship between $\\mathrm{CAOS}_T$/$\\mathrm{CAOS}_X$ and $\\mathrm{CAOS}_K$: models that hallucinate frequent training objects more also hallucinate image- and context-grounded objects less, which implies distinct hallucination mechanisms.","Out-of-domain hallucinated objects can be detected and labeled, and they too show measurable semantic influence from the most frequent MSCOCO objects, so restricting evaluation to in-domain vocabulary understates the problem.","CAOS scores are largely stable across 14 instruction prompts, indicating the model ordering is not caused by a single prompt.","MiniGPT-4 shows competitive precision, POPE-F1, and high $\\mathrm{CAOS}_{T/X}$, suggesting that when it does hallucinate, its errors tend to be semantically tied to the image's actual contents."],"supporting_citations":[{"why":"Provides the CHAIR metrics and rule-based object parsing that CAOS extends, and appears as the baseline for hallucination evaluation.","marker":"Rohrbach et al. 2018"},{"why":"Introduces POPE and the 2000-image MSCOCO validation subset that the paper reuses; the main object-statistics baseline CAOS compares against.","marker":"Li et al. 2023"},{"why":"Supplies the MSCOCO dataset, its ground-truth object labels, and the object frequency statistics used for CAOS_K.","marker":"Lin et al. 2014"},{"why":"Provides GloVe word embeddings, one of the two embedding models used to compute the cosine similarities in CAOS.","marker":"Pennington, Socher, and Manning 2014"},{"why":"Provides MiniLM-L6 embeddings, the second embedding model used in the CAOS calculations.","marker":"Wang et al. 2020"},{"why":"Supplies LLaMA-2-7B, the LLM used with five-shot in-context learning for augmented object identification.","marker":"Touvron et al. 2023b"},{"why":"Provides InstructBLIP, which is both an oracle ensemble member and one of the evaluated LVLMs.","marker":"Dai et al. 2024"},{"why":"Provides LLaVA, used as an oracle member and an evaluated model whose captions are studied.","marker":"Liu et al. 2024a"},{"why":"Provides mPLUG-Owl, an oracle ensemble member and an evaluated LVLM.","marker":"Ye et al. 2023"},{"why":"Provides MiniGPT-4, an oracle ensemble member and the model that shows relatively context-grounded hallucinations.","marker":"Zhu et al. 2023"}],"fun_headline_variants":["Three similarity axes trace why LVLMs hallucinate objects","CAOS ranks hallucination causes, not just error rates","Context-aware metric splits vision models by hallucination type","New method detects out-of-vocabulary object hallucinations","Similarity decomposition explains object hallucination in LVLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the oracle ensemble of LVLMs correctly decides whether an out-of-domain object is present or absent in the query image; if the ensemble frequently confirms as 'present' an object that is actually absent, hallucinated objects get mislabeled as genuine and the resulting CAOS scores are distorted. The paper itself acknowledges that errors in the rule-based parser, the LLM, or the oracle can lead to inaccuracies.","fun_headline_variants_meta":{"raw":{"variants":["Three similarity axes trace why LVLMs hallucinate objects","CAOS ranks hallucination causes, not just error rates","Context-aware metric splits vision models by hallucination type","New method detects out-of-vocabulary object hallucinations","Similarity decomposition explains object hallucination in LVLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000231,"raw_usage":{"total_tokens":1556,"prompt_tokens":1083,"completion_tokens":473,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":699,"completion_tokens_details":{"reasoning_tokens":394}},"tokens_in":699,"tokens_out":473,"duration_ms":4993,"temperature":1.0,"reasoning_tokens":394,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T14:40:14.658222+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a fresh sample of, say, 300 MSCOCO validation images, have human annotators label the presence or absence of every out-of-domain object that the LLM-extraction step proposes, and compare those labels with the ensemble oracle's votes. If the oracle's accuracy falls well below the reported 93.43% on this larger sample, or if its 'Present' votes systematically align with hallucinations produced by models that are members of the ensemble, then CAOS's out-of-domain detection and any conclusions drawn from it would not stand. Excluding each ensemble member from the oracle and re-computing scores for that member would also settle whether oracle self-membership biases the results.","supporting_citations":[],"review_version":1}