{"id":"02b22296-0397-4cda-adaa-ad63a7d84cff","arxiv_id":"2412.15484","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"CapMAS improves factual accuracy of long image captions by atomic claim decomposition and verification, and the paper adds a multimodal factuality metric and a coverage QA benchmark.","lead":"This paper introduces CapMAS, a plug-and-play system that uses a language model and a vision-language model to split detailed captions into small claims, verify each against the image, and rewrite the caption to remove false details. It also contributes a GPT-4o-based factuality metric and a 19,899-question coverage benchmark, and shows that VQA leaderboard rankings do not predict detailed captioning quality.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The factuality metric and the fact-checker in CapMAS share the same GPT-4o decomposition mechanism; if that metric is unreliable on hyper-detailed captions, the central claim of significant factuality improvement is not independently established.","rationale":"The reader's weakest assumption identifies the same load-bearing point: the primary outcome metric is validated only on a small, filtered human study and shares its decomposition mechanism with the method being evaluated. I agree this is the most load-bearing concern. I do not see an internal inconsistency in the method as described; rather, the risk is external validity and independence of the evaluation. The paper would be strengthened by a larger human study on the actual corrected captions, public release of exact π values, and error bars or variance estimates. The central claim could still hold even if the metric is imperfect, but with current evidence the superiority claim is conditional on the metric's reliability. Therefore the CONDITIONAL verdict is appropriate; REJECT is not warranted because there is no demonstrated internal flaw and the proposed check could confirm the claim.","tokens_in":20452,"tokens_out":1412,"duration_ms":11774,"concrete_test":"Run a blind human preference study on N ≥ 100 matched caption pairs (generated caption vs. CapMAS-corrected caption) across the models in Table 4, asking annotators to judge which caption is more factual and which contains more hallucinations. Then compute the correlation between human rankings and the paper's GPT-4o factuality metric on these pairs, and report whether CapMAS's advantage holds under human judgment. If human judgments confirm the improvement and the metric correlates well (e.g., Spearman rho > 0.6) on these pairs, the central claim stands.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that CapMAS significantly improves factuality of hyper-detailed captions, measured by the paper's own GPT-4o-based factuality metric. That metric decomposes captions into atomic propositions and judges each proposition against the image and reference caption. CapMAS uses the same decomposition-plus-verification mechanism, with the MLLM fact-checker scoring propositions. The metric's validation is limited: only 100 DOCCI images, with caption pairs of similar factuality excluded (Appendix A), and the only CapMAS hyperparameter π is tuned on five examples (Section 4.1). If GPT-4o's proposition decomposition or truth judgments are biased toward the style or content of corrected captions, the measured gains could be inflated without reflecting genuine hallucination reduction. The meta-evaluation in Table 2 uses only synthetic hallucination injections and shows the Clean caption score is only 62.8, so the metric's absolute scores are not calibrated. A human evaluation on the actual corrected captions is needed to confirm that the reported 70.2 Spearman correlation transfers to the setting where CapMAS's improvement is claimed.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CapMAS, a training-free multiagent pipeline that decomposes a hyper-detailed image caption into atomic propositions, verifies each proposition with an MLLM against the image, and uses an LLM to rewrite the caption so that only verified content is retained. The paper also introduces a factuality evaluation metric built on GPT-4o that decomposes captions into atomic propositions and judges each against the image and a reference caption, and a coverage benchmark of multiple-choice questions derived from IIW-400 with human-validated answers. Experiments claim that CapMAS improves factuality across several MLLMs including GPT-4V, that the proposed metric correlates better with human judgments than FaithScore and FactScore, that existing hallucination mitigation methods underperform on detailed captions, and that VQA-centric benchmark rankings do not predict detailed captioning ability.","tokens_in":20649,"tokens_out":8331,"duration_ms":60396,"significance":"If the results hold, the paper would make three useful contributions: a plug-and-play, training-free correction method that appears to reduce hallucination in long captions; a dual factuality/coverage evaluation framework with a new human-annotated QA benchmark; and evidence that VQA-centric benchmarking is insufficient for assessing detailed captioning. The code and data are released, and the metric is validated with human judgments, which is a concrete step beyond purely model-based evaluation. The main caveat is that the validation is narrowly scoped (100 filtered DOCCI images for the metric, five examples for the threshold), so the headline improvements rest on a metric whose behavior on corrected captions has not been independently checked; this limits the current significance but is addressable with additional human evaluation.","major_comments":[{"comment":"The central claim that CapMAS significantly enhances factuality is measured entirely with the proposed GPT-4o factuality metric, yet the metric's human validation (Table 3, Appendix A) is based on only 100 DOCCI images from which caption pairs of similar factuality were excluded, and it compares only original LLaVA-v1.5-7B vs InstructBLIP captions. No human evaluation is reported for original vs CapMAS-corrected captions. Because both CapMAS and the metric rely on atomic-proposition decomposition and truth verification, the observed gains in Table 4 (e.g., LLaVA-NeXT-7B factuality from 59.9 to 72.2) could be inflated by GPT-4o favoring the style or content of corrected captions rather than by a genuine reduction in hallucination. Please provide a human evaluation on a sample of corrected vs original captions, or otherwise demonstrate that the metric's judgments transfer to the correction setting.","section":"Section 4.3 / Table 4 / Appendix A"},{"comment":"The paper uses the word \"significantly\" repeatedly, but no confidence intervals, standard errors, or hypothesis tests are reported. The hyperparameter π is tuned on only five examples (Section 4.1), and the Spearman correlations in Table 3 (70.2 vs 67.9 for FactScore) may not be statistically distinguishable given the small filtered sample. Please report bootstrap confidence intervals and significance tests for the correlation differences and for the factuality/coverage improvements in Tables 4 and 5.","section":"Section 4.1 / Section 4.2 / Tables 3-6"},{"comment":"The comparison with VisualFactChecker (VFC) is not a direct experiment. The text states that VFC could not be reproduced and that the authors \"infer\" superiority from two design differences: VFC targets only object hallucinations and uses an object detector, whose detection AUROC is lower than the Isolation method in Table 1. This inference conflates component-level hallucination detection with end-to-end caption quality and is insufficient to support the claim that CapMAS outperforms VFC. Please either include VFC results if resources become available, or restrict the claim to a design-level qualitative statement.","section":"Section 4.4"},{"comment":"The meta-evaluation reveals that the proposed metric assigns a factuality score of 62.8 to clean, human-written DOCCI captions. This means GPT-4o considers approximately 37% of atomic propositions in human captions to be false, indicating poor absolute calibration. Since Table 4 reports absolute factuality gains, it is important to discuss this calibration issue and to consider reporting relative improvements or a calibration-corrected score, especially because the gains could partly reflect changes in caption style rather than a reduction of hallucinations.","section":"Section 3.3 / Table 2"},{"comment":"The claim that VQA-centric benchmark performance \"may not correlate\" with detailed captioning ability is based on a visual comparison of ranks across only eight models, with no correlation coefficient or statistical test reported. Please compute a rank correlation (e.g., Spearman) between the detailed captioning average and OpenCompass or individual VQA benchmarks, and report its value and significance; otherwise, soften the claim to a qualitative counterexample.","section":"Section 4.5 / Table 6"}],"minor_comments":[{"comment":"The hallucination score formula uses min(p(T|x,Q(u)) - p(F|x,Q(u)), epsilon). Since p(T)-p(F) can be negative, taking the min with a small positive epsilon can produce a negative argument to the log. It appears that max(p(T)-p(F), epsilon) was intended to avoid taking the log of a non-positive number. Please correct or clarify.","section":"Section 3.2 / Eq. (1)"},{"comment":"The claim that hallucinations after the 192nd token are undetectable is based on visual inspection of the plotted means without error bars. Please report the dataset size, the number of hallucination/non-hallucination objects per bin, and consider adding confidence bands.","section":"Section 3.1 / Figure 2"},{"comment":"The text says an LLM answers the multiple-choice questions using only the captions, but the model and the prompt for this coverage evaluation step are not specified in Section 4.1 or in the prompt templates of Appendix G. Please specify the model and provide the prompt template.","section":"Section 3.3 / Coverage"},{"comment":"The sentence \"We adopt LLaVA-v1.5-7B, LLaVA-NeXT-7B, LLaVA-NeXT-13B, InternVL-Chat-V1.5, and GPT-4V as the models for both captioning and CapMAS's fact-checking\" is ambiguous, because Table 4 also uses different MLLMs as fact-checkers for GPT-4V captions. Please clarify which models are used for each role.","section":"Section 4.1"},{"comment":"The methods are evaluated on different base models (LLaVA-v1.5-7B for VCD/OPERA/SPARC, MiniGPT-4 for LURE/LRV), which confounds the comparison. Please acknowledge this limitation explicitly or report all methods on a common base model.","section":"Section 4.4 / Table 5"},{"comment":"The paper contains typographical and formatting issues, including \"V olcano\" with an extra space in Section 4.4 and Table 5, and inconsistent capitalization of \"LLaVA\" and \"FactScore/FaithScore\". Please proofread and standardize the notation.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a vision-language venue, and the multiagent correction idea plus the coverage benchmark are useful contributions. However, the validation gaps—small filtered human set, threshold tuning on five examples, no human check on corrected captions, and lack of statistical reporting—currently prevent the central improvement claim from being fully established. I would urge the editor to require a human evaluation on corrected captions and proper confidence intervals before publication. The indirect VFC comparison and the calibration issue with the factuality metric should also be addressed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What should you know? This is a solid, honest empirical paper about hallucination correction and evaluation for hyper-detailed image captions. The genuinely new pieces: CapMAS, a training-free LLM-MLLM correction pipeline; a factuality metric that decomposes captions into atomic propositions and checks them against both image and reference; and a 19,899-question coverage benchmark built from human-refined GPT-4o questions. Code and data are public. The paper also demonstrates something useful: existing hallucination detection degrades sharply for captions longer than ~190 tokens, and VQA-style benchmarks do not predict detailed captioning quality.\n\nWhat it does well: the motivating analysis is concrete, the baseline comparisons are reasonable, and the authors are transparent about limitations. They tune the only hyperparameter pi on five examples (Section 4.1) and validate the metric on 100 DOCCI images, excluding pairs of similar factuality (Appendix A). Those limitations are disclosed, not hidden. The metric is calibrated against human judgments on a small dataset, and the reported Spearman rho (70.2) beats FaithScore and FactScore. The coverage benchmark is a real resource; the QA-based coverage evaluation makes sense.\n\nSoft spots, in proportion: the factuality metric and CapMAS's fact-checker both rely on atomic-proposition decomposition, so the main evaluation is not fully independent of the method's mechanism. This is mitigated because the evaluator (GPT-4o) differs from the fact-checker MLLMs used inside CapMAS, and the metric has human-validated correlation, but the shared decomposition remains a conceptual weakness. The metric's absolute scores are not calibrated: in Table 2, the Clean caption gets only 62.8, meaning GPT-4o rejects a third of true atomic propositions. That does not disqualify the metric for relative comparisons, but it matters when interpreting absolute factuality gains. The VFC comparison is inferential, not direct, because VFC is not reproducible. Tables lack error bars, and the coverage loss with weaker fact-checkers is significant (e.g., ~2.7 points when LLaVA-NeXT-7B checks GPT-4V captions in Table 4). None of these are load-bearing flaws; they are caveats.\n\nWho it is for: people working on MLLM hallucination, caption evaluation, and accessibility. It deserves a serious referee; a good review would ask for a larger human study on actual corrected captions and a more direct comparison with VFC, but the core contribution is worth engaging.","headline":"Solid empirical paper with a useful training-free caption correction method and a coverage benchmark; the main caveat is that the factuality metric and the method share a decomposition mechanism, but the metric has independent human validation and the limitations are honestly disclosed.","tokens_in":21194,"tokens_out":2472,"would_cite":true,"duration_ms":20023,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Hyper-detailed image captions can be made more factual by splitting them into atomic claims, verifying each against the image, and rewriting the caption to keep only the verified claims.","keywords":["hyper-detailed image captioning","hallucination detection","multiagent system","atomic proposition verification","factuality evaluation","coverage evaluation","multimodal large language models","VQA benchmark evaluation"],"falsifier":"Take a fresh set of several hundred detailed captions from multiple MLLMs, have human annotators label every atomic proposition as true or false, and compute the proposed metric's per-caption scores from those labels without excluding close pairs. If the Spearman correlation with human preferences drops to the level of FaithScore or FactScore, or if proposition-level agreement with humans is poor, the central evaluation claim fails; the same experiment would also show whether the threshold $\\pi$ generalizes beyond five tuning examples.","tokens_in":20249,"feed_emoji":"🖼️","tokens_out":6905,"duration_ms":53527,"temperature":0.7,"pith_summary":"The paper argues that hallucinations in long, hyper-detailed image captions are less a decoding problem than a detection problem: once a caption runs hundreds of tokens, standard hallucination scores stop working and the model drifts from the image. Its proposed system, CapMAS, has an LLM decompose a caption into atomic propositions, an MLLM fact-check each proposition against the image, and an LLM rewrite the caption from only the verified propositions. The paper also introduces a factuality metric that uses GPT-4o to decompose captions and judge each claim against both the image and a reference caption, plus a coverage benchmark of nearly 20,000 human-refined multiple-choice questions answered from the caption alone. It reports that this metric matches human factuality judgments better than FaithScore and FactScore, and that CapMAS improves factuality for every captioner tested, including GPT-4V, while sacrificing little coverage. A separate finding is that rankings on VQA benchmarks do not predict detailed-captioning ability, which calls into question VQA-centric model evaluation.","feed_headline":"Multiagent pipeline lifts caption factuality, even for GPT-4V","feed_subtitle":"Decomposing captions into checkable claims and verifying each one beats existing factuality metrics.","key_machinery":"The load-bearing object is the atomic proposition, a claim that must be either true or false. CapMAS's pipeline surrounds it with three agents: a decomposer LLM that splits a caption into propositions, a fact-checker MLLM that converts each proposition into a 'True or False?' question and reads hallucination from the probability gap between the True and False tokens, and a corrector LLM that removes unverified content. The threshold $\\pi$ controls a loose factuality-coverage trade-off, with smaller $\\pi$ giving stricter fact-checking and lower coverage. For evaluation, the paper adds a GPT-4o-based factuality score that reuses the same decomposition-plus-verification logic with both image and reference caption, and a coverage score computed by having an LLM answer human-refined multiple-choice questions using only the caption.","core_discovery":"The central discovery is that decomposing a long caption into atomic propositions and verifying each proposition in isolation exposes hallucinations that token-confidence and consistency methods miss, especially after the 192nd token. CapMAS operationalizes this with an LLM decomposer, an MLLM fact-checker that scores each unit by $-\\log(\\min(p(\\text{T}|x,Q(u))-p(\\text{F}|x,Q(u)),\\epsilon))$, a threshold $\\pi$ that separates true from false units, and an LLM corrector that rewrites the caption using only the true units. The same decomposition idea drives the evaluation metric, where GPT-4o judges atomic propositions against both image and reference; the paper reports a Spearman correlation of 70.2 with human factuality preferences, versus 62.5 for FaithScore and 67.9 for FactScore. The paper further claims that decoding-based and training-based hallucination fixes validated on VQA can fail or even backfire on detailed captions, while CapMAS improves factuality for LLaVA, InternVL, and GPT-4V.","pith_inferences":["The same decompose-verify-rewrite loop could be applied to other long multimodal generations, such as video descriptions or dense scene graphs, where the latter half of the output is the most hallucination-prone.","A direct testable extension is to measure whether caption-only accuracy on the proposed coverage benchmark predicts downstream usability for blind users, connecting coverage scores to the accessibility outcome the paper motivates but does not measure.","Because the factuality metric's judge is GPT-4o, the method is tied to a closed model; an obvious extension is to train a smaller open verifier on the paper's atomic-proposition labels and check whether the correlation with human judgments is preserved.","The paper's dissociation between VQA ranking and captioning ranking suggests that benchmark suites should add long-form generation tasks; one could test this by checking whether composite rankings that include captioning scores predict human preference better than VQA-only rankings."],"forward_implications":["Factuality fixes validated on short-answer VQA tasks should be re-audited on long-form captioning; the paper shows that applying VCD actually reduces factuality there.","CapMAS can be dropped onto any captioning model without training, including a closed model like GPT-4V, and it raises factuality while holding coverage roughly constant.","Model leaderboards built from VQA benchmarks should not be read as captioning ability; Idefics2 and MiniCPM-V rank well on VQA but poorly on detailed captioning relative to GPT-4V.","Factuality and coverage are separable evaluation axes with a loose trade-off controlled by one threshold, so a caption can be fully factual and still miss most of the image's content.","The proposed factuality metric should be preferred over reference-only or image-only metrics because it combines both sources and avoids the stylistic bias of reference-only scoring."],"supporting_citations":[{"why":"Supplies the DOCCI images and human-annotated detailed captions used for the meta-evaluation of factuality metrics and for the human correlation study.","marker":"Onoe et al., 2024"},{"why":"Supplies the IIW-400 hyper-detailed reference captions used to build the coverage QA benchmark and to expose reference-only metric bias.","marker":"Garg et al., 2024"},{"why":"FaithScore is the closest baseline the proposed factuality metric must beat in the human-correlation comparison.","marker":"Jing et al., 2024"},{"why":"FactScore is the reference-only factuality baseline compared against in both the correlation and bias experiments.","marker":"Min et al., 2023"},{"why":"VCD is the decoding-based hallucination mitigation baseline that the paper shows fails on hyper-detailed captions.","marker":"Leng et al., 2024"},{"why":"OPERA is the attention-based decoding baseline compared in the detailed-captioning experiments.","marker":"Huang et al., 2024"},{"why":"LURE is the corrector-based baseline, and its Confidence method is one of the late-hallucination detectors the paper analyzes.","marker":"Zhou et al., 2024"},{"why":"VisualFactChecker is the prior object-hallucination-focused revision pipeline the paper contrasts with CapMAS and whose summarization prompt it adapts.","marker":"Ge et al., 2024"}],"fun_headline_variants":["Fact-check captions claim by claim, beating existing metrics","Decompose captions into claims to improve factuality over GPT-4V","New evaluation metric aligns better with human judgments on factuality","Long captions verified claim-by-claim: multiagent fix for hallucinations"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's headline results assume that GPT-4o can decompose any caption into atomic propositions and judge each one correctly against the image and a reference caption; this was validated only on 100 DOCCI images with caption pairs of similar factuality excluded, and the CapMAS threshold $\\pi$ was tuned on five examples.","fun_headline_variants_meta":{"raw":{"variants":["Fact-check captions claim by claim, beating existing metrics","Decompose captions into claims to improve factuality over GPT-4V","New evaluation metric aligns better with human judgments on factuality","Long captions verified claim-by-claim: multiagent fix for hallucinations"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000281,"raw_usage":{"total_tokens":1684,"prompt_tokens":982,"completion_tokens":702,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":598,"completion_tokens_details":{"reasoning_tokens":627}},"tokens_in":598,"tokens_out":702,"duration_ms":6337,"temperature":1.0,"reasoning_tokens":627,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:23:04.587732+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a fresh set of several hundred detailed captions from multiple MLLMs, have human annotators label every atomic proposition as true or false, and compute the proposed metric's per-caption scores from those labels without excluding close pairs. If the Spearman correlation with human preferences drops to the level of FaithScore or FactScore, or if proposition-level agreement with humans is poor, the central evaluation claim fails; the same experiment would also show whether the threshold $\\pi$ generalizes beyond five tuning examples.","supporting_citations":[{"cited_title":"Analyzing and mitigating object hallucination in large vision-language models","cited_arxiv_id":null,"evidence_quote":"LURE is the corrector-based baseline, and its Confidence method is one of the late-hallucination detectors the paper analyzes."}],"review_version":1}