{"id":"d7c989ef-68ff-4d78-8780-a66eea8a2051","arxiv_id":"1909.02050","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"TIGEr is a caption evaluation metric that uses a pre-trained text-to-image grounding model to compare candidate captions with human references via region rank and weight distribution similarity.","lead":"This paper introduces TIGEr, an automatic metric for judging image captions by comparing how strongly captions are grounded in image regions against human-written references. It reports higher correlation with human judgments than standard metrics such as BLEU, CIDEr, and SPICE on three caption evaluation datasets.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported gains may reflect test-set tuning of τ in Eq. 13 and post hoc averaging in Eq. 14, not the metric itself; no metric-vs-metric significance tests support the headline.","rationale":"The reader's weakest assumption is that SCAN grounding vectors faithfully reflect semantic relevance. I agree that is a genuine limitation, and the paper's Figure 7 and Appendix A.2 support it. But I do not think it is the most load-bearing threat to the central claim. If SCAN noise corrupts grounding, it lowers TIGEr's ceiling and makes the claimed improvements harder to obtain; it does not explain why the reported numbers are as high as they are. Protocol degrees of freedom do explain that: τ is tunable, the aggregation in Eq. 14 was chosen after observing RRS/WDS variance, the tie rule in Section 4.3 deviates from prior convention in a direction that favors a continuous metric, and no significance tests compare TIGEr with baselines. The central claim is an empirical superiority claim, and superiority claims need a protocol that excludes selection. The reader's conditional verdict is the right level: the concern is addressable by fixing τ or disclosing its selection, pre-specifying the aggregation and tie rule, and reporting bootstrap intervals. The released code and standard datasets are helpful, and the paper honestly states limitations, but code availability does not reveal which hyperparameters were tried. Since my analysis reinforces the conditional verdict rather than moving it, I recommend no change.","tokens_in":11357,"tokens_out":7706,"duration_ms":72491,"concrete_test":"Recompute the headline numbers with a pre-registered protocol: set τ=1 (or select τ on a disjoint validation split of each dataset, never on the test pairs), fix Eq. 14 to equal weights before seeing results, and count PASCAL-50S ties both as incorrect and as correct (Anderson et al.'s convention); then compare TIGEr with the best baseline via 10,000 bootstrap resamples over candidate pairs, reporting 95% confidence intervals for the difference. If TIGEr does not exceed the best baseline outside the interval on at least two of the three datasets, the claimed advantage is not established. Also check whether the 2007 Composite MS-COCO images appear in SCAN's train/val split; if they do, rerun Composite on non-overlapping images.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline claim is that TIGEr has higher consistency with human judgments than existing metrics. That claim depends on the evaluation protocol being a fair, pre-specified estimator of metric quality. The protocol is not fully specified. Equation 13 defines WDS with a temperature τ described only as 'to-be-tuned' (Section 3.2.2); no search range, selection criterion, or fixed value is reported. If τ was chosen on the same test instances that produce Tables 1 and 2, the gains can be inflated by selection. Equation 14 averages RRS and WDS, and Section 4.3 says the arithmetic mean was selected 'by empirically observing the value variance between RRS and WDS in the [0,1] interval', which is a post hoc modeling choice made after seeing data. In PASCAL-50S, ties are counted as incorrect (Section 4.3), which penalizes discrete-valued baselines such as SPICE while the continuous TIGEr score rarely ties. No confidence intervals or metric-vs-metric significance tests are reported, so the 2.72-point aggregate advantage over METEOR in Table 2 is not shown to be outside sampling noise. The paper's own Appendix A.2 concedes that TIGEr is limited with semantically close objects and imaginative captions, and Figure 7 shows weakness on action differences; these limitations do not refute the headline by themselves, but they make it important to exclude protocol artifacts before claiming broad superiority. SCAN noise is secondary: it would tend to weaken TIGEr uniformly, whereas these degrees of freedom can specifically manufacture the observed advantage.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TIGEr, a new automatic metric for image caption evaluation. TIGEr computes text-to-image grounding vectors for a candidate caption and for human references using a pre-trained SCAN model, then compares the vectors using two scores: RRS, a normalized discounted cumulative gain that compares the ranking of image regions by grounding relevance, and WDS, a sigmoid-mapped combination of KL divergence between grounding softmax distributions and a log-ratio of vector norms. The final score is the arithmetic mean of RRS and WDS. The authors evaluate TIGEr against BLEU, METEOR, ROUGE-L, CIDEr, and SPICE on three datasets: Composite, Flickr8K, and PASCAL-50S, reporting correlations with human scores (Kendall's tau and Spearman's rho) on the first two and pairwise accuracy on PASCAL-50S. They claim that TIGEr achieves higher consistency with human judgments than existing metrics, particularly for machine-machine caption pairs and when few references are available. Code is released.","tokens_in":11701,"tokens_out":5850,"duration_ms":47174,"significance":"If the empirical claims hold, TIGEr offers a useful new direction for caption evaluation by incorporating image content through an external grounding model, moving beyond pure text-matching. The decomposition into ranking-based and distribution-based similarities is interpretable, and the component analysis (Section 4.6) gives insight into which caption aspects the metric captures. The release of code and the analysis of reference-number sensitivity are strengths. However, the headline result depends on an evaluation protocol with undisclosed free parameters and no significance testing, so the significance is conditional on those issues being resolved.","major_comments":[{"comment":"The temperature τ is described only as 'to-be-tuned,' and the paper never reports its value, its search range, or the criterion used to select it. If τ was chosen by optimizing agreement with human judgments on the test sets behind Tables 1 and 2, the reported gains (e.g., TIGEr's 80.70% vs. METEOR's 77.98% in Table 2) could be inflated by test-set selection. Please disclose the exact τ used for each dataset, the selection procedure (e.g., a held-out validation split or cross-validation), and a sensitivity analysis over τ.","section":"3.2.2, Eq. (13)"},{"comment":"The arithmetic mean of RRS and WDS is justified by 'empirically observing the value variance between RRS and WDS in the [0,1] interval,' which is a post hoc choice made after inspecting the test data. The paper does not report whether alternative combination weights (e.g., weighted averages) would change the conclusions. Please select the combination weight on a validation set or provide evidence that the ranking of metrics is stable across a range of weights.","section":"3.2.3, Eq. (14)"},{"comment":"The paper reports no confidence intervals or significance tests for the differences between TIGEr and the baselines. For instance, the 2.72-point aggregate accuracy advantage over METEOR in Table 2, and the τ differences in Table 1 (TIGEr 0.454 vs. SPICE 0.419 on Composite), are not assessed for statistical significance. Please report bootstrap confidence intervals for the correlation coefficients and paired significance tests (e.g., McNemar's test for pairwise accuracy) to support the claim that TIGEr has 'higher consistency with human judgments.'","section":"Tables 1 and 2"},{"comment":"The protocol counts equally-scored candidate pairs as incorrect, whereas Anderson et al. (2016) counted them as correct. This choice systematically penalizes discrete-valued baselines like SPICE (which can produce ties) while TIGEr's continuous score rarely ties. The reported accuracy differences on PASCAL-50S, especially the 2.72% aggregate gain, may be partly attributable to this tie-handling difference. Please report results under both conventions or justify the chosen one with a sensitivity analysis.","section":"Section 4.3"},{"comment":"The grounding model SCAN is pre-trained on MS-COCO, and the Composite test set draws images from MS-COCO; meanwhile the paper's own component analysis shows TIGEr is weak on action differences, and Appendix A.2 concedes difficulty with semantically close objects and imaginative captions. As a result, the abstract's unqualified claim of 'higher consistency with human judgments' should be scoped to the evaluated domain and caption types, or supported by out-of-domain results that show the metric retains its advantage.","section":"Sections 3.1, 4.6, Appendix A.2"}],"minor_comments":[{"comment":"The phrase 'higher consistency with human judgments' should be qualified with 'on the datasets and caption types tested,' given the limitations disclosed in Appendix A.2.","section":"Abstract and Conclusion"},{"comment":"The Composite dataset description lists 2007 MS-COCO images, 997 Flickr 8k pictures, and 991 Flickr 30k images, but does not explain how these map to the 11,985 candidates; please clarify the sampling.","section":"Section 4.1"},{"comment":"The statement 'All p-values < 0.01' is ambiguous; specify which test produced these p-values and whether they are corrected for multiple comparisons.","section":"Table 1"},{"comment":"The reference 'Aditya et al., 2015a' appears twice with identical content, and 'Karpathy and Li, 2015' and 'Karpathy and Fei-Fei, 2015' refer to the same CVPR paper; duplicate references should be merged.","section":"References"},{"comment":"The figure legend is embedded in the caption; adding a legend inside the plot or directly labeling lines would improve readability.","section":"Figure 5"},{"comment":"The mapping from continuous TIGEr scores to score groups is described only vaguely; please specify the binning procedure (e.g., quantiles vs. equal-width) since the error analysis depends on it.","section":"Appendix A.1"},{"comment":"There are minor typos: 'Flick8K' in Section 4.4 should be 'Flickr8K', and 'matching-generated captions' in Section 4.1 should be 'machine-generated captions'.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper's core idea is sound and the code availability is commendable. However, the evaluation protocol has several degrees of freedom that are not disclosed, and the reported gains may be selection artifacts. I would encourage the editor to request a revised version that addresses the protocol issues before considering acceptance. There is also a duplicate reference that should be cleaned up."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper proposes a new image caption evaluation metric, TIGEr, that uses a pre-trained text-image grounding model (SCAN) to compare a candidate caption and reference captions via region-level grounding vectors. It's a genuine new composition, not just borrowed pieces, and the empirical study is broader than most metric papers: three datasets, multiple baselines, component analysis, and released code. I'd send it to review, but the authors need to tighten the evaluation protocol before I'd trust the numbers.\n\nWhat's actually new: instead of n-gram matching or end-to-end learned scoring, TIGEr takes the grounding vectors produced by SCAN and compares them with NDCG (region rank) and a KL-divergence + norm-ratio term called WDS. That combination is new, and the idea of making the metric interpretable by showing which regions ground which words is a real plus. The qualitative examples and the sensitivity analysis over object/action/property distinctions are useful and honest.\n\nThe soft spots are all in the experimental protocol. Equation 13 has a temperature tau described only as 'to-be-tuned'; no value, no search range, no selection criterion is reported. If tau was picked on the same test data that produced Tables 1 and 2, the reported gains are partly selection artifacts. The arithmetic mean in Equation 14 was chosen 'by empirically observing the value variance between RRS and WDS in the [0,1] interval' — that is post hoc on the evaluation data. In PASCAL-50S, ties are counted as incorrect, which systematically hurts discrete metrics like SPICE that tie often, while TIGEr's continuous score rarely ties. There are no significance tests or confidence intervals for any of the correlation or accuracy differences, so a 0.454 vs 0.419 Kendall tau difference is not shown to be real. And SCAN was pretrained on COCO train/val, while Composite contains COCO test images, so part of the advantage on that dataset could come from seeing the same distribution. These are not fatal, but they are enough to make the headline 'higher consistency with human judgments' premature without a controlled re-run.\n\nI'd like to see a version where tau is fixed on a validation split (or shown not to matter), the averaging and tie rules are preregistered or justified, and differences are reported with bootstrap intervals. The central idea is sound and the code is out, so this deserves referee time, not a desk reject.","headline":"TIGEr is a plausible and interpretable new grounding-based caption metric, but its reported gains are weakened by undisclosed tuning of tau, post hoc averaging, and missing significance tests.","tokens_in":12196,"tokens_out":2331,"would_cite":false,"duration_ms":24579,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"TIGEr: a caption metric that checks the image as well as the text","keywords":["image captioning evaluation","text-to-image grounding","semantic similarity","region rank similarity","weight distribution similarity","human judgment correlation","SCAN"],"falsifier":"Build a test set of images paired with reference captions and two candidate captions that differ only by an action verb (e.g., 'walk' vs. 'run' for the same scene), and score them with human raters and TIGEr. If TIGEr fails to prefer the human-chosen caption at rates well above chance on such action pairs, the paper's claim that TIGEr has higher consistency with human judgments would be undermined. A cheaper check: compute TIGEr's caption-level correlation with human scores on a held-out set of non-photo images (drawings or abstract scenes), where SCAN grounding was not trained.","tokens_in":11196,"feed_emoji":"🖼️","tokens_out":4894,"duration_ms":46813,"temperature":0.7,"pith_summary":"The paper proposes TIGEr, an automatic metric for judging image captions that goes beyond word overlap with reference captions. It argues that existing metrics such as BLEU and CIDEr are biased because references only partially cover an image and natural language is ambiguous. TIGEr instead uses a pre-trained text-to-image grounding model to map both the candidate caption and human references onto image regions, then scores the candidate by how similarly its grounding pattern ranks and weights those regions. Across three benchmark datasets, the reported correlations with human judgments are higher and more stable than those of six existing metrics, with the largest gains on machine-generated caption pairs and when only a few references are available. The paper frames TIGEr as interpretable: the comparison is defined by two explicit similarity measures rather than a black-box scorer.","feed_headline":"Image-aware caption metric beats text-only scores","feed_subtitle":"TIGEr compares how captions and human references attend to image regions, matching human judgments better across three benchmarks.","key_machinery":"The load-bearing object is the grounding vector $s(V,C)$, one scalar per image region, produced by a pre-trained Stacked Cross Attention Network (SCAN): word embeddings from an RNN attend over 36 bottom-up region features, and each region's grounding score is the cosine similarity between the region vector and the caption's attention-weighted feature. The comparison stage rests on two identities applied to these vectors: RRS treats the grounding scores as a relevance ranking and applies normalized discounted cumulative gain; WDS converts the vectors to attention distributions and combines KL divergence with a log-ratio of vector magnitudes. Averaging the two yields TIGEr. The machinery replaces exact n-gram matching with semantic region-level matching, which is what allows the metric to catch captions that reuse reference words but describe the wrong image content.","core_discovery":"TIGEr's central claim is that a caption should be evaluated by whether it attends to the image the way human references do, not by whether it shares words with them. For a candidate caption $C$, an image $V$, and references $R$, the score averages two comparisons of grounding vectors $s(V,C)$ and $s(V,R)$: Region Rank Similarity (RRS), a normalized discounted cumulative gain over image regions ordered by grounding strength, and Weight Distribution Similarity (WDS), a softmax-normalized KL divergence augmented by a log-ratio of grounding-vector norms. The result is a score in $[0,1]$. On the Composite, Flickr 8K, and PASCAL-50S datasets, the paper reports that TIGEr's caption-level correlations with human scores exceed those of BLEU-1/4, ROUGE-L, METEOR, CIDEr, and SPICE, and that its pairwise accuracy on PASCAL-50S is highest overall, improving machine-machine pair accuracy by about ten absolute points over the best prior metric while being less sensitive to the number of references.","pith_inferences":["The grounding-model dependence cuts both ways: replacing SCAN with a stronger grounding model that handles actions and attributes should directly raise TIGEr's ceiling, a testable extension the paper itself points toward.","One could use TIGEr as a reward or loss during caption training, but that invites metric gaming, and whether TIGEr stays reliable under optimization is an open question the paper does not test.","The same region-level comparison could be adapted to other image-to-text tasks such as visual storytelling or text-to-image generation evaluation, where reference coverage is even more incomplete.","A useful stress test would be evaluating TIGEr on abstract or imaginative captions, which the appendix identifies as hard, to see where human-TIGEr agreement breaks down."],"forward_implications":["If TIGEr's reported correlations hold, captioning systems can be ranked without relying on many reference captions, since performance stays stable as references drop from 50 to 1.","The metric's strength on machine-machine pairs means it can discriminate between two generated captions for the same image, the setting where text-overlap metrics are weakest.","Because TIGEr detects object-level mismatches better than action-level ones, its use would encourage systems to get objects right while remaining less sensitive to verb changes.","The interpretable decomposition into rank and distribution similarity gives developers a diagnostic: a low RRS means attention is on the wrong regions, a low WDS means the attention profile is too concentrated or diffuse.","Existing text-only metrics could be augmented with TIGEr as a second signal, making evaluation of open-domain captioning less dependent on reference coverage."],"supporting_citations":[{"why":"Supplies the pre-trained SCAN text-to-image grounding model that produces the region-level grounding vectors on which both RRS and WDS are computed.","marker":"(Lee et al., 2018)"},{"why":"Provides the Composite dataset of human-graded candidate captions used to measure TIGEr's correlation with human judgments.","marker":"(Aditya et al., 2015a)"},{"why":"Provides the Flickr 8K dataset and its human relevance scores, the second caption-level correlation benchmark.","marker":"(Hodosh et al., 2013)"},{"why":"Provides the PASCAL-50S pairwise human judgments and defines the CIDEr baseline that TIGEr is compared against.","marker":"(Vedantam et al., 2015)"},{"why":"Provides the SPICE baseline and the evaluation protocol for measuring metric accuracy on PASCAL-50S.","marker":"(Anderson et al., 2016)"},{"why":"Defines BLEU, one of the rule-based text-matching baselines TIGEr must outperform.","marker":"(Papineni et al., 2002)"},{"why":"Defines METEOR, a baseline metric that was previously the strongest on several TIGEr comparison settings.","marker":"(Banerjee and Lavie, 2005)"},{"why":"Defines ROUGE, a recall-oriented n-gram overlap baseline included in the comparison.","marker":"(Lin, 2004)"}],"fun_headline_variants":["Image-aware captions get a sharper score","TIGEr judges captions by image grounding","Caption evaluation that sees the picture","Beyond text: New metric uses image grounding"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"TIGEr's scores are only as trustworthy as the pre-trained SCAN grounding model, which was trained on MS-COCO photos; if that model does not accurately ground captions in an evaluation image, both the region ranking and weight-distribution comparisons inherit the error.","fun_headline_variants_meta":{"raw":{"variants":["Image-aware captions get a sharper score","TIGEr judges captions by image grounding","Caption evaluation that sees the picture","Beyond text: New metric uses image grounding"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000511,"raw_usage":{"total_tokens":2462,"prompt_tokens":901,"completion_tokens":1561,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":517,"completion_tokens_details":{"reasoning_tokens":1507}},"tokens_in":517,"tokens_out":1561,"duration_ms":12734,"temperature":1.0,"reasoning_tokens":1507,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T05:00:49.002927+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a test set of images paired with reference captions and two candidate captions that differ only by an action verb (e.g., 'walk' vs. 'run' for the same scene), and score them with human raters and TIGEr. If TIGEr fails to prefer the human-chosen caption at rates well above chance on such action pairs, the paper's claim that TIGEr has higher consistency with human judgments would be undermined. A cheaper check: compute TIGEr's caption-level correlation with human scores on a held-out set of non-photo images (drawings or abstract scenes), where SCAN grounding was not trained.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the SPICE baseline and the evaluation protocol for measuring metric accuracy on PASCAL-50S."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines METEOR, a baseline metric that was previously the strongest on several TIGEr comparison settings."}],"review_version":1}