Pith. sign in

REVIEW 4 major objections 6 minor 36 references

CHIRP: A Fine-Grained Benchmark for Open-Ended Response Evaluation in Vision-Language Models

T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Standard VLM benchmarks give similarly scoring models statistically indistinguishable scores; CHIRP, a 104-question open-ended pairwise benchmark, separates them and reveals clear language-model scaling that those benchmarks hide.

desk verdict CHIRP is a useful benchmark and the Robin suite is a solid empirical contribution, but the LLM-scaling signal is likely confounded by response fluency and the statistical support is thinner than the claims. read the letter →

arxiv 2501.09672 v3 pith:6PTZFORR submitted 2025-01-16 cs.CV cs.AI

Alexis Roger , Prateek Humane , Daniel Z. Kaplan , Kshitij Gupta , Qi Sun , George Adamopoulos , Jonathan Siu Chi Lim , Quentin Anthony
show 2 more authors
Edwin Fennell Irina Rish
This is my paper · ORCID
classification cs.CVcs.AI
keywords vision-languagemodelsbenchmarkdesignopen-endedevaluationpairwisepreferenceEloratingscalinglawsLLMasjudgehuman
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that standard vision-language benchmarks grade short, right-or-wrong answers so coarsely that models of genuinely different quality end up with statistically indistinguishable scores, and introduces CHIRP to measure what those benchmarks miss. CHIRP is a benchmark of 104 open-ended questions (with generated images and no fixed correct answers) on which two models' responses are compared side by side by human raters or a VLM judge across five criteria, and the preferences are turned into bootstrapped Elo ratings. Evaluated on Robin, a suite of 20 models varying only in language-model size and vision-encoder size, CHIRP shows what the existing benchmarks do not: clear improvement as the language model grows, a strictly rising human-preference trend with vision-encoder size, and an optimal vision-to-language size ratio. The paper also traces why older benchmarks are blind, estimating that about 9% of sampled GQA questions and 5% of sampled TextVQA questions have faulty or ambiguous ground truths, and showing that switching to longer responses changes which questions a model gets right. The sympathetic reading is that human-centered, open-ended, pairwise evaluation is a more sensitive tool for telling models apart, and that AI judges such as GPT-4V can reproduce its overall trends at lower cost.

What carries the argument

The carrying object is CHIRP itself: a 104-question, eight-category benchmark of open-ended image-question pairs whose images were generated rather than scraped, so models cannot have memorized them, graded by pairwise preference, one evaluator compares two models' responses per question on overall preference, relevance and completeness, understanding and reasoning, hallucinations, and details, and the results are collapsed into Elo ratings through 500 bootstrap iterations. The pairing is essential: it converts evaluation from absolute accuracy against a fixed answer into a relative judgment, which is where size-dependent differences appear. Two supporting machines carry the argument: the Robin suite of 20 LLaVA-style models with independently varied language-model and vision-encoder sizes, which makes controlled single-parameter comparisons possible; and a set of diagnostic experiments (long-versus-short response scoring, LLM and VLM grading against human grades, and manual auditing of ground-truth quality) that attribute the insensitivity of older benchmarks to response length, grader rigidity, and faulty ground truths.

What would settle it

A direct check would be to run the dense version of the human study: recruit a fresh panel of raters to judge all 190 pairwise matchups of the 20 Robin models on all 104 CHIRP questions, several raters per matchup, and see whether the clear LLM-size scaling and the vision-encoder sweet spot still appear in the bootstrapped Elo rankings; if the ordering shifts materially, the headline result is an artifact of the sparse five-matchup sampling. A complementary check is to re-run the GQA and TextVQA comparisons after deleting the roughly 9% and 5% of questions flagged as faulty, since if the LLM-size trend still fails to appear on the cleaned sets, the faulty-ground-truth explanation for benchmark insensitivity would be weakened.

Watch

Extended reading notes

Core claim

On the paper's terms, the discovery is that the way a benchmark is designed, whether it asks short factual answers matched against a ground truth, is what hides real differences between vision-language models, not the underlying model itself. Using the Robin suite of 20 models in which only the language model (Pythia 410M to 12B) or the vision encoder (CLIP Base to ViT-g) changes, the authors find that standard benchmarks show no clear vision-encoder trend and only a weak language-model trend, while their own CHIRP benchmark, which asks 104 open-ended questions and records pairwise human and VLM preferences on five criteria, shows a clear language-model scaling effect in every evaluation. Human preferences on CHIRP also rise monotonically with vision-encoder size and expose a preferred ratio between vision and language scale. The paper attributes the gap to three design flaws in existing benchmarks, short responses that under-determine quality, graders too rigid to accept correct phrasings, and ground truths that are themselves wrong or ambiguous, and it measures the last of these directly: in a 100-question sample from GQA, 9% of questions had incorrect ground truths (95% confidence interval 3.4% to 14.6%), which is large relative to the typical few-percent gap between competing models.

Load-bearing premise

The central claim rests on the assumption that preference judgments collected from a modest pool of crowd-sourced English-speaking raters, each seeing a random subset of model pairings on 104 questions, yield Elo rankings stable enough to reveal true differences between models, and that 100-question samples are representative enough to estimate the error rates of GQA and TextVQA, while the paper's own Limitations section acknowledges the deliberately small size and the strong reliance on evaluator language proficiency.

Editorial extensions

If this is right

  • Two models that score within roughly 3% of each other on GQA, or 0.7% on TextVQA, are statistically indistinguishable on those benchmarks, so small leaderboard gaps there should not be read as real capability differences.
  • Long-form and short-form responses draw on different skills, since the same model answers a different set of questions correctly in each mode, so benchmarks must be deliberately designed for the response format they claim to measure.
  • A GPT-4V judge that reasons before choosing reproduces the overall CHIRP trends seen with human raters and correlates with training loss (distance correlation 0.96 versus 0.91 for humans), making AI-based evaluation a viable low-cost substitute for broad trend detection, though per-case agreement with humans is only slight to fair.
  • Human preferences on CHIRP reveal an optimal vision-encoder-to-language-model ratio for each language-model size, a pattern the standard benchmarks did not surface.
  • Performance on CHIRP tracks training loss more closely than performance on the other benchmarks tested, which the paper takes as evidence that CHIRP measures a distinct, human-valued skill that existing suites do not cover.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If CHIRP's sensitivity is real, the same pairwise-preference machinery could be extended into a continuously updated public VLM leaderboard in which any two models are matched on a rotating pool of open-ended questions, since CHIRP already shows that Elo ratings from sparse matchups correlate with the model property, training loss, that scaling research cares about.
  • The paper's error audit method, sampling 100 questions and hand-checking ground truths, transfers directly to any other harvested VQA dataset, and applying it to newer benchmarks would immediately show whether the multi-percent noise floor is widespread.
  • Because near-chance agreement (48.7%) between GPT-4V's preferences and token log-probabilities indicates the judge responds to image content rather than fluency, a natural next test is whether CHIRP rankings predict which model a user prefers in a genuinely interactive setting, such as asking follow-up questions about an image.
  • Because every CHIRP image was generated for the benchmark, the dataset cannot be contaminated by training exposure today, but this also means its questions may not reflect the distribution of natural images; a natural-image twin of CHIRP would show whether the extra sensitivity survives outside the generated domain.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces CHIRP, a 104-question open-ended vision-language benchmark scored by pairwise preference judgments from humans or VLMs, together with Robin, a suite of 20 LLaVA-style models that systematically vary vision-encoder and LLM sizes. The authors claim that standard benchmarks (ScienceQA, GQA, VQAv2, TextVQA, MM-Vet, LLaVA-Bench) fail to reveal scaling trends in model quality, while CHIRP shows a clear LLM-size trend. They support this with heatmaps, Elo ratings over bootstrapped pairwise preferences, an audit of GQA/TextVQA ground-truth errors, and an analysis of AI-judge agreement with human raters.

Significance. If the central claim holds, CHIRP would be a valuable complement to existing VLM benchmarks, and the Robin suite would provide a controlled testbed for studying scale effects in open-ended response quality. The paper's strengths include the public release of training code, model suite, benchmark, and hand-validated images; the use of bootstrap resampling for Elo scores; and the concrete audit of ground-truth errors in GQA and TextVQA. However, the current evidence base is too thin and too confounded to establish the central claim: the human-preference samples are small, no significance tests are reported, and the LLM-size ablation is not controlled for response length or fluency.

major comments (4)
  1. [§5.1, Figure 5] The central claim that CHIRP reveals LLM-size scaling rests on small, sparse preference samples with no inferential statistics. The LLM-size study consists of 104 questions × 5 matchups = 520 human judgments and the VE-size study of 312 judgments, with matchups randomly sampled per question; the Elo trajectories in Figure 5 are therefore averages over non-independent, sparse data. I ask for bootstrap confidence intervals on Elo differences between adjacent sizes, a permutation or sign test for monotonicity, and a statement of the minimum effect size detectable at this sample size. Without such tests, the visual contrast between the CHIRP heatmaps and the existing-benchmark heatmaps in Figure 2 does not establish that CHIRP reliably discriminates scales.
  2. [§6.3, Appendix C.2.7] The LLM-size trend is confounded with response length and fluency. In the LLM-size ablation the vision encoder is fixed and only the Pythia LLM varies; larger Pythia models typically produce longer, more grammatical, and more fluent text, and the paper itself cites Wu & Aji (2023) for the finding that human and LLM evaluators prefer fluent but flawed content. The only related check, Appendix C.2.7, tests whether GPT-4V (R) preferences agree with token log-probabilities (48.7%, near chance); this does not control for length, lexical diversity, or grammaticality, and it does not apply to human raters. I ask for a matched-response analysis, such as truncating responses to equal length or including length and fluency as covariates in a preference model, before interpreting the CHIRP trend as evidence of improved image-grounded understanding.
  3. [§6.2.1, Table 1; §6.3] The claim that AI evaluations can serve as proxies for human evaluations is not supported by the reported agreement statistics. Cohen's Kappa values of 0.10, 0.114, 0.204, and 0.216 fall into the 'slight' to 'fair' range, yet the text states that GPT-4V evaluations 'still exhibit very similar trends to human evaluations.' Low instance-level agreement can coincide with similar aggregate trends by construction, so I ask for a formal trend-level test: report bootstrap distributions of the Elo trajectory for each evaluator and test whether the human and AI trajectories have the same shape, rather than relying on visual similarity.
  4. [§5.2.3] The extrapolation from 9/100 incorrect GQA ground truths to '770,769 to 3,309,773 questions' uses a normal-approximation confidence interval (3.4%–14.6%) that is very wide and assumes a simple random sample. The subsequent statement that two SoTA models scoring within 3% 'could very well be equal' extrapolates from this fragile estimate and is used to motivate CHIRP. I ask for either a larger audit sample (e.g., 500 questions) or a substantially softened conclusion, since the current interval does not support a tight bound on the error rate.
minor comments (6)
  1. [Abstract and §1] The abstract in the paper body differs from the metadata abstract and contains typos such as 'langauge' and 'identifiying'; unify the two versions and proofread the text.
  2. [§3.4 and Figure 5 caption] The number of Elo bootstrap iterations is reported as 500 in §3.4 but as '1000 samples' in the Figure 5 caption; make the numbers consistent.
  3. [Appendix B.3] The GPT-4 evaluation prompt states that the model should respond only with 'Correct' or 'Incorrect', but the surrounding text allows 'Partially Correct'; clarify how partial-credit responses are scored.
  4. [Appendix B.3 and Figure 20/21] The pseudocode contains several typos ('asssisstant', 'correspondig', 'exmaple') and inconsistent capitalization; clean up the prompts before publication.
  5. [Figure 22 caption] The caption says the first row concerns the VE-size ablation and the second row the LLM-size ablation, while Figure 5 presents the top row as LLM size and the bottom row as VE size; align the captions to avoid confusion.
  6. [§5.2.3] The displayed confidence-interval formula 'p∈p±z∗√(p∗(1−p))/n' should be written as a standard interval for a proportion, and the source of the population total 22,669,678 should be stated explicitly.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: CHIRP is an empirical benchmark and evaluation study; Elo ratings and scaling trends are measured outcomes, not derived from fitted inputs or self-citations.

full rationale

I walked the paper's claimed derivation chain. The paper does not derive any target quantity from fitted parameters or from a self-citation chain. CHIRP's Elo ratings are computed from pairwise human and VLM preferences via standard Elo and bootstrapping; the claimed LLM-scaling trend is a descriptive summary of those preferences, not a quantity defined to match them. The only author-overlap citation is Pythia (Biderman et al. 2023, co-authored by Quentin Anthony), used as the public base LLM suite for the Robin models; this is code-reproduced external work and is not invoked as an unverified premise that forces CHIRP's conclusions. The comparisons against GQA and TextVQA use independently sampled questions and human grading; the logits-agreement check in Appendix C.2.7 is a confound test rather than a circular reduction. Potential concerns such as response-fluency confounding or the small pool of human raters are validity and robustness issues, not instances where an output is equivalent to an input by construction. Accordingly, no circular steps are identified.

Assumptions & free parameters 0 free parameters · 4 assumptions · 1 invented entities

The ledger is nearly empty because the paper is an empirical construction rather than a derivation. The benchmark itself is an invented artifact with independent usability. Main axioms are domain assumptions about human evaluators and sample representativeness; no free parameters are fit to make the central claims.

assumptions (4)
  • domain assumption Human preference judgments are a reliable ground truth for evaluating open-ended responses.
    The entire CHIRP framework treats human preferences as the gold standard; the paper acknowledges in Limitations that this relies on evaluator language proficiency. Section 3.2.
  • domain assumption The 100-question samples from GQA and TextVQA are representative of the full datasets.
    The paper extrapolates error rates from a sample of 100 questions to millions, assuming representativeness. Section 5.2.3.
  • domain assumption The five evaluation criteria (overall, relevance, understanding, hallucination, detail) capture the relevant dimensions of response quality.
    Criteria chosen based on 'empirical evidence' not shown; they structure the benchmark. Section 3.2.
  • standard math Bootstrapped Elo ratings provide stable rankings given sparse pairwise data.
    The paper uses 500-1000 bootstrap iterations to estimate Elo scores, relying on standard statistical resampling. Section 3.4.
invented entities (1)
  • CHIRP benchmark independent evidence
    purpose: Evaluate long-form responses of VLMs via pairwise preference comparisons
    The benchmark is released publicly (though via an anonymous link) and can be used to evaluate any VLM, providing a falsifiable handle for model evaluation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CHIRP: A Fine-Grained Benchmark for Open-Ended Response Evaluation in Vision-Language Models." pith.science (2026). https://pith.science/paper/6PTZFORR

@misc{pith2026250109672,
  author       = {Pith},
  title        = {Pith review of: CHIRP: A Fine-Grained Benchmark for Open-Ended Response Evaluation in Vision-Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6PTZFORR}},
  note         = {Machine review of arXiv:2501.09672}
}
read the original abstract

The proliferation of Vision-Language Models (VLMs) in the past several years calls for rigorous and comprehensive evaluation methods and benchmarks. This work analyzes existing VLM evaluation techniques, including automated metrics, AI-based assessments, and human evaluations across diverse tasks. We first introduce Robin - a novel suite of VLMs that we built by combining Large Language Models (LLMs) and Vision Encoders (VEs) at multiple scales, and use Robin to identify shortcomings of current evaluation approaches across scales. Next, to overcome the identified limitations, we introduce CHIRP - a new long form response benchmark we developed for more robust and complete VLM evaluation. We provide open access to the Robin training code, model suite, and CHIRP benchmark to promote reproducibility and advance VLM research.

Figures

Figures reproduced from arXiv: 2501.09672 by the authors.

Figure 1
Figure 1. Experimental Setup Pipeline. Detailed pipeline used to evaluate CHIRP: from the left we have the models trained with the 3 different ablations, then in the middle we evaluate said models on different benchmarks, including CHIRP, and finally we compare the model rankings obtained on the different benchmarks. By running experiments across these Robin models, we aim to demonstrate CHIRP’s ability to capture subtle diff… view at source ↗
Figure 2
Figure 2. All Robin model performance on existing benchmarks vs CHIRP. Left. Heatmap showing the scaled average score detailed in Section 4. Center. Elo calculated from GPT-4V (R) on CHIRP. Right. Elo calculated from human survey on CHIRP. 5.2 Investigating differences between CHIRP and other benchmarks Empirically testing our models we saw that model performance aligned more closely to the trends indicated using CHIRPs evalu… view at source ↗
Figure 3
Figure 3. Accuracy of long vs short responses on GQA sample for [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (21 more)
Figure 4
Figure 4. Figure 4: Left. Evaluating GQA and textVQA across the LLM size and VE size ablations using automated string matching programs, LLMs, VLMs, and humans evaluators. Solid lines are evaluations of responses where models were prompted for short responses. Dashed lines have no such in…
Figure 5
Figure 5. Figure 5: Mean Elo calculated over LLM Size (top row) and VE Size (bottom row) using different evaluators [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Percentage of CHIRP questions graded with a contradiction of preferences within a specific criteria. [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: Comparison of loss scaling and evaluation scores for the Robin models. [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: Log-log plots showing the scaling laws with [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: Heatmaps showing the performance of the different models of the scaling suite on the different [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 10
Figure 10. Figure 10: Heatmaps showing the performance of the different models of the scaling suite on the different [PITH_FULL_IMAGE:figures/full_fig_p019_10.png]
Figure 11
Figure 11. Figure 11: Accuracy of the Robin suite of models on the 100 GQA question sample calculated using different [PITH_FULL_IMAGE:figures/full_fig_p020_11.png]
Figure 12
Figure 12. Figure 12: Accuracy of the Robin suite of models on the 100 TextVQA question sample calculated using [PITH_FULL_IMAGE:figures/full_fig_p020_12.png]
Figure 13
Figure 13. Figure 13: Confusion matrices of the different evaluation methods on the LLaVA1.5-7B responses. [PITH_FULL_IMAGE:figures/full_fig_p021_13.png]
Figure 14
Figure 14. Figure 14: Accuracy of different evaluation methods on a sample of 100 questions from both GQA and [PITH_FULL_IMAGE:figures/full_fig_p022_14.png]
Figure 15
Figure 15. Figure 15: Prompt passed to GPT-4 for the LLM evaluation of both long and short responses on GQA and [PITH_FULL_IMAGE:figures/full_fig_p023_15.png]
Figure 16
Figure 16. Figure 16: Prompt used for LLaVA-34B evaluation of both long and short responses on GQA and textVQA. [PITH_FULL_IMAGE:figures/full_fig_p023_16.png]
Figure 17
Figure 17. Figure 17: Prompt used for GPT-4V evaluation of long responses on GQA and textVQA. We first asked [PITH_FULL_IMAGE:figures/full_fig_p024_17.png]
Figure 18
Figure 18. Figure 18: CHIRP single question category distribution. [PITH_FULL_IMAGE:figures/full_fig_p031_18.png]
Figure 19
Figure 19. Figure 19: Example of survey questions displayed to a human evaluator on Cloud Research. The same [PITH_FULL_IMAGE:figures/full_fig_p033_19.png]
Figure 20
Figure 20. Figure 20: Prompt used for LLaVA-34B evaluation of model responses on the CHIRP benchmark. [PITH_FULL_IMAGE:figures/full_fig_p034_20.png]
Figure 21
Figure 21. Figure 21: Prompts used for GPT-4V evaluation of model responses on the CHIRP benchmark. The [PITH_FULL_IMAGE:figures/full_fig_p035_21.png]
Figure 22
Figure 22. Figure 22: Elo scores calculated over LLM size and VE size on the 5 different evaluation criteria of CHIRP [PITH_FULL_IMAGE:figures/full_fig_p036_22.png]
Figure 23
Figure 23. Figure 23: Visualization of model preferences over multiple human evaluators for a single question. Arrows [PITH_FULL_IMAGE:figures/full_fig_p038_23.png]
Figure 24
Figure 24. Figure 24: Visualization of model preferences over multiple human evaluators for the CHIRP benchmark. [PITH_FULL_IMAGE:figures/full_fig_p039_24.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

36 extracted references · 20 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Scaling laws for generative mixed-modal language models

    Armen Aghajanyan, Lili Yu, Alexis Conneau, Wei-Ning Hsu, Karen Hambardzumyan, Susan Zhang, Stephen Roller, Naman Goyal, Omer Levy, and Luke Zettlemoyer. Scaling laws for generative mixed-modal language models. In International Conference on Machine Learning, pp.\ 265--279. PMLR, 2023

    2023

  3. [3]

    Lawrence Zitnick, Dhruv Batra, and Devi Parikh

    Aishwarya Agrawal, Jiasen Lu, Stanislaw Antol, Margaret Mitchell C. Lawrence Zitnick, Dhruv Batra, and Devi Parikh. VQA: Visual Question Answering , 2016

    work page 2016

  4. [4]

    Pythia: A suite for analyzing large language models across training and scaling

    Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling . In International Conference on Machine Learning, pp.\ 2397--2430. PMLR, 2023

    work page 2023

  5. [5]

    Pali: A jointly-scaled multilingual language-image model

    Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, et al. Pali: A jointly-scaled multilingual language-image model. arXiv preprint arXiv:2209.06794, 2022

    arXiv 2022

  6. [6]

    Reproducible scaling laws for contrastive language-image learning

    Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning . In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 2818--2829, 2023

    work page 2023

  7. [7]

    A coefficient of agreement for nominal scales

    Jacob Cohen. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20 0 (1): 0 37--46, 1960. doi:10.1177/001316446002000104. URL https://doi.org/10.1177/001316446002000104

    doi:10.1177/001316446002000104 1960

  8. [8]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answering

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering . In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 6904--6913, 2017

    work page 2017

Show all 36 references
  1. [9]

    Sugarcrepe: Fixing hackable benchmarks for vision-language compositionality

    Cheng-Yu Hsieh, Jieyu Zhang, Zixian Ma, Aniruddha Kembhavi, and Ranjay Krishna. Sugarcrepe: Fixing hackable benchmarks for vision-language compositionality. Advances in Neural Information Processing Systems, 36, 2024

    2024

  2. [10]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-Rank Adaptation of Large Language Models , 2021

    2021

  3. [11]

    Hudson and Christopher D

    Drew A. Hudson and Christopher D. Manning. GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering , 2019

    2019

  4. [12]

    Scaling laws for downstream task performance of large language models

    Berivan Isik, Natalia Ponomareva, Hussein Hazimeh, Dimitris Paparas, Sergei Vassilvitskii, and Sanmi Koyejo. Scaling laws for downstream task performance of large language models. arXiv preprint arXiv:2402.04177, 2024

    2024

  5. [13]

    Large language models as automated aligners for benchmarking vision-language models

    Yuanfeng Ji, Chongjian Ge, Weikai Kong, Enze Xie, Zhengying Liu, Zhengguo Li, and Ping Luo. Large language models as automated aligners for benchmarking vision-language models. arXiv preprint arXiv:2311.14580, 2023

    2023 arXiv

  6. [14]

    Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling Laws for Neural Language Models , 2020

    2020

  7. [15]

    BENCHMARKING COGNITIVE BIASES IN LARGE LANGUAGE MODELS AS EVALUATORS , 2023

    Ryan Koo, Minhwa Lee, Vipul Raheja, Jong Inn Park, Zae Myung Kim, and Dongyeop Kang. BENCHMARKING COGNITIVE BIASES IN LARGE LANGUAGE MODELS AS EVALUATORS , 2023. Available on arXiv: https://arxiv.org/abs/2309.17012

    2023 arXiv

  8. [16]

    Shamma, Michael S

    Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Fei-Fei Li. Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations , 2016

    2016

  9. [17]

    Richard Landis and Gary G

    J. Richard Landis and Gary G. Koch. The measurement of observer agreement for categorical data. Biometrics, 33 0 (1): 0 159--174, 1977. ISSN 0006341X, 15410420. doi:10.2307/2529310. URL http://www.jstor.org/stable/2529310

    1977

  10. [18]

    Prometheus-vision: Vision-language model as a judge for fine-grained evaluation

    Seongyun Lee, Seungone Kim, Sue Hyun Park, Geewook Kim, and Minjoon Seo. Prometheus-vision: Vision-language model as a judge for fine-grained evaluation. arXiv preprint arXiv:2401.06591, 2024

    2024 arXiv

  11. [19]

    Hashimoto

    Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-following models. https://github.com/tatsu-lab/alpaca_eval, 2023 a

    2023

  12. [20]

    Monkey: Image resolution and text label are important things for large multi-modal models

    Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. Monkey: Image resolution and text label are important things for large multi-modal models. arXiv preprint arXiv:2311.06607, 2023 b

    2023 arXiv

  13. [21]

    Lawrence Zitnick, and Piotr Dollár

    Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. Microsoft COCO: Common Objects in Context , 2015

    2015

  14. [22]

    Improved Baselines with Visual Instruction Tuning , 2023 a

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved Baselines with Visual Instruction Tuning , 2023 a

    2023

  15. [23]

    Visual Instruction Tuning , 2023 b

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual Instruction Tuning , 2023 b

    2023

  16. [24]

    Learn to explain: Multimodal reasoning via thought chains for science question answering

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering . Advances in Neural Information Processing Systems, 35: 0 250...

    2022

  17. [25]

    OCR-VQA: Visual Question Answering by Reading Text in Images

    Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. OCR-VQA: Visual Question Answering by Reading Text in Images . In 2019 International Conference on Document Analysis and Recognition (ICDAR), pp.\ 947--952, 2019. doi:10.1109/ICDAR.2019.00156

    2019

  18. [26]

    Gpt-4v(ision) system card

    OpenAI. Gpt-4v(ision) system card. https://cdn.openai.com/papers/GPTV\_System\_Card.pdf, 2023

    2023

  19. [27]

    Openai model documentation

    OpenAI. Openai model documentation. https://platform.openai.com/docs/models, 2024

    2024

  20. [28]

    Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever

    Alec Radford, Jong Wook Kim, Chris Hallacy, A. Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision . In ICML, 2021

    2021

  21. [29]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024

    2024

  22. [30]

    Revisiting the train loss: an efficient performance estimator for neural architecture search, 06 2020

    Binxin Ru, Clare Lyle, Lisa Schut, Mark Wilk, and Yarin Gal. Revisiting the train loss: an efficient performance estimator for neural architecture search, 06 2020

    2020

  23. [31]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 8317--8326, 2019

    2019

  24. [32]

    Generative multimodal models are in-context learners

    Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Zhengxiong Luo, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, et al. Generative multimodal models are in-context learners. arXiv preprint arXiv:2312.13286, 2023

    2023 arXiv

  25. [33]

    Székely, Maria L

    Gábor J. Székely, Maria L. Rizzo, and Nail K. Bakirov. Measuring and testing dependence by correlation of distances. The Annals of Statistics, 35 0 (6), December 2007. ISSN 0090-5364. doi:10.1214/009053607000000505. URL http://dx.doi.org/10.1214/009053607000000505

    2007 doi

  26. [34]

    Style Over Substance: Evaluation Biases for Large Language Models , 2023

    Minghao Wu and Alham Fikri Aji. Style Over Substance: Evaluation Biases for Large Language Models , 2023. Available on arXiv: https://arxiv.org/abs/2307.03025

    2023 arXiv

  27. [35]

    MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities , 2023

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities , 2023

    2023

  28. [36]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena , 2023

    2023

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.