VLR-Bench: Multilingual Benchmark Dataset for Vision-Language Retrieval Augmented Generation

Dongjae Shin; Haneol Jang; HyeonSeok Lim; InHo Won; Junghun Yuk; KyungTae Lim; Minjun Kim; Seohyun Song

REVIEW 4 major objections 6 minor 16 references

VLR-Bench: Multilingual Benchmark Dataset for Vision-Language Retrieval Augmented Generation

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper introduces a five-passage benchmark for vision-language RAG and a training set that improves passage selection.

desk verdict A genuinely useful small benchmark with a five-passage design that deserves referee time, but the paper's central claim about VLR-IF's effect on passage selection lacks a format-matched control. read the letter →

arxiv 2412.10151 v1 pith:7X5TGFYZ submitted 2024-12-13 cs.CV cs.AIcs.CL

Hyeonseok Lim , Dongjae Shin , Seohyun Song , Inho Won , Minjun Kim , Junghun Yuk , Haneol Jang , KyungTae Lim This is my paper · ORCID

classification cs.CVcs.AIcs.CL

keywords visualquestionansweringretrievalaugmentedgenerationbenchmarkdatasetpassageselectionmultilingualevaluationinstructionfollowingexternalknowledgevision-languagemodels

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes VLR-Bench, a 300-sample multilingual visual question answering benchmark for retrieval-augmented generation, in which each query arrives with five passages: two gold, two silver, and one bronze, so that only two of the five contain information needed to answer. It argues that this tests a capability missing from earlier external-knowledge VQA datasets, which provide a single gold passage: deciding which retrieved passage is actually useful. The paper also presents VLR-IF, 32,000 automatically generated instruction-following examples per language, built by pairing valid passages with invalid passages sampled from other examples, and reports that fine-tuning on VLR-IF improves both passage selection and answer quality. The benchmark is constructed as a parallel corpus in English, Chinese, and Korean, with human review of questions, passages, answers, and keywords.

What carries the argument

The load-bearing object is the five-passage structure: each VLR-Bench item contains two gold passages that directly support the answer, two silver passages that are topic-related but not central, and one bronze passage that is unrelated, together with two manually selected keywords used for keyword-matching scoring alongside ROUGE, BLEU, and BERTScore. For training, VLR-IF is built by taking an image, generating one valid passage, and randomly drawing invalid passages from other examples to create four contrastive patterns: valid only, invalid only, valid plus one invalid, and valid plus two invalid. This contrastive design is what teaches the model both to use the useful passage and to refuse when no useful passage is present.

What would settle it

Run a controlled comparison where the same image-question pairs are answered with no passages but with explicit instructions to produce an equally long, descriptive answer; if the gap between the with-passages and without-passages conditions collapses, the benchmark is measuring response-format effects rather than a need for external knowledge.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that passage selection for RAG can be isolated as a measurable skill and improved by training. In an English passage-selection task, fine-tuning on VLR-IF raises exact match from 2.0 to 28.0 and F1 from 15.9 to 41.0, and the same training lifts performance on the InfoSeek benchmark from 42.9 to 44.5. Correlation analysis shows that gold passages overlap with ground-truth answers much more than silver or bronze passages, while no passage type is more correlated with the question alone, which the paper reads as evidence that both the image and the passages are needed to answer correctly. The paper also reports that removing the passages drops ROUGE-2 sharply, averaging 37.72% for one English model, and takes this as evidence that the benchmark requires external knowledge.

Load-bearing premise

The load-bearing assumption is that the score drop when passages are withheld reflects the absence of external knowledge, not a change in prompt format or in the length and phrasing of the responses the models produce.

Editorial extensions

If this is right

If the benchmark measures what it claims, RAG evaluation for vision-language models should include distractor passages rather than a single gold passage, because real retrieval returns a mix of useful and irrelevant results.
Fine-tuning on contrastively constructed valid/invalid passage data measurably improves passage selection, with exact match rising from 2.0 to 28.0 and F1 from 15.9 to 41.0 on the paper's selection task.
The same training transfers to an existing external-knowledge VQA benchmark, raising InfoSeek performance from 42.9 to 44.5, so the effect is not confined to the proposed benchmark's format.
The parallel English, Chinese, and Korean construction allows the same underlying questions to be compared across languages, which is useful for studying how passage selection behaves in multilingual RAG.
The finding that removing passages sharply lowers scores, with one English model dropping an average of 37.72% in ROUGE-2, supports the benchmark's design goal of requiring external knowledge.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The gold, silver, and bronze passage types could be reused as an automatic evaluation protocol for retrievers: a good retriever should surface gold passages before silver or bronze ones, giving a graded measure of retrieval quality without new annotation.
The VLR-IF construction recipe, which draws invalid passages randomly from other examples, is cheap enough to extend to additional languages and domains, and its refusal behavior when only invalid passages are present is a testable way to measure hallucination resistance.
The observed flat correlation between passage types and questions suggests that passage selection cannot be solved from text alone, which in turn predicts that models with stronger visual grounding should gain more from gold passages than models that rely mostly on language priors.
One could probe whether the reported score gaps reflect true knowledge use by holding answer length and prompt format constant while varying only the content of the passages, a control the paper does not report.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The core idea here is real: VLR-Bench gives each query five passages (two gold, two silver, one bronze), so a model must select what is useful rather than trust a single retrieved gold passage. That is a genuine departure from InfoSeek and Encyclopedic VQA, and the multilingual parallel construction (EN/ZH/KO) is a practical asset. The data is public, the annotation process is described in reasonable detail, and the InfoSeek transfer result (42.9 to 44.5) is at least external evidence that VLR-IF does something beyond fitting the benchmark's own distribution.

The soft spots are in proportion. Table 9's passage-selection jump (EM 2.0 to 28.0) is the strongest evidence for VLR-IF, but the baseline LLaVA-Llama-3 was presumably never asked to output passage selections, while the VLR-IF model was trained to do exactly that. Without a format-matched baseline—same instruction, same output schema, no VLR-IF training—the gain could be mostly the model learning to produce 'passage 1, passage 3' rather than learning to judge relevance. That is a load-bearing concern, and the paper does not address it. The 300-item test set, single runs, and no confidence intervals also make the headline numbers fragile. The validity check (with vs. without passages) assumes the performance gap measures external knowledge need, but prompt format and response length are confounded; the paper itself concedes KMS is length-biased, and the R-2 drop for X-LLaVA (37.72%) is reported without control conditions. GPT-4o generates the questions, passages, and training data, and also serves as judge, so distribution matching is a real worry; the InfoSeek transfer mitigates but does not eliminate it.

Minor issues: Table 7's caption says 'search engine as a passage retriever' while the text says Oracle was used, and Table 10's Qualitative/Quantitative columns appear mislabeled. These are fixable.

Who this is for: anyone building or evaluating RAG-capable VLMs, especially in multilingual settings. The benchmark is a useful resource even if the training-data claim needs tightening. I would send it to review; a good referee can push for the format-matched control and multi-run evidence. The paper is honest about its limitations, and the central benchmark contribution is solid enough to warrant that effort.

Referee Report

4 major / 6 minor

Summary. The paper proposes VLR-Bench, a multilingual (English/Chinese/Korean) visual question answering benchmark for retrieval-augmented generation (RAG), comprising 300 image-query-answer instances, each paired with five passages (two gold, two silver, one bronze). It also introduces VLR-IF, a 32K-example instruction-following dataset designed to teach vision-language models to select and use relevant passages. Experiments evaluate several open VLMs and GPT-4o with and without passages, report that VLR-IF training improves performance on VLR-Bench and transfers to InfoSeek, and include a passage-selection experiment in which EM/F1 rise from 2.0/15.9 to 28.0/41.0 after VLR-IF fine-tuning.

Significance. If the results hold, VLR-Bench fills a genuine gap: existing external-knowledge VQA benchmarks provide a single gold passage, whereas real RAG systems present top-k results with distractors. The five-passage design with graded gold/silver/bronze relevance is a useful evaluation instrument, and the multilingual parallel construction is a valuable addition. The release of the benchmark and training data, and the positive InfoSeek transfer, are concrete strengths. However, the main evidence for the central claim that VLR-IF improves passage selection rests on a single experiment (Table 9) that lacks a format-matched control, and other headline numbers come from single runs on 300 items without confidence intervals. The validity correlation (Table 3) is partly by construction because GPT-4o generates both the gold passages and the ground-truth answers. The contribution is promising, but the evaluation currently overclaims the strength of the evidence.

major comments (4)

[§4.1, Table 1]
[§D.2, Table 9]
[§3.1, Table 3]
[§4, Tables 1 and 7]

minor comments (6)

[Table 7 caption]
[Figure 12]
[Table 10]
[§3.1]
[§3.2]
[§D.1, Table 8]

Circularity Check

1 steps flagged · score 4.0 of 10

Gold passages are defined as the passages used to write the ground-truth answer, making Table 3's Gold > Silver > Bronze correlation a built-in artifact; InfoSeek keeps the central claim partially independent.

self definitional [Section 3.1 (Generation of Additional External Knowledge) and Section 4.2 (Table 3 analysis)]
"Among these, two are directly referenced when generating answers for the actual images and questions, referred to as ‘Gold Passage’... The results presented in Table 3 show that the ground truth output correlates most strongly with the Gold - Silver - Bronze Passage in descending order. This trend substantiates the effective use of gold passages in deducing answers to the VLR-BENCH..."

The gold passages are, by construction, the exact passages that GPT-4o used to write the ground-truth answer during dataset generation. Section 4.2 then reports that the ground-truth output correlates most strongly with gold passages and takes this ordering as evidence that appropriate use of gold passages is crucial. That ordering is not an empirical discovery: an answer generated from the gold passages will necessarily resemble those passages more than the silver or bronze distractors. The analysis therefore confirms the construction procedure rather than independently validating the benchmark's passage-selection premise, so this particular validity argument reduces to its own input.

full rationale

The only concrete circularity is in the benchmark-validity argument built around Table 3. The paper defines the two gold passages as the passages directly referenced when generating the ground-truth answers, and those answers were produced by GPT-4o from those passages. The correlation analysis in Section 4.2 therefore measures the construction mechanism rather than an independent property of the dataset. This does not make the whole benchmark circular: the gold/silver/bronze labels were manually reviewed, the benchmark is a real artifact, and the InfoSeek evaluation in Table 7 provides external grounding for the VLR-IF training effect, although it tests generation with a retrieved passage rather than selection among multiple distractors. The remaining concerns—the with/without-passages comparison lacking a format-matched control, GPT-4o serving as both data generator and qualitative judge, and single-annotator keyword collection acknowledged in the Limitations—are validity and reproducibility issues rather than reductions of the central claim to its inputs. On balance, the score is 4: one self-definitional analysis step is present, but the central benchmark claim retains substantial independent content.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The benchmark's design choices (passage count and composition, keyword count) are hand-set and untested against alternatives. The analysis relies on the assumption that GPT-4o-generated content plus light human review yields valid ground truth, that gold/silver/bronze categories are meaningful, and that the with/without-passage gap isolates knowledge use. No machine-checked proofs or independent data support these assumptions.

free parameters (2)

Passage composition ratio = 2 gold, 2 silver, 1 bronze
The benchmark's structure is defined by this hand-chosen ratio; it is not derived from any analysis, and the central claim that five passages test passage selection depends on this composition.
Number of keywords per query = 2
The keyword-matching score (KMS) counts a response as correct only when both manually chosen keywords appear; this threshold is a design choice and affects all quantitative results in Table 1.

assumptions (4)

domain assumption GPT-4o-generated question-answer-passage candidates, after human review, provide a valid ground truth for external-knowledge VQA.
The entire benchmark and training data are constructed by prompting GPT-4o (Section 3.1, Figure 6); the paper treats the resulting outputs as correct without comparing against independently verified sources beyond annotator judgment.
domain assumption Gold, silver, and bronze passage categories are meaningful and can be reliably assigned by annotators.
Section A.2 states annotators used personal judgment to assign passage types; the benchmark's validity analysis (Table 3) assumes this ordering is a property of the data rather than of the annotation process.
domain assumption The performance drop in the without-passages condition is attributable to the need for external knowledge.
Section 4.1 interprets the drop (e.g., X-LLaVA R-2 -37.72%) as evidence that queries require external knowledge, without controlling for prompt differences or answer-length effects.
domain assumption A 300-item test set is sufficient to produce stable model rankings.
The benchmark contains 300 parallel samples per language; all conclusions are drawn from single-run evaluations on this size.

how reviews work

0 comments

Cite this review

Pith. "Pith review of VLR-Bench: Multilingual Benchmark Dataset for Vision-Language Retrieval Augmented Generation." pith.science (2026). https://pith.science/paper/7X5TGFYZ

@misc{pith2026241210151,
  author       = {Pith},
  title        = {Pith review of: VLR-Bench: Multilingual Benchmark Dataset for Vision-Language Retrieval Augmented Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7X5TGFYZ}},
  note         = {Machine review of arXiv:2412.10151}
}

read the original abstract

We propose the VLR-Bench, a visual question answering (VQA) benchmark for evaluating vision language models (VLMs) based on retrieval augmented generation (RAG). Unlike existing evaluation datasets for external knowledge-based VQA, the proposed VLR-Bench includes five input passages. This allows testing of the ability to determine which passage is useful for answering a given query, a capability lacking in previous research. In this context, we constructed a dataset of 32,000 automatically generated instruction-following examples, which we denote as VLR-IF. This dataset is specifically designed to enhance the RAG capabilities of VLMs by enabling them to learn how to generate appropriate answers based on input passages. We evaluated the validity of the proposed benchmark and training data and verified its performance using the state-of-the-art Llama3-based VLM, the Llava-Llama-3 model. The proposed VLR-Bench and VLR-IF datasets are publicly available online.

Figures

Figures reproduced from arXiv: 2412.10151 by the authors.

**Figure 1.** An example of VLR-BENCH data sample. In addition, we introduce a construction method for the VLR-IF dataset designed to enhance the ability of VLMs to select external knowledge. 3.1 VLR-Bench Dataset VLR-BENCH was constructed to evaluate whether VLMs can use the correct external knowledge to generate accurate responses to query. We constructed a parallel corpus of 300 datasets: 150 based on general knowledge and 15… view at source ↗

**Figure 2.** Examples of the created VLR-Bench data. (English culture) [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗

**Figure 3.** Examples of the created VLR-Bench data. (commonsense knowledge) [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

Figures from the paper (9 more)

**Figure 4.** Figure 4: Examples of the created VLR-Bench data. (Korean culture) [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]

**Figure 5.** Figure 5: Examples of the created VLR-Bench data. (commonsense knowledge) [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]

**Figure 6.** Figure 6: Overview of the VLR-BENCH dataset construction process. Overview of Data Construction Procedure The image samples used in the dataset are sourced from the BOK-VQA (Kim et al., 2024) dataset, ensuring a wide range of visual content. The construction process involves few…

**Figure 7.** Figure 7: VLR-BENCH annotation tool [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]

**Figure 8.** Figure 8: Examples of the few-shot sample [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]

**Figure 9.** Figure 9: First example of the created VLR-IF data. [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]

**Figure 10.** Figure 10: Second example of the created VLR-IF data. [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]

**Figure 11.** Figure 11: The process of constructing the VLR-IF dataset. [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]

**Figure 12.** Figure 12: Examples of prompts used with GPT models in qualitative evaluations. [PITH_FULL_IMAGE:figures/full_fig_p018_12.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

16 extracted references · 9 canonical work pages

[2]

XTuner Contributors

Can pre-trained vision and language mod- els answer visual information-seeking questions? Preprint, arXiv:2302.11713. XTuner Contributors

arXiv
[7]

In Proceedings of the 2022 Conference on Empiri- cal Methods in Natural Language Processing, pages 11238–11254

Retrieval augmented visual question answering with outside knowledge. In Proceedings of the 2022 Conference on Empiri- cal Methods in Natural Language Processing, pages 11238–11254. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee

work page 2022
[8]

In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 6417–6431

Weakly-supervised visual-retriever- reader for knowledge-based question answering. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 6417–6431. Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi

work page 2021
[10]

In Findings of the Association for Computational Linguistics: ACL 2022, pages 2497– 2511, Dublin, Ireland

xGQA: Cross-lingual visual ques- tion answering. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2497– 2511, Dublin, Ireland. Association for Computational Linguistics. Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi

work page 2022
[11]

Preprint, arXiv:2206.01718

A-okvqa: A benchmark for visual question answering using world knowledge. Preprint, arXiv:2206.01718. Sanket Shah, Anand Mishra, Naganand Yadati, and Partha Pratim Talukdar

arXiv
[12]

Preprint, arXiv:2403.11399

X-llava: Op- timizing bilingual large vision-language alignment. Preprint, arXiv:2403.11399. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain...

arXiv
[13]

In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online

Transform- ers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics. Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi

work page 2020
[14]

gold passage

Bertscore: Eval- uating text generation with bert. In International Conference on Learning Representations. Appendix A VLR-Bench 9 A.1 VLR-Bench Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 A.2 VLR-B ENCH Construction Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 A.3 VLR-B ENCH Few-shot Setup Exampl...

work page 1927

Show all 16 references

[16]

Works of Antoni Gaudí

dataset, ensuring a wide range of visual content. The construction process involves few-shot learning and initial generation, annotator review and selection, passage expansion, and final review. GPT-4o generates candidate question-answer-passage sets based on few-shot examples...

1984
[1941]

이 조각상은 네 명의 미국 대통령의 60피트 크기 머리를 특징으로 합니다.4)러시모어산조각의뒤에는구츠온보글럼이있었고, 이프로젝트에는400명이상의노동자들이참여했습니다

George Washington's face was the first to be carved, starting in 1927.5)The Eiffel Tower in Paris, France, was completed in 1889 and was the tallest man-made structure in the world until the completion of the Chrysler Building in New York City in 1930.1)拉什莫尔山上描绘的其他总统是托马斯·杰斐逊、西...

1927
[2015]

Preprint, arXiv:1405.0312

Microsoft coco: Common objects in con- text. Preprint, arXiv:1405.0312. Weizhe Lin and Bill Byrne

arXiv
[2020]

In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781

Dense passage retrieval for open- domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781. MinJun Kim, SeungWoo Song, YouHan Lee, Haneol Jang, and KyungTae Lim

2020
[2021]

Preprint, arXiv:2005.11401

Retrieval-augmented generation for knowledge- intensive nlp tasks. Preprint, arXiv:2005.11401. Chin-Yew Lin

2005 arXiv
[2022]

In 2022 IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pages 5057–5067

Transform-retrieve-generate: Natural language-centric outside-knowledge visual question answering. In 2022 IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pages 5057–5067. IEEE. Drew A. Hudson and Christopher D. Manning

2022
[2023]

Preprint, arXiv:2308.12966

Qwen-vl: A versatile vision- language model for understanding, localization, text reading, and beyond. Preprint, arXiv:2308.12966. Yang Chen, Hexiang Hu, Yi Luan, Haitian Sun, So- ravit Changpinyo, Alan Ritter, and Ming-Wei Chang

arXiv
[2024]

Preprint, arXiv:2303.08774

Gpt-4 technical report. Preprint, arXiv:2303.08774. Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu

arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [2]

XTuner Contributors

Can pre-trained vision and language mod- els answer visual information-seeking questions? Preprint, arXiv:2302.11713. XTuner Contributors

arXiv

[2] [7]

In Proceedings of the 2022 Conference on Empiri- cal Methods in Natural Language Processing, pages 11238–11254

Retrieval augmented visual question answering with outside knowledge. In Proceedings of the 2022 Conference on Empiri- cal Methods in Natural Language Processing, pages 11238–11254. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee

work page 2022

[3] [8]

In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 6417–6431

Weakly-supervised visual-retriever- reader for knowledge-based question answering. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 6417–6431. Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi

work page 2021

[4] [10]

In Findings of the Association for Computational Linguistics: ACL 2022, pages 2497– 2511, Dublin, Ireland

xGQA: Cross-lingual visual ques- tion answering. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2497– 2511, Dublin, Ireland. Association for Computational Linguistics. Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi

work page 2022

[5] [11]

Preprint, arXiv:2206.01718

A-okvqa: A benchmark for visual question answering using world knowledge. Preprint, arXiv:2206.01718. Sanket Shah, Anand Mishra, Naganand Yadati, and Partha Pratim Talukdar

arXiv

[6] [12]

Preprint, arXiv:2403.11399

X-llava: Op- timizing bilingual large vision-language alignment. Preprint, arXiv:2403.11399. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain...

arXiv

[7] [13]

In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online

Transform- ers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics. Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi

work page 2020

[8] [14]

gold passage

Bertscore: Eval- uating text generation with bert. In International Conference on Learning Representations. Appendix A VLR-Bench 9 A.1 VLR-Bench Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 A.2 VLR-B ENCH Construction Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 A.3 VLR-B ENCH Few-shot Setup Exampl...

work page 1927