PRGB Benchmark: A Robust Placeholder-Assisted Algorithm for Benchmarking Retrieval-Augmented Generation

Dan Yang; Duolin Sun; Jian Wang; Jie Feng; Jinjie Gu; Lei Liu; Peng Wei; Yihan Jiao; Yue Shen; Zhehao Tan

REVIEW 5 major objections 6 minor 19 references

PRGB Benchmark: A Robust Placeholder-Assisted Algorithm for Benchmarking Retrieval-Augmented Generation

T0 review · 5 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper introduces a placeholder-based benchmark that replaces answer values in retrieved documents with type-consistent placeholders, so that a model's score reflects its ability to use external information rather than its memorized…

desk verdict A well-built RAG benchmark whose headline 'decoupling' claim needs a control experiment before it fully lands. read the letter →

arxiv 2507.22927 v1 pith:CZEB3NGE submitted 2025-07-23 cs.CL

Zhehao Tan , Yihan Jiao , Dan Yang , Lei Liu , Jie Feng , Duolin Sun , Yue Shen , Jian Wang

show 2 more authors

Peng Wei Jinjie Gu

This is my paper

classification cs.CL

keywords retrieval-augmentedgenerationbenchmarkplaceholdersubstitutionparametricknowledgedecouplingmulti-levelfilteringcombinationabilityreferencereasoningLLMevaluation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper is trying to establish a way to measure whether an LLM actually uses retrieved documents in a RAG system, rather than falling back on knowledge memorized during pre-training. It introduces a benchmark whose questions are built from Wikipedia entity triplets and graded along three progressive dimensions: multi-level filtering of noise, combination of scattered facts, and reference reasoning across documents. The central move is a dynamic placeholder substitution: the answer value in a document is replaced with an arbitrary but type-consistent placeholder, and each question is repeated with several candidate values. The authors argue this makes evaluation robust because the model cannot rely on having seen the answer before; its score then reflects genuine document utilization. If this works, it gives the community a reproducible way to compare RAG generation models on their actual grounding ability rather than on memorization.

What carries the argument

The central object is the dynamic placeholder substitution: for each golden triplet $(e, p, v)$, the answer value $v$ is replaced by a placeholder to form $(e, p, \text{Placeholder})$, documents are generated around that placeholder, and multiple candidate values of the same datatype are produced. The evaluation pipeline repeats each question with several placeholder values and averages the scores, which is what blocks a model from guessing the correct answer from memory. The benchmark's task taxonomy—multi-level filtering (weak, moderate, hard noise), combination (explicit, multi-value, multi-scenario), and reference reasoning (comparative, deductive, comparative-deductive)—is built on triplet-based metadata from parent and child entities. Together, the placeholder mechanism and the graded task dimensions are what the paper claims makes the benchmark a fine-grained and robust test of document utilization.

What would settle it

Take a sample of benchmark questions and run them twice: once with the original answers in the documents and once with placeholder substitutions. If models answer the original versions correctly even when no document is supplied, or if accuracy on the placeholder versions shifts when the candidate values are chosen to be common versus rare in pretraining data, then the decoupling is incomplete.

Watch

Extended reading notes

Core claim

The paper's central claim is that a placeholder-based benchmark can decouple an LLM's parametric knowledge from the external knowledge in retrieved documents, so that measured performance 'solely relies on their ability to utilize external information.' To support this, it constructs 3,887 English and 3,387 Chinese evaluation samples from 224 parent entities, 2,272 child entities, and 16,033 triplets, and organizes tasks into multi-level filtering, combination, and reference reasoning. It reports experiments on a range of state-of-the-art models showing consistent limitations in error resilience and context faithfulness, and it shows that filter ability does not scale monotonically with model size while combination and reasoning do favour larger models.

Load-bearing premise

The central assumption is that replacing an answer with an arbitrary but type-consistent placeholder value removes the model's memorized knowledge without changing the difficulty of the question.

Editorial extensions

If this is right

If the placeholder decoupling works, any LLM can be retested as new models appear, and scores will continue to measure grounding rather than memorization.
The benchmark's noise library can be reused to evaluate retrieval models, since the same noise types (similar entities, generalizing parent cases) stress retrieval as well as generation.
The finding that filter ability does not scale with model size suggests that smaller models can be competitive in needle-in-a-haystack settings, which bears on model selection for RAG deployment.
Reasoning-mode and non-reasoning-mode differences indicate that chain-of-thought style reasoning changes how models handle noisy documents, a factor RAG evaluation should report explicitly.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If placeholder substitution truly removes parametric memory, the same trick could be adapted to test knowledge contamination over time: a benchmark whose answers are periodically replaced with fresh placeholders should stay valid even as models absorb the original data.
The method implicitly assumes candidate values are neutral; a practical extension would be to measure and correct for distributional bias, for example by checking whether models prefer candidate values that are statistically more frequent in their training data.
The approach could transfer to non-textual RAG settings, such as retrieval over code or tabular data, wherever a 'value' can be swapped while preserving the surrounding structure.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Dear colleague,

Quick take: this is a solid, well-scoped benchmark paper for RAG evaluation, but the central claim about placeholder substitution needs a control experiment before I'd trust the decoupling story.

What's genuinely new: the three-way split of document-utilization skills (multi-level filtering, combination, reference reasoning) is more granular than RGB or RAGBench, and the placeholder substitution idea—replacing answer values with type-matched arbitrary values so models can't rely on memorized facts—is a clever workaround for the parametric-knowledge confound. The construction pipeline is explicit: triplet seeds from Wikipedia, propagated child entities, per-task noise levels, manual validation of a chunk of the data. Two languages, 7k+ samples, and a reasonable set of ablations on noise ratios and placeholder counts. That's real work, and it's reproducible in principle.

The soft spots are concentrated around the main claim. There is no experiment that compares the same items with original values versus placeholders. The placeholder section shows that changing placeholders changes outcomes for ~30% of partially-correct cases, but that only shows models sometimes follow the document—it doesn't show parametric knowledge is suppressed. Without a control, a model could still be guessing from a distributionally-biased candidate set (generated by GPT-4o and Qwen2.5-MAX). Second, GPT-4o both generates data, judges answers, and is itself evaluated; that's a self-reference loop that should at least be acknowledged with an external judge or agreement analysis. Third, no head-to-head comparison against existing benchmarks (RGB, RAGBench, RAGAS) is provided, so it's hard to tell whether the new dimensions measure something other benchmarks miss. Also, the abstract says code is available, the conclusion says it will be released later, and the numbers in Table 1 look off for Gemma3-12b (89.92 for combination on ZH?); needs a pass.

These are all addressable. The core architecture is sensible, the data is probably useful, and the benchmark could become a solid evaluation tool if the authors add the original-vs-placeholder control, switch to an external judge or at least report agreement, fix the inconsistencies, and release the dataset.

Who it's for: people building or evaluating RAG systems, especially those who care about separating what the model knows from what it reads. I'd give it a serious referee rather than desk reject, but I'd push for a major revision.

Recommendation: send it to review, conditional on the control experiment being added.

Referee Report

5 major / 6 minor

Summary. The paper introduces PRGB, a bilingual (English/Chinese) benchmark for evaluating retrieval-augmented generation (RAG) with three fine-grained dimensions—multi-level filtering, combination, and reference reasoning—built from a triplet-based metadata graph. Its central methodological proposal is a dynamic placeholder substitution algorithm that replaces answer values in golden documents with type-consistent placeholders and candidate values, intended to suppress LLMs' parametric knowledge so that evaluated performance 'solely relies on their ability to utilize external information.' The authors report construction statistics (3,887 English and 3,387 Chinese samples), a GPT-4 and Qwen-assisted synthesis pipeline with manual validation, and experiments on a range of closed- and open-source LLMs, concluding that reasoning models handle noisy contexts better while non-reasoning models are prone to misleading answers. The paper also includes ablation-style analyses of noise levels and placeholder sensitivity.

Significance. If the decoupling claim were validated, PRGB would be a useful contribution to RAG evaluation: it targets LLM-specific document-utilization abilities rather than end-to-end RAG scores, provides a finer-grained task taxonomy than RGB or RAGBench, and ships public code and data. The explicit triplet-to-task construction, the bilingual scale, and the manual validation step are genuine strengths, and the observation that model rankings differ across filtering, combination, and reasoning is interesting. However, the central methodological claim—that placeholder substitution removes parametric-knowledge bias—is asserted but not directly tested, and the evaluation pipeline has a self-referential component because GPT-4o is both a data generator/judge and an evaluated model. These issues must be addressed before the benchmark's headline conclusions can be accepted.

major comments (5)

[Dynamic Placeholder Substitution] The paper's core claim that placeholder substitution makes performance 'solely rely on their ability to utilize external information' (Abstract and Section 'Dynamic Placeholder Substitution') is not directly tested. The section asserts that replacing values (a) reduces bias from internal knowledge and (b) lowers the probability of guessing, but no experiment compares the same items with original values against placeholder versions, and no leakage metric is reported. The 'Placeholders' discussion and Figure 2 only show that changing placeholders changes outcomes for about 30% of partially correct cases; sensitivity to placeholder changes is consistent with models partially following the document and does not establish that parametric knowledge is suppressed. Please add a control condition (for example, original-value items versus placeholder items, stratified by entity popularity or recency) and report agreement between placeholder-based scores and scores on held-out original items.
[Evaluation Metrics / Table 1] GPT-4o is used to generate placeholder documents and candidate values ('Placeholder-based Candidate Values Generation'), to judge answers ('Evaluation Metrics': 'we directly let GPT4o judge whether the answer is correct'), and is itself one of the evaluated models (Table 1, GPT4o-1120 row). This creates a self-referential evaluation loop: the benchmark's lead model may be rewarded for answers that match GPT-4o's preferred phrasing or its own generated candidate distributions. Please either exclude the judge model from the evaluated set, use a judge not involved in data generation, or report per-model agreement between Covered Exact Match and GPT-4o judgment to show that the judgment is not a GPT-4o self-preference artifact.
[Abstract / Conclusion / Evaluation Metrics] The abstract claims that experiments demonstrate limitations 'particularly in error resilience and context faithfulness,' but no context-faithfulness metric is defined anywhere in the paper. The only metrics described in 'Evaluation Metrics' are covered exact-match accuracy and GPT-4o correctness judgment; no faithfulness or error-resilience measure is specified or reported. Please either add and report the missing faithfulness metrics or revise the abstract and conclusion to state precisely which capabilities were measured.
[Algorithm 1 / Experimental Setup / Benchmark Statistics] Algorithm 1 takes n < N candidate placeholders as input but computes Score = AVG(metric(result_{1,...,N}, GT)) over N, while the main experiments set three placeholders despite the datasets averaging 4.39–4.68 candidate values per sample (Benchmark Statistics). This inconsistency makes the exact scoring procedure ambiguous. Please clarify how the averaging is computed and whether using n=3 introduces sampling variance, and report standard errors or bootstrap intervals for the headline scores in Table 1, especially for models separated by only a few points.
[Data Quality Validation] The manual validation step starts from samples that 'multiple models fail to answer correctly' under the simplest condition and then corrects approximately 30% of the dataset. Because the same model families are later evaluated on the corrected data, this creates a model-dependent data-selection step; the paper gives no evidence that the corrections do not systematically favor or penalize particular architectures. Please report the fraction of corrected samples per model family and check whether headline rankings are stable after excluding corrected samples.

minor comments (6)

[Introduction] The text has several typographical errors, including 'toevaluate' and 'isinsufficient' in the Introduction; these should be corrected.
[Related Work] In the Related Work section, REALM is cited as '(Lewis et al. 2020)', but the reference list correctly attributes REALM to Guu et al. (2020); please fix this citation.
[Benchmark Statistics] The abstract and conclusion state different emphases: the abstract mentions 'error resilience and context faithfulness' while the conclusion emphasizes 'mitigating the interference of knowledge stored in the model's internal parameters'; please align these statements with the actual metrics used.
[Experimental Setup] Model names are written inconsistently (for example, 'GPT4o' versus 'GPT-4o' and 'Qwen' versus 'Qwen3'); please standardize the notation throughout.
[Evaluation Metrics] The 'Covered Exact Match' metric is described informally; a formal definition, including how the 'or' and 'and' operators are applied to keyword sets, would improve reproducibility.
[Placeholders] The 'Placeholders' subsection reports that about 30% of partially correct answers are affected by placeholder changes, but this result is presented without confidence intervals or a description of how many samples underlie Figure 2; please add this information.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: PRGB directly measures model outputs, and the placeholder decoupling claim is an unsupported empirical assumption rather than a definitional reduction.

full rationale

I walked the claimed derivation chain: triplet metadata yields fine-grained tasks; the placeholder substitution replaces answer values with type-consistent candidates; scores are computed by averaging measured inference results. No step defines an output in terms of an input, fits a parameter and then renames it a prediction, or imports a load-bearing result solely from the authors' prior work. The central claim that placeholder substitution makes performance 'solely rely on their ability to utilize external information' is asserted via two mechanisms: (a) substituting critical information 'reduces bias from the model's internal knowledge during evaluation' and (b) repeated testing 'lowers the probability of models guessing correct answers.' These are intended mechanisms, not definitions that make the conclusion true by construction. The paper reports no control comparing original-value versus placeholder-value items, so the decoupling claim is empirically unverified; that is a validity or robustness gap, not a circularity. The self-citation to HIRAG (Jiao et al. 2025) is used only as related-work contrast and is not load-bearing. GPT-4o is used both to generate candidate values and to judge answers while also being an evaluated model, which is a self-referential evaluation design and a possible source of bias; however, it does not make any reported score equal to an input by construction, and in Table 1 GPT-4o ranks relatively low, so no inflation is evident. Accordingly, no circular step meeting the quoted-reduction standard is present.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central benchmark rests on several domain assumptions rather than mathematical derivation. The free parameters are evaluation design choices, and no new physical or formal entities are introduced.

free parameters (4)

l1_noise_doc = 4
Number of weak noise documents per sample, chosen by the authors as an evaluation hyperparameter.
l2_noise_doc = 4
Number of moderate noise documents per sample, chosen by the authors.
l3_noise_doc = 1
Number of hard noise documents per sample, chosen by the authors.
placeholder_count = 3
Three placeholder substitutions per data point in the main experiments; this is a user-chosen evaluation depth, not derived from theory.

assumptions (4)

domain assumption Triplet-based synthetic documents generated by GPT-4o and Qwen2.5-MAX accurately represent the properties of real retrieved RAG contexts.
The entire benchmark relies on this assumption; it is stated in the data construction pipeline but not validated against authentic retrieval corpora.
domain assumption Replacing the answer value with a type-consistent placeholder eliminates the model's reliance on parametric knowledge.
This is the core methodological premise in the 'Dynamic Placeholder Substitution' section; the text asserts it ensures performance 'solely relies on external information', but no experiment verifies the elimination.
domain assumption GPT-4o can reliably judge answer correctness for these tasks.
GPT evaluation is used as a primary metric, yet GPT-4o is also among the evaluated models in Table 1, creating a potential self-favoring bias.
ad hoc to paper Manual validation and correction of about 30% of samples does not introduce systematic bias toward certain models.
The Data Quality Validation section says samples that multiple models failed are manually corrected; the effect of this procedure on benchmark difficulty and fairness is not analyzed.

how reviews work

0 comments

Cite this review

Pith. "Pith review of PRGB Benchmark: A Robust Placeholder-Assisted Algorithm for Benchmarking Retrieval-Augmented Generation." pith.science (2026). https://pith.science/paper/CZEB3NGE

@misc{pith2026250722927,
  author       = {Pith},
  title        = {Pith review of: PRGB Benchmark: A Robust Placeholder-Assisted Algorithm for Benchmarking Retrieval-Augmented Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CZEB3NGE}},
  note         = {Machine review of arXiv:2507.22927}
}

read the original abstract

Retrieval-Augmented Generation (RAG) enhances large language models (LLMs) by integrating external knowledge, where the LLM's ability to generate responses based on the combination of a given query and retrieved documents is crucial. However, most benchmarks focus on overall RAG system performance, rarely assessing LLM-specific capabilities. Current benchmarks emphasize broad aspects such as noise robustness, but lack a systematic and granular evaluation framework on document utilization. To this end, we introduce \textit{Placeholder-RAG-Benchmark}, a multi-level fine-grained benchmark, emphasizing the following progressive dimensions: (1) multi-level filtering abilities, (2) combination abilities, and (3) reference reasoning. To provide a more nuanced understanding of LLMs' roles in RAG systems, we formulate an innovative placeholder-based approach to decouple the contributions of the LLM's parametric knowledge and the external knowledge. Experiments demonstrate the limitations of representative LLMs in the RAG system's generation capabilities, particularly in error resilience and context faithfulness. Our benchmark provides a reproducible framework for developing more reliable and efficient RAG systems. Our code is available in https://github.com/Alipay-Med/PRGB.

Figures

Figures reproduced from arXiv: 2507.22927 by the authors.

**Figure 1.** Visualization for Evaluation Dimensions of Placeholder-RAG-Eval. Starting from triplet-based metadata, three finer [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

**Figure 2.** Qwen2.5 Series Model Performance Comparison, [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

19 extracted references · 2 canonical work pages

[1]

, " * write output.state after.block = add.period write newline

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

H.; Meade, N.; and Reddy, S

Adlakha, V.; BehnamGhader, P.; Lu, X. H.; Meade, N.; and Reddy, S. 2024. Evaluating correctness and faithfulness of instruction-following models for question answering. Transactions of the Association for Computational Linguistics, 12: 681--699

2024
[4]

B.; Lespiau, J.-B.; Damoc, B.; Clark, A.; et al

Borgeaud, S.; Mensch, A.; Hoffmann, J.; Cai, T.; Rutherford, E.; Millican, K.; Van Den Driessche, G. B.; Lespiau, J.-B.; Damoc, B.; Clark, A.; et al. 2022. Improving language models by retrieving from trillions of tokens. In International conference on machine learning, 2206--2240. PMLR

2022
[5]

Chen, J.; Lin, H.; Han, X.; and Sun, L. 2023. Benchmarking Large Language Models in Retrieval-Augmented Generation. arXiv:2309.01431

arXiv 2023
[6]

Chen, J.; Lin, H.; Han, X.; and Sun, L. 2024. Benchmarking large language models in retrieval-augmented generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, 17754--17762

2024
[7]

E.; and Schockaert, S

Es, S.; James, J.; Anke, L. E.; and Schockaert, S. 2024. Ragas: Automated evaluation of retrieval augmented generation. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, 150--158

2024
[8]

Friel, R.; Belyi, M.; and Sanyal, A. 2025. RAGBench: Explainable Benchmark for Retrieval-Augmented Generation Systems. arXiv:2407.11005

arXiv 2025

Show all 19 references

[9]

Guu, K.; Lee, K.; Tung, Z.; Pasupat, P.; and Chang, M. 2020. Retrieval augmented language model pre-training. In International conference on machine learning, 3929--3938. PMLR

2020
[10]

Jiao, Y.; Tan, Z.; Yang, D.; Sun, D.; Feng, J.; Wang, J.; and Wei, P. 2025. HIRAG: Hierarchical-Thought Instruction-Tuning Retrieval-Augmented Generation. arXiv:2507.05714

2025 arXiv
[11]

S.; and Zettlemoyer, L

Joshi, M.; Choi, E.; Weld, D. S.; and Zettlemoyer, L. 2017. TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. arXiv:1705.03551

2017 arXiv
[12]

u ttler, H.; Lewis, M.; Yih, W.-t.; Rockt \

Lewis, P.; Perez, E.; Piktus, A.; Petroni, F.; Karpukhin, V.; Goyal, N.; K \"u ttler, H.; Lewis, M.; Yih, W.-t.; Rockt \"a schel, T.; et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems, 33: 9459--9474

2020
[13]

Mallen, A.; Asai, A.; Zhong, V.; Das, R.; Khashabi, D.; and Hajishirzi, H. 2023. When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories. arXiv:2212.10511

2023 arXiv
[14]

Min, S.; Michael, J.; Hajishirzi, H.; and Zettlemoyer, L. 2020. AmbigQA: Answering ambiguous open-domain questions. arXiv preprint arXiv:2004.10645

2020 arXiv
[15]

Ram, O.; Levine, Y.; Dalmedigos, I.; Muhlgay, D.; Shashua, A.; Leyton-Brown, K.; and Shoham, Y. 2023. In-context retrieval-augmented language models. Transactions of the Association for Computational Linguistics, 11: 1316--1331

2023
[16]

Saad-Falcon, J.; Khattab, O.; Potts, C.; and Zaharia, M. 2023. Ares: An automated evaluation framework for retrieval-augmented generation systems. arXiv preprint arXiv:2311.09476

2023 arXiv
[17]

Shi, Z.; Sun, W.; Gao, S.; Ren, P.; Chen, Z.; and Ren, Z. 2024. Generate-then-ground in retrieval-augmented generation for multi-hop question answering. arXiv preprint arXiv:2406.14891

2024 arXiv
[18]

Trivedi, H.; Balasubramanian, N.; Khot, T.; and Sabharwal, A. 2022. Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions. arXiv preprint arXiv:2212.10509

2022 arXiv
[19]

Voorhees, E. M. 1998. Variations in relevance judgments and the measurement of retrieval effectiveness. In Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, 315--323

1998

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

, " * write output.state after.block = add.period write newline

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

H.; Meade, N.; and Reddy, S

Adlakha, V.; BehnamGhader, P.; Lu, X. H.; Meade, N.; and Reddy, S. 2024. Evaluating correctness and faithfulness of instruction-following models for question answering. Transactions of the Association for Computational Linguistics, 12: 681--699

2024

[4] [4]

B.; Lespiau, J.-B.; Damoc, B.; Clark, A.; et al

Borgeaud, S.; Mensch, A.; Hoffmann, J.; Cai, T.; Rutherford, E.; Millican, K.; Van Den Driessche, G. B.; Lespiau, J.-B.; Damoc, B.; Clark, A.; et al. 2022. Improving language models by retrieving from trillions of tokens. In International conference on machine learning, 2206--2240. PMLR

2022

[5] [5]

Chen, J.; Lin, H.; Han, X.; and Sun, L. 2023. Benchmarking Large Language Models in Retrieval-Augmented Generation. arXiv:2309.01431

arXiv 2023

[6] [6]

Chen, J.; Lin, H.; Han, X.; and Sun, L. 2024. Benchmarking large language models in retrieval-augmented generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, 17754--17762

2024

[7] [7]

E.; and Schockaert, S

Es, S.; James, J.; Anke, L. E.; and Schockaert, S. 2024. Ragas: Automated evaluation of retrieval augmented generation. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, 150--158

2024

[8] [8]

Friel, R.; Belyi, M.; and Sanyal, A. 2025. RAGBench: Explainable Benchmark for Retrieval-Augmented Generation Systems. arXiv:2407.11005

arXiv 2025