A Systems-Level Analysis of Sensitivity, Robustness, and Stability in Retrieval-Augmented Generation

Bharath Simha Reddy Muthyam

REVIEW 2 major objections 2 minor 1 cited by

A Systems-Level Analysis of Sensitivity, Robustness, and Stability in Retrieval-Augmented Generation

T0 review · 2 major / 2 minor · reviewed 2026-06-30 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read RAG final answer accuracy often changes non-monotonically when chunk size or retrieval depth varies, so evaluation must track failures at each stage.

desk verdict Controlled sweeps on one RAG setup show non-monotonic downstream scores and stage-specific failures, but the single-corpus design leaves the evaluation advice on shaky ground. read the letter →

arxiv 2606.28337 v1 pith:RTRHHNN4 submitted 2026-05-29 cs.IR cs.AI

Bharath Simha Reddy Muthyam This is my paper · ORCID

classification cs.IRcs.AI

keywords retrieval-augmentedgenerationRAGevaluationsensitivityanalysisrobustnessstabilitymulti-stagefailureempiricalstudychunksize

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper runs 56 controlled experiments on a fixed set of 500 questions linked to 20,958 corpus contexts to test how RAG systems respond to changes in chunk size, retrieval depth, reranking, and injected noise. Retrieval metrics rise with broader settings, yet exact-match and F1 scores at the final answer stage frequently rise then fall or show high variance. Smaller chunks lose answer-bearing text before retrieval even begins, and added retrieval noise causes steady degradation. The authors conclude that measuring only the end answer hides where and why the system fails.

What carries the argument

Multi-stage failure tracking that separately measures retrieval success, context packing, and generation under controlled changes to chunk size, top-k depth, reranking, and probabilistic noise.

What would settle it

If final answer accuracy increased monotonically with every increase in retrieval depth or chunk size across several new corpora and models, the argument for mandatory multi-stage analysis would lose force.

Watch

Extended reading notes

Core claim

Across the 56 runs, retrieval-oriented metrics improved under broader retrieval settings, while downstream exact-match and F1 scores often behaved non-monotonically. Preprocessing-induced answer loss appeared under smaller chunk sizes, progressive degradation occurred under retrieval corruption, and higher variance was observed in broader retrieval regimes. These patterns indicate that RAG evaluation must incorporate sensitivity, robustness, stability, and multi-stage failure analysis rather than final answer accuracy alone.

Load-bearing premise

The non-monotonic score changes and variance patterns seen on this 500-question subset and 20,958-context corpus will appear with other corpora, models, and query distributions.

Editorial extensions

If this is right

Retrieval success rates rise when more chunks or higher top-k values are used.
Final exact-match and F1 scores frequently fail to follow the same upward trend.
Smaller chunk sizes discard answer text during preprocessing before retrieval occurs.
Added retrieval noise produces steady drops in end-to-end performance.
Variance across repeated runs grows under broader retrieval settings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Evaluation suites for RAG should log per-stage success rates rather than only the final string match.
The same staged checks could be applied to other composite systems that combine retrieval with generation.
Optimal chunk and depth settings may need to be tuned per query type instead of chosen globally.
Repeating the sweeps on larger or more diverse corpora would test whether the non-monotonic pattern persists.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

read the letter

The main takeaway is that this paper ran 56 controlled experiments varying chunk size, top-k, reranking, noise injection, and seeds on a fixed 500-question subset drawn from one 20,958-context corpus. Retrieval metrics improved with broader settings while exact-match and F1 scores often moved non-monotonically, and they also logged preprocessing losses at small chunks plus higher variance at larger k.

What the work does well is track intermediate stages instead of stopping at final accuracy. Measuring answer loss from chunking, progressive degradation under retrieval corruption, and variance differences gives a clearer picture of where the pipeline breaks. That kind of staged logging is still uncommon and the observations are direct measurements rather than fitted quantities.

The soft spot is exactly what the stress-test note flags: everything sits on one question set and one corpus. The non-monotonic patterns and variance increases could easily be tied to how answers are phrased or how contexts overlap in this particular collection. The abstract gives no error bars, statistical tests, or exclusion criteria, so it is hard to tell whether the divergences are reliable even within this setup. Generalization to other corpora or real query distributions is untested.

This is the sort of paper that would interest people who actually build and tune RAG systems and want empirical data on sensitivity. It shows honest engagement with the pipeline stages. I would send it to peer review so the methods and any additional runs can be checked; the design is clear enough that referees could usefully ask for cross-corpus checks or basic stats rather than desk-rejecting it outright.

Referee Report

2 major / 2 minor

Summary. The manuscript reports a controlled empirical study of Retrieval-Augmented Generation (RAG) systems across 56 experimental runs on a fixed 500-question QA subset mapped to a 20,958-context corpus. It systematically varies chunk size, retrieval depth (top-k), embedding reranking, probabilistic retrieval noise, and repeated seeded runs, measuring effects on retrieval metrics, context packing, and downstream generation (exact-match and F1). Key observations include non-monotonic behavior in final-answer metrics despite improving retrieval scores, preprocessing-induced answer loss at small chunk sizes, progressive degradation under noise, and higher variance in broader retrieval regimes. The authors conclude that RAG evaluation should incorporate sensitivity, robustness, stability, and multi-stage failure analysis rather than relying solely on final-answer accuracy.

Significance. If the reported patterns prove robust, the work would usefully demonstrate concrete limitations of accuracy-only RAG evaluation and supply a template for multi-stage analysis that isolates preprocessing, retrieval, and generation failures. The controlled design with intermediate metrics and repeated runs is a clear strength, offering reproducible examples of where and why performance diverges. The single-corpus, single-question-set scope, however, constrains how far the prescriptive recommendation can be taken without further validation.

major comments (2)

[Abstract] Abstract and conclusion: The recommendation that RAG evaluation 'should include sensitivity, robustness, stability, and multi-stage failure analysis' is grounded exclusively in results from one 500-question subset and one 20,958-context corpus. The non-monotonic downstream metrics and rising variance under broader retrieval could be specific to this question distribution, context overlap, or answer phrasing; no cross-corpus or cross-query-set experiments are reported to test whether the divergence between retrieval and generation metrics generalizes.
[Results] Experimental design and results sections: The 56 runs are presented without statistical significance tests, error bars, or explicit exclusion criteria for questions or runs. This weakens the ability to assess whether the claimed non-monotonic behaviors and variance patterns are reliable or sensitive to the particular 500-question sample.

minor comments (2)

A summary table listing the exact parameter settings for each of the 56 runs would improve reproducibility and allow readers to map specific configurations to the reported trends.
[Results] Figures illustrating non-monotonic trends and variance would benefit from explicit variance bands or per-run scatter to make the stability claims visually clearer.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the detailed and constructive report. We address each major comment below, clarifying the scope of our claims and the experimental controls already present while noting where revisions can strengthen the manuscript.

read point-by-point responses

Referee: [Abstract] Abstract and conclusion: The recommendation that RAG evaluation 'should include sensitivity, robustness, stability, and multi-stage failure analysis' is grounded exclusively in results from one 500-question subset and one 20,958-context corpus. The non-monotonic downstream metrics and rising variance under broader retrieval could be specific to this question distribution, context overlap, or answer phrasing; no cross-corpus or cross-query-set experiments are reported to test whether the divergence between retrieval and generation metrics generalizes.

Authors: We agree that the empirical patterns are demonstrated on a single corpus and question set. The manuscript frames the contribution as a controlled case study that isolates specific failure modes (preprocessing loss, non-monotonicity, variance under noise) rather than claiming universality. The prescriptive recommendation follows from the observation that final-answer accuracy alone missed these behaviors in this reproducible setting; it is offered as a template for multi-stage analysis, not as a proven requirement for every RAG system. We will revise the abstract and conclusion to explicitly qualify the scope and note that broader validation across corpora would be valuable future work. revision: partial
Referee: [Results] Experimental design and results sections: The 56 runs are presented without statistical significance tests, error bars, or explicit exclusion criteria for questions or runs. This weakens the ability to assess whether the claimed non-monotonic behaviors and variance patterns are reliable or sensitive to the particular 500-question sample.

Authors: The design already incorporates repeated seeded runs (five seeds per configuration) to quantify variance, and the full 500-question set was used with no exclusions. We will add error bars derived from the repeated runs to all relevant figures and tables, and we will include a brief statement on the absence of question-level filtering. While formal hypothesis tests were not performed, the repeated-run variance already provides a direct measure of stability; we can add paired significance tests on the key non-monotonic comparisons if the editor deems it necessary. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: purely empirical measurements on fixed dataset

full rationale

The paper conducts a controlled empirical study consisting of 56 experimental runs on one fixed 500-question QA subset and 20,958-context corpus. All reported behaviors (non-monotonic downstream scores, preprocessing loss, variance patterns, degradation under noise) are direct observations from these runs rather than quantities derived from equations, fitted parameters renamed as predictions, or self-citation chains. No derivation chain exists; the central recommendation follows from the measured divergence between retrieval and generation metrics. This is the most common honest non-finding for measurement-focused work.

Assumptions & free parameters 0 free parameters · 1 assumptions · 0 invented entities

The recommendation for multi-stage evaluation rests on the assumption that the controlled sweeps on this fixed QA subset capture representative failure modes; no free parameters are fitted to produce the headline claim.

assumptions (1)

domain assumption The 500-question subset mapped to 20,958 contexts is sufficiently representative for drawing general conclusions about RAG behavior.
The study fixes this subset as the evaluation basis for all 56 runs.

how reviews work

0 comments

Cite this review

Pith. "Pith review of A Systems-Level Analysis of Sensitivity, Robustness, and Stability in Retrieval-Augmented Generation." pith.science (2026). https://pith.science/paper/RTRHHNN4

@misc{pith2026260628337,
  author       = {Pith},
  title        = {Pith review of: A Systems-Level Analysis of Sensitivity, Robustness, and Stability in Retrieval-Augmented Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RTRHHNN4}},
  note         = {Machine review of arXiv:2606.28337}
}

read the original abstract

Retrieval-Augmented Generation (RAG) systems are often evaluated using final answer accuracy, even though their failures can originate from preprocessing, retrieval, context packing, or generation. This paper presents a controlled empirical study of RAG sensitivity, robustness, and stability across 56 experimental runs. We evaluate how chunk size, retrieval depth (top k), embedding-based reranking, probabilistic retrieval noise, and repeated seeded runs affect retrieval, context packing, and generation behavior. Using a fixed 500-question QA subset mapped to 20,958 unique corpus contexts, we analyze both final answer metrics and intermediate failure modes. Across these experiments, retrieval-oriented metrics improved under broader retrieval settings, while downstream exact-match and F1 scores often behaved non-monotonically. We also observe preprocessing-induced answer loss under smaller chunk sizes, progressive degradation under retrieval corruption, and higher observed variance in broader retrieval regimes. These findings suggest that RAG evaluation should include sensitivity, robustness, stability, and multi-stage failure analysis rather than relying only on final answer accuracy.

Figures

Figures reproduced from arXiv: 2606.28337 by the authors.

**Figure 1.** Pipeline architecture for the controlled RAG evaluation framework. The study separates preprocessing, retrieval, packing, generation, and failure analysis so final answer errors can be traced to pipeline stages. 3 Methodology and Evaluation Framework 3.1 Pipeline Overview [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗

**Figure 2.** Preprocessing-induced QA answer loss across chunk sizes [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 3.** Baseline Hit@k increases with retrieval depth. representative configurations that make this tradeoff visible without listing all 32 sensitivity runs. This pattern is one of the clearest examples of nonmonotonic RAG behavior in the study. The system retrieves more gold-context evidence at higher top k, but the generator does not consistently convert that additional evidence into better answers. Higher top k increas… view at source ↗

Figures from the paper (4 more)

**Figure 5.** Figure 5: Relationship between input token count and F1, supporting the context-overload interpretation [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]

**Figure 4.** Figure 4: Baseline F1 shows non-monotonic behavior as retrieval depth increases. 5.3 Configuration Tradeoffs Rather Than a Universal Optimum The strongest EM configuration in the baseline grid was chunk size 120 with top k=3, for both rerank off and rerank on. The strongest F1 …

**Figure 6.** Figure 6: F1 degradation under increasing retrieval corruption. 5.4 Robustness Under Retrieval Noise Retrieval corruption caused progressive degradation. Averaged across the four robustness anchors, Hit@k decreased from 0.711 at 10% noise to 0.561 at 30% noise. Mean F1 decreased…

**Figure 8.** Figure 8: F1 standard deviation across repeated seeded runs. 5.6 Multi-Stage Failure Analysis Failure analysis is one of the core contributions of this study. A final answer failure can arise from at least three distinct stages: • Retrieval failure: no retrieved chunk comes from…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Locating Failure in Multi-Page Visually Rich Document Understanding: An Empirical Attribution
cs.AI 2026-08 conditional novelty 6.0 of 10

A controlled attribution study of multi-page document understanding finds that missing evidence and cross-page integration failures dominate over distractor noise and text extraction quality.

Reference graph

Works this paper leans on

14 extracted references · 14 canonical work pages · cited by 1 Pith paper

[1]

Retrieval-Augmented Generation for Knowledge-Intensive

Lewis, Patrick and Perez, Ethan and Piktus, Aleksandra and Petroni, Fabio and Karpukhin, Vladimir and Goyal, Naman and Kuttler, Heinrich and Lewis, Mike and Yih, Wen-tau and Rockt. Retrieval-Augmented Generation for Knowledge-Intensive. Advances in Neural Information Processing Systems , volume =

work page
[2]

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing , pages =

Dense Passage Retrieval for Open-Domain Question Answering , author =. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing , pages =

work page 2020
[3]

Sentence-bert: Sentence embeddings using siamese bert-networks

Reimers, Nils and Gurevych, Iryna , booktitle =. Sentence-. doi:10.18653/v1/D19-1410 , url =

work page doi:10.18653/v1/d19-1410
[4]

Wang, Wenhui and Wei, Furu and Dong, Li and Bao, Hangbo and Yang, Nan and Zhou, Ming , booktitle =

work page
[5]

IEEE Transactions on Big Data , author =

Johnson, Jeff and Douze, Matthijs and J. Billion-Scale Similarity Search with. doi:10.1109/TBDATA.2019.2921572 , year =

work page doi:10.1109/tbdata.2019.2921572 2019
[6]

Journal of Machine Learning Research , volume =

Scaling Instruction-Finetuned Language Models , author =. Journal of Machine Learning Research , volume =

work page
[7]

Transactions of the Association for Computational Linguistics , volume =

Lost in the Middle: How Language Models Use Long Contexts , author =. Transactions of the Association for Computational Linguistics , volume =

work page
[8]

SQ u AD : 100,000+ Questions for Machine Comprehension of Text

Rajpurkar, Pranav and Zhang, Jing and Lopyrev, Konstantin and Liang, Percy , booktitle =. doi:10.18653/v1/D16-1264 , url =

work page doi:10.18653/v1/d16-1264

Show all 14 references

[9]

arXiv preprint arXiv:2312.10997 , year =

Retrieval-Augmented Generation for Large Language Models: A Survey , author =. arXiv preprint arXiv:2312.10997 , year =

arXiv
[10]

doi:10.18653/v1/2024.naacl-long.20 , url =

Saad-Falcon, Jon and Khattab, Omar and Potts, Christopher and Zaharia, Matei , booktitle =. doi:10.18653/v1/2024.naacl-long.20 , url =

2024 doi
[11]

Proceedings of the AAAI Conference on Artificial Intelligence , volume =

Benchmarking Large Language Models in Retrieval-Augmented Generation , author =. Proceedings of the AAAI Conference on Artificial Intelligence , volume =. doi:10.1609/aaai.v38i16.29728 , url =

doi
[12]

Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (

Investigating the Robustness of Retrieval-Augmented Generation at the Query Level , author =. Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (
[13]

Sun, Jiashuo and Zhong, Xianrui and Zhou, Sizhe and Han, Jiawei , eprint =
[14]

2602.03689 , archivePrefix =

Rethinking the Reranker: Boundary-Aware Evidence Selection for Robust Retrieval-Augmented Generation , author =. 2602.03689 , archivePrefix =

Pith tools

Reviewed June 30, 2026 · model on record in the stance chip above.

[1] [1]

Retrieval-Augmented Generation for Knowledge-Intensive

Lewis, Patrick and Perez, Ethan and Piktus, Aleksandra and Petroni, Fabio and Karpukhin, Vladimir and Goyal, Naman and Kuttler, Heinrich and Lewis, Mike and Yih, Wen-tau and Rockt. Retrieval-Augmented Generation for Knowledge-Intensive. Advances in Neural Information Processing Systems , volume =

work page

[2] [2]

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing , pages =

Dense Passage Retrieval for Open-Domain Question Answering , author =. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing , pages =

work page 2020

[3] [3]

Sentence-bert: Sentence embeddings using siamese bert-networks

Reimers, Nils and Gurevych, Iryna , booktitle =. Sentence-. doi:10.18653/v1/D19-1410 , url =

work page doi:10.18653/v1/d19-1410

[4] [4]

Wang, Wenhui and Wei, Furu and Dong, Li and Bao, Hangbo and Yang, Nan and Zhou, Ming , booktitle =

work page

[5] [5]

IEEE Transactions on Big Data , author =

Johnson, Jeff and Douze, Matthijs and J. Billion-Scale Similarity Search with. doi:10.1109/TBDATA.2019.2921572 , year =

work page doi:10.1109/tbdata.2019.2921572 2019

[6] [6]

Journal of Machine Learning Research , volume =

Scaling Instruction-Finetuned Language Models , author =. Journal of Machine Learning Research , volume =

work page

[7] [7]

Transactions of the Association for Computational Linguistics , volume =

Lost in the Middle: How Language Models Use Long Contexts , author =. Transactions of the Association for Computational Linguistics , volume =

work page

[8] [8]

SQ u AD : 100,000+ Questions for Machine Comprehension of Text

Rajpurkar, Pranav and Zhang, Jing and Lopyrev, Konstantin and Liang, Percy , booktitle =. doi:10.18653/v1/D16-1264 , url =

work page doi:10.18653/v1/d16-1264