Engineering RAG Systems for Real-World Applications: Design, Development, and Evaluation

Ayman Asad Khan; Kai-Kristian Kemell; Md Toufique Hasan; Mika Saari; Muhammad Waseem; Pekka Abrahamsson

REVIEW 4 major objections 4 minor 34 references

Engineering RAG Systems for Real-World Applications: Design, Development, and Evaluation

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Five RAG systems built for governance, cybersecurity, agriculture, engineering, and medicine were rated by 100 users, and the paper claims these ratings show retrieval-augmented generation is practical in real, multilingual settings.

desk verdict A modest but honest experience report: five real RAG deployments and a 100-user study, with weaknesses the paper itself mostly acknowledges—still worth refereeing. read the letter →

arxiv 2506.20869 v3 pith:EVJFWSUW submitted 2025-06-25 cs.SE cs.AIcs.IR

Md Toufique Hasan , Muhammad Waseem , Kai-Kristian Kemell , Ayman Asad Khan , Mika Saari , Pekka Abrahamsson This is my paper

classification cs.SEcs.AIcs.IR

keywords retrieval-augmentedgenerationRAGsystemsdomain-specificapplicationsuserevaluationmultilingualOCRLLMgroundinglessonslearnedsoftwareengineering

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that retrieval-augmented generation can be moved from clean benchmarks into working tools for real, knowledge-heavy settings. The authors built five RAG systems for municipal governance, cybersecurity, agriculture, industrial research, and medical diagnostics, each pairing multilingual OCR, vector-based retrieval, and a domain-adapted language model, and evaluated them with 100 users. Their claim is that such systems are practical now: users rated ease of use, relevance, transparency, responsiveness, and accuracy well enough that most would recommend the tools to colleagues. They also argue that the engineering effort surfaces recurring lessons around OCR noise, chunking, scaling, data cleaning, source transparency, and dataset bias that should guide future RAG development. The evidence of accuracy is user perception, and the paper itself flags that only a minority of raters were domain experts.

What carries the argument

The central mechanism is a two-phase RAG pipeline: a retrieval phase in which queries are embedded and matched against vector embeddings of domain documents, followed by a generation phase in which retrieved chunks are concatenated with the query and passed to an LLM. Around this, the paper places multilingual OCR preprocessing to bring scanned documents into the index, and a web-based evaluation instrument with six Likert-scale dimensions. This machinery carries the argument because it is what makes the five systems comparable and what lets the authors attribute user ratings to design choices such as chunk size, model selection, and transparency decisions.

What would settle it

Have domain experts fact-check a fixed set of system answers from each of the five systems and compare their correctness scores against the accuracy ratings given by the non-expert users on the same answers; if expert scores are markedly lower, the perceived-accuracy evidence would not establish factual reliability.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central discovery is that a single two-phase RAG architecture—embedding user queries, retrieving relevant chunks from a vector database, then passing chunks plus query to an LLM—can be adapted to five different high-stakes domains and receive positive user evaluations. Perceived accuracy was favorable, ranging from roughly 62 to 91 percent across systems, ease of use was consistently strong, and recommendation rates ranged from about 48 to 64 percent. The paper further claims that twelve lessons learned, drawn from development experience and user feedback, identify concrete technical, operational, and ethical levers: domain-specific models beat general ones for Finnish and specialized content, OCR noise degrades retrieval, chunk sizes of 200 to 500 tokens balance speed and relevance, metadata filtering helps at scale, clean data improves answers without model changes, and transparency trades against security. The authors position this as evidence that RAG can be engineered for real-world, multilingual, domain-specific use, while acknowledging that the evaluation captures interaction experience more than factual correctness.

Load-bearing premise

The whole argument rests on treating users' self-reported accuracy ratings as evidence of real-world performance, even though only 17% of the 100 participants were domain experts and the paper itself cautions that non-expert ratings reflect interaction experience rather than factual correctness.

Editorial extensions

If this is right

A modular RAG pipeline with multilingual OCR, vector retrieval, and a domain-adapted LLM can be reused across governance, agriculture, cybersecurity, engineering, and medicine without changing the core architecture.
Design choices measurably shape user trust: hiding sources for security lowers transparency ratings, while showing source filenames builds confidence.
Engineering decisions such as 200 to 500 token chunk sizes, metadata filtering for large corpora, and clean source data improve both retrieval speed and answer relevance.
User feedback alone is insufficient to catch factual errors, so reliable high-stakes deployments will need an automated evaluation agent that flags weak responses before they reach the user.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural testable extension is to have domain experts fact-check the same queries the 100 users rated; if expert fact-checking scores fall well below the perceived-accuracy ratings, the true reliability of these systems is lower than the user study suggests.
The authors' architecture separates preprocessing, retrieval, and generation, which means the same evaluation instrument could be applied to test whether stronger OCR or different chunking changes retrieval quality in a controlled way.
The lessons imply that data cleaning and domain-specific model choice may matter more than the retrieval architecture itself; this is an ordering the paper documents but does not isolate experimentally.
The proposed evaluation agent is falsifiable: running the five systems with and without it and comparing factual error rates would show whether automated response checking adds measurable reliability.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Briefly, what you should know: this is a genuine experience report from five RAG deployments with real partner organizations, not a lab demo. It ships no code, but it gives an honest account of what worked and what didn't. The evaluation is a 100-person user survey across five systems, and the paper is upfront about its limits. The novelty is thin—standard OCR plus embeddings plus vector store plus LLM pipelines—but the user-perception data and the twelve lessons are a real, citable addition to the practitioner literature.

What it does well: the system descriptions are concrete and domain-specific (municipal records, cybersecurity, Finnish agriculture, engineering documents, aneurysm diagnostics). The lessons learned read like genuine engineering experience: OCR noise degrading FAISS, chunk-size tradeoffs, FAISS latency at scale, fragile scraping pipelines, GDPR-driven self-hosting, source transparency except when security requires hiding it. The paper also concedes that user feedback alone is insufficient, and proposes an evaluation-agent mechanism for future work. That is the right attitude.

The soft spots are the ones the stress-test flags. Accuracy ratings come mostly from non-experts (17% domain experts), and the paper itself says those ratings may not reflect factual correctness. Per-system samples are 17–22, with no statistical tests, no baseline, no ground truth. So the specific percentages (e.g., 91% accuracy for the city AI) should not be read as objective accuracy. That said, the paper is careful to call them 'perceived accuracy' in places, and it lists this limitation in Section 7. The abstract is a bit more assertive than the evidence supports, but not dishonestly so. The bigger issue is the absence of any reproducibility artifact: no data, no code, no retrieval metrics. For a software-engineering experience report, that is a common limitation, but it means the lessons are trust-me claims based on the authors' narrative.

Who is this for? Practitioners building RAG systems in similar domains, and researchers collecting experience reports on LLM-based systems. It is not a technical innovation paper. With a serious referee who asks for a clearer separation of perceived accuracy from actual accuracy, a comparison to at least one baseline retrieval system, and an appendix with the survey instrument and per-system response distributions, this could be a solid SEAA-style paper. I would engage with it fairly, and I would not desk-reject it.

My recommendation: send it to peer review, with the expectation of moderate revision. The central message—RAG in practice is about data cleaning, chunking, and honest evaluation—holds up even if the statistics are soft.

Referee Report

4 major / 4 minor

Summary. The paper reports an experience-based study of five domain-specific RAG systems built with collaborating organizations: Kankaanpää City AI (governance), Disarm RAG (cybersecurity), AgriHubi AI Assist (agriculture), FEMMa Oracle (industrial research), and Assist Doctor (medical diagnostics). Each system combines multilingual OCR, embedding-based retrieval, a vector store, and a domain-adapted LLM. The evaluation is a web-based survey of 100 participants rating each system on six Likert-scale dimensions (Ease of Use, Relevance, Transparency, Responsiveness, Accuracy, Recommendation), supplemented by open-ended feedback and developer notes. Based on this material, the authors distill twelve lessons across technical, operational, and ethical categories. The paper's central claim is that these deployments and the user ratings provide evidence of real-world usability and perceived accuracy, and that the lessons are useful for engineering practical RAG systems.

Significance. The paper addresses a genuine gap: most RAG research is benchmark-based, whereas this is a multi-domain deployment experience report with real stakeholder collaborations. Its strengths are the breadth of domains, the concrete engineering details (OCR cleanup, chunk-size trade-offs, FAISS scaling, self-hosting for GDPR, source-reference transparency), and the unusually candid limitations section. The user survey, however, can support only perceived-usability and perceived-accuracy claims; the quantitative accuracy evidence is weak because most raters were not domain experts, samples per system are small, and no objective ground truth or baseline is provided. If the accuracy claims are appropriately reframed, and the statistical limitations are reported, the paper is a useful contribution for software engineering practitioners. Machine-checked proofs are not applicable here, but the concreteness of the system descriptions and the explicit statement of limitations are positive features.

major comments (4)

[§5.1, §5.2 (Figure 4), §7] The central evaluation claim in RQ2 and in the second contribution—that user ratings demonstrate 'real-world performance' on accuracy—is not supported by the data as reported. Only 17% of the 100 participants were domain experts (§5.1), and the authors themselves concede in §7 that accuracy ratings by non-experts 'might not reliably reflect the factual correctness of the systems' outputs.' Percentages such as the 91% 'Accuracy of Answers' for Kankaanpää City AI are therefore measures of perceived accuracy, not factual accuracy. The accuracy-related claims should either be restricted to perceived accuracy throughout the abstract, contributions, and conclusions, or supplemented by expert-annotated ground-truth evaluation for at least a sample of queries.
[§5.2] Per-system ratings rest on very small samples—22, 20, 20, 17, and 21 participants—and the paper reports no confidence intervals, raw counts, or significance tests. As a result, comparative statements such as 'FEMMa Oracle performed well across all criteria' (based on 17 responses) or 'Transparency was mixed' (45.5% vs. 45.5% of 22 participants) are presented without any estimate of uncertainty. The authors should report exact response counts, confidence intervals, and, if cross-system comparisons are intended, appropriate statistical tests or explicit non-comparative framing.
[§4, §5.2] The introduction states that RAG 'improves factual accuracy' and the contributions claim a user-centred evaluation of 'real-world performance,' but the study includes no baseline condition—neither manual search, nor a non-RAG LLM, nor an existing organizational tool. Without such a comparison, the data cannot establish that the RAG architecture itself is responsible for the observed ratings. I recommend adding at least a small comparative condition or weakening the causal/architectural claims and describing the study as a descriptive evaluation of five deployed systems.
[§6, §7] The twelve lessons in Section 6 are presented as findings, yet Section 7 explicitly states that they 'do not result from formal empirical analysis.' This is an internal tension: the abstract presents them as documented lessons learned, and the conclusion calls them 'twelve lessons learned,' while the limitations disclaim formal status. Please relabel these as experience-based hypotheses or observations, and state which lessons, if any, are corroborated by the survey data (e.g., the influence of source visibility on Transparency ratings).

minor comments (4)

[§5.2] In the FEMMa Oracle results, the percentages '88.9% found it transparent' and 'fast by (50%) and average by (28.6%)' do not align with the stated 17 participants; please clarify the response counts and whether the percentages are of all participants or of those who answered each item.
[§3.2] The paper describes the survey but does not provide the full instrument or an appendix link; adding the exact Likert question wording and response scales would improve reproducibility.
[§4] The statement that 'All developed systems comply with GDPR' is not substantiated; a sentence on data processing agreements, anonymization, or data retention would strengthen this claim.
[§1] The term 'real-world' is used for live demo environments and a web-based survey; please clarify the extent of actual organizational deployment versus demonstration use, since this affects how readers interpret the evaluation.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular reasoning: the paper makes no derived predictions or fitted-parameter claims; its empirical evaluation is self-contained and its limitations are explicitly acknowledged.

full rationale

This paper is an empirical engineering and evaluation study, not a derivation. It reports the design, implementation, and user evaluation of five RAG systems, along with twelve qualitative lessons learned. There is no formal chain in which an output quantity is defined in terms of an input quantity, no parameter fitted to one subset of data and then 'predicted' for a closely related quantity, and no forced derivation resting on the paper's own prior results. The only self-citation (Khan et al., reference [12]) is used in the Related Work section as background on PDF-focused retrieval and is not load-bearing for any of the paper's claims. The accuracy evaluation is explicitly based on self-reported user ratings, and the paper itself concedes that non-expert ratings 'might not reliably reflect the factual correctness of the systems' outputs' (Section 7). That is a validity limitation, not circularity, because the paper does not present these ratings as a derived consequence of the system design; it presents them as user-perception data. The lessons learned are likewise described as practical insights from development experience rather than as formally derived results. No circularity score is warranted.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper is an empirical study without mathematical derivations, so no fitted free parameters or invented entities are used. The axioms are the implicit assumptions behind the user-study design and the generalizability of its findings.

assumptions (3)

domain assumption Participant ratings of answer accuracy are a valid proxy for factual correctness.
The evaluation uses self-reported accuracy from users, only 17% of whom are domain experts, to claim real-world performance. Section 5.1 and Section 7 acknowledge this limits the validity of accuracy scores.
domain assumption The five selected domains and use cases are representative enough to support generalizable lessons for RAG engineering.
The paper generalizes lessons from five specific systems to broader RAG practice without a formal sampling or comparison framework; see Section 3.1 and Section 6.
domain assumption The survey responses accurately reflect participants' actual interaction experience.
Standard self-report assumption; not independently verified. Section 5.2 describes the anonymous web survey.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Engineering RAG Systems for Real-World Applications: Design, Development, and Evaluation." pith.science (2026). https://pith.science/paper/EVJFWSUW

@misc{pith2026250620869,
  author       = {Pith},
  title        = {Pith review of: Engineering RAG Systems for Real-World Applications: Design, Development, and Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EVJFWSUW}},
  note         = {Machine review of arXiv:2506.20869}
}

read the original abstract

Retrieval-Augmented Generation (RAG) systems are emerging as a key approach for grounding Large Language Models (LLMs) in external knowledge, addressing limitations in factual accuracy and contextual relevance. However, there is a lack of empirical studies that report on the development of RAG-based implementations grounded in real-world use cases, evaluated through general user involvement, and accompanied by systematic documentation of lessons learned. This paper presents five domain-specific RAG applications developed for real-world scenarios across governance, cybersecurity, agriculture, industrial research, and medical diagnostics. Each system incorporates multilingual OCR, semantic retrieval via vector embeddings, and domain-adapted LLMs, deployed through local servers or cloud APIs to meet distinct user needs. A web-based evaluation involving a total of 100 participants assessed the systems across six dimensions: (i) Ease of Use, (ii) Relevance, (iii) Transparency, (iv) Responsiveness, (v) Accuracy, and (vi) Likelihood of Recommendation. Based on user feedback and our development experience, we documented twelve key lessons learned, highlighting technical, operational, and ethical challenges affecting the reliability and usability of RAG systems in practice.

Figures

Figures reproduced from arXiv: 2506.20869 by the authors.

**Figure 1.** Overview of the research methodology – Generation Phase: The retrieved text chunks are concatenated with the original user query and passed into a large language model (LLM), such as GPT-4o, LLaMA 2 Uncensored, or Poro-34B, to synthesize contextually relevant responses. This approach improves factual accuracy, minimizes hallucinations, and delivers insights that are well aligned with domain-specific needs. Core Co… view at source ↗

**Figure 2.** System architecture of five RAG-based systems, showing data processing, vector storage, retrieval, and LLM-based response generation. system integrates red team techniques (e.g., phishing, deep-fake disinformation, privilege escalation) and blue team strategies (e.g., bot detection, misinformation control, network forensics), grounded in the Disarm Framework. It supports queries such as “How would you create a dee… view at source ↗

**Figure 3.** Participant profiles and interaction with the RAG systems. either moderately (41%) or very familiar (34%) with AI-based RAG systems. However, since many participants were not domain experts in the specific fields covered by the systems (e.g., healthcare, cybersecurity), their feedback primarily reflects their interaction experience with RAG rather than deep subject-matter validation. 4. Experience with AI-Based RAG:… view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: presents the aggregated user ratings across six evaluation criteria for all five RAG systems, offering a comparative perspective on system performance. To capture both measurable and descriptive insights, we employed a survey combining Likert-scale questions (1–5 scale…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

34 extracted references · 9 canonical work pages

[1]

In: 2024 IEEE International Conference on Soft- ware Maintenance and Evolution (ICSME), pp

Abrahamyan, D., Fard, F.H.: StackRAG agent: improving developer answers with retrieval-augmented generation. In: 2024 IEEE International Conference on Soft- ware Maintenance and Evolution (ICSME), pp. 893–897. IEEE Computer Society, Los Alamitos (2024). https://doi.org/10.1109/ICSME58944.2024.00098. https:// doi.ieeecomputersociety.org/10.1109/ICSME58944....

arXiv 2024
[2]

In: 2024 6th Novel Intelligent and Leading Emerging Sciences Con- ference (NILES), pp

Ahmed, M., et al.: Codeqa: advanced programming question-answering using llm agent and rag. In: 2024 6th Novel Intelligent and Leading Emerging Sciences Con- ference (NILES), pp. 494–499 (2024). https://doi.org/10.1109/NILES63360.2024. 10753267

work page arXiv 2024
[3]

Towards Interpretable Radiology Report Generation via Concept Bottlenecks using a Multi-Agentic RAG

Alam, H.M.T., Srivastav, D., Kadir, M.A., Sonntag, D.: Towards interpretable radi- ology report generation via concept bottlenecks using a multi-agentic rag (2025). https://arxiv.org/abs/2412.16086 Engineering RAG Systems for Real-World Applications 157

work page Pith review arXiv 2025
[4]

In: Proceed- ings of the IEEE/ACM 3rd International Conference on AI Engineering - Software Engineering for AI, CAIN ’24, pp

Barnett, S., Kurniawan, S., Thudumu, S., Brannelly, Z., Abdelrazek, M.: Seven fail- ure points when engineering a retrieval augmented generation system. In: Proceed- ings of the IEEE/ACM 3rd International Conference on AI Engineering - Software Engineering for AI, CAIN ’24, pp. 194–199. Association for Computing Machinery, New York (2024). https://doi.org...

arXiv 2024
[5]

Document Haystacks: Vision-Language Reasoning Over Piles of 1000+ Documents

Chen, J., Xu, D., Fei, J., Feng, C.M., Elhoseiny, M.: Document haystacks: vision- language reasoning over piles of 1000+ documents (2024). https://arxiv.org/abs/ 2411.16740

work page Pith review arXiv 2024
[6]

In: Li, S., et al

Chirkova, N., Rau, D., Déjean, H., Formal, T., Clinchant, S., Nikoulina, V.: Retrieval-augmented generation in multilingual settings. In: Li, S., et al. (eds.) Proceedings of the 1st Workshop on Towards Knowledgeable Language Mod- els (KnowLLM 2024), pp. 177–188. Association for Computational Linguis- tics, Bangkok (2024). https://doi.org/10.18653/v1/2024...

doi:10.18653/v1/2024.knowllm-1.15 2024
[7]

In: Proceedings of the 30th ACM SIGKDD Conference on Knowl- edge Discovery and Data Mining, KDD ’24, pp

Fan, W., et al.: A survey on rag meeting llms: towards retrieval-augmented large language models. In: Proceedings of the 30th ACM SIGKDD Conference on Knowl- edge Discovery and Data Mining, KDD ’24, pp. 6491–6501. Association for Com- puting Machinery, New York (2024). https://doi.org/10.1145/3637528.3671470

arXiv 2024
[8]

In: 2024 IEEE International Conference on Indus- trial Technology (ICIT), pp

Gamage, G., et al.: Multi-agent rag chatbot architecture for decision support in net- zero emission energy systems. In: 2024 IEEE International Conference on Indus- trial Technology (ICIT), pp. 1–6 (2024). https://doi.org/10.1109/ICIT58233.2024. 10540920

arXiv 2024

Show all 34 references

[9]

https://arxiv.org/abs/2410.05779

Guo, Z., Xia, L., Yu, Y., Ao, T., Huang, C.: Lightrag: simple and fast retrieval- augmented generation (2024). https://arxiv.org/abs/2410.05779

2024 arXiv
[10]

https://arxiv.org/abs/2410.12837

Gupta, S., Ranjan, R., Singh, S.N.: A comprehensive survey of retrieval-augmented generation (rag): evolution, current landscape and future directions (2024). https://arxiv.org/abs/2410.12837

2024 arXiv
[11]

In: Proceedings of the 2024 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Paciﬁc Region, SIGIR-AP 2024, pp

Jang, J., Li, W.S.: Au-rag: agent-based universal retrieval augmented generation. In: Proceedings of the 2024 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Paciﬁc Region, SIGIR-AP 2024, pp. 2–11. Association for Comp...

2024
[12]

https://arxiv.org/abs/2410.15944

Khan, A.A., Hasan, M.T., Kemell, K.K., Rasku, J., Abrahamsson, P.: Developing retrieval augmented generation (rag) based llm systems from pdfs: an experience report (2024). https://arxiv.org/abs/2410.15944

2024 arXiv
[13]

https://arxiv.org/abs/2409.12941

Krishna, S., et al.: Fact, fetch, and reason: a uniﬁed evaluation of retrieval- augmented generation (2025). https://arxiv.org/abs/2409.12941

2025 arXiv
[14]

https://arxiv.org/abs/2005.11401

Lewis, P., et al.: Retrieval-augmented generation for knowledge-intensive nlp tasks (2021). https://arxiv.org/abs/2005.11401

2021 arXiv
[15]

In: Rambow, O., Wanner, L., Apidianaki, M., Al-Khalifa, H., Eugenio, B.D., Schockaert, S

Li, S., Stenzel, L., Eickhoﬀ, C., Bahrainian, S.A.: Enhancing retrieval-augmented generation: a study of best practices. In: Rambow, O., Wanner, L., Apidianaki, M., Al-Khalifa, H., Eugenio, B.D., Schockaert, S. (eds.) Proceedings of the 31st International Conference on Computa...

2025
[16]

https://arxiv.org/abs/2411.19710

de Lima, R.T., et al.: Know your rag: dataset taxonomy and generation strategies for evaluating rag systems (2024). https://arxiv.org/abs/2411.19710

2024 arXiv
[17]

In: The Twelfth Inter- national Conference on Learning Representations (2024)

Liu, X., et al.: Agentbench: evaluating LLMs as agents. In: The Twelfth Inter- national Conference on Learning Representations (2024). https://openreview.net/ forum?id=zAdUB0aCTQ 158 M. T. Hasan et al

2024
[18]

https://arxiv.org/abs/2411.17073

Naeem, A., et al.: Path-rag: knowledge-guided key region retrieval for open-ended pathology visual question answering (2024). https://arxiv.org/abs/2411.17073

2024 arXiv
[19]

https://arxiv.org/abs/2310.18648

Nguyen-Duc, A., et al.: Generative artiﬁcial intelligence for software engineering – a research agenda (2023). https://arxiv.org/abs/2310.18648

2023 arXiv
[20]

https:// arxiv.org/abs/2411.19804

Pesl, R.D., Mathew, J.G., Mecella, M., Aiello, M.: Advanced system integration: analyzing openapi chunking for retrieval-augmented generation (2024). https:// arxiv.org/abs/2411.19804

2024 arXiv
[21]

In: Proceedings of the 5th ACM International Conference on AI in Finance, ICAIF ’24, pp

Sarmah, B., Mehta, D., Hall, B., Rao, R., Patel, S., Pasquali, S.: Hybridrag: inte- grating knowledge graphs and vector retrieval augmented generation for eﬃcient information extraction. In: Proceedings of the 5th ACM International Conference on AI in Finance, ICAIF ’24, pp. 6...

2024
[22]

In: 2024 IEEE LLM Aided Design Workshop (LAD), pp

Shi, L., Kazda, M., Sears, B., Shropshire, N., Puri, R.: Ask-eda: a design assistant empowered by llm, hybrid rag and abbreviation de-hallucination. In: 2024 IEEE LLM Aided Design Workshop (LAD), pp. 1–5 (2024). https://doi.org/10.1109/ LAD62341.2024.10691824

2024
[23]

https://arxiv.org/abs/2501.09136

Singh, A., Ehtesham, A., Kumar, S., Khoei, T.T.: Agentic retrieval-augmented generation: a survey on agentic rag (2025). https://arxiv.org/abs/2501.09136

2025 arXiv
[24]

In: 2024 6th International Conference on Electrical, Control and Instrumentation Engineering (ICECIE), pp

Sree, Y.B., Sathvik, A., Hema Akshit, D.S., Kumar, O., Pranav Rao, B.S.: Retrieval-augmented generation based large language model chatbot for improving diagnosis for physical and mental health. In: 2024 6th International Conference on Electrical, Control and Instrumentation E...

2024
[25]

https://doi.org/10.1007/s11704-024-40231-1

Wang, L., et al.: A survey on large language model based autonomous agents (2024). https://doi.org/10.1007/s11704-024-40231-1

2024 doi
[26]

In: Al-Onaizan, Y., Bansal, M., Chen, Y.N

Wang, X., et al.: Searching for best practices in retrieval-augmented generation. In: Al-Onaizan, Y., Bansal, M., Chen, Y.N. (eds.) Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 17716–17736. Associ- ation for Computational Linguist...

2024 doi
[27]

In: Proceedings of the 33rd ACM International Conference on Informa- tion and Knowledge Management, CIKM ’24, pp

Xu, A., et al.: Generative ai and retrieval-augmented generation (rag) systems for enterprise. In: Proceedings of the 33rd ACM International Conference on Informa- tion and Knowledge Management, CIKM ’24, pp. 5599–5602. Association for Com- puting Machinery, New York (2024). h...

2024
[28]

https://arxiv.org/abs/2401.15884

Yan, S.Q., Gu, J.C., Zhu, Y., Ling, Z.H.: Corrective retrieval augmented generation (2024). https://arxiv.org/abs/2401.15884

2024 arXiv
[29]

https://arxiv.org/abs/2410.10594

Yu, S., et al.: Visrag: vision-based retrieval-augmented generation on multi- modality documents (2024). https://arxiv.org/abs/2410.10594

2024 arXiv
[30]

In: Ku, L.W., Martins, A., Srikumar, V

Zeng, A., et al.: AgentTuning: enabling generalized agent abilities for LLMs. In: Ku, L.W., Martins, A., Srikumar, V. (eds.) Findings of the Association for Com- putational Linguistics: ACL 2024, pp. 3053–3077. Association for Computational Linguistics, Bangkok (2024). https:/...

2024 doi
[31]

https://arxiv.org/abs/2411.14572

Zeng, S., et al.: Towards knowledge checking in retrieval-augmented generation: a representation perspective (2024). https://arxiv.org/abs/2411.14572

2024
[32]

https://arxiv.org/abs/2412.02592

Zhang, J., et al.: Ocr hinders rag: evaluating the cascading impact of ocr on retrieval-augmented generation (2024). https://arxiv.org/abs/2412.02592

2024 arXiv
[33]

https://arxiv.org/abs/2402.19473

Zhao, P., et al.: Retrieval-augmented generation for ai-generated content: a survey (2024). https://arxiv.org/abs/2402.19473

2024 arXiv
[34]

https://arxiv.org/ abs/2411.19463

Zhao, S., Huang, Y., Song, J., Wang, Z., Wan, C., Ma, L.: Towards understanding retrieval accuracy and prompt quality in rag systems (2024). https://arxiv.org/ abs/2411.19463

2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

In: 2024 IEEE International Conference on Soft- ware Maintenance and Evolution (ICSME), pp

Abrahamyan, D., Fard, F.H.: StackRAG agent: improving developer answers with retrieval-augmented generation. In: 2024 IEEE International Conference on Soft- ware Maintenance and Evolution (ICSME), pp. 893–897. IEEE Computer Society, Los Alamitos (2024). https://doi.org/10.1109/ICSME58944.2024.00098. https:// doi.ieeecomputersociety.org/10.1109/ICSME58944....

arXiv 2024

[2] [2]

In: 2024 6th Novel Intelligent and Leading Emerging Sciences Con- ference (NILES), pp

Ahmed, M., et al.: Codeqa: advanced programming question-answering using llm agent and rag. In: 2024 6th Novel Intelligent and Leading Emerging Sciences Con- ference (NILES), pp. 494–499 (2024). https://doi.org/10.1109/NILES63360.2024. 10753267

work page arXiv 2024

[3] [3]

Towards Interpretable Radiology Report Generation via Concept Bottlenecks using a Multi-Agentic RAG

Alam, H.M.T., Srivastav, D., Kadir, M.A., Sonntag, D.: Towards interpretable radi- ology report generation via concept bottlenecks using a multi-agentic rag (2025). https://arxiv.org/abs/2412.16086 Engineering RAG Systems for Real-World Applications 157

work page Pith review arXiv 2025

[4] [4]

In: Proceed- ings of the IEEE/ACM 3rd International Conference on AI Engineering - Software Engineering for AI, CAIN ’24, pp

Barnett, S., Kurniawan, S., Thudumu, S., Brannelly, Z., Abdelrazek, M.: Seven fail- ure points when engineering a retrieval augmented generation system. In: Proceed- ings of the IEEE/ACM 3rd International Conference on AI Engineering - Software Engineering for AI, CAIN ’24, pp. 194–199. Association for Computing Machinery, New York (2024). https://doi.org...

arXiv 2024

[5] [5]

Document Haystacks: Vision-Language Reasoning Over Piles of 1000+ Documents

Chen, J., Xu, D., Fei, J., Feng, C.M., Elhoseiny, M.: Document haystacks: vision- language reasoning over piles of 1000+ documents (2024). https://arxiv.org/abs/ 2411.16740

work page Pith review arXiv 2024

[6] [6]

In: Li, S., et al

Chirkova, N., Rau, D., Déjean, H., Formal, T., Clinchant, S., Nikoulina, V.: Retrieval-augmented generation in multilingual settings. In: Li, S., et al. (eds.) Proceedings of the 1st Workshop on Towards Knowledgeable Language Mod- els (KnowLLM 2024), pp. 177–188. Association for Computational Linguis- tics, Bangkok (2024). https://doi.org/10.18653/v1/2024...

doi:10.18653/v1/2024.knowllm-1.15 2024

[7] [7]

In: Proceedings of the 30th ACM SIGKDD Conference on Knowl- edge Discovery and Data Mining, KDD ’24, pp

Fan, W., et al.: A survey on rag meeting llms: towards retrieval-augmented large language models. In: Proceedings of the 30th ACM SIGKDD Conference on Knowl- edge Discovery and Data Mining, KDD ’24, pp. 6491–6501. Association for Com- puting Machinery, New York (2024). https://doi.org/10.1145/3637528.3671470

arXiv 2024

[8] [8]

In: 2024 IEEE International Conference on Indus- trial Technology (ICIT), pp

Gamage, G., et al.: Multi-agent rag chatbot architecture for decision support in net- zero emission energy systems. In: 2024 IEEE International Conference on Indus- trial Technology (ICIT), pp. 1–6 (2024). https://doi.org/10.1109/ICIT58233.2024. 10540920

arXiv 2024