All for law and law for all: Adaptive RAG Pipeline for Legal Research

Aravindh Manivannan; Dion Fernandes; Faisal Ahmad; Figarri Keisha; Ilham Wicaksono; Pallavi; Prince Singh; Wiem Ben Rim

REVIEW 4 major objections 4 minor 1 cited by

All for law and law for all: Adaptive RAG Pipeline for Legal Research

T0 review · 4 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that a fully open-source, adaptively tuned legal RAG pipeline can match proprietary retrieval quality and, with a custom legal-grounded prompt, generate more faithful answers than baseline prompting.

desk verdict The open-source retrieval result is plausible and useful, but the intro oversells it, and the generation win rests on dropping two metrics after they disagreed with the preferred conclusion—send to review, expect heavy revision. read the letter →

arxiv 2508.13107 v2 pith:2BP5BYL6 submitted 2025-08-18 cs.CL cs.IR

Figarri Keisha , Prince Singh , Pallavi , Dion Fernandes , Aravindh Manivannan , Ilham Wicaksono , Faisal Ahmad , Wiem Ben Rim This is my paper · ORCID

classification cs.CLcs.IR

keywords retrieval-augmentedgenerationlegalresearchquerytranslationopen-sourceembeddingsBenchRAGfaithfulnessevaluationpromptengineering

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that legal retrieval-augmented generation does not need proprietary services: a deliberately tuned pipeline built from open-source components can rival paid retrieval and, with a custom legal-grounded prompt, produce more faithful answers than standard prompts. The claimed retrieval gains are large—Recall@K improves by 30–95% and Precision@K by roughly 2.5× for K>4 against the LegalBenchRAG reference—while the generation experiments indicate that prompt design matters more than model choice. A sympathetic reader would care because legal research is high-stakes and error-prone, and if the claim holds, organisations could deploy cheaper, reproducible, and legally grounded question answering without sacrificing quality.

What carries the argument

The load-bearing mechanism is the query translator sitting in front of a retrieval and generation loop. It rewrites the user's question, extracts any file path or document reference so retrieval can be narrowed to that document, and labels the query as expert versus layperson and vague versus verbose; those labels set retrieval depth (K) and the response style. Retrieval then uses cosine similarity over RCTS-chunked documents embedded with SBERT (GTE as a close alternative), and generation uses a custom legal-grounded prompt that restricts the model to answering from the retrieved chunks. The pipeline's three-way evaluation—RAGAS Faithfulness, BERTScore-F1, and ROUGE-Recall—is designed to separate semantic grounding from lexical overlap.

What would settle it

Re-run the Section 6.2 generation comparison on a fresh LegalBenchRAG sample keeping the original RAGAS answer relevancy score with its non-committal multiplier and ROUGE-Recall as decision metrics: if the baseline prompt then ties or beats the custom prompt on those metrics, or if faithfulness continues to rise beyond K=5, the paper's central generation claim fails.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central discovery is that component-level adaptation carries legal RAG performance: a context-aware query translator that separates document references from natural-language questions and classifies expertise and specificity, combined with RCTS chunking, open-source SBERT or GTE embeddings, cosine retrieval, and a hand-tuned legal prompt, reaches or exceeds the retrieval quality of the proprietary text-embedding-3-large model used in LegalBenchRAG on precision for all evaluated k and matches it at low k, while lifting Recall@K by 30–95% and Precision@K by roughly 2.5× for K>4. In generation, the paper finds GPT-4o-mini with the custom prompt consistently outscores the baseline prompt and zero-shot chain-of-thought on RAGAS Faithfulness and BERTScore-F1, with K=5 as the chosen operating point because faithfulness plateaus beyond it. The paper also argues that two common metrics—RAGAS answer relevancy and ROUGE-Recall—should be discarded in this setting, on the grounds that answer relevancy mostly reflects the rate of non-committal replies and ROUGE penalises valid paraphrase.

Load-bearing premise

The generation conclusion rests on the assumption that RAGAS Faithfulness and BERTScore-F1 are the right quality measures, and that the discarded answer-relevancy and ROUGE-Recall metrics are genuinely uninformative rather than revealing of a real quality difference.

Editorial extensions

If this is right

An all-open-source legal RAG stack can be deployed without per-query embedding costs while matching proprietary retrieval on precision and low-k recall.
Prompt design, specifically a legal-grounded instruction that forces answers to be derived from retrieved chunks, is a first-order lever on faithfulness—larger than swapping the underlying LLM.
K=5 is a defensible operating point for legal research assistants: beyond it retrieval precision falls and faithfulness stops improving.
Using RAGAS answer relevancy or ROUGE-Recall alone would mislead legal RAG tuning, because non-committal answers and valid paraphrase are penalised respectively.
Task-aware, component-level tuning (query translation, chunking, embedding, prompt) is what delivers the reported gains, not any single model.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial: because the paper shows non-committal answers drop as k grows, a 'decisiveness' score could be added to legal RAG evaluation as a first-class metric alongside faithfulness, giving a concrete way to trade off recall and answer confidence.
Editorial: the same query-translator-plus-open-retriever recipe should transfer to other dense, jargon-heavy domains such as regulatory compliance or medical records, where paraphrase-tolerant semantic metrics matter more than n-gram overlap.
Editorial: a natural follow-up experiment is to test whether the expert/layperson classification changes real user outcomes—e.g., comprehension or task completion—rather than only output length and detail, since the paper's qualitative examples show detail increases for expert queries.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is a sincere engineering paper with one genuinely new piece—the query translator that classifies expertise and specificity and adjusts retrieval depth and response style. The retrieval comparison is the strongest part: SBERT/GTE with RCTS chunking and cosine similarity, unranked, slightly beats OpenAI's text-embedding-3-large on precision and is comparable on recall. That is a useful, non-obvious practitioner result, and the paper is honest enough to show it in the body (Section 6.1) as 'slightly better', not the 30–95% Recall@K and ~2.5x Precision@K the intro claims. That gap is the first thing to fix: a 30–95% claim does not match 'very similar for initial k, lower as k increases'. Either the numbers come from a different comparison, or the abstract and intro need rewriting.

The soft spot is the generation evaluation. Section 6.2 reports four metrics. Faithfulness and BERTScore-F1 favor GPT-4o-mini with the custom prompt; ROUGE-Recall favors Llama-3-8B with the custom prompt; answer relevancy favors the baseline GPT prompt. The paper then spends two appendices arguing why ROUGE and answer relevancy are unsuitable—after seeing that they disagree. The reasons have some merit (ROUGE is lexical; answer relevancy is heavily affected by the non-committal multiplier, and they show that removing the multiplier erases the gap). But a reader cannot tell whether the exclusion was pre-planned or post-hoc, because there is no held-out benchmark, no pre-registered endpoints, and no code or data to check. K=5 is also selected from the same faithfulness curves used to report the result. So the central claim—'consistently more faithful and contextually relevant answers'—is plausible but not convincingly established as written. The answer relevancy result matters especially because it is the only metric that checks whether the answer addresses the user's question, and it favors the baseline.

Also, no code or data is released despite the 'reproducible' claim, and the manuscript text I saw is partial (sections 2–4 missing), so I could not inspect the classifier details or the training procedure.

Who this is for: practitioners building legal RAG, especially people weighing open-source embeddings against paid APIs. They will get a useful configuration and a cautionary example of how metric choice can drive conclusions. As a research contribution it is incremental; as an engineering report it deserves a serious referee. My recommendation: send to peer review, but require the authors to (a) reconcile the intro's quantitative claims with Section 6.1, (b) pre-specify the metric set or evaluate on a true held-out split with K and prompt chosen on a validation set, and (c) release code and data. If those are addressed, the retrieval finding alone warrants publication.

Referee Report

4 major / 4 minor

Summary. The paper presents an end-to-end retrieval-augmented generation pipeline for legal research. It claims three contributions: a context-aware query translator that classifies user expertise and query specificity and rewrites queries; an open-source retrieval stage using SBERT and GTE embeddings compared against LegalBenchRAG's OpenAI text-embedding-3-large and the RetroMAE model; and a generation/evaluation framework using GPT-4o-mini and LLaMA-3-8B under three prompting strategies, evaluated with RAGAS Faithfulness, answer relevancy, BERTScore-F1 with LegalBERT, and ROUGE-Recall. The authors conclude that open-source retrieval can rival proprietary embeddings and that a custom legal-grounded prompt with GPT-4o-mini and K=5 yields the most faithful and contextually relevant answers. The final pipeline includes a readability/complexity classifier that adapts response detail to the predicted user expertise. The evaluation is conducted on sampled subsets of LegalBenchRAG.

Significance. If the results held up, the paper would provide valuable evidence that cost-effective, open-source components can match proprietary RAG systems in the legal domain, and that prompt design can matter as much as model choice. The paper has strengths: it systematically compares chunking, embedding, and similarity-search configurations; it uses a domain-specific LegalBERT for semantic similarity; and it provides qualitative examples of the adaptive response behavior. However, as written, the headline generation claim relies on selectively retained metrics and same-benchmark tuning, and the retrieval claim in the introduction conflicts with the results section. These issues must be resolved before the conclusions can be accepted.

major comments (4)

[Introduction, Contribution 2, and Section 6.1] The introduction states that the tailored retrieval 'improves Recall@K by 30–95% and Precision@K by ~2.5× for K>4' against LegalBenchRAG, but Section 6.1 reports that the best open-source configuration performs only 'slightly better on precision for all k-values and very similar for recall for initial k-values, which gets lower as k-value increases' relative to LegalBenchRAG's OpenAI model. These statements are inconsistent. Please state the exact comparison baseline, report the numerical results for the specific configurations, and reconcile the 30–95% figure with the 'slightly better' description.
[Section 6.2 and Appendices D and E] The generation evaluation reports four metrics. RAGAS answer relevancy favors the baseline GPT prompt, ROUGE-Recall favors LLaMA-3-8B with the custom prompt, and only Faithfulness and BERTScore-F1 favor GPT-4o-mini with the custom prompt. The paper excludes answer relevancy and ROUGE-Recall after observing this disagreement, arguing in Appendices D and E that these metrics are unsuitable. This is post-hoc metric selection: the retained metrics both measure grounding in the retrieved context, and neither checks whether the answer actually addresses the user's question, which is precisely what answer relevancy measures. With no pre-specified primary endpoint and no held-out evaluation, the claim that the custom prompt is 'consistently more faithful and contextually relevant' is not empirically established.
[Section 6.2 and final pipeline] The optimal K value is selected by inspecting the same faithfulness and BERT-F1 curves that are then used to demonstrate the pipeline's success ('faithfulness stagnates after K=5 ... we choose K=5'), and the same LegalBenchRAG subsets appear to be used for prompt tuning, retrieval tuning, and final evaluation. No held-out set is described. This selection-on-the-test-set inflates the apparent advantage of the advertised configuration and prevents the comparison from being an independent test.
[Section 5 and Appendix F] The Readability and Complexity (R&C) classifier is a central component of the claimed adaptive behavior (expert vs. non-expert, vague vs. verbose), but the manuscript reports no quantitative evaluation of its classification accuracy. Appendix F provides only qualitative examples. If the classifier misclassifies expertise, the adaptive response claims do not follow. Please report its accuracy or a confusion matrix on the sampled queries, or re-scope the claim as a proof-of-concept.

minor comments (4)

[Abstract] The phrase 'consistently produces more faithful and contextually relevant answers' is too strong given that the answer relevancy metric favored the baseline prompt; please qualify the claim to reflect the metric-selection issues described in Section 6.2.
[Introduction, Contribution 2] There is a typo in 'Text Embeding 3 Large'; it should be 'text-embedding-3-large'.
[Section 6.1] The label 'SoTA model (RetroMAE)' is used without a citation; please provide a reference for this state-of-the-art claim.
[Abstract and throughout] The paper promises reproducibility but does not state whether code and data will be released; please include an availability statement or explicitly note limitations.

Circularity Check

0 steps flagged · score 0.0 of 10

No construction-level circularity: the central claims are empirical benchmark comparisons on external LegalBenchRAG, not derivations from their own inputs. The notable weaknesses are in-sample configuration selection and post-hoc metric pruning, which are correctness and validity risks rather than circular reasoning.

full rationale

I examined the paper for the six circularity patterns. The paper contains no formal derivation chain in which a predicted quantity is equal by construction to a fitted input. The query translator, retrieval configuration, K-value, and prompt are selected by experimentation, and the same LegalBenchRAG subsets are reused for the reported results; this is an overfitting and selection-bias concern, not a tautology, because the reported scores are measured outcomes rather than consequences of the selection rule. No load-bearing self-citation appears: the benchmark (LegalBenchRAG) and the SoTA comparison (RetroMAE) are external. No uniqueness theorem, ansatz-by-citation, or renaming-of-known-result pattern is present. The post-hoc exclusion of answer relevancy and ROUGE-Recall after they contradicted the preferred conclusion (Section 6.2, Appendices D and E) is a serious methodological and reporting risk, but it is not a definitional equivalence: Faithfulness and BERTScore-F1 are still independent measurements of the generated outputs and could, in principle, have favored the baseline. These weaknesses should be assessed as correctness risk, not circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical or formal entities; it is an engineering system. Free parameters include the retrieval depth k, the dataset-specific thresholds for the query translator, and the number of artificial questions in the answer relevancy metric. These parameters are tuned on the benchmark used for evaluation, which raises selection concerns. The main axioms are the validity of the benchmark and the retained evaluation metrics.

free parameters (3)

Retrieval depth k = K=5 default; K=10 for expert queries
Selected in Section 6.2 from the same faithfulness and BERT-F1 curves used for evaluation; K=5 chosen because faithfulness stagnates after K=5.
Dataset-specific file-path detection thresholds = Not specified
Section 5 says thresholds are dataset-specific; values are not reported, making the adaptive query translation hard to reproduce or audit.
Number of artificial questions for answer relevancy = 3
Appendix D compares 3 vs 5 questions and chooses 3 because differences are minuscule; this parameter affects the answer relevancy metric.

assumptions (4)

domain assumption LegalBenchRAG provides a valid gold standard for legal QA retrieval and generation evaluation.
The paper compares all retrieval and generation results against LegalBenchRAG data and baselines in Sections 1 and 6.
ad hoc to paper RAGAS Faithfulness and BERTScore-F1 are valid quality measures, while ROUGE-Recall and answer relevancy are not.
This is asserted in Section 6.2 and Appendices D-E after observing the discarded metrics disagree with the preferred conclusion.
domain assumption The query complexity classifier correctly estimates user expertise and query specificity without reported accuracy evaluation.
The pipeline adapts retrieval depth and response style based on this classifier in Section 5 and Appendix F, but no accuracy or error analysis for the classifier is provided.
domain assumption RCTS chunking is appropriate for legal documents and preserves semantic boundaries.
RCTS is used as the best chunking method based on retrieval results in Section 6.1, following LegalBenchRAG.

how reviews work

0 comments

Cite this review

Pith. "Pith review of All for law and law for all: Adaptive RAG Pipeline for Legal Research." pith.science (2026). https://pith.science/paper/2BP5BYL6

@misc{pith2026250813107,
  author       = {Pith},
  title        = {Pith review of: All for law and law for all: Adaptive RAG Pipeline for Legal Research},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2BP5BYL6}},
  note         = {Machine review of arXiv:2508.13107}
}

read the original abstract

Retrieval-Augmented Generation (RAG) has transformed how we approach text generation tasks by grounding Large Language Model (LLM) outputs in retrieved knowledge. This capability is especially critical in the legal domain. In this work, we introduce a novel end-to-end RAG pipeline that improves upon previous baselines using three targeted enhancements: (i) a context-aware query translator that disentangles document references from natural-language questions and adapts retrieval depth and response style based on expertise and specificity, (ii) open-source retrieval strategies using SBERT and GTE embeddings that achieve substantial performance gains while remaining cost-efficient, and (iii) a comprehensive evaluation and generation framework that combines RAGAS, BERTScore-F1, and ROUGE-Recall to assess semantic alignment and faithfulness across models and prompt designs. Our results show that carefully designed open-source pipelines can rival proprietary approaches in retrieval quality, while a custom legal-grounded prompt consistently produces more faithful and contextually relevant answers than baseline prompting. Taken together, these contributions demonstrate the potential of task-aware, component-level tuning to deliver legally grounded, reproducible, and cost-effective RAG systems for legal research assistance.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Evaluating RAG for French immigration law: a benchmark and baseline study
cs.IR 2026-07 conditional novelty 5.0 of 10

Dense RAG improves French immigration permit-type accuracy over parametric Qwen baselines on a 52-profile public benchmark, with weaker gains on documents and citations.

Reference graph

Works this paper leans on

12 extracted references · 2 canonical work pages · cited by 1 Pith paper

[1]

Chalkidis, M

I. Chalkidis, M. Fergadiotis, P. Malakasiotis, N. Aletras, and I. Androutsopoulos. 2020. https://doi.org/10.18653/v1/2020.findings-emnlp.261 LEGAL-BERT : The muppets straight out of law school . In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2898--2904, Online. Association for Computational Linguistics

doi:10.18653/v1/2020.findings-emnlp.261 2020
[2]

S. Es, J. James, L. Espinosa-Anke, and S. Schockaert. 2023. https://doi.org/10.48550/arXiv.2309.15217 Ragas: Automated evaluation of retrieval augmented generation . CoRR, abs/2309.15217

arXiv doi:10.48550/arxiv.2309.15217 2023
[3]

Exploding Gradients . 2025. GitHub - explodinggradients/ragas: Supercharge Your LLM Application Evaluations . https://github.com/explodinggradients/ragas/tree/main. Accessed: 2025-04-07

work page 2025
[4]

N. Guha, M. Lamm, B. Wu, A. Zhang, J. Yin, Y. Taori, Y. Zhang, S. S. Schoenholz, R. G. Krishnan, and C. D. Manning. 2023. https://doi.org/10.48550/arXiv.2308.11462 Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models . CoRR, abs/2308.11462

arXiv doi:10.48550/arxiv.2308.11462 2023
[5]

Y. Han, A. Ceross, and J. H. M. Bergmann. 2024. http://arxiv.org/abs/2411.09497 The use of readability metrics in legal text: A systematic literature review . CoRR, abs/2411.09497

arXiv 2024
[6]

Jeong, J

S. Jeong, J. Baek, S. Cho, S. J. Hwang, and J. C. Park. 2024. https://doi.org/10.48550/arXiv.2403.14403 Adaptive-rag: Learning to adapt retrieval-augmented large language models through question complexity . CoRR, abs/2403.14403

arXiv doi:10.48550/arxiv.2403.14403 2024
[7]

Kalra, Z

R. Kalra, Z. Wu, A. Gulley, A. Hilliard, X. Guan, A. Koshiyama, and P. Treleaven. 2025. http://arxiv.org/abs/2409.09046 Hypa-rag: A hybrid parameter adaptive retrieval-augmented generation system for ai legal and policy applications . CoRR, abs/2409.09046

arXiv 2025
[8]

Pipitone and G

N. Pipitone and G. H. Alami. 2024. https://doi.org/10.48550/arXiv.2408.10343 Legalbench-rag: A benchmark for retrieval-augmented generation in the legal domain . CoRR, abs/2408.10343

arXiv doi:10.48550/arxiv.2408.10343 2024

Show all 12 references

[9]

D. Rau, S. Chau, R. Kalra, A. Wang, B. Faltings, P. Treleaven, and A. Koshiyama. 2024. https://doi.org/10.48550/arXiv.2407.01102 Bergen: A benchmarking library for retrieval-augmented generation . CoRR, abs/2407.01102

2024 doi arXiv
[10]

Zheng and H

J. Zheng and H. Yu. 2017. https://doi.org/10.2196/jmir.6962 Readability formulas and user perceptions of electronic health records difficulty: A corpus study . Journal of Medical Internet Research, 19(3):e59

2017 doi
[11]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
[12]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

Chalkidis, M

I. Chalkidis, M. Fergadiotis, P. Malakasiotis, N. Aletras, and I. Androutsopoulos. 2020. https://doi.org/10.18653/v1/2020.findings-emnlp.261 LEGAL-BERT : The muppets straight out of law school . In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2898--2904, Online. Association for Computational Linguistics

doi:10.18653/v1/2020.findings-emnlp.261 2020

[2] [2]

S. Es, J. James, L. Espinosa-Anke, and S. Schockaert. 2023. https://doi.org/10.48550/arXiv.2309.15217 Ragas: Automated evaluation of retrieval augmented generation . CoRR, abs/2309.15217

arXiv doi:10.48550/arxiv.2309.15217 2023

[3] [3]

Exploding Gradients . 2025. GitHub - explodinggradients/ragas: Supercharge Your LLM Application Evaluations . https://github.com/explodinggradients/ragas/tree/main. Accessed: 2025-04-07

work page 2025

[4] [4]

N. Guha, M. Lamm, B. Wu, A. Zhang, J. Yin, Y. Taori, Y. Zhang, S. S. Schoenholz, R. G. Krishnan, and C. D. Manning. 2023. https://doi.org/10.48550/arXiv.2308.11462 Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models . CoRR, abs/2308.11462

arXiv doi:10.48550/arxiv.2308.11462 2023

[5] [5]

Y. Han, A. Ceross, and J. H. M. Bergmann. 2024. http://arxiv.org/abs/2411.09497 The use of readability metrics in legal text: A systematic literature review . CoRR, abs/2411.09497

arXiv 2024

[6] [6]

Jeong, J

S. Jeong, J. Baek, S. Cho, S. J. Hwang, and J. C. Park. 2024. https://doi.org/10.48550/arXiv.2403.14403 Adaptive-rag: Learning to adapt retrieval-augmented large language models through question complexity . CoRR, abs/2403.14403

arXiv doi:10.48550/arxiv.2403.14403 2024

[7] [7]

Kalra, Z

R. Kalra, Z. Wu, A. Gulley, A. Hilliard, X. Guan, A. Koshiyama, and P. Treleaven. 2025. http://arxiv.org/abs/2409.09046 Hypa-rag: A hybrid parameter adaptive retrieval-augmented generation system for ai legal and policy applications . CoRR, abs/2409.09046

arXiv 2025

[8] [8]

Pipitone and G

N. Pipitone and G. H. Alami. 2024. https://doi.org/10.48550/arXiv.2408.10343 Legalbench-rag: A benchmark for retrieval-augmented generation in the legal domain . CoRR, abs/2408.10343

arXiv doi:10.48550/arxiv.2408.10343 2024