{"id":"bcba3a12-819d-47bd-94fc-659ab08b252b","arxiv_id":"2508.13107","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"An adaptive RAG pipeline for legal research with query classification and open-source embeddings rivals a proprietary baseline on retrieval, and a custom legal prompt improves faithfulness.","lead":"This paper builds a legal question-answering system that retrieves the relevant contract clauses and adjusts the answer to the user's expertise. It shows free open-source language tools can match a paid commercial system, while a custom prompt keeps answers more faithful to the documents.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claim that the custom prompt yields more faithful and contextually relevant answers rests on post-hoc retention of two metrics after answer relevancy and ROUGE contradicted it, with no held-out evaluation run.","rationale":"The reader's weakest assumption correctly identifies the generation evaluation as the fragile part of the paper. The paper's strongest advertised claim is that the custom prompt produces more faithful and contextually relevant answers, and the only direct evidence for that is the two retained metrics. The discarded metrics do not merely show noise; they point in opposite directions, with answer relevancy favoring the baseline and ROUGE-Recall favoring the Llama model with the custom prompt. The paper's justification for exclusion is qualitative and was developed after seeing the numerical outcomes, which introduces a selection effect that cannot be removed by argument alone. The absence of a held-out evaluation compounds the problem: the same benchmark was used to tune K, compare prompts, choose models, and select the R&C classifier, so the reported numbers likely overstate the benefit. The retrieval claim is also overstated relative to the figures, but the generation claim is the more central and more weakly supported. A conditional verdict is appropriate because the pipeline design and the open-source retrieval comparison are plausible and could be confirmed with a cleaner evaluation; the current evidence, however, does not yet establish the headline generation result.","tokens_in":5782,"tokens_out":5729,"duration_ms":63469,"concrete_test":"Run a pre-registered head-to-head of GPT-4o-mini with the baseline prompt versus the custom legal-grounded prompt on a held-out LegalBenchRAG domain (e.g., CUAD) that was not used in any prompt, K, or metric selection step, reporting all four metrics (Faithfulness, answer relevancy, BERTScore-F1, ROUGE-Recall) as primary endpoints. If answer relevancy still favors the baseline or ROUGE still favors the custom prompt on this held-out split, the claim of consistently more faithful and contextually relevant answers is not robust to the metrics the authors chose to discard.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 6.2 reports four metrics for prompt/model comparison. RAGAS answer relevancy favors the baseline GPT prompt, and ROUGE-Recall favors Llama-3-8B with the custom prompt; only Faithfulness and BERTScore-F1 favor the advertised GPT-4o-mini + custom prompt combination. Appendices D and E then argue that answer relevancy and ROUGE are unsuitable, after observing that they disagree with the preferred conclusion. The retained two metrics both measure how well the generated answer is grounded in the retrieved context; neither checks whether the answer actually addresses the user's question. Answer relevancy is precisely the metric designed for that, and it was excluded. The same LegalBenchRAG subsets were used to tune the prompt, K, and the R&C classifier, then reused as the demonstration, so the comparison is not an independent test. With no held-out benchmark, no pre-registered endpoints, and no released code or data, the central generation claim is not empirically established as written.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents an end-to-end retrieval-augmented generation pipeline for legal research. It claims three contributions: a context-aware query translator that classifies user expertise and query specificity and rewrites queries; an open-source retrieval stage using SBERT and GTE embeddings compared against LegalBenchRAG's OpenAI text-embedding-3-large and the RetroMAE model; and a generation/evaluation framework using GPT-4o-mini and LLaMA-3-8B under three prompting strategies, evaluated with RAGAS Faithfulness, answer relevancy, BERTScore-F1 with LegalBERT, and ROUGE-Recall. The authors conclude that open-source retrieval can rival proprietary embeddings and that a custom legal-grounded prompt with GPT-4o-mini and K=5 yields the most faithful and contextually relevant answers. The final pipeline includes a readability/complexity classifier that adapts response detail to the predicted user expertise. The evaluation is conducted on sampled subsets of LegalBenchRAG.","tokens_in":5971,"tokens_out":5724,"duration_ms":52975,"significance":"If the results held up, the paper would provide valuable evidence that cost-effective, open-source components can match proprietary RAG systems in the legal domain, and that prompt design can matter as much as model choice. The paper has strengths: it systematically compares chunking, embedding, and similarity-search configurations; it uses a domain-specific LegalBERT for semantic similarity; and it provides qualitative examples of the adaptive response behavior. However, as written, the headline generation claim relies on selectively retained metrics and same-benchmark tuning, and the retrieval claim in the introduction conflicts with the results section. These issues must be resolved before the conclusions can be accepted.","major_comments":[{"comment":"The introduction states that the tailored retrieval 'improves Recall@K by 30–95% and Precision@K by ~2.5× for K>4' against LegalBenchRAG, but Section 6.1 reports that the best open-source configuration performs only 'slightly better on precision for all k-values and very similar for recall for initial k-values, which gets lower as k-value increases' relative to LegalBenchRAG's OpenAI model. These statements are inconsistent. Please state the exact comparison baseline, report the numerical results for the specific configurations, and reconcile the 30–95% figure with the 'slightly better' description.","section":"Introduction, Contribution 2, and Section 6.1"},{"comment":"The generation evaluation reports four metrics. RAGAS answer relevancy favors the baseline GPT prompt, ROUGE-Recall favors LLaMA-3-8B with the custom prompt, and only Faithfulness and BERTScore-F1 favor GPT-4o-mini with the custom prompt. The paper excludes answer relevancy and ROUGE-Recall after observing this disagreement, arguing in Appendices D and E that these metrics are unsuitable. This is post-hoc metric selection: the retained metrics both measure grounding in the retrieved context, and neither checks whether the answer actually addresses the user's question, which is precisely what answer relevancy measures. With no pre-specified primary endpoint and no held-out evaluation, the claim that the custom prompt is 'consistently more faithful and contextually relevant' is not empirically established.","section":"Section 6.2 and Appendices D and E"},{"comment":"The optimal K value is selected by inspecting the same faithfulness and BERT-F1 curves that are then used to demonstrate the pipeline's success ('faithfulness stagnates after K=5 ... we choose K=5'), and the same LegalBenchRAG subsets appear to be used for prompt tuning, retrieval tuning, and final evaluation. No held-out set is described. This selection-on-the-test-set inflates the apparent advantage of the advertised configuration and prevents the comparison from being an independent test.","section":"Section 6.2 and final pipeline"},{"comment":"The Readability and Complexity (R&C) classifier is a central component of the claimed adaptive behavior (expert vs. non-expert, vague vs. verbose), but the manuscript reports no quantitative evaluation of its classification accuracy. Appendix F provides only qualitative examples. If the classifier misclassifies expertise, the adaptive response claims do not follow. Please report its accuracy or a confusion matrix on the sampled queries, or re-scope the claim as a proof-of-concept.","section":"Section 5 and Appendix F"}],"minor_comments":[{"comment":"The phrase 'consistently produces more faithful and contextually relevant answers' is too strong given that the answer relevancy metric favored the baseline prompt; please qualify the claim to reflect the metric-selection issues described in Section 6.2.","section":"Abstract"},{"comment":"There is a typo in 'Text Embeding 3 Large'; it should be 'text-embedding-3-large'.","section":"Introduction, Contribution 2"},{"comment":"The label 'SoTA model (RetroMAE)' is used without a citation; please provide a reference for this state-of-the-art claim.","section":"Section 6.1"},{"comment":"The paper promises reproducibility but does not state whether code and data will be released; please include an availability statement or explicitly note limitations.","section":"Abstract and throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper contains useful empirical comparisons and a sensible motivation, but the headline generation claim is not supported as written because two of four metrics were discarded after observing disagreement and because tuning was performed on the same benchmark used for the final comparison. The 30–95% retrieval gain claim in the introduction also does not match the 'slightly better' statement in Section 6.1. I recommend major revision with concrete requirements: report all metrics, add a held-out evaluation, and correct or qualify the retrieval claims. Releasing code and data would strengthen the reproducibility argument."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a sincere engineering paper with one genuinely new piece—the query translator that classifies expertise and specificity and adjusts retrieval depth and response style. The retrieval comparison is the strongest part: SBERT/GTE with RCTS chunking and cosine similarity, unranked, slightly beats OpenAI's text-embedding-3-large on precision and is comparable on recall. That is a useful, non-obvious practitioner result, and the paper is honest enough to show it in the body (Section 6.1) as 'slightly better', not the 30–95% Recall@K and ~2.5x Precision@K the intro claims. That gap is the first thing to fix: a 30–95% claim does not match 'very similar for initial k, lower as k increases'. Either the numbers come from a different comparison, or the abstract and intro need rewriting.\n\nThe soft spot is the generation evaluation. Section 6.2 reports four metrics. Faithfulness and BERTScore-F1 favor GPT-4o-mini with the custom prompt; ROUGE-Recall favors Llama-3-8B with the custom prompt; answer relevancy favors the baseline GPT prompt. The paper then spends two appendices arguing why ROUGE and answer relevancy are unsuitable—after seeing that they disagree. The reasons have some merit (ROUGE is lexical; answer relevancy is heavily affected by the non-committal multiplier, and they show that removing the multiplier erases the gap). But a reader cannot tell whether the exclusion was pre-planned or post-hoc, because there is no held-out benchmark, no pre-registered endpoints, and no code or data to check. K=5 is also selected from the same faithfulness curves used to report the result. So the central claim—'consistently more faithful and contextually relevant answers'—is plausible but not convincingly established as written. The answer relevancy result matters especially because it is the only metric that checks whether the answer addresses the user's question, and it favors the baseline.\n\nAlso, no code or data is released despite the 'reproducible' claim, and the manuscript text I saw is partial (sections 2–4 missing), so I could not inspect the classifier details or the training procedure.\n\nWho this is for: practitioners building legal RAG, especially people weighing open-source embeddings against paid APIs. They will get a useful configuration and a cautionary example of how metric choice can drive conclusions. As a research contribution it is incremental; as an engineering report it deserves a serious referee. My recommendation: send to peer review, but require the authors to (a) reconcile the intro's quantitative claims with Section 6.1, (b) pre-specify the metric set or evaluate on a true held-out split with K and prompt chosen on a validation set, and (c) release code and data. If those are addressed, the retrieval finding alone warrants publication.","headline":"The open-source retrieval result is plausible and useful, but the intro oversells it, and the generation win rests on dropping two metrics after they disagreed with the preferred conclusion—send to review, expect heavy revision.","tokens_in":6505,"tokens_out":2315,"would_cite":false,"duration_ms":21990,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a fully open-source, adaptively tuned legal RAG pipeline can match proprietary retrieval quality and, with a custom legal-grounded prompt, generate more faithful answers than baseline prompting.","keywords":["retrieval-augmented generation","legal research","query translation","open-source embeddings","LegalBenchRAG","faithfulness evaluation","prompt engineering","RAG evaluation"],"falsifier":"Re-run the Section 6.2 generation comparison on a fresh LegalBenchRAG sample keeping the original RAGAS answer relevancy score with its non-committal multiplier and ROUGE-Recall as decision metrics: if the baseline prompt then ties or beats the custom prompt on those metrics, or if faithfulness continues to rise beyond K=5, the paper's central generation claim fails.","tokens_in":5563,"feed_emoji":"⚖️","tokens_out":8720,"duration_ms":76926,"temperature":0.7,"pith_summary":"This paper tries to show that legal retrieval-augmented generation does not need proprietary services: a deliberately tuned pipeline built from open-source components can rival paid retrieval and, with a custom legal-grounded prompt, produce more faithful answers than standard prompts. The claimed retrieval gains are large—Recall@K improves by 30–95% and Precision@K by roughly 2.5× for K>4 against the LegalBenchRAG reference—while the generation experiments indicate that prompt design matters more than model choice. A sympathetic reader would care because legal research is high-stakes and error-prone, and if the claim holds, organisations could deploy cheaper, reproducible, and legally grounded question answering without sacrificing quality.","feed_headline":"Open-source legal RAG rivals proprietary retrieval on legal benchmarks","feed_subtitle":"A task-aware query translator plus a legal-grounded prompt beats baseline prompting on faithfulness and matches paid retrieval quality.","key_machinery":"The load-bearing mechanism is the query translator sitting in front of a retrieval and generation loop. It rewrites the user's question, extracts any file path or document reference so retrieval can be narrowed to that document, and labels the query as expert versus layperson and vague versus verbose; those labels set retrieval depth (K) and the response style. Retrieval then uses cosine similarity over RCTS-chunked documents embedded with SBERT (GTE as a close alternative), and generation uses a custom legal-grounded prompt that restricts the model to answering from the retrieved chunks. The pipeline's three-way evaluation—RAGAS Faithfulness, BERTScore-F1, and ROUGE-Recall—is designed to separate semantic grounding from lexical overlap.","core_discovery":"On its own terms, the paper's central discovery is that component-level adaptation carries legal RAG performance: a context-aware query translator that separates document references from natural-language questions and classifies expertise and specificity, combined with RCTS chunking, open-source SBERT or GTE embeddings, cosine retrieval, and a hand-tuned legal prompt, reaches or exceeds the retrieval quality of the proprietary text-embedding-3-large model used in LegalBenchRAG on precision for all evaluated k and matches it at low k, while lifting Recall@K by 30–95% and Precision@K by roughly 2.5× for K>4. In generation, the paper finds GPT-4o-mini with the custom prompt consistently outscores the baseline prompt and zero-shot chain-of-thought on RAGAS Faithfulness and BERTScore-F1, with K=5 as the chosen operating point because faithfulness plateaus beyond it. The paper also argues that two common metrics—RAGAS answer relevancy and ROUGE-Recall—should be discarded in this setting, on the grounds that answer relevancy mostly reflects the rate of non-committal replies and ROUGE penalises valid paraphrase.","pith_inferences":["Editorial: because the paper shows non-committal answers drop as k grows, a 'decisiveness' score could be added to legal RAG evaluation as a first-class metric alongside faithfulness, giving a concrete way to trade off recall and answer confidence.","Editorial: the same query-translator-plus-open-retriever recipe should transfer to other dense, jargon-heavy domains such as regulatory compliance or medical records, where paraphrase-tolerant semantic metrics matter more than n-gram overlap.","Editorial: a natural follow-up experiment is to test whether the expert/layperson classification changes real user outcomes—e.g., comprehension or task completion—rather than only output length and detail, since the paper's qualitative examples show detail increases for expert queries."],"forward_implications":["An all-open-source legal RAG stack can be deployed without per-query embedding costs while matching proprietary retrieval on precision and low-k recall.","Prompt design, specifically a legal-grounded instruction that forces answers to be derived from retrieved chunks, is a first-order lever on faithfulness—larger than swapping the underlying LLM.","K=5 is a defensible operating point for legal research assistants: beyond it retrieval precision falls and faithfulness stops improving.","Using RAGAS answer relevancy or ROUGE-Recall alone would mislead legal RAG tuning, because non-committal answers and valid paraphrase are penalised respectively.","Task-aware, component-level tuning (query translation, chunking, embedding, prompt) is what delivers the reported gains, not any single model."],"supporting_citations":[{"why":"Supplies the LegalBenchRAG datasets and the proprietary retrieval baseline (text-embedding-3-large) that the paper's Recall@K and Precision@K gains are measured against.","marker":"Pipitone and Alami, 2024"},{"why":"Cited in Appendix E to establish ROUGE's extractive-summarization origins and justify discarding ROUGE-Recall for paraphrased legal answers.","marker":"Chalkidis et al., 2020"},{"why":"Provides BERTScore, the embedding-based semantic similarity metric used alongside RAGAS Faithfulness to select the custom prompt and K=5.","marker":"Exploding Gradients, 2025"}],"fun_headline_variants":["Open-source legal RAG rivals proprietary retrieval","Legal RAG: custom prompt boosts faithfulness","Adaptive legal RAG: open-source beats proprietary on recall","Cost-effective legal RAG matches paid retrieval quality","Legal RAG: task-aware tuning rivals proprietary models"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The generation conclusion rests on the assumption that RAGAS Faithfulness and BERTScore-F1 are the right quality measures, and that the discarded answer-relevancy and ROUGE-Recall metrics are genuinely uninformative rather than revealing of a real quality difference.","fun_headline_variants_meta":{"raw":{"variants":["Open-source legal RAG rivals proprietary retrieval","Legal RAG: custom prompt boosts faithfulness","Adaptive legal RAG: open-source beats proprietary on recall","Cost-effective legal RAG matches paid retrieval quality","Legal RAG: task-aware tuning rivals proprietary models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000287,"raw_usage":{"total_tokens":1707,"prompt_tokens":990,"completion_tokens":717,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":606,"completion_tokens_details":{"reasoning_tokens":644}},"tokens_in":606,"tokens_out":717,"duration_ms":7402,"temperature":1.0,"reasoning_tokens":644,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:15:24.223337+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the Section 6.2 generation comparison on a fresh LegalBenchRAG sample keeping the original RAGAS answer relevancy score with its non-committal multiplier and ROUGE-Recall as decision metrics: if the baseline prompt then ties or beats the custom prompt on those metrics, or if faithfulness continues to rise beyond K=5, the paper's central generation claim fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides BERTScore, the embedding-based semantic similarity metric used alongside RAGAS Faithfulness to select the custom prompt and K=5."}],"review_version":1}