{"id":"24f60a67-7605-40d3-aa6b-f44b449298f9","arxiv_id":"2504.16121","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"An advanced RAG pipeline with an extra relevance-check and query-refinement model improves Bangla legal document QA over a vanilla RAG baseline on a 168-pair test set.","lead":"This paper builds a question answering system for Bangladesh Police Gazettes, combining retrieval with a second language model that checks whether retrieved passages are relevant and rewrites unclear queries. The system beats a basic retrieval-augmented pipeline on a 168 question test set, but only that baseline is compared.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"OCR fidelity is unverified; if Tesseract mangles Bangla legal text, both the test set and retrieval index are built on a corrupted source, so the measured advantage may not transfer to actual gazette QA.","rationale":"The reader's weakest-assumption analysis correctly identifies OCR fidelity as foundational. The paper's entire evaluation chain, from the document store to the GPT-4o-generated ground truth to the cosine-similarity reference, depends on the Tesseract transcript being accurate. If OCR corrupts Bangla legal terms, then the test set itself is not a valid measure of legal information retrieval, regardless of the RAG architecture. This concern is more load-bearing than the absence of significance testing or the abstract overclaim, because those issues affect the strength of the evidence, while OCR fidelity affects whether the evidence measures the intended object at all. The proposed concrete test, manual transcription plus a clean-corpus re-evaluation, would settle the concern directly. The reader's verdict of CONDITIONAL remains appropriate: the paper's narrow internal comparison may hold, but the broader claim requires verification that the OCR text is faithful. No change to the verdict is needed; the conditional acceptance already depends on this assumption being addressed.","tokens_in":12159,"tokens_out":8536,"duration_ms":85302,"concrete_test":"Manually transcribe a stratified sample of at least 20 pages covering all 13 gazettes, both Bangla and English portions, and both scanned and digitally born PDFs. Run the exact Tesseract preprocessing from Section III.B and compute aligned per-language character error rate and word error rate against the manual transcription. Then rebuild the document store from the clean transcription and re-evaluate all 168 QA pairs with both Vanilla and Advanced RAG. Decision rule: if Bangla WER exceeds roughly 5-10%, or if the Advanced-minus-Vanilla gap in Table IV shrinks by more than half on the clean corpus, the reported advantage is inseparable from OCR errors and the paper's real-world claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is that Advanced RAG improves QA over Vanilla RAG on Bangladesh Police Gazettes. That claim presupposes that the OCR-derived text in Section III.B is a faithful transcript of the legal PDFs. The paper states only that \"Tesseract OCR is configured to recognize both Bangla and English texts from the images\" and reports no character error rate, no word error rate, no language-specific accuracy, and no manual spot-check. Since the same OCR output is used (a) to build the vector index, (b) as the source for GPT-4o to generate the 168 ground-truth QA pairs, and (c) as the reference for both semantic similarity and human evaluation, any systematic OCR corruption of Bangla legal terms is propagated into every component of the evaluation. The Vanilla-vs-Advanced comparison itself is not automatically invalid, because both pipelines see the same corrupted corpus, but the headline claim is about answering questions on legal documents, not on OCR artifacts. If Bangla legal terminology is dropped or garbled, the ground-truth answers may not correspond to the actual gazette content, and improvements in cosine similarity or human scores would not establish that the method retrieves correct legal information. This is the least secure link in the chain because Tesseract is known to be weak on mixed-script and low-resource text, and no evidence is offered that it works here.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents LegalRAG, a bilingual retrieval-augmented generation system for question answering on Bangladesh Police Gazettes, which contain mixed English and Bangla content. The authors build a vanilla RAG pipeline (bge-m3 embeddings, ChromaDB, MMR retrieval, and a generator LLM) and an advanced RAG pipeline that inserts a Llama 3.2 (3B) model to check whether retrieved chunks are relevant and, if not, to refine the query iteratively. They evaluate both pipelines on a self-curated set of 168 GPT-4o-generated Bangla question-answer pairs using mean cosine similarity and average human scores from three legal-domain evaluators. The central empirical claim is that the advanced pipeline outperforms the vanilla pipeline on this test set, which is supported by Tables IV and V, while the abstract makes the broader and less supported claim that the approach 'consistently outperforms existing methods across all evaluation metrics.'","tokens_in":12368,"tokens_out":4449,"duration_ms":40702,"significance":"If the narrow result holds, this is a modest but useful proof-of-concept: a lightweight relevance-check and query-refinement stage improves retrieval and answer quality for low-resource bilingual legal documents. The paper's strengths are its focus on a genuinely low-resource domain (Bangla legal gazettes), the inclusion of realistic query types such as dialectal and misspelled inputs, the use of both automatic and human evaluation, and ablations of temperature and prompt language. The significance is limited, however, by the internally generated benchmark, the absence of any non-vanilla RAG baselines, the lack of statistical testing or inter-annotator agreement, and the unverified OCR quality that feeds every evaluation component. No code, data, or OCR output is released, so the results are not independently reproducible as reported.","major_comments":[{"comment":"The OCR stage is not validated in any quantitative way. The paper states only that 'Tesseract OCR is configured to recognize both Bangla and English texts from the images' and reports no character error rate, word error rate, per-language accuracy, or manual spot-check. Because the same OCR output is used to build the vector index, to generate the 168 ground-truth QA pairs with GPT-4o, and as the reference for semantic similarity and human evaluation, any systematic corruption of Bangla legal terminology propagates into every component. The vanilla-versus-advanced comparison is not automatically invalid, but the headline claim is about answering questions on legal documents, not on OCR artifacts; the authors should add an OCR accuracy assessment on a held-out page sample and report language-specific error rates.","section":"Section III.B (Data Preprocessing)"},{"comment":"The abstract claims that 'our approach consistently outperforms existing methods across all evaluation metrics,' but the experiments compare only the proposed Advanced RAG against a single Vanilla RAG baseline on a self-constructed test set. None of the RAG variants discussed in Section II (e.g., Self-RAG, query-document alignment, or Blended RAG) is used as a baseline, and no public benchmark is used. Either the claim should be restricted to 'outperforms the vanilla RAG pipeline on this test set,' or the required baselines and standard datasets should be added.","section":"Abstract and Sections IV.A/IV.B"},{"comment":"No statistical significance is reported for any comparison in Tables IV and V. The differences in human scores (e.g., Llama 3.1 8B: 3.41 vs. 3.70) and cosine similarity (0.76 vs. 0.82) are given as point estimates without confidence intervals, paired tests, or per-question variance. Given the small test set (n=168) and the multiple comparisons across three LLMs and eight domains, the observed differences could be within sampling noise. The authors should provide paired significance tests (e.g., bootstrap or permutation tests) and effect sizes, or explicitly label the comparisons as descriptive only.","section":"Section IV.B, Tables IV and V"},{"comment":"The sampling temperature is selected on the same test set used for final evaluation. Figure 5(a) reports mean cosine similarity for temperatures 0.1, 0.4, 0.7, and 1.0, and the paper then uses 0.1 for all reported results, selecting the best value on the test set. This is a form of test-set overfitting. There is no validation/tuning split, and other fixed hyperparameters (retrieval chunk count k, maximum refinement iterations, MMR settings) are not varied. The authors should either use a development/test split and choose hyperparameters on the development split, or report the sensitivity of the main conclusions to these choices.","section":"Section IV.C and Figure 5(a)"},{"comment":"The human evaluation reports average scores from three evaluators but provides no inter-annotator agreement statistic. Since the ratings are subjective and the pipeline differences are small (at most 0.4 on a 1-5 scale), an agreement measure such as Krippendorff's alpha or quadratic weighted kappa is needed to establish that the signal is not driven by a single rater. In addition, the paper does not state whether evaluators were blinded to which pipeline generated each response; this should be clarified.","section":"Section IV.A, Evaluation, and Table III"}],"minor_comments":[{"comment":"The claim that vanilla RAG is acceptable for science and finance domains is supported only by Table II, but no details are given for how those semantic similarity numbers were computed or on what data; this table should either be moved to an appendix with full methodology or removed.","section":"Section III.C, Table II"},{"comment":"The sentence 'The retriever then measured the similarity' mixes tenses; it should read 'The retriever then measures the similarity.'","section":"Section III.B, step 5"},{"comment":"Figure 1 is dense and the relationship between the generated answers, the relevance labels, and the retrieved chunks is hard to follow; a simplified diagram or a table with aligned rows would improve readability.","section":"Figure 1"},{"comment":"Reference formatting is inconsistent: arXiv identifiers, access dates, and conference styles are mixed, and at least one reference (Ref. [5]) duplicates the same work as Ref. [2]; the list should be normalized.","section":"References"},{"comment":"The ethics statement says that 'no personal or sensitive data were collected or stored,' yet Table III reports age and gender summaries; the authors should clarify that only de-identified demographic summaries were retained.","section":"Ethics Statement"}],"recommendation":"major_revision","confidential_remarks":"This is a solid engineering report but not yet a complete research contribution as written. The main issues are the unvalidated OCR chain, the internally generated benchmark, the absence of non-vanilla baselines, and the lack of statistical rigor in the evaluation. The narrow claim is defensible and the authors should be encouraged to tighten the claims, add significance testing and inter-annotator agreement, and release the test set and OCR preprocessing pipeline so that the community can reproduce the results."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nLegalRAG is a straightforward application of known RAG mechanisms to a genuinely under-served setting: bilingual Bangla/English police gazettes from Bangladesh. The genuinely new pieces are the specific pipeline combination (relevance check plus query refinement on top of vanilla RAG) and a 168-pair QA test set built from those gazettes. That is a legitimate empirical contribution, modest in scope. The central comparison—advanced versus vanilla RAG—is supported by Tables IV and V across three LLMs and several question categories. The pipeline description is clear, the domain is real, and the human evaluation, though small, uses raters with legal backgrounds.\n\nThe soft spots are real but mostly addressable. First, the abstract overclaims: \"consistently outperforms existing methods\" when the only baseline is vanilla RAG. The body is more careful, but the abstract sets a bad frame. Second, OCR fidelity is never checked. The same Tesseract output feeds both the retrieval index and the ground-truth generation, so if Bangla legal terms are garbled, the whole evaluation is built on a corrupted source. The stress-test note is on target here. The vanilla-versus-advanced comparison might survive that, since both pipelines see the same text, but the headline about legal QA would not. Third, no significance testing, no inter-annotator agreement, and the sampling temperature is selected on the same test set. The test set itself is GPT-4o-generated, then checked by two authors. None of this kills the narrow claim, but it caps the strength of the claim.\n\nI agree with the reader's conditional verdict. The paper would benefit from a revised abstract, a significance test or at least bootstrapped confidence intervals, a reporting of OCR accuracy or a spot-check, and ideally a release of the test set and code so others can build on it.\n\nWho is this for? People working on low-resource legal information retrieval. It is a useful data point, not a breakthrough. I would send it to peer review—it deserves referee time—but I would expect major revision, mainly to tone down claims and tighten the evaluation. If the authors fix those issues, the narrow result is publishable.\n\nRecommendation: engage with it, but require the fixes before acceptance.","headline":"A modest but legitimate empirical contribution: a relevance-check/query-refinement RAG variant evaluated on a new Bangla legal QA set, with real soft spots around OCR fidelity and overclaimed baselines.","tokens_in":12972,"tokens_out":1473,"would_cite":false,"duration_ms":14072,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A relevance-check and query-refinement step improves bilingual legal question answering.","keywords":["Retrieval-Augmented Generation","bilingual question answering","Bangla legal documents","government gazettes","relevance check","query refinement","low-resource NLP","information retrieval"],"falsifier":"Run the same question set against manually corrected transcripts of a sample of gazette pages and compare retrieval and answer accuracy with the OCR-derived versions; if the corrected transcripts give substantially better scores, then at least part of the claimed gain rests on OCR quality rather than the RAG design itself.","tokens_in":11919,"feed_emoji":"⚖️","tokens_out":6729,"duration_ms":57067,"temperature":0.7,"pith_summary":"This paper argues that a relatively small change in how retrieved text is selected can make retrieval-augmented question answering work for low-resource legal documents. The authors build a pipeline that reads bilingual Bangladeshi police gazettes, retrieves relevant passages for a Bangla question, and then uses an extra language-model step to check whether those passages are truly relevant and to rephrase the question if they are not. On a curated set of 168 question-answer pairs covering factual, temporal, statistical, dialectal, and out-of-context queries, they report that this extra step improves both human ratings and semantic similarity to reference answers compared with a standard retrieval pipeline, across all three answer generators tested. If the claim holds, a modest two-stage filtering loop can make legal information from low-resource official documents considerably more accessible.","feed_headline":"Relevance-check filter lifts Bangla legal QA accuracy","feed_subtitle":"A hybrid RAG pipeline with a query-refinement loop beats its vanilla counterpart on bilingual police gazettes.","key_machinery":"The mechanism that carries the argument is the added relevance-check and query-refinement stage, implemented as a separate small language model placed between the retriever and the final answer generator. After the retriever returns top chunks for a user query, this model decides whether each chunk is actually relevant; if it decides they are not, it rewrites the query while preserving the user's intent, and the system retrieves again. The loop repeats at most three times, and only chunks that pass the check reach the generator, so irrelevant retrieved material is filtered out before answer writing. This stage is what distinguishes the advanced pipeline from the vanilla one and is credited with the measured improvements.","core_discovery":"The paper's central empirical claim is that inserting a relevance-check and query-refinement component into a retrieval-augmented question-answering pipeline improves retrieval and answer quality on a low-resource legal corpus. Working from the Bangladesh Police Gazettes, the authors preprocess 13 mixed English–Bangla gazette documents, build a vector index of chunks, and compare a standard pipeline that sends retrieved chunks straight to an answer generator with an advanced pipeline that first asks a separate language model whether the chunks are relevant, reformulating the query up to three times when they are not. On a curated 168-pair Bangla test set spanning factual, temporal, statistical, dialectal, spelling-error, and out-of-context questions, the advanced pipeline attains higher human evaluation scores and higher mean cosine similarity with lower standard deviation across all three answer generators; the best configuration reaches 3.70 out of 5 on human evaluation and 0.82 mean cosine similarity.","pith_inferences":["The same two-stage filter could plausibly transfer to other code-mixed official records beyond police gazettes, but the paper does not test that transfer.","Because the question-answer pairs were drafted from the same gazette text the system retrieves from, the benchmark may under-represent how real users phrase legal queries; a separate user study would test that gap.","The reported aggregate scores do not separate gains from discarding irrelevant chunks from gains due to query rephrasing; logging retrieval decisions per query would settle which mechanism carries the improvement."],"forward_implications":["Adding the relevance-check and query-refinement stage raises the average human evaluation score and the mean cosine similarity for all three generation models tested, while also lowering the spread of cosine-similarity scores.","The advanced pipeline yields higher mean cosine similarity than vanilla RAG in every tested question category, including factual questions, temporal-change questions, statistical questions, Bangla dialect questions, and grammar or spelling-error questions.","Out-of-context questions remain the weakest category for both pipelines, so a RAG system alone does not reliably detect when a query falls outside the document collection.","Lowering sampling temperature to 0.1 gives the highest mean cosine similarity for all three generation models, consistent with legal question answering needing exact answers rather than creative variation.","Using Bangla or English for the instruction prompt makes little difference to the scores, suggesting the multilingual embedding keeps the pipeline robust to prompt-language variation."],"supporting_citations":[{"why":"Introduces retrieval-augmented generation, the paradigm both pipelines build on.","marker":"[7]"},{"why":"Provides the model family used for answer generation and for the relevance-check and query-refinement step.","marker":"[17]"},{"why":"Supplies the language model that drafted the 168 question-answer pairs from gazette text.","marker":"[23]"},{"why":"Gives the multilingual embedding model that maps English and Bangla chunks into a shared vector space.","marker":"[28]"},{"why":"Provides the OCR engine that converts gazette PDF pages into text for indexing.","marker":"[29]"},{"why":"Describes the quantization method enabling large generation models to run under tight GPU memory.","marker":"[30]"}],"fun_headline_variants":["LegalRAG's relevance check sharpens Bangla legal QA","Hybrid RAG with query refinement wins on police gazettes","Bilingual legal retrieval improved by relevance-check loop","Bangla police gazette QA gets accuracy lift from RAG loop","Better legal answers via RAG with relevance filter for Bangla"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The pipeline depends on the OCR step transcribing mixed Bangla–English gazette pages faithfully enough to preserve legal terms, page references, and numbers, yet the paper reports no measurement of OCR accuracy.","fun_headline_variants_meta":{"raw":{"variants":["LegalRAG's relevance check sharpens Bangla legal QA","Hybrid RAG with query refinement wins on police gazettes","Bilingual legal retrieval improved by relevance-check loop","Bangla police gazette QA gets accuracy lift from RAG loop","Better legal answers via RAG with relevance filter for Bangla"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000266,"raw_usage":{"total_tokens":1575,"prompt_tokens":871,"completion_tokens":704,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":487,"completion_tokens_details":{"reasoning_tokens":620}},"tokens_in":487,"tokens_out":704,"duration_ms":6252,"temperature":1.0,"reasoning_tokens":620,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:54:02.941502+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same question set against manually corrected transcripts of a sample of gazette pages and compare retrieval and answer accuracy with the OCR-derived versions; if the corrected transcripts give substantially better scores, then at least part of the claimed gain rests on OCR quality rather than the RAG design itself.","supporting_citations":[{"cited_title":"Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation,","cited_arxiv_id":null,"evidence_quote":"Gives the multilingual embedding model that maps English and Bangla chunks into a shared vector space."},{"cited_title":"QLoRA: Efficient finetuning of quantized LLMs,","cited_arxiv_id":null,"evidence_quote":"Describes the quantization method enabling large generation models to run under tight GPU memory."}],"review_version":1}