The reviewed record of science sign in
Pith

arxiv: 2606.07523 · v1 · pith:M3BJPQ2T · submitted 2026-04-20 · cs.CL · cs.AI

Retrieval Augmented Generation Framework for the Nepali Legal Domain Question Answering

Samir Wagle , Abiral Adhikari , Reewaj Khanal , Batsal Bhandari , Prashant Manandhar , Praveen Acharya , Bal Krishna Bal This is my paper

Reviewed by Pith2026-07-05 15:51 UTCglm-5.2pith:M3BJPQ2Topen to challenge →

classification cs.CL cs.AI
keywords legalpercentnepaliansweringquestiongenerationlanguagesmodel
0
0 comments X

The pith

Simple keyword search beats AI embeddings for Nepali legal documents

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper builds the first retrieval-augmented generation (RAG) pipeline for answering legal questions in Nepali, a low-resource language with no existing legal NLP benchmarks. The authors scrape 10,265 Supreme Court case documents from the Nepal Kanun Patrika digital archive, chunk them into 500-token passages, and compare five retrieval methods against 100 expert-annotated queries. The central finding is that BM25 — a decades-old term-frequency ranking algorithm — retrieves the correct document 91% of the time at rank one on chunked text and 88% on full documents, while the best dense embedding model (multilingual-e5-large) manages only 75%. The authors attribute this gap to the formulaic, lexically rigid nature of Nepali legal prose, where exact statutory phrasing makes keyword overlap more informative than semantic approximation. When paired with GPT-o3 using a two-step prompt that first extracts verbatim legal evidence then generates an answer, the BM25 pipeline achieves 85% truthfulness and 74% groundedness on the benchmark. The paper argues that for low-resource legal domains, a well-structured retrieval pipeline with classical search can substitute for the fine-tuned legal LLMs that data scarcity makes infeasible.

Core claim

BM25 sparse retrieval substantially outperforms multilingual dense embedding models on Nepali legal text, achieving Precision@1 of 0.91 on chunked documents versus 0.75 for the best dense retriever, because the formulaic and lexically constrained nature of legal language makes exact term matching more effective than semantic approximation. This retrieval quality directly determines generation quality: when BM25-retrieved context feeds a two-step evidence-extraction prompt with GPT-o3, the pipeline produces 85% truthful and 74% grounded answers, establishing a working baseline for legal QA in a language with no prior legal NLP infrastructure.

What carries the argument

The pipeline has three load-bearing components. First, BM25 retrieval on the full document corpus, which ranks documents by term frequency and document length normalization. Second, a two-step prompting strategy where the language model first extracts verbatim legal sentences from the retrieved context as evidence, then generates a final answer grounded only in that evidence, with an explicit refusal instruction when evidence is insufficient. Third, an LLM-as-judge evaluation using binary yes/no prompts for groundedness (all claims verifiable in source) and truthfulness (semantic equivalence with a reference answer), cross-checked against human evaluation. The corpus itself is a contribution

If this is right

  • For other low-resource languages with formulaic legal traditions, BM25 may remain competitive with or superior to dense retrieval, reducing the barrier to building legal QA systems without large annotated training sets.
  • The two-step evidence-extraction-then-generation prompt pattern could be adopted in other high-stakes domains where hallucination risk must be minimized, as it forces verifiable grounding before answer synthesis.
  • The publicly released dataset of 10,265 Nepali Supreme Court case documents creates a foundation for future legal NLP benchmarks in Nepali, including hybrid retrieval, re-ranking, and fine-tuning experiments.
  • The gap between BM25 and dense retrieval on legal text suggests that domain-adaptive pretraining of multilingual embeddings on legal corpora could narrow the performance difference, pointing to a concrete training target.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The 100-query evaluation set with single ground-truth documents per query means the BM25 advantage may shrink on multi-document or ambiguous queries where semantic matching could help; the paper's conclusion that sparse retrieval is inherently better for this domain should be tested on a larger, multi-relevance benchmark.
  • The refusal strategy (answering only when sufficient evidence is present) means the 92% successful answer generation rate also functions as a recall ceiling; the 8% of refused queries represent cases where retrieval failed, and analyzing those failure modes could reveal whether dense retrieval complements BM25 on the queries it misses.
  • The latency difference between BM25 on chunks (1.2s) versus full documents (0.1s) suggests a hybrid approach using full-document BM25 for initial filtering followed by chunk-level re-ranking on top candidates could capture most of the accuracy gain at a fraction of the latency cost.

Load-bearing premise

The entire performance comparison rests on 100 expert-annotated queries, each mapped to exactly one ground-truth document, with no held-out validation or cross-validation. A different or larger query set could shift the reported gap between BM25 and dense retrieval, and the single-document-per-query design does not reflect realistic legal QA where multiple documents may be relevant.

What would settle it

Construct a larger test set (say 500+ queries) of Nepali legal questions where multiple documents may be relevant, and re-run all five retrieval methods. If BM25's Precision@1 drops substantially or if dense retrieval closes the gap, the central claim that sparse retrieval is inherently superior for Nepali legal text would be weakened.

read the original abstract

Legal domains in high-resource languages like English have widely adopted artificial intelligence for legal question answering. However, data scarcity in low resource languages such as Nepali has limited the training of large language models on Nepali legal texts. This study presents the first application of a Retrieval Augmented Generation based model for Nepali legal question answering using case laws extracted from the Nepal Kanun Patrika digital archive. Using BM25 on chunked documents, the approach achieved a top precision at one of 91 percent, and up to 75 percent with the multilingual E5 large model. Evaluation of generated answers showed 74 percent groundedness, 85 percent truthfulness according to an automated judge model, and 84 percent human evaluated truthfulness when using BM25 document retrieval, with a 92 percent successful answer generation rate. These results demonstrate that the RAG pipeline can effectively address the gap in legal question answering for low resource languages and provide a foundation for reliable AI systems in the Nepali legal domain.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 8 minor

Summary. This paper presents the first RAG-based question-answering pipeline for the Nepali legal domain, using Supreme Court case law documents from the Nepal Kanun Patrika (NKP) digital archive. The authors compare sparse retrieval (BM25 at document and chunk level) against dense retrieval (multilingual-e5-large, multilingual-e5-base, LaBSE) on a 100-query expert-annotated test set, and evaluate end-to-end generation using GPT-o3 with a two-step evidence-extraction prompt. The headline finding is that BM25 on chunked documents achieves Precision@1 of 0.91, substantially outperforming the best dense retriever (multilingual-e5-large at 0.75), and that the full pipeline achieves 85% truthfulness and 74% groundedness. The dataset (10,265 documents, 110,626 chunks) and code are publicly available, which is a notable strength for reproducibility in a low-resource setting.

Significance. The paper addresses a genuine gap: there is no prior systematic evaluation of retrieval strategies for Nepali legal QA, and the curated corpus of Supreme Court case law is a contribution in itself. The public release of both dataset and code is commendable and strengthens reproducibility. The finding that BM25 outperforms off-the-shelf dense retrievers in this domain is consistent with prior work on formulaic legal text [16, 21] and provides a useful baseline for future research. However, the significance of the headline BM25-vs-dense comparison is tempered by the evaluation methodology concerns detailed below.

major comments (3)
  1. §III.A and §V: The 100-query test set is constructed by legal domain experts who had access to the source documents, and each query maps to exactly one ground-truth document. The paper itself notes that Nepali legal text is 'lexically constrained and highly formulaic' (§V), which it frames as a domain property explaining BM25's advantage. However, this framing overlooks a methodological confound: expert-authored queries likely mirror the exact statutory terminology and formulaic phrasing of the source documents, systematically favoring BM25's term-frequency matching over dense semantic retrieval. The paper's stated goal of serving 'public engagement in the legal domain' (§I) implies non-expert users, yet the evaluation uses expert-formulated queries that structurally resemble document language. This is a load-bearing concern because the headline finding (P@1: 0.91 vs 0.75) may not hold —
  2. §IV.B and Table V: The generation evaluation reports truthfulness (85%), groundedness (74%), and successful answer rate (92%) for BM25_Docs, but these are computed on the same 100-query test set used for retrieval evaluation with no held-out split or cross-validation. The paper acknowledges this limitation in §VI ('hinders broad statistical generalization'), but the generation quality claims — particularly the comparison between BM25_Docs, e5-large, and the Count method — rest entirely on this single evaluation. A different query set could shift the relative ranking of generation methods. At minimum, the authors should report confidence intervals or bootstrap estimates for the generation metrics in Table V to indicate the uncertainty introduced by the small sample.
  3. §III.E and Appendix C: The groundedness evaluation uses Gemini 2.5 Pro as an LLM-as-judge with binary (0/1) scoring, but the paper does not report any validation of the judge's reliability — no inter-annotator agreement with the human evaluators, no calibration against a gold-standard subset. Given that the 74% groundedness figure is a headline result, the absence of judge validation is a gap. The human evaluation (84% truthfulness) is compared to the LLM judge (85% truthfulness), but the paper does not report whether these were on the same queries or how agreement was computed. Please clarify the relationship between the human and automated evaluations and report agreement statistics.
minor comments (8)
  1. The model referred to as 'GPT-o3' or 'OpenAI's 03 model' (§III.D, §V) appears to reference OpenAI's o3 model, but the citation [4] (Cao et al., 2025) is about retrieval robustness evaluation, not the o3 model itself. Please cite the appropriate technical report for the model used.
  2. Table IV (latency comparison) reports BM25_Chunks at 1.2003s/query and BM25_Docs at 0.1042s/query, but the text in §IV.A states the latency difference is '11 times higher.' The ratio is approximately 11.5, which is close, but the text should state the precise figure or round consistently.
  3. §III.D: The 'Count Method' is described as selecting 'the most frequent document after aggregating the top 5 documents retrieved by BM25_Docs, BM25_Chunks, and e5-large.' It is unclear whether this means the top-5 from each method (15 total) or the top-5 overall. Please clarify the aggregation procedure.
  4. The document text contains numerous OCR or encoding artifacts (e.g., 'Bal Kr 1108. 381' in the author list, 'Kathmandu Universit;' in the affiliation, 'Fmail' instead of 'Email', 'tem' instead of 'system' in §I). These should be corrected for the camera-ready version.
  5. §III.A: The test set composition is described as 79% objective (87% fact/definition, 13% party identification) and 21% subjective (58% procedure, 42% rationale). Please include example queries from each category in an appendix or supplementary material to help readers assess query difficulty and lexical overlap with source documents.
  6. Reference [16] (Mori et al., 2025) is cited as supporting the finding that BM25 outperforms dense retrieval in formulaic legal text. This appears to be a preprint (arXiv:2506.12895). Please verify whether it has been peer-reviewed and cite its preprint status.
  7. §V: The paper states 'BM25_Chunks provided 8 slight performance improvement of 3.4%' over BM25_Docs. The '8' appears to be a typo for 'a'. Also, the 3.4% figure should be verified: 0.91 vs 0.88 is a 3 percentage-point difference, which is a 3.4% relative improvement — please state whether this is absolute or relative.
  8. The paper does not report any statistical significance tests for the retrieval performance differences in Table III. Given the small test set (n=100), reporting whether the BM25 vs e5-large difference is statistically significant (e.g., via a paired test) would strengthen the claims.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for a careful and constructive review. The referee raises three major concerns: (1) expert-authored queries may systematically favor BM25 over dense retrievers, confounding the headline comparison; (2) generation metrics lack confidence intervals or cross-validation on the 100-query test set; and (3) the LLM-as-judge (Gemini 2.5 Pro) for groundedness lacks validation against human annotators. We agree that all three points identify genuine methodological gaps. We will address (2) and (3) directly by adding bootstrap confidence intervals and inter-annotator agreement statistics in the revision. For (1), we partially agree—the expert-query confound is real—but we explain why the finding remains informative as a baseline while acknowledging the limitation more explicitly. We cannot fully resolve (1) without collecting a non-expert query set, which is beyond the scope of a revision but is planned as future work.

read point-by-point responses

  1. Referee: Expert-authored queries likely mirror the exact statutory terminology and formulaic phrasing of the source documents, systematically favoring BM25 over dense retrieval. The stated goal of serving public engagement implies non-expert users, yet the evaluation uses expert-formulated queries.

    Authors: The referee raises a valid and important methodological concern. We agree that expert-authored queries are more likely to mirror the lexical and formulaic phrasing of the source documents, which can systematically favor BM25's term-frequency matching. This is a genuine confound in our experimental design. We will revise the manuscript to acknowledge this limitation explicitly in both the Discussion (§V) and the methodology description (§III.A), rather than framing the BM25 advantage solely as a domain property. That said, we note two points that partially mitigate the concern. First, our finding is consistent with prior work on formulaic legal text in other languages (refs [16, 21, 25] in the manuscript), where BM25 advantages were observed on non-expert-constructed benchmarks as well. Second, 21% of our test queries are subjective (assessing legal procedure and reasoning), which are less likely to directly mirror document terminology. However, we acknowledge that 79% objective queries may still exhibit lexical overlap bias even when not directly quoting statutory language. We cannot fully resolve this confound within the scope of a revision—collecting a non-expert query set with verified ground-truth annotations requires substantial expert validation, which we have planned as future work. We will add this as an explicit limitation and adjust our claims to scope the BM25-vs-dense comparison to expert-formulated queries rather than general user queries. revision: partial

  2. Referee: Generation metrics (truthfulness, groundedness, successful answer rate) are computed on the same 100-query test set with no held-out split or cross-validation. Confidence intervals or bootstrap estimates should be reported for Table V.

    Authors: We agree. The 100-query test set is small, and reporting point estimates without uncertainty intervals overstates the precision of the generation metrics. We will add bootstrap 95% confidence intervals for all metrics in Table V (truthfulness, groundedness, and successful answer rate) across all three methods (BM25_Docs, e5-large, Count). We will also clarify in §IV.B that the retrieval and generation evaluations use the same 100-query set—there is no separate held-out split—and explain why this was necessary given the cost and expertise required for expert annotation of legal ground-truth answers. We acknowledge that a different query set could shift the relative ranking of generation methods, and the confidence intervals will make this uncertainty visible to readers. We will also note in the revision that the comparison between BM25_Docs and e5-large (85% vs 78% truthfulness) should be interpreted in light of the overlapping or non-overlapping confidence intervals once computed. revision: yes

  3. Referee: The groundedness evaluation uses Gemini 2.5 Pro as an LLM-as-judge with binary scoring, but no validation of the judge's reliability is reported—no inter-annotator agreement with human evaluators, no calibration against a gold-standard subset. The relationship between human (84% truthfulness) and automated (85% truthfulness) evaluations is unclear.

    Authors: We agree that this is a gap. The LLM-as-judge evaluation lacks validation against human annotations, and the relationship between the human truthfulness evaluation (84%) and the automated truthfulness evaluation (85%) is not clearly specified. We will address this in the revision as follows. First, we will clarify that the human and automated truthfulness evaluations were conducted on the same 100 queries for BM25_Docs, and we will report inter-annotator agreement (Cohen's kappa or percentage agreement) between the human evaluators and the Gemini 2.5 Pro judge on the truthfulness dimension. Second, we will create a gold-standard subset (approximately 30-50 queries) where human annotators also perform the groundedness evaluation, and report agreement statistics between the human groundedness judgments and the LLM judge's binary groundedness scores. This will provide calibration for the 74% groundedness headline figure. We acknowledge that if agreement is low, the groundedness number should be interpreted with caution. We will report these results transparently regardless of the outcome. revision: yes

Circularity Check

0 steps flagged

No circularity found — the paper applies standard IR methods to a new domain with an independently curated test set

full rationale

The paper's derivation chain is straightforward and self-contained. BM25 and dense retrieval (multilingual-e5-large, e5-base, LaBSE) are standard, externally defined methods applied without fitting parameters to the test set. The 100-query test set was independently curated by legal domain scholars at Kathmandu University School of Law, mapped to single ground-truth documents. Retrieval metrics (Precision@1, Hit@k, MRR@k) are standard IR metrics computed against this test set — no metric is defined in terms of the retrieval method's output. Generation quality (truthfulness, groundedness) is evaluated by an independent LLM-as-judge (Gemini 2.5 Pro) and human evaluators using prompts that compare generated answers to annotator-curated reference answers. No parameter is fitted to a subset of data and then 'predicted' on related data. The paper cites external work [16, Mori et al.] supporting the finding that BM25 outperforms dense retrieval on formulaic legal text, but this is independent confirmation, not a self-citation chain. The authors do not cite themselves at all. The skeptic's concern about expert-formulated queries potentially favoring BM25 is a validity/generalization concern (correctly classified under correctness risk), not a circularity issue — the test set construction does not define or force the retrieval results by construction.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 0 invented entities

The paper introduces no new entities (particles, forces, dimensions, etc.). It uses only existing tools (BM25, FAISS, LangChain, GPT-o3, Gemini 2.5 Pro, multilingual-e5 models). The free parameters are standard design choices for a RAG pipeline, not fitted to the evaluation set. The most consequential axiom is the single-ground-truth-document assumption for the test set, which simplifies evaluation but limits generalizability.

free parameters (6)
  • Chunk size (500 tokens) = 500
    Chosen to fit within the 512-token limit of embedding models; not tuned on the evaluation set but is a design choice that affects retrieval performance.
  • Chunk overlap (10%) = 0.10
    Selected to maintain contextual continuity; not tuned.
  • Temperature (0.5) = 0.5
    Set for GPT-o3 generation to balance factuality and fluency; not tuned on the evaluation set.
  • Top-p (1.0) = 1.0
    Generation parameter for GPT-o3; standard default.
  • Max output length (2048 tokens) = 2048
    Cap on generation length; not tuned.
  • Top-K for retrieval evaluation (k=1,3,5,10,15,20) = various
    Standard IR evaluation cutoffs; not tuned but reported across all values.
axioms (5)
  • domain assumption BM25 is appropriate for legal domain retrieval due to strong performance on TREC Legal Track and COLIEE benchmarks
    Invoked in §II and §III.B to justify the choice of BM25; supported by cited prior work [13, 21, 25].
  • domain assumption Nepali legal language is formulaic and lexically constrained, making exact-term overlap more informative than semantic approximation
    Invoked in §V (Discussion) to explain why BM25 outperforms dense retrieval; supported by analogy to prior work [16, 17] but not independently verified for Nepali.
  • ad hoc to paper Each test query corresponds to exactly one relevant case document
    Stated in §III.A: 'each query strictly corresponds to a single relevant case document.' This simplifies evaluation but is an artificial constraint that may not reflect real legal QA scenarios.
  • domain assumption LLM-as-judge (Gemini 2.5 Pro) can reliably evaluate groundedness and truthfulness of generated answers
    Invoked in §III.E and §IV.B; partially validated by comparison with human evaluation (84% vs 85% truthfulness), but inter-annotator agreement is not reported.
  • domain assumption GPT-o3 can generate accurate Nepali legal answers without domain-specific fine-tuning when provided with retrieved context
    Invoked in §III.D and §V; supported by the experimental results but assumed a priori.

pith-pipeline@v1.1.0-glm · 16111 in / 3442 out tokens · 254291 ms · 2026-07-05T15:51:41.956750+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

6 extracted references · 6 canonical work pages · 3 internal anchors

  1. [1]

    Exploring the state of the art in legal QA systems,

    A. Abdallah, उ. Piryani, and A. Jatowt, “Exploring the state of the art in legal QA systems,” Journal of Big Data, vol. 10, 2023. [Online]. Available: https://doi.org/10.1186/s405: 00802-8

    work page doi:10.1186/s405: 2023

  2. [2]

    Natural Language Processing for the Legal Domain: A Survey of Tasks, Datasets, Models, and Challenges,

    F. Ariai and G. Demartini, “Natural Language Processing for the Legal Domain: A Survey of Tasks, Datasets, Models, and Challenges,” 2025. [Online]. Available: https:/ /arxiv.org/abs/2410.21306. [छ] T.N. Ba, V. D. The, T. P. Quang, and T. T. Van, “Vietnamese Legal Information Retrieval in Question-Answering System,” 2024. [Online]. Available: https://arxiv...

    work page arXiv 2025

  3. [3]

    Evaluating the Retrieval Robustness of Large Language Models

    5. Cao, K. Radhakrishnan, D. Rosenberg, S. Lu, P. Cheng, L. Wang, and S. Zhang, “Evaluating the Retrieval Robust- ness of Large Language Models,” 2025. [Onlinc]. Available: https:/ /arxiv.org/abs/2505.21870. 5 6] [7] 8 [9] 70 1] [2] 13) 14] छ] 7] [7] छ] 7" 20] [21] L. Chalkidis, M. Fergadiotis, P. Malakasiotis, N. Aletras, and L. Androutsopoulos, “LEGAL-B...

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.47164/i- 2025

  4. [4]

    Retrieval Augmented Generation Evaluation in the Era of Large Language Models: A Comprehensive Survey

    [Online]. Available: https://arxiv.org/abs/2504.14891. Karpukhin, उ. Oguz, S. Min, P. Lewis, L. Wu, 5. Edunov, D. लाला, and W.-t. Yih, “Dense Passage Retricval for Open-Domain Question Answering,” 2020. [Online]. Available: https://arxiv.org/abs/2004.04906. M.-Y. Kim, J. Rabelo, K. Okeke, and R. Goebel, “Legal Infor- mation Retrieval and Entailment Based ...

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.1007/s12626-022-00103-1 2020

  5. [5]

    Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

    Riedel, and D. Kiela, “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks,” CoRR, abs/2005.11401, 2020. [Online]. Available: https://arxiv.org/abs/2005.11401. H. Ma, Y. Lu, Z. Xiao, J. Feng, छ. Zhang, and J. Yu, “SDD- LawLLM: Advancing Intelligent Legal Systems Through Syn- thetic Data-Driven Fine-Tuning of Large Language Models,” Electronic...

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.1145/3626772.3657717 2005

  6. [6]

    Legal Information Retrieval,

    छ. Rabiou, “Legal Information Retrieval,” पाए Handbook ला Law and Technology, Edward Elgar Publishing, 2021, pp. 198-215. [Online]. Available: https://doi.org/10.4337/9781788972826.00016. S. Robertson and H. Zaragoza, “The Probabilistic Relevance Framework: BM25 and Beyond,” Foundations and Trends in [22) [23] थ्व न्‌ एल शत Information Retrieval, vol 3-38...

    work page doi:10.4337/9781788972826.00016 2021