1-800-SHARED-TASKS at RegNLP: Lexical Reranking of Semantic Retrieval (LeSeR) for Regulatory Question Answering

Drishti Sharma; Jebish Purbey; Khawaja Murad; Ram Mohan Rao Kadiyala; Siddartha Pullakhandam; Siddhant Gupta

REVIEW 3 major objections 4 minor 16 references

1-800-SHARED-TASKS at RegNLP: Lexical Reranking of Semantic Retrieval (LeSeR) for Regulatory Question Answering

T0 review · 3 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A two-stage dense-then-BM25 retrieval beats both pure approaches on regulatory QA.

desk verdict A competent shared-task system description whose headline gain rests on an unspecified fusion weight; worth a conditional accept, not a pass. read the letter →

arxiv 2412.06009 v1 pith:E5WBU7HF submitted 2024-12-08 cs.CL cs.IRcs.LG

Jebish Purbey , Drishti Sharma , Siddhant Gupta , Khawaja Murad , Siddartha Pullakhandam , Ram Mohan Rao Kadiyala This is my paper · ORCID

classification cs.CLcs.IRcs.LG

keywords regulatoryquestionansweringhybridretrievallexicalrerankingBM25denseMNSRfine-tuningObliQAretrieval-augmentedgeneration

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper is a system description for the RIRAG regulatory question-answering challenge, and its central claim is that retrieval improves when the job is split into two phases: a dense semantic search that gathers candidate passages and a lexical BM25 rerank that fixes their order. On the ObliQA regulatory benchmark, this LeSeR pipeline reaches a recall@10 of 0.8201 and a mean average precision@10 of 0.6655, beating both the pure lexical baseline BM25 (0.7611 and 0.6237) and the best purely dense model (0.8068 and 0.1077). The authors argue this is worth caring about because regulatory texts depend on exact terminology, so a model that can retrieve broadly but cannot rank precisely is not enough for compliance work. They also note, in their own Limitations section, that dense models alone underperform on ranking precision and that answer generation might improve if only the most relevant retrieved contexts were fed to the generator.

What carries the argument

LeSeR (Lexical reranking of Semantic Retrieval) is a two-stage pipeline that separates recall from precision: a fine-tuned embedding model returns the top 20 passages by cosine similarity, and BM25 sparse scores are then combined with those dense scores in a weighted aggregation, with the top 10 kept as the final ranking. The decoupling is the load-bearing design: dense matching supplies broad recall, and lexical matching supplies exact-term precision. The quality of the final ranking therefore depends on how the two score types are mixed.

What would settle it

On a held-out set of regulatory queries, fix the fusion weights using only the development split, then sweep the weight from 0 to 1 and record mAP@10; if no interior weight is better than the pure dense and pure BM25 endpoints, the LeSeR advantage disappears, and if the optimum is at an endpoint, the hybrid claim reduces to whichever single retriever won.

Watch

Extended reading notes

Core claim

On its own terms, the paper's finding is that lexical reranking repairs the ranking weakness of dense retrieval in a legal domain. The authors fine-tune several open embedding models with a contrastive loss that treats every other example in the batch as a negative, retrieve the top 20 passages by cosine similarity of the embeddings, and then reorder those candidates by combining the dense scores with BM25 sparse scores in a weighted aggregation, keeping only the top 10. The fine-tuned BGE-small model, called BGE_MNSR, is the strongest candidate generator of the models tested, and after this reranking step, BGE_LeSeR outperforms every other system on both recall@10 and mAP@10, including the BM25 baseline. The same retrieval pipeline, when paired with the Qwen2.5 7B language model, also gives the best answer-generation score on the RePASs metric among the four LLMs compared.

Load-bearing premise

The reported superiority of LeSeR rests on the way the dense cosine scores and BM25 scores are merged, but the paper does not give the weights, normalisation, or tuning procedure, so if that merger was chosen arbitrarily or fitted to the test set, the advantage would not transfer.

Editorial extensions

If this is right

A hybrid that reranks dense candidates lexically can beat both BM25 and dense-only retrieval on the same benchmark: BGE_LeSeR gives recall@10 0.8201 and mAP@10 0.6655, against 0.7611/0.6237 for BM25 and 0.8068/0.1077 for the best dense-only model.
Fine-tuning a small general embedding model with symmetric contrastive loss makes it the strongest candidate generator among the tested dense models, even before reranking.
For answer generation, the same retrieved context works best with Qwen2.5 7B, which reaches the top RePASs score (0.4340) of the four LLMs tested.
Regulatory question-answering systems should not rely on a dense retriever alone when rankings matter, because the dense models here show high recall but very low mAP.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper's numerical gain in mAP@10 appears to come almost entirely from the fusion step rather than from the dense retriever; a natural extension is to sweep the fusion weight between pure dense and pure BM25 on a fixed split and check whether any interior mixture beats both extremes.
Since the paper does not report the fusion weights, normalisation, or tuning procedure, a reproducibility test is to re-derive the weights on the development split and see whether the 0.6655 mAP@10 holds on unseen queries.
The authors' own Limitations section concedes that only the most relevant contexts should be appended to the generator prompt; a natural follow-up is to replace the top-10 context with the reranked top 3-5 passages and measure RePASs again.
The same two-stage recipe should transfer to other terminology-heavy domains such as medical or financial compliance, where exact phrases carry regulatory weight, but that is a testable guess rather than a claim of the paper.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Punchline: this is a solid shared-task system description wrapped in slightly inflated language. The actual method is a standard dense-retrieve-then-BM25-rerank hybrid, and the paper's central number—mAP jumping from 0.1077 for the dense model to 0.6655 for LeSeR—is real only if the undisclosed fusion weights are what make it work.

The paper does several things well. It evaluates five embeddings, fine-tunes with MNSR loss, reports both recall and mAP, and includes a limitations section that honestly concedes dense models underperform on ranking precision. The recall gain from 0.8068 to 0.8201 is modest but plausible, and the answer-generation comparisons are straightforward.

The soft spots are real and load-bearing. The weighted aggregation of BM25 and dense scores is never specified: no weights, no normalization, no tuning procedure. Because BM25 alone reaches mAP@10 of 0.6237, any fusion with even modest BM25 weight will reproduce most of that number. The stress-test concern is on target: as written, the dense component's contribution to the final ranking is unverified. The paper's own limitation statement says dense models 'underperformed in ranking precision.' That is not a flaw in itself, but it raises the bar for showing that the dense scores add anything beyond candidate recall. Also missing are error bars, significance tests, and code or data release; for a shared-task paper these are less critical but still worth noting.

The 'novel approach' claim is overstated. Decoupling dense retrieval and lexical reranking is a routine extension of hybrid retrieval. The paper is best read as a description of what worked on the RIRAG benchmark, not a conceptual advance.

Recommendation: this deserves a serious referee, but the referee should condition acceptance on the authors disclosing the fusion weights and ideally an ablation with BM25-only reranking on the same candidate set. For a shared-task system description, that is a reasonable bar. I would bring it to a reading group only if someone is working on regulatory QA or hybrid retrieval recipes.

Referee Report

3 major / 4 minor

Summary. This paper is a system description for the COLING 2025 RegNLP RIRAG shared task. For passage retrieval, the authors fine-tune several open-source dense embedding models (Stella, BGE, CDE, MedEmbed, MPNet) with MNSR loss on ObliQA, retrieve top-20 candidates with FAISS, and then rerank by a 'weighted aggregation' of dense cosine and BM25 scores (LeSeR). On the test split, the best system BGE_LeSeR reports recall@10 = 0.8201 and mAP@10 = 0.6655, compared with the BM25 baseline (0.7611/0.6237) and dense-only BGE_MNSR (0.8068/0.1077). For answer generation, the paper evaluates Qwen2.5 7B, Mistral 7B, Mistral Nemo 12B, and Gemma-2 9B using the RePASs metric, with Qwen2.5 7B performing best.

Significance. If the reported numbers are taken at face value, the paper offers a useful practical recipe for regulatory passage retrieval: dense candidates provide high recall and lexical reranking restores precision. The use of a public benchmark (ObliQA), standard metrics, and a systematic comparison across five embedding models are strengths, and a shared-task system description is a legitimate contribution to the workshop literature. However, the central methodological novelty is not currently verifiable: the fusion step is described only as a 'weighted aggregation' with no weights, normalization, or tuning protocol, and the reported gain in mAP is exactly the quantity that this unspecified component controls. The significance of the paper therefore depends on the authors disclosing the fusion details and providing an ablation; as written, the LeSeR contribution cannot be assessed independently of the unspecified fusion hyperparameters.

major comments (3)

[§3, Methodology; Table 2] The 'weighted aggregation' that defines LeSeR is never specified. The paper does not state how dense cosine similarities and BM25 scores are normalized before combination, what weights are used, or how the weights were chosen. This matters because BM25 alone already achieves mAP@10 = 0.6237, while dense-only BGE_MNSR achieves 0.1077; a fusion that is even mostly lexical would trivially reproduce most of the reported 0.6655. As written, the central claim that BGE_LeSeR 'outperforms all other models' cannot be checked.
[§3, dev/test protocol; Table 2] The manuscript does not disclose whether the fusion weights, the candidate pool size of 20, or the fine-tuning checkpoints were selected on the development split or on the test split. Since Table 2 reports test-set numbers and the core claim is that LeSeR generalizes, the selection protocol must be stated. If test-set information was used to choose any hyperparameter, the reported numbers would be circular; if the dev set was used, that should be said explicitly.
[§4, Limitations; Table 2] The paper's own Limitations section concedes that dense models 'underperformed in ranking precision,' and the recall improvement from BGE_MNSR to BGE_LeSeR is only 0.0133 (0.8068 to 0.8201). To support the mechanistic story that semantic retrieval supplies high-recall candidates and lexical reranking supplies precision, the authors should report an ablation: dense-only, BM25-only, dense-top-20 with BM25 reranking, and dense-plus-BM25 at several fusion weights and candidate-pool sizes. Without this, the contribution of the dense stage to the final ranking is unverified.

minor comments (4)

[Title, §1, §3] LeSeR is expanded inconsistently: the title and abstract say 'Lexical Reranking of Semantic Retrieval,' while §3 introduces 'LeSeR (Lexical-Semantic Retrieval).' Please unify the expansion.
[Table 2, §4, Abstract] There are several typos and formatting inconsistencies: 'reprsent' in the Table 2 caption, 'baselines model' in §4, and 'map@10' in the abstract should be 'mAP@10'.
[§3, Figure 1] Figure 1 is cited in §3 as 'shown in Figure 1' but does not appear in the manuscript text; the reference should be either implemented or removed.
[Overall, Reproducibility] No code, configuration files, or data availability statement are included. For a system description, even a short statement on release of the retrieval/reranking code and fine-tuning configuration would substantially aid reproducibility.

Circularity Check

1 steps flagged · score 2.0 of 10

LeSeR's fusion result is a genuine test-set measurement, but the base model is selected using test-set recall, so the 'best dense model' claim is true by selection rather than by independent prediction; this is a minor, partially load-bearing circularity.

fitted input called prediction [Section 4, Results (Table 2 discussion)]
"During the fine-tuning phase, various retrieval models were assessed on the test dataset to identify the top-performing systems for the retrieval task (Table 2). ... Because of its highest recall score, BGE_MNSR is implemented in LeSeR approach."

The base dense model for LeSeR is chosen by reading Recall@10 off the test set, and the paper then uses the same test set to report that BGE_MNSR is the best dense model and that BGE_LeSeR outperforms all others. The statement that BGE_MNSR has the highest recall is not an independent prediction; it is the selection criterion itself (the argmax over the test column). This makes the base-model comparison true by construction rather than by out-of-sample evidence. The final LeSeR mAP@10 is a separate measurement not used for that selection, so the circularity is confined to the base-model choice and does not by itself force the fusion result.

full rationale

The paper is an empirical system description whose central claim, BGE_LeSeR's recall@10 of 0.8201 and mAP@10 of 0.6655, is a measurement on the RIRAG test set rather than a quantity derived from fitted parameters. The dense and lexical components (fine-tuned BGE and BM25) are externally defined, and the citations to ObliQA, RePASs, BM25, and FAISS involve no author overlap with this paper, so no self-citation chain is load-bearing. The undisclosed weighted aggregation of dense and BM25 scores is a reproducibility gap, not a circular step, because the paper does not state that the weights were tuned on the test set and no equation reduces the output to the input. The one genuine circular element is the model-selection procedure: the paper explicitly says models were assessed on the test set to choose the top performer, and then reports that chosen model as best using that same test set. This makes the 'BGE_MNSR is best' comparison a selection tautology rather than a prediction. However, the LeSeR fusion result is a new measurement that was not used as the selection criterion, so the central claim retains independent content; a score of 2 reflects this partial, non-central circularity.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim rests on the undocumented fusion of two retrieval signals. The free parameters (especially the fusion weights) are not reported, making it impossible to judge whether the improvement is robust or overfit.

free parameters (3)

score fusion weight
The weighted aggregation of dense and BM25 scores is never specified; if tuned on the dev/test set, it is a fitted parameter that determines the reported mAP jump.
candidate pool size = 20
The top 20 dense candidates are reranked; this is a manually chosen hyperparameter that affects the upper bound of recall.
fine-tuning epochs = 10 (max)
Maximum epochs for contrastive fine-tuning; checkpoint selected by dev performance.

assumptions (3)

domain assumption ObliQA annotations are ground truth
The entire evaluation relies on the correctness of the dataset's relevance labels.
domain assumption BM25 and dense scores can be combined linearly without normalization
The paper assumes a simple weighted sum is a valid fusion, but does not justify scale matching between cosine distances and BM25 scores.
domain assumption The shared-task test set is representative of real regulatory queries
Generalization claims depend on this assumption.

how reviews work

0 comments

Cite this review

Pith. "Pith review of 1-800-SHARED-TASKS at RegNLP: Lexical Reranking of Semantic Retrieval (LeSeR) for Regulatory Question Answering." pith.science (2026). https://pith.science/paper/E5WBU7HF

@misc{pith2026241206009,
  author       = {Pith},
  title        = {Pith review of: 1-800-SHARED-TASKS at RegNLP: Lexical Reranking of Semantic Retrieval (LeSeR) for Regulatory Question Answering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/E5WBU7HF}},
  note         = {Machine review of arXiv:2412.06009}
}

read the original abstract

This paper presents the system description of our entry for the COLING 2025 RegNLP RIRAG (Regulatory Information Retrieval and Answer Generation) challenge, focusing on leveraging advanced information retrieval and answer generation techniques in regulatory domains. We experimented with a combination of embedding models, including Stella, BGE, CDE, and Mpnet, and leveraged fine-tuning and reranking for retrieving relevant documents in top ranks. We utilized a novel approach, LeSeR, which achieved competitive results with a recall@10 of 0.8201 and map@10 of 0.6655 for retrievals. This work highlights the transformative potential of natural language processing techniques in regulatory applications, offering insights into their capabilities for implementing a retrieval augmented generation system while identifying areas for future improvement in robustness and domain adaptation.

Figures

Figures reproduced from arXiv: 2412.06009 by the authors.

**Figure 1.** System design workflow erate clear and concise information from varying sources to fully address the compliance and obligation requirement of the query. RIRAG utilizes the ObliQA Dataset (Gokhan et al., 2024) which is a RegNLP resource built from the Abu Dhabi Global Markets (ADGM) regulations. The dataset incorporates comprehensive and meticulously organized documents, preserving the intricate structure and term… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

16 extracted references · 6 canonical work pages

[1]

Sallam Abualhaija, Chetan Arora, Amin Sleimi, and Lionel C. Briand. 2022. https://doi.org/10.1109/RE54965.2022.00011 Automated question answering for improved understanding of compliance requirements: A multi-document study . In 2022 IEEE 30th International Requirements Engineering Conference (RE), pages 39--50

work page arXiv 2022
[2]

Abhinand Balachandran. 2024. https://github.com/abhinand5/MedEmbed Medembed: Medical-focused embedding models

work page 2024
[3]

Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre-Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. 2024. https://arxiv.org/abs/2401.08281 The faiss library . Preprint, arXiv:2401.08281

arXiv 2024
[4]

GemmaTeam. 2024. https://arxiv.org/abs/2403.08295 Gemma: Open models based on gemini research and technology . Preprint, arXiv:2403.08295

arXiv 2024
[5]

Tuba Gokhan, Kexin Wang, Iryna Gurevych, and Ted Briscoe. 2024. https://arxiv.org/abs/2409.05677 Regnlp in action: Facilitating compliance through automated information retrieval and answer generation . Preprint, arXiv:2409.05677

arXiv 2024
[6]

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. https://arxiv.org/abs/2310.0...

arXiv 2023
[7]

MistralAI. 2024. https://mistral.ai/news/mistral-nemo/ Mistral nemo

work page 2024
[8]

Morris and Alexander M

John X. Morris and Alexander M. Rush. 2024. https://arxiv.org/abs/2410.02525 Contextual document embeddings . Preprint, arXiv:2410.02525

arXiv 2024

Show all 16 references

[9]

Qwen Team . 2024. https://qwenlm.github.io/blog/qwen2.5/ Qwen2.5: A party of foundation models

2024
[10]

Stephen Robertson, Steve Walker, Susan Jones, Micheline Hancock-Beaulieu, and Mike Gatford. 1994. Okapi at trec-3. In 3rd Text REtrieval Conference (TREC-3), page 109–126

1994
[11]

Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2020. https://arxiv.org/abs/2004.09297 Mpnet: Masked and permuted pre-training for language understanding . Preprint, arXiv:2004.09297

2020 arXiv
[12]

UnslothAI. 2024. https://unsloth.ai/ Fastlanguage: A fast nlp model . Accessed: 2024-12-02

2024
[13]

Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. 2023. https://arxiv.org/abs/2309.07597 C-pack: Packaged resources to advance general chinese embedding . Preprint, arXiv:2309.07597

2023 arXiv
[14]

Dun Zhang. 2024. https://huggingface.co/dunzhang/stella_en_400M_v5 Stella: A multilingual text embedding model . Accessed: 2024-11-25

2024
[15]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
[16]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

Sallam Abualhaija, Chetan Arora, Amin Sleimi, and Lionel C. Briand. 2022. https://doi.org/10.1109/RE54965.2022.00011 Automated question answering for improved understanding of compliance requirements: A multi-document study . In 2022 IEEE 30th International Requirements Engineering Conference (RE), pages 39--50

work page arXiv 2022

[2] [2]

Abhinand Balachandran. 2024. https://github.com/abhinand5/MedEmbed Medembed: Medical-focused embedding models

work page 2024

[3] [3]

Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre-Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. 2024. https://arxiv.org/abs/2401.08281 The faiss library . Preprint, arXiv:2401.08281

arXiv 2024

[4] [4]

GemmaTeam. 2024. https://arxiv.org/abs/2403.08295 Gemma: Open models based on gemini research and technology . Preprint, arXiv:2403.08295

arXiv 2024

[5] [5]

Tuba Gokhan, Kexin Wang, Iryna Gurevych, and Ted Briscoe. 2024. https://arxiv.org/abs/2409.05677 Regnlp in action: Facilitating compliance through automated information retrieval and answer generation . Preprint, arXiv:2409.05677

arXiv 2024

[6] [6]

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. https://arxiv.org/abs/2310.0...

arXiv 2023

[7] [7]

MistralAI. 2024. https://mistral.ai/news/mistral-nemo/ Mistral nemo

work page 2024

[8] [8]

Morris and Alexander M

John X. Morris and Alexander M. Rush. 2024. https://arxiv.org/abs/2410.02525 Contextual document embeddings . Preprint, arXiv:2410.02525

arXiv 2024