REVIEW 4 major objections 7 minor 16 references
Optimizing Legal Document Retrieval in Vietnamese with Semi-Hard Negative Mining
T0 review · 4 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read A lightweight two-stage Vietnamese legal retriever reaches 79.11% MRR@10 locally and 77.54% on the private test, matching far larger ensembles.
desk verdict A useful Vietnamese legal retrieval system paper with a solid-looking negative-mining comparison, but the local validation split likely leaks question IDs and the 'semi-hard' negatives are closer to easy ones than the name suggests. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the two-stage pipeline: a Bi-Encoder quickly embeds the query and each legal document independently, allowing cosine search over the full 261,446-document corpus to return 90 candidates, and a Cross-Encoder then scores each query-candidate pair jointly and returns the top 10. Candidate negatives for the re-ranker are mined from the Bi-Encoder's own top-90 lists: after removing correct answers, hard negatives take the highest-scoring wrong documents, semi-hard negatives randomly sample from the remaining candidates, and easy negatives sample from the whole corpus. Exist@m is the metric that makes the retriever's job explicit, checking only whether the correct document is present in the candidate set rather than its rank, since the re-ranker will do the final ordering.
What would settle it
Group the processed training questions by their question ID before the 90/10 split and re-run fine-tuning: if MRR@10 falls from 79.11% toward 77.54%, the local split was leaking. Independently, train the re-ranker on easy negatives matched to the same mean cosine similarity as the claimed semi-hard negatives: if MRR@10 stays near 79%, the paper's mechanism is not the semi-hard difficulty band.
Extended reading notes
Core claim
The paper's central discovery is an engineering result: a two-stage retrieval and re-ranking pipeline, built from a fine-tuned Vietnamese Bi-Encoder and a pretrained Vietnamese Cross-Encoder, can place in the top three of a legal document retrieval competition while using far fewer parameters than ensemble approaches. The retriever is trained with MultipleNegativesRankingLoss and evaluated with a new metric, Exist@m, which measures whether at least one correct document appears among the top m candidates; the fine-tuned retriever reaches roughly 97% Exist@90, beating BM25. The re-ranker is trained with binary cross-entropy on positive pairs plus negatives mined from the retriever's top-90 candidates. Among hard, semi-hard, and easy negative mining, only semi-hard negatives give large and reliable gains, and increasing the number of such negatives from 2 to 10 steadily improves MRR@10, while hard negatives are unstable until the sample size becomes large.
Load-bearing premise
The local 79.11% result depends on the 90/10 split of processed training questions not letting the same question appear in both training and validation; if that fails, only the 77.54% private-test score is independent evidence.
Editorial extensions
If this is right
- If the result holds, a single fine-tuned Bi-Encoder plus Cross-Encoder is enough for competitive specialized legal retrieval, so expensive ensembles are not a prerequisite for top performance.
- Retrieval stages in a two-stage pipeline can be tuned for candidate coverage with Exist@m instead of MRR@10, which should make retriever development faster and more directly aligned with the re-ranker's needs.
- Semi-hard negatives mined from the retriever's own candidates appear to be a reliable and cheap training signal for re-rankers, with more negatives per question consistently improving performance.
- The 23% relative MRR@10 gain suggests that data preparation steps like replacing truncated answers with full legal documents, splitting multi-answer questions, and Vietnamese word segmentation are not incidental but carry substantial value.
- A lightweight pipeline of this kind is more practical for real-time or large-scale legal information retrieval than ensemble systems, because only 90 pairs per query reach the expensive Cross-Encoder.
Reading between the lines
- Editorial extension: the paper's local 79.11% score would be more convincing if the 90/10 split grouped by question ID; the authors do not describe such grouping, so a reader should treat the private-test 77.54% as the stronger evidence.
- Editorial extension: Table 3 shows the random top-90 negatives have mean cosine similarity 0.2072, close to the easy-negative distribution, so the paper's label 'semi-hard' may overstate how close these negatives are to true positives; the benefit could come more from sampling candidates on the retrieval frontier than from the difficulty band itself.
- Editorial extension: a natural transfer test is to apply the same retrieval-plus-rerank recipe with retriever-mined semi-hard negatives to other low-resource domains or languages, where ensembles are less affordable.
- Editorial extension: the authors attribute hard-negative instability to large gradients from near-positive samples; this could be tested directly by monitoring gradient norms or positive-negative score margins during training under each mining strategy.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper describes a two-stage pipeline for Vietnamese legal document retrieval: a fine-tuned Vietnamese Bi-Encoder retrieves top-90 candidates, and a PhoRanker Cross-Encoder re-ranks them. The authors propose a recall-style metric Exist@m for the retrieval stage and compare three negative-mining strategies for the re-ranker, reporting that semi-hard negatives (random candidates from the Bi-Encoder top-90) outperform hard and easy negatives, with 79.11% MRR@10 on a local eval split and 77.54% on the SoICT 2024 private test.
Significance. If the empirical comparisons are valid, the main contribution is a practical demonstration that, for a legal QA retrieval task, training a cross-encoder with negatives sampled from the retriever's candidate distribution gives large gains over corpus-wide easy negatives and over the hardest in-candidate negatives, with consistent results across three random seeds. The paper is clearly written and does not overstate the architecture's novelty relative to ensemble competitors. However, the local evaluation is compromised by the data-splitting pipeline (Section 5.2 followed by Section 5.4) and by an ambiguous statement that the Bi-Encoder was fine-tuned using both train_df and eval_df (Section 7.1); until those issues are fixed, the headline local numbers cannot be regarded as evidence for generalization, and the private-test score lacks the protocol details needed to carry the comparison alone.
major comments (4)
- [§7.1, Table 1] The sentence 'The Bi-Encoder is fine-tuned using the preprocessed train_df and eval_df datasets' is, if taken literally, direct training-data evaluation: the model would be evaluated on eval_df after training on it, invalidating all Exist@90 and MRR@10 values in Table 1 and the choice of the 11-epoch retriever. Please clarify whether this is a typo (i.e., trained on train_df and evaluated on eval_df) or re-run the experiments with a strict separation; this also affects the negative-mining step in Section 7.2 because the Bi-Encoder used to mine negatives would have seen eval_df.
- [§5.2, §5.4, Table 2] The local validation split is made after Section 5.2 expands multi-answer questions into separate rows, and Section 5.4 does not state that the 90/10 split groups by qid. Since roughly 10% of questions are linked to multiple cids, rows with the same qid can fall on both sides of the split, so eval_df is not a held-out set for those questions. Both the Bi-Encoder (MultipleNegativesRankingLoss) and the Cross-Encoder (BCEWithLogitsLoss) are trained on question–document pairs, so a validation row whose qid appeared in training can be memorized; this inflates the local 79.11% MRR@10 and the Exist@90 values in Tables 1–3, and it undermines the strategy-level comparison in Table 2, which is the paper's central empirical claim. Please re-split by qid (or report both row-level and qid-level splits) and re-run all local tables; alternatively, provide per-strategy results on the private test.
- [§7.2, Table 3] The operational definition of semi-hard negatives as uniform random draws from the Bi-Encoder top-90 after removing correct answers produces negatives with a mean cosine similarity of 0.2072 (79.44% below 0.5), which is much closer to the easy-negative distribution (mean 0.0008) than to the 'similar to the correct answers but still incorrect' definition stated in Section 3.3. Table 3 therefore does not demonstrate that the improvement in Table 2 is due to semi-hardness; it demonstrates only that negatives from the candidate set help. The paper should rename the strategy or provide an analysis that isolates hardness (for example, sampling from specified cosine-similarity bands), and the mechanistic claims in Sections 6.3 and 8 should be moderated accordingly.
- [Abstract / Conclusion / private test] The 77.54% MRR@10 private-test score is the only independent evidence for the headline result, but it is reported without any evaluation protocol: number of submitted runs, selection procedure (e.g., best-of-k), and whether the same negative-mining strategy was used. Because the local eval_df comparison is compromised by the qid split, this single number cannot independently validate the Table 2 strategy comparisons. Please add a subsection describing the private-test submission and, if possible, report private-test scores for at least the main negative-mining variants.
minor comments (7)
- [§6.2, Eq. (1)] Exist@m is recall@m for multi-document questions; the novelty claim in Section 1 should be calibrated by relating it to standard recall-oriented IR metrics.
- [§7.2, Table 3] The text says 'Using PhoRanker, we measured the initial cosine similarity', but PhoRanker is a Cross-Encoder and does not naturally produce query–document cosine similarities; specify the model and how the embeddings or scores were obtained.
- [§7.1, Table 1] The BM25Plus configuration with b=0 is described as 'tuned k1 and b' in the text, but the table lists many configurations; clarify whether these are a grid search and how the best configuration was selected to avoid selection-on-test concerns.
- [Figure 3] Figure 3 places the 0.9/0.1 split after the 'separate if multiple answers' step; annotate the figure to make the qid-grouping choice explicit, or redraw the pipeline to split before expansion.
- [§8, Conclusion] The conclusion states the model 'achieves same performance compared to competitors', but no table with other teams' private-test scores is provided; please add the competition leaderboard excerpt or precise ranking context.
- [§4.4] Section 4.4 says models with 1024-token capacity may be beneficial, but no such experiment is reported; either remove the speculation or add a sentence clarifying it is outside the scope.
- [General] There is no statement about code or data availability; for reproducibility of an empirical benchmark paper, please add one.
Circularity Check
No significant circularity: the empirical claims are measured on held-out and private data, and the self-referential metric and split concerns are validity issues rather than circular derivations.
full rationale
This is an empirical systems paper rather than a mathematical derivation, so the circularity burden is low. The Bi-Encoder and Cross-Encoder are trained on processed training rows and evaluated on a 90/10 split plus an external private test; the headline MRR@10 and the 23% relative improvement from semi-hard negatives are computed from trained model outputs, not forced by the definitions of the losses or metrics. Exist@m is a metric the authors define and then use for retriever selection, but reporting Exist@90 measured after training is not a prediction that reduces to its own input. The operationalization of semi-hard negatives as random draws from the Bi-Encoder top-90 is arguably a naming/mechanism concern, especially because Table 3 shows those negatives have mean cosine similarity 0.2072, close to easy negatives, but the Table 2 comparisons are empirical outcomes rather than identities. The most serious issue is the local evaluation split: Section 5.2 expands multi-answer questions into multiple rows before the Section 5.4 90/10 split, with no qid-level grouping described, so rows with the same qid could appear in both training and validation and inflate the 79.11% MRR@10. That is a data-split validity risk, not a circularity-by-construction, and the 77.54% private-test score remains independent external evidence. No load-bearing self-citation is present.
Assumptions & free parameters
free parameters (5)
- top_m candidate count =
90
- negative sample count n =
2, 5, 10
- BM25 k1 and b =
k1 in {0.8, 1.2, 2.0}, b in {0, 0.75, 1}
- fine-tuning epochs =
7, 9, 11 for Bi-Encoder, 2 for Cross-Encoder
- random seeds =
28, 42, 2025
assumptions (4)
- domain assumption Pre-trained Vietnamese Bi-Encoder and PhoRanker checkpoints are suitable starting points for legal retrieval fine-tuning.
- domain assumption Random negatives sampled from the Bi-Encoder top-90 candidates are semi-hard negatives.
- domain assumption The 90/10 train/validation split is a fair estimate of generalization.
- domain assumption Pyvi tokenization preserves semantic content needed for retrieval.
Cite this review
Pith. "Pith review of Optimizing Legal Document Retrieval in Vietnamese with Semi-Hard Negative Mining." pith.science (2026). https://pith.science/paper/JXIUSSWI
@misc{pith2026250714619,
author = {Pith},
title = {Pith review of: Optimizing Legal Document Retrieval in Vietnamese with Semi-Hard Negative Mining},
year = {2026},
howpublished = {\url{https://pith.science/paper/JXIUSSWI}},
note = {Machine review of arXiv:2507.14619}
}
read the original abstract
Large Language Models (LLMs) face significant challenges in specialized domains like law, where precision and domain-specific knowledge are critical. This paper presents a streamlined two-stage framework consisting of Retrieval and Re-ranking to enhance legal document retrieval efficiency and accuracy. Our approach employs a fine-tuned Bi-Encoder for rapid candidate retrieval, followed by a Cross-Encoder for precise re-ranking, both optimized through strategic negative example mining. Key innovations include the introduction of the Exist@m metric to evaluate retrieval effectiveness and the use of semi-hard negatives to mitigate training bias, which significantly improved re-ranking performance. Evaluated on the SoICT Hackathon 2024 for Legal Document Retrieval, our team, 4Huiter, achieved a top-three position. While top-performing teams employed ensemble models and iterative self-training on large bge-m3 architectures, our lightweight, single-pass approach offered a competitive alternative with far fewer parameters. The framework demonstrates that optimized data processing, tailored loss functions, and balanced negative sampling are pivotal for building robust retrieval-augmented systems in legal contexts.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[1]
Retrieval-Augmented Gen- eration for Knowledge-Intensive NLP Tasks
Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K ¨uttler, Mike Lewis, Wen-tau Yih, Tim Rockt¨aschel, Sebastian Riedel, and Douwe Kiela. “Retrieval-Augmented Gen- eration for Knowledge-Intensive NLP Tasks”. In: Advances in Neural Informa- tion Processing Systems 33: Annual Conference on Neural ...
work page 2020
-
[2]
A statistical interpretation of term specificity and its appli- cation in retrieval
Karen Sp ¨arck Jones. “A statistical interpretation of term specificity and its appli- cation in retrieval”. In: J. Documentation 60.5 (2004), pp. 493–502
work page 2004
-
[3]
Some Simple Effective Approxima- tions to the 2-Poisson Model for Probabilistic Weighted Retrieval
Stephen E. Robertson and Steve Walker. “Some Simple Effective Approxima- tions to the 2-Poisson Model for Probabilistic Weighted Retrieval”. In: Proceed- ings of the 17th Annual International ACM-SIGIR Conference on Research and Development in Information Retrieval. Dublin, Ireland, 3-6 July 1994 (Special Issue of the SIGIR Forum) . Ed. by W . Bruce Croft...
work page 1994
-
[4]
Sentence-BERT: Sentence Embeddings us- ing Siamese BERT-Networks
Nils Reimers and Iryna Gurevych. “Sentence-BERT: Sentence Embeddings us- ing Siamese BERT-Networks”. In: Proceedings of the 2019 Conference on Em- pirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). Ed. by Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan. Hong Kong, ...
work page 2019
-
[5]
Universal Sentence Encoder for English
Daniel Cer, Yinfei Y ang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Brian Strope, and Ray Kurzweil. “Universal Sentence Encoder for English”. In: Pro- ceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. Ed. by Eduardo Blan...
work page 2018
-
[6]
Legal Document Retrieval using Document Vector Embeddings and Deep Learning
Keet Sugathadasa, Buddhi Ayesha, Nisansa de Silva, Amal Shehan Perera, Vin- dula Jayawardana, Dimuthu Lakmal, and Madhavi Perera. “Legal Document Retrieval using Document Vector Embeddings and Deep Learning”. In: CoRR abs/1805.10685 (2018). arXiv: 1805.10685
work page Pith review arXiv 2018
-
[7]
Attentive deep neural networks for legal doc- ument retrieval
Ha-Thanh Nguyen, Manh-Kien Phi, Xuan-Bach Ngo, Vu D. Tran, Le-Minh Nguyen, and Minh-Phuong Tu. “Attentive deep neural networks for legal doc- ument retrieval”. In: Artif. Intell. Law 32.1 (2024), pp. 57–86
work page 2024
-
[8]
Enhancing Legal Document Retrieval: A Multi-Phase Approach with Large Language Models
Hai-Long Nguyen, Duc-Minh Nguyen, Tan-Minh Nguyen, Ha-Thanh Nguyen, Thi-Hai-Y en Vuong, and Ken Satoh. “Enhancing Legal Document Retrieval: A Multi-Phase Approach with Large Language Models”. In: CoRR abs/2403.18093 (2024). arXiv: 2403.18093
work page Pith review arXiv 2024
Show all 16 references
-
[9]
Learning Dense Representa- tions for Entity Retrieval
Daniel Gillick, Sayali Kulkarni, Larry Lansing, Alessandro Presta, Jason Baldridge, Eugene Ie, and Diego Garcia-Olano. “Learning Dense Representa- tions for Entity Retrieval”. In: Proceedings of the 23rd Conference on Compu- tational Natural Language Learning (CoNLL) . Ed. by ...
2019
-
[10]
Ef- ficient Natural Language Response Suggestion for Smart Reply
Matthew L. Henderson, Rami Al-Rfou, Brian Strope, Yun-Hsuan Sung, Lás- zló Lukács, Ruiqi Guo, Sanjiv Kumar, Balint Miklos, and Ray Kurzweil. “Ef- ficient Natural Language Response Suggestion for Smart Reply”. In: CoRR abs/1705.00652 (2017). arXiv: 1705.00652
2017 arXiv
-
[11]
Multi-Stage Document Ranking with BERT
Rodrigo Nogueira, Wei Y ang, Kyunghyun Cho, and Jimmy Lin. “Multi-Stage Document Ranking with BERT”. In: CoRR abs/1910.14424 (2019). arXiv: 1910.14424
2019 arXiv
-
[12]
In De- fense of Cross-Encoders for Zero-Shot Retrieval
Guilherme Rosa, Luiz Henrique Bonifacio, Vitor Jeronymo, Hugo Queiroz Abonizio, Marzieh Fadaee, Roberto A. Lotufo, and Rodrigo Nogueira. “In De- fense of Cross-Encoders for Zero-Shot Retrieval”. In: CoRR abs/2212.06121 (2022). arXiv: 2212.06121
2022 arXiv
-
[13]
T wente-BMS-NLP at PerspectiveArg 2024: Combining Bi-Encoder and Cross-Encoder for Argument Retrieval
Leixin Zhang and Daniel Braun. “T wente-BMS-NLP at PerspectiveArg 2024: Combining Bi-Encoder and Cross-Encoder for Argument Retrieval”. In: Pro- ceedings of the 11th Workshop on Argument Mining (ArgMining 2024) . Ed. by Y amen Ajjour, Roy Bar-Haim, Roxanne El Baff, Zhexiong Li...
2024
-
[14]
ACORD: An Expert-Annotated Retrieval Dataset for Legal Contract Drafting
Steven H. Wang, Maksim Zubkov, Kexin Fan, Sarah Harrell, Yuyang Sun, Wei Chen, Andreas Plesner, and Roger Wattenhofer. “ACORD: An Expert-Annotated Retrieval Dataset for Legal Contract Drafting”. In:CoRR abs/2501.06582 (2025). arXiv: 2501.06582
2025
-
[15]
Towards Comprehensive Viet- namese Retrieval-Augmented Generation and Large Language Models
Nguyen Quang Duc, Le Hai Son, Nguyen Duc Nhan, Nguyen Dich Nhat Minh, Le Thanh Huong, and Dinh Viet Sang. “Towards Comprehensive Viet- namese Retrieval-Augmented Generation and Large Language Models”. In: CoRR abs/2403.01616 (2024). arXiv: 2403.01616
2024 arXiv
-
[16]
PhoRanker: A Cross-encoder Model for Vietnamese Text Rank- ing
Dai Nguyen Ba. PhoRanker: A Cross-encoder Model for Vietnamese Text Rank- ing. https://huggingface.co/itdainb/PhoRanker. 2024
2024
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.