REVIEW 4 major objections 5 minor 16 references
A Data Science Approach to Calcutta High Court Judgments: An Efficient LLM and RAG-powered Framework for Summarization and Similar Cases Retrieval
T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Fine-tuned Pegasus with a two-step Abs-Ext pipeline summarizes legal judgments and retrieves similar cases.
desk verdict Plausible legal NLP pipeline, but the summarization claims rest on LLM-generated references with no documented validation, and the text contradicts its own tables. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the two-step Abs-Ext summarization pipeline, built on two named components. Pegasus is a transformer model pre-trained with extracted gap sentences for abstractive summarization; the paper fine-tunes it on case headnote summaries to adapt it to legal language. BERTSum is the extractive summarizer that selects key sentences from the Pegasus output. The paper's argument is that this order, abstract first and extract second, preserves the legal context that a vector database needs, so the final stage, retrieval-augmented generation, can take a user query, embed it, find the top three matching summaries, and generate a grounded answer.
What would settle it
Re-run the fine-tuning and evaluation with a strictly partitioned train/test split of the 3,100 annotated cases, and have a second legal expert independently write headnotes for a random sample of test cases; if fine-tuned Abs-Ext no longer beats the without-fine-tuning baseline, or its ROUGE F1 gains shrink to noise, the central claim fails.
Extended reading notes
Core claim
The central claim is that legal summarization and similar-case retrieval for Calcutta High Court judgments can be handled by one framework: scrape roughly 130,000 judgments, use an LLM to annotate about 3,100 with structured summaries, fine-tune Pegasus on those headnotes, and then summarize each judgment in two steps, abstractive generation first with Pegasus and extractive selection next with BERTSum, before embedding the summaries for similarity search. The paper reports that fine-tuning Pegasus raises recall, and that Abs-Ext summarization consistently beats Ext-Abs summarization across ROUGE-1, ROUGE-2, and ROUGE-L while balancing precision and recall. On the paper's account, the ordering matters: generating abstractively first preserves the legal context, the extractive stage then foregrounds the sentences that carry it, and that preserved context is what makes the vector database accurate for retrieval. The RAG stage retrieves the top three matching summaries and lets a generation LLM compose answers from the query plus the retrieved context.
Load-bearing premise
The evaluation rests on the roughly 3,100 LLM-written headnote summaries being an accurate gold standard, yet the paper gives no annotation protocol, no inter-annotator agreement, and no explicit guarantee that fine-tuning and test examples were kept separate.
Editorial extensions
If this is right
- A legal researcher could type a query in natural language and receive the three most similar past cases, each with case name, date, citations, acts and sections invoked, outcome, and a summary.
- Fine-tuned abstractive summarization would capture more relevant content from a judgment than off-the-shelf abstractive summarization, at the cost of some precision and F1 score.
- Legal research tool builders should prefer abstract-then-extractive ordering over extract-then-abstract ordering, since the paper's ROUGE results favor Abs-Ext.
- The same framework could be pointed at another court only after a comparable headnote-annotated dataset is built, because fine-tuning on local legal language is what adapts the model to the jurisdiction.
Reading between the lines
- The paper leaves the quality of RAG retrieval unmeasured; a natural follow-up would score retrieval precision and recall against a gold set of known similar-case pairs rather than relying on end-to-end user queries.
- Because ROUGE compares against the LLM-generated headnotes, the reported gains may partly reflect how well the model reproduces that annotation style; a human-expert evaluation would test whether the summaries are actually more useful in practice.
- The two-step ordering claim is tested on one court's corpus, so whether abstract-first is universally better or an artifact of this dataset would need to be re-checked on other jurisdictions and legal languages.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents a framework for summarizing Calcutta High Court judgments and retrieving similar cases. It scrapes roughly 130,000 raw judgment texts from indiankanoon.org, annotates about 3,100 of them using the Mistral LLM, fine-tunes Pegasus on the annotated headnote summaries, applies a two-step (abstractive-then-extractive) summarization pipeline, and builds a vector database to support a RAG system backed by Llama-2. The reported experiments compare extractive, abstractive, Ext-Abs, and Abs-Ext summarization variants with and without fine-tuning using ROUGE-1, ROUGE-2, and ROUGE-L precision, recall, and F1. The central claims are that fine-tuning Pegasus yields significant improvements in legal summarization and that the RAG framework efficiently retrieves similar cases.
Significance. If the claims were substantiated, the contribution would be a useful domain-specific resource: a sizable corpus of Indian court judgments, an LLM-annotated dataset, and an end-to-end summarization and retrieval system for legal research. The paper also lays out a concrete pipeline for combining fine-tuned Pegasus with extractive post-processing and a RAG component, which could be a practical blueprint for similar legal NLP applications. However, the quantitative support for the central summarization claim is undermined by internal contradictions in the reported results and by the use of an unvalidated, possibly overlapping evaluation reference. The retrieval half of the contribution has no evaluation at all. As a result, the significance is currently prospective rather than demonstrated.
major comments (4)
- [Section 5, Tables 3-5] The prose in Section 5 is contradicted by the tables it cites. It states that 'Extractive summarization shows higher F1 scores across all ROUGE metrics compared to abstractive summarization,' but Table 5 reports abstractive summarization without fine-tuning at ROUGE-1 F1 = 0.56 versus extractive ROUGE-1 F1 = 0.53, and abstractive also exceeds extractive on ROUGE-L F1 (0.29 vs 0.28). It also states that 'Abs-Ext summarization consistently outperforms Ext-Abs summarization across all metrics,' but Table 4 shows a tie in ROUGE-1 precision (0.69 for both) and Table 3 shows Abs-Ext below Ext-Abs on ROUGE-L recall (0.30 vs 0.28? Actually Table 3: Ext-Abs with FT ROUGE-L 0.28, Abs-Ext with FT ROUGE-L 0.30, so this particular example is not contradictory, but the precision tie already violates 'across all metrics'). These inconsistencies make the central comparative claims unreliable as stated.
- [Sections 3.1, 4.1, and 5] The evaluation of the summarization models uses as references the 3,100 Mistral-generated headnote summaries described in Section 3.1 and Section 4.1. No annotation protocol, inter-annotator agreement measure, or independent human-verified test set is reported. The paper states that legal experts verified the annotations, but gives no details of how many experts were involved, what instructions they followed, or what fraction of annotations they checked. Furthermore, no train/test split is described for the fine-tuning of Pegasus, so it is unclear whether the texts evaluated in Tables 3-5 were also used to fine-tune the model. If the evaluation documents overlap the training set, the ROUGE scores are inflated; even without overlap, ROUGE against an LLM-produced reference measures fidelity to that annotation model's style, not intrinsic summary quality. The summarization claim is therefore not established.
- [Section 5, Tables 3-5] The abstract and Section 1 claim that fine-tuning Pegasus on case headnote summaries yields 'significant improvements' in summarization, but the F1 results in Table 5 do not consistently support this. For abstractive summarization, fine-tuning lowers ROUGE-1 F1 from 0.56 to 0.50 and leaves ROUGE-L F1 unchanged at 0.29; the improvement appears only for the two-step variants. No significance tests, confidence intervals, or error bars are reported, so the word 'significant' is used without statistical support. The central claim as stated in the abstract is thus not backed by the presented evidence.
- [Section 3.5] The RAG-based similar-case retrieval component, which is half of the paper's central contribution, is never evaluated. Section 3.5 describes the mechanism, and Section 4.2 reports that the top 3 documents are retrieved, but the paper presents no retrieval metrics such as recall@k, mean reciprocal rank, or human relevance judgments, and no qualitative example of a query and its retrieved cases. The claim in the abstract and conclusion that the framework 'efficiently retrieves similar cases' is therefore unsupported by any experimental evidence.
minor comments (5)
- [Section 1, reference [6]] Reference [6], cited for RAG in the introduction, is Liu and Lapata's 'Text summarization with pretrained encoders,' which is not a retrieval-augmented generation paper; the appropriate citation for RAG is [4] (Lewis et al.). Please correct the citation or the claim.
- [Sections 3.3 and 5] The two-step procedure in Section 3.3 is described as abstractive followed by extractive, and Table 5 uses the labels 'Abs-Ext' and 'Ext-Abs,' but the mapping between the prose description and these labels is never made explicit. Please define both acronyms when they are first introduced and indicate which one corresponds to the implemented pipeline.
- [Section 4.1] The dataset is described as approximately 130,000 raw text files, yet only about 3,100 examples are annotated. The paper does not report how the 3,100 were sampled from the 130,000, so the reader cannot assess selection bias or the representativeness of the annotated subset.
- [Section 4.2, Table 2] The table lists 'RAM 32GB' without specifying whether this is GPU memory or system memory; the preceding sentence separately mentions a Tesla V100-PCIE GPU with 32GB, so the table entry is redundant and ambiguous. Also, the correspondence between 'Max-chunk-size for summarization (200 words)' and the earlier statement about sentence boundaries should be clarified.
- [Throughout] The manuscript would benefit from a careful proofread for formatting and typographical issues, including the broken table captions 'T able 1' and 'T able 2,' the malformed email address 'waakcs rs@caluniv.ac.in,' and the repeated footnote markers for the indiankanoon.org URL.
Circularity Check
No demonstrated circularity: the main claim is a standard fine-tuning-plus-ROUGE evaluation; the central weakness is an unreported train/test split and an LLM-generated reference, which is an evidence-quality issue rather than a reduction of the result to its inputs.
full rationale
The claimed derivation chain is: scrape judgments from indiankanoon.org; use Mistral to produce structured annotations including summaries; fine-tune Pegasus on those 'case head note' summaries; apply two-step summarization; and evaluate with ROUGE (Tables 3-5). I can find no equation, definition, or explicit statement in the paper that makes a reported result equal to its input. The ROUGE references are not explicitly identified as the same documents used for fine-tuning, and no train/test split is reported; if the evaluation set overlaps the training set, the reported improvements would be inflated, but the paper itself does not exhibit that reduction, so under the hard rule I cannot call it circular. The concern that the gold standard is an LLM-generated summary, verified only by an opaque 'legal expert' check, is a real external-validity problem, not a by-construction circularity. Reference [16] is a self-citation by the authors, but it appears only in the related-work section and is not used to justify the main summarization or retrieval results. Accordingly, no significant circularity is demonstrated; the score reflects the minor self-citation and the unresolved evaluation-reference risk, not a demonstrated circular step.
Assumptions & free parameters
free parameters (5)
- Learning rate =
1.8e-5
- Number of epochs =
20
- Max chunk size for summarization =
200 words with 2-sentence overlap
- Chunk size for vector database =
1024 tokens with 100-token overlap
- Top-k retrieved documents =
3
assumptions (3)
- ad hoc to paper Mistral-generated structured annotations, claimed to be verified by legal experts, are an acceptable gold standard for fine-tuning and ROUGE evaluation.
- domain assumption The 3,100 annotated judgments are representative of the 130,000-judgment corpus and sufficient for fine-tuning and retrieval.
- domain assumption Embedding similarity between summary chunks is a valid proxy for legal case similarity in the RAG retrieval step.
Cite this review
Pith. "Pith review of A Data Science Approach to Calcutta High Court Judgments: An Efficient LLM and RAG-powered Framework for Summarization and Similar Cases Retrieval." pith.science (2026). https://pith.science/paper/CY5DYA7U
@misc{pith2026250701058,
author = {Pith},
title = {Pith review of: A Data Science Approach to Calcutta High Court Judgments: An Efficient LLM and RAG-powered Framework for Summarization and Similar Cases Retrieval},
year = {2026},
howpublished = {\url{https://pith.science/paper/CY5DYA7U}},
note = {Machine review of arXiv:2507.01058}
}
read the original abstract
The judiciary, as one of democracy's three pillars, is dealing with a rising amount of legal issues, needing careful use of judicial resources. This research presents a complex framework that leverages Data Science methodologies, notably Large Language Models (LLM) and Retrieval-Augmented Generation (RAG) techniques, to improve the efficiency of analyzing Calcutta High Court verdicts. Our framework focuses on two key aspects: first, the creation of a robust summarization mechanism that distills complex legal texts into concise and coherent summaries; and second, the development of an intelligent system for retrieving similar cases, which will assist legal professionals in research and decision making. By fine-tuning the Pegasus model using case head note summaries, we achieve significant improvements in the summarization of legal cases. Our two-step summarizing technique preserves crucial legal contexts, allowing for the production of a comprehensive vector database for RAG. The RAG-powered framework efficiently retrieves similar cases in response to user queries, offering thorough overviews and summaries. This technique not only improves legal research efficiency, but it also helps legal professionals and students easily acquire and grasp key legal information, benefiting the overall legal scenario.
Reference graph
Works this paper leans on
-
[1]
arXiv preprint arXiv:1602.06023 (2016)
Nallapati, R., Zhou, B., Gulcehre, C., Xiang, B., et al.: Abstractive text summarization using sequence-to-sequence rnns and beyond. arXiv preprint arXiv:1602.06023 (2016)
arXiv 2016
-
[2]
arXiv preprint arXiv:2004.12158 (2020)
Zhong, H., Xiao, C., Tu, C., Zhang, T., Liu, Z., Sun, M.: How does nlp benefit legal system: A summary of legal artificial intelligence. arXiv preprint arXiv:2004.12158 (2020)
arXiv 2020
-
[3]
In: Legal Knowledge and Information Systems, pp
Filtz, E., Navas-Loro, M., Santos, C., Polleres, A., Kirrane, S.: Events matter: Extraction of events from court decisions. In: Legal Knowledge and Information Systems, pp. 33–42. IOS Press, ??? (2020)
work page 2020
-
[4]
Advances in neural information processing systems 33, 9459–9474 (2020)
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., K¨ uttler, H., Lewis, M., Yih, W.-t., Rockt¨ aschel, T.,et al.: Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems 33, 9459–9474 (2020)
2020
-
[5]
In: International Conference on Machine Learning, pp
Zhang, J., Zhao, Y., Saleh, M., Liu, P.: Pegasus: Pre-training with extracted gap- sentences for abstractive summarization. In: International Conference on Machine Learning, pp. 11328–11339 (2020). PMLR
work page 2020
-
[6]
arXiv preprint arXiv:1908.08345 (2019)
Liu, Y., Lapata, M.: Text summarization with pretrained encoders. arXiv preprint arXiv:1908.08345 (2019)
arXiv 2019
-
[7]
Named Entity Recognition in Indian court judgments
Kalamkar, P., Agarwal, A., Tiwari, A., Gupta, S., Karn, S., Raghavan, V.: Named entity recognition in indian court judgments. arXiv preprint arXiv:2211.03442 (2022)
work page Pith review arXiv 2022
-
[8]
Processing Long Legal Documents with Pre-trained Transformers: Modding LegalBERT and Longformer
Mamakas, D., Tsotsi, P., Androutsopoulos, I., Chalkidis, I.: Processing long legal documents with pre-trained transformers: Modding legalbert and longformer. arXiv preprint arXiv:2211.00974 (2022)
work page Pith review arXiv 2022
Show all 16 references
-
[9]
arXiv preprint arXiv:2004.05150 (2020)
Beltagy, I., Peters, M.E., Cohan, A.: Longformer: The long-document trans- former. arXiv preprint arXiv:2004.05150 (2020)
2020 arXiv
-
[10]
arXiv preprint arXiv:2010.02559 (2020) 11
Chalkidis, I., Fergadiotis, M., Malakasiotis, P., Aletras, N., Androutsopou- los, I.: Legal-bert: The muppets straight out of law school. arXiv preprint arXiv:2010.02559 (2020) 11
2020 arXiv
-
[11]
Information Processing & Management 59(1), 102780 (2022)
Lyu, Y., Wang, Z., Ren, Z., Ren, P., Chen, Z., Liu, X., Li, Y., Li, H., Song, H.: Improving legal judgment prediction through reinforced criminal element extraction. Information Processing & Management 59(1), 102780 (2022)
2022
-
[12]
arXiv preprint arXiv:2201.13125 (2022)
Kalamkar, P., Tiwari, A., Agarwal, A., Karn, S., Gupta, S., Raghavan, V., Modi, A.: Corpus for automatic structuring of legal documents. arXiv preprint arXiv:2201.13125 (2022)
2022 arXiv
-
[13]
Journal of King Saud University-Computer and Information Sciences 34(5), 2141–2150 (2022)
Anand, D., Wagh, R.: Effective deep learning approaches for summarization of legal texts. Journal of King Saud University-Computer and Information Sciences 34(5), 2141–2150 (2022)
2022
-
[14]
In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (volume 1: Long Papers), pp
Feng, Y., Li, C., Ng, V.: Legal judgment prediction via event extraction with constraints. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (volume 1: Long Papers), pp. 648–664 (2022)
2022
-
[15]
In: Proceedings of the Nineteenth International Conference on Artificial Intelligence and Law, pp
Paul, S., Mandal, A., Goyal, P., Ghosh, S.: Pre-trained language models for the legal domain: a case study on indian law. In: Proceedings of the Nineteenth International Conference on Artificial Intelligence and Law, pp. 187–196 (2023)
2023
-
[16]
In: National Conference on CONTROL INSTRUMENTATION SYSTEM CONFERENCE, pp
Mazumdar, A., Banerjee, P., Ansar, W., Goswami, S., Chakrabarti, A.: An effi- cient graph-based summarization approach for judicial case type prediction using bert. In: National Conference on CONTROL INSTRUMENTATION SYSTEM CONFERENCE, pp. 677–687 (2023). Springer 12
2023
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.