{"id":"62c2304f-d4bd-4786-adcf-755969010a62","arxiv_id":"2412.08385","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"A new large corpus of Indian court cases and a legal LLaMA model report very high judgment-prediction accuracy, but the evaluation leaks the outcome from the input text.","lead":"This paper introduces a large dataset of Indian court judgments and a legal-tuned language model, reporting about 90% F1 for predicting case outcomes. But the model is given the full judgment text, including the final decision, so the high accuracy may simply reflect reading the answer.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The ~90% F1 claim is not interpretable because labels are extracted from the final 750 words of the same full judgment fed to the model, so the reported numbers mix genuine legal prediction with copying the operative clause; the paper's own Appendix E.1 and Figure 3 confirm this leakage.","rationale":"The reader's weakest_assumption is exactly the load-bearing concern: the evaluation assumes the keyword-derived labels are valid ground truth and that the full judgment text contains no outcome leakage. The paper's own Appendix E.1 and Figure 3 confirm that labels come from the final 750 words of the same document that is fed to the model, so the ~90% F1 number does not measure predictive ability in any legally meaningful sense. This is not a mere disagreement with consensus; it is a correctness risk in the central claim. The independent evidence in the paper (temporal test sets, expert evaluation, and the PredEx/ILDC comparisons) does not rescue the headline claim, because the same leakage applies to the temporal test inputs and the explanation task is evaluated on a different dataset (PredEx) than the prediction claim. The ternary results showing class-2 F1 near 0 further undermine the three-way framing and the reliability of the automatically generated labels. For these reasons the reader's REJECT verdict is appropriate, and the recommended concrete test would settle the concern decisively if the authors were to rerun the experiments with truncated inputs.","tokens_in":30549,"tokens_out":1912,"duration_ms":19087,"concrete_test":"Construct a 'fact-only' version of the NyayaAnumana-SCI binary test split by stripping the final ~750 words (the label window described in Appendix E.1) from each test document, and optionally any sentence containing outcome keywords. Retrain or fine-tune the same InLegalBERT/XLNet models on the same truncated training inputs and evaluate on the truncated test inputs, using the original labels. Compare the resulting macro-F1/accuracy to the ~0.83-0.87 SCI figures in Table 5. If the drop is large (e.g., >10 points) or the model falls to near majority-class performance, the reported 90% numbers are driven by extracting the operative clause rather than by legal prediction. As a second check, manually audit a random sample of 100 labeling decisions from Appendix E.1 against the actual final paragraph of each judgment, and report agreement on 'clear acceptance vs. partial vs.","verdict_should_be":"REJECT","load_bearing_attack":"The central empirical claim (Abstract; Section 8.1; Appendix A: 'around 90% F1') depends on treating the automatically derived labels as ground truth and on the input containing no direct statement of the answer. Both conditions fail. Appendix E.1 states that labels are generated by keyword heuristics applied to the last 750 words of each document, where 'decisions are summarized,' with a context window around 'appeal,' 'petition,' or 'case' and a label-flipping rule for negations. The task setup (Section 3; Figure 3) feeds the entire judgment text as input to the classifier (Section 6.1.1 chunking; Appendix C.1 training on the full train set). Thus a model can reach high F1 by locating the final operative clause (e.g., 'appeal is dismissed' or 'appeal is allowed') rather than by predicting the outcome from case facts. This is not an internal inconsistency per se, but it invalidates the paper's headline interpretation that 'domain-specific models... significantly outperform generic LLMs' and that 'diverse court data significantly boosts model accuracy.' The temporal-split results (Table 4; Section 4.2.3) are genuinely predictive in spirit, but even there the final 750 words of a future judgment normally contain the outcome, so leakage persists. The ternary results compound the problem: Table 3 and Tables 24-26 show class-2 F1 collapsing to 0.00-0.31 as the partial-acceptance distribution shifts, which is inconsistent with the claimed 90% three-way ability and indicates the heuristic labels are not capturing the intended 'partial acceptance' semantics. The paper itself flags related data-quality problems when discussing ILDC (Section 8.1), showing the authors are aware of preprocessing-induced artifacts, yet no similar validation of NyayaAnumana's own labels or leakage is provided.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces NyayaAnumana, a large corpus of Indian legal judgments (about 703k preprocessed cases) with binary and ternary outcome labels, and INLegalLlama, a LLaMA-2-7B model adapted via continued pretraining and supervised finetuning for judgment prediction and explanation. The authors report roughly 90% F1 for binary prediction on several court-level configurations, claim that diverse court data improves accuracy, and provide expert-based evaluation of generated explanations. The dataset and code are released for reproducibility.","tokens_in":30865,"tokens_out":3303,"duration_ms":36299,"significance":"If the reported performance reflected genuine legal judgment prediction from case facts, the dataset would be a substantial resource for Indian legal NLP, and the findings on data diversity and domain adaptation would be useful to the community. The paper also ships a large, publicly released corpus and a reproducible training pipeline, which are real contributions. However, the central quantitative claims are compromised by a label-generation procedure that makes the prediction target recoverable from the input text, and the temporal and ternary results raise additional validity concerns that must be addressed before the main claims can be accepted.","major_comments":[{"comment":"The evaluation setup has a fundamental target-leakage problem. Appendix E.1 states that labels are generated by keyword heuristics applied to the last 750 words of each judgment, where the decision is summarized, with a context window around terms like 'appeal,' 'petition,' or 'case' and a label-flipping rule for negations. The model input, however, is the full judgment text: Section 6.1.1 chunks each document into 512-token windows with overlap, and the training procedure (Appendix C.1) uses the full train set. Figure 3 illustrates that the label is extracted from the same judgment text that is fed to the model. Consequently, a model can achieve high F1 by locating the operative clause (e.g., 'appeal dismissed' or 'appeal allowed') rather than by predicting the outcome from case facts. The reported ~90% F1 in Table 5 and Appendix A therefore does not measure predictive ability in the intended sense; it measures the model's ability to copy or recognize the concluding decision. This invalidates the headline claim that 'incorporating diverse court data significantly boosts model accuracy' in judgment prediction.","section":"§4.2.3 / §6.1.1 / Appendix E.1 / Figure 3"},{"comment":"The temporal test set is not explicitly shown to be disjoint from the training data. Section 4.2.3 says the 'single' dataset is split into train/validation/test in a 70:10:20 ratio and then adds that the authors 'also tested the model performance on temporal data... from January 2020 to April 2024,' but the paper does not state that documents dated after January 2020 were excluded from the training split. Without such a statement, the temporal results in Table 4 and Tables 5, 15-21 could be contaminated by training-set overlap, making the claimed 'future or unseen data' evaluation uninterpretable. Even if the split is temporally exclusive, the final-750-word leakage described above persists, because a future judgment still contains its own concluding decision in its last 750 words. The authors should clarify the exact construction of the temporal split and, more importantly, re-evaluate on a setting where the input excludes the decision-containing final section.","section":"§4.2.3 / Table 4"},{"comment":"The ternary classification results contradict the paper's general claim that adding diverse court data boosts accuracy. In Tables 24-26, the 'partially accepted' class (class 2) has macro F1 of 0.00 for all three models when Tribunals and Daily Orders/District Courts are included, and at most 0.31 for the SCI-only setting. This means the models never (or almost never) predict partial acceptance in the larger configurations, despite the dataset containing roughly 1% partial-acceptance examples in those settings (Table 3). The abstract and Appendix A claim 'approximately 90% F1-score in prediction tasks' and that diversity and volume of training data are crucial for high accuracy; these statements are not supported for the three-way task. The paper should either restrict the headline claim to binary classification or provide an analysis of why the ternary model fails on class 2 and what the limits of the proposed dataset are for partial-outcome prediction.","section":"Tables 24-26 / Section 8.1 / Appendix A"},{"comment":"The expert evaluation is too small and too narrow to support the strength of the claims made. Section 8.2.3 states that the instruction-tuned model 'in some instances even surpasses the quality of explanations provided by human legal experts, achieving an impressive rating score of 4.' However, Table 28 shows that this conclusion rests on 27 (PredEx) and 28 (ILDC_expert) ratings of 4 out of 50 and 54 total cases, respectively, with no rating of 5 on PredEx and only 4 on ILDC_expert. With such small samples, the difference between models (e.g., INLegalLlama versus LLaMa-2 SFT on ILDC_expert: 3.67 versus 3.30) is not shown to be statistically reliable, and the claim that the model surpasses human experts is not justified. The authors should report variance or significance tests, or temper the claim to a qualitative observation.","section":"Section 8.2.3 / Table 8 / Table 28"}],"minor_comments":[{"comment":"The title contains a typo: 'INL EGAL LLAMA' should be 'INLegalLlama.'","section":"Title"},{"comment":"The description 'largest Indian legal corpus for judgment prediction' is potentially misleading because Table 1 lists CAIL as having 2.67 million cases; the qualifiers 'Indian' and 'English-language' should be stated more explicitly in the abstract and introduction.","section":"Table 1"},{"comment":"The sentence 'Suggests that the inclusion of domain-specific data significantly enhances performance' is a fragment; it should be integrated with the preceding sentence.","section":"Section 8.1"},{"comment":"The distinction between the 'single' and 'multi' splits is used throughout the results but defined only implicitly in Appendix E.1; a clearer definition in Section 4.2 would improve readability.","section":"Section 4.2"},{"comment":"The continued pretraining is described as using 'a comprehensive Indian legal corpus' in the abstract, but Section 5.1 (and Section 4.2.1) explains that only a subset (38,321 SCI cases and 100,000 High Court cases) was used; the text should be consistent in describing the scale of the CPT corpus.","section":"Section 5.1"}],"recommendation":"reject","confidential_remarks":"The reader's report and this assessment converge on the central problem: the evaluation is circular because the label is derived from the final 750 words of the same judgment that the model reads. This is not a minor flaw; it invalidates the main empirical claims of the paper. The dataset and code release are useful resources, and a revised paper that re-frames the task as outcome extraction or that re-evaluates on a facts-only setting could be a positive contribution. However, as submitted, the load-bearing results cannot be trusted, and the ternary and expert-evaluation issues further weaken the claims. I recommend rejection, while noting that the authors might be encouraged to resubmit a substantially revised version that addresses the leakage."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"NyayaAnumana is a real resource: 702k Indian judgments across SCI, High Courts, Tribunals, District Courts, and Daily Orders, with preprocessing and label generation described in enough detail to reproduce. That alone makes the paper worth engaging with. The INLegalLlama model is a standard CPT + LoRA SFT recipe, so the modeling novelty is modest, but the corpus is the headline.\n\nThe reader's leakage concern lands. Appendix E.1 says labels are derived from the last 750 words of each judgment, where decisions are summarized, using keyword heuristics and negation flipping. The model gets the full judgment as input, chunked into 512-token windows for classifiers and full text for LLMs. A model can therefore reach high F1 by locating the operative clause, such as \"appeal dismissed\" or \"appeal allowed\", rather than predicting from case facts. The ~90% F1 in the abstract is not interpretable as legal judgment prediction. The temporal split does not fix this: it is not described as excluded from training, and even if it were, the final 750 words of a future judgment still state the outcome. The ternary results are worse: class 2 F1 drops to 0.00-0.31 as the partial-acceptance distribution shifts, which contradicts the three-way claim and suggests the heuristic labels do not capture partial acceptance reliably.\n\nCredit where due: the authors flag preprocessing artifacts in ILDC, describe their own label heuristics openly, and provide data and code. That transparency is good. The flaw is not concealment; it is that the evaluation design answers a different question from the one claimed.\n\nWho is this for? Researchers building Indian legal NLP resources and anyone using LJP benchmarks. The dataset may still be useful for fact-based prediction if re-labeled or if the decision portion is stripped from inputs. As is, the headline result should not be cited as evidence of 90% predictive accuracy. I would send it to a serious referee, with the expectation of major revision: re-run the prediction task on inputs that exclude the final operative section, clarify or truly hold out the temporal split, and either fix or drop the ternary claim.","headline":"Large and genuinely new Indian legal corpus, but the ~90% F1 is inflated by target leakage since labels come from the final 750 words of the same full judgment fed to the model.","tokens_in":31501,"tokens_out":2197,"would_cite":true,"duration_ms":25528,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a corpus of 702,945 Indian judgments spanning all court levels, paired with a legal-domain-tuned LLaMa-2 model, achieves roughly 90% macro F1 on judgment prediction while generating expert-rated explanations.","keywords":["legal judgment prediction","Indian legal corpus","NyayaAnumana","INLegalLlama","domain-specific language model","continued pretraining","explainable AI","court-level diversity"],"falsifier":"Take a random sample of, say, 500 NyayaAnumana test judgments, have legal experts assign labels independently, and also run the trained models on versions of the same judgments with the final 750 words removed. If expert labels disagree with the heuristic labels, or if the redacted-input F1 falls far below the reported ~90%, the headline result is measuring pattern recognition of the decision paragraph rather than prediction from case reasoning.","tokens_in":30364,"feed_emoji":"⚖️","tokens_out":8168,"duration_ms":79079,"temperature":0.7,"pith_summary":"NyayaAnumana is a corpus of 702,945 Indian court judgments spanning the Supreme Court, High Courts, tribunals, district courts, and daily orders, assembled from a raw collection of over 2.2 million documents. The paper's central claim is that this court-level diversity, not model size alone, drives a legal judgment prediction baseline to roughly 90% macro F1, a jump over the prior ~79% results on Indian data. On top of the corpus, the paper builds INLegalLlama, a LLaMa-2-based generative model that is first continued-pretrained on Indian judgments and then supervised-finetuned to produce both a binary or ternary outcome and a rationale. The authors argue that this combination yields predictions and explanations that legal experts rate as accurate and relevant, making the work a step toward AI assistance for a legal system with a large case backlog.","feed_headline":"700,000 Indian judgments push prediction score to 90%","feed_subtitle":"A court-diverse dataset and a legal-tuned model produce explainable predictions for India's case backlog.","key_machinery":"The load-bearing object is the NyayaAnumana corpus itself, produced by cleaning and labeling 7,02,945 judgments collected from a public legal search engine. Labels come from a keyword heuristic applied to the final 750 words of each judgment: words like \"allowed\" and \"granted\" signal acceptance, \"rejected\" and \"dismissed\" signal rejection, and negation near these keywords flips the label; cases with mixed outcomes become the \"partial\" class. The model machinery is INLegalLlama, a LLaMa-2-7B base trained in two phases: continued pretraining on 38,321 Supreme Court and 100,000 High Court judgments, then LoRA-based supervised finetuning on 12,178 PredEx cases with 16 prediction/explanation instruction sets. Ablation tables that train on progressively larger court subsets are what carry the diversity claim, with the full corpus configuration reaching the reported ~90% F1.","core_discovery":"The central discovery is that adding lower-court and daily-order data to Supreme Court and High Court judgments improves macro F1 from about 0.83 to about 0.90 in binary prediction, and that this improvement holds on temporally later test cases from 2020–2024. The authors also find that INLegalLlama's prediction-with-explanation model, trained with 16 instruction templates on expert-annotated PredEx cases, produces explanations whose expert ratings average near 4 on a 1–5 scale, occasionally matching or exceeding the human reference. The paper presents the NyayaAnumana corpus as the resource that makes these gains possible: it is roughly twenty times larger than ILDC and the first Indian LJP corpus to include tribunals, daily orders, and district courts.","pith_inferences":["Editorial inference: the 90% F1 likely overstates substantive outcome prediction, because the model sees the full judgment text including its final decision; rerunning with the final 750 words removed would measure how much of the score is legal reasoning versus verdict-pattern recognition.","Editorial inference: court-level diversity may help partly because each court type has distinctive phrasing and label base rates; a cross-court transfer test (train on High Court only, test on tribunals) would separate vocabulary effects from substantive generalization.","Editorial inference: the same keyword-plus-negation labeling recipe could be ported to other common-law judgment archives to create large LJP corpora, but any such corpus should be validated against human labels before being used to benchmark models.","Editorial inference: explanation quality rests on a small expert sample (50 and 54 cases), so a larger annotation study with agreement statistics would be needed to confirm that the ratings reflect stable model behavior."],"forward_implications":["If the diversity claim is correct, adding more court levels to a legal corpus is a reliable lever for improving LJP accuracy, so future dataset efforts should prioritize coverage over raw size.","Trained classifiers keep macro F1 above 0.85 on temporal test sets from 2020–2024, suggesting the gains are not an artifact of the training era.","INLegalLlama's joint prediction-and-explanation training yields expert Likert ratings that average near 4 out of 5, supporting the use of generative models for explainable legal AI.","Ternary prediction works on Supreme Court cases, but the partial-acceptance class collapses when broader court data is added, so the reliable setting remains binary."],"supporting_citations":[{"why":"Supplies the prior Indian LJP corpus ILDC and the expert-labeled ILDC_expert test set that NyayaAnumana is benchmarked against.","marker":"Malik et al., 2021b"},{"why":"Supplies PredEx, the expert-annotated prediction-and-explanation dataset used for supervised finetuning and for the expert evaluation test cases.","marker":"Nigam et al., 2024b"},{"why":"Supplies the LLaMa-2-7B base model and 2K context architecture on which INLegalLlama is built.","marker":"Touvron et al., 2023"},{"why":"Provides the LoRA low-rank adaptation method used for supervised finetuning under tight compute constraints.","marker":"Hu et al., 2021"},{"why":"Supplies the few-shot prompts used for prediction and explanation inference from the LLM.","marker":"Vats et al., 2023"},{"why":"Provides InCaseLaw, a legal-domain pretrained baseline that the corpus helps outperform.","marker":"Paul et al., 2023"},{"why":"Provides the XLNet-large baseline that the diverse corpus matches or beats.","marker":"Yang et al., 2019"}],"fun_headline_variants":["Court diversity lifts legal AI to 90% F1","700k Indian judgments power explainable legal AI","India's largest legal corpus yields 90% prediction score","INLegalLlama models 700k cases to hit 90% F1"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that labels scraped by keyword heuristics from the final 750 words of each judgment are correct ground truth, and that the full judgment text the model reads does not already contain the outcome.","fun_headline_variants_meta":{"raw":{"variants":["Court diversity lifts legal AI to 90% F1","700k Indian judgments power explainable legal AI","India's largest legal corpus yields 90% prediction score","INLegalLlama models 700k cases to hit 90% F1"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000193,"raw_usage":{"total_tokens":1378,"prompt_tokens":1003,"completion_tokens":375,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":619,"completion_tokens_details":{"reasoning_tokens":305}},"tokens_in":619,"tokens_out":375,"duration_ms":4225,"temperature":1.0,"reasoning_tokens":305,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:51:41.185897+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of, say, 500 NyayaAnumana test judgments, have legal experts assign labels independently, and also run the trained models on versions of the same judgments with the final 750 words removed. If expert labels disagree with the heuristic labels, or if the redacted-input F1 falls far below the reported ~90%, the headline result is measuring pattern recognition of the decision paragraph rather than prediction from case reasoning.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the few-shot prompts used for prediction and explanation inference from the LLM."}],"review_version":1}