{"id":"91452668-35a4-42ca-b34e-f83b3b0e3efd","arxiv_id":"2501.09768","paper_version":3,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"Fine-tuning a small LLaMA model on a new Arabic legal dataset yields near-par performance with a larger model, but the generalization claim is tested on the same instructions used during training.","lead":"This paper builds a new Arabic dataset of Saudi commercial court judgments and tests whether open-source language models can predict the final judgment text from case facts and reasons. The authors find that a small fine-tuned model performs nearly as well as a larger one, but the task may be extraction rather than prediction.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's central claim assumes that 'Facts + Reasons' does not already contain the final judgment; if the judgment text overlaps with the reasons in Saudi court documents, the benchmark measures extraction and copying, not outcome prediction (Secs. III.A, III.D.1).","rationale":"The reader's weakest assumption matches my read exactly: the task may be extraction rather than prediction. I checked the methodology and found that the dataset construction splits judgments into parts, and the zero-shot prompt explicitly says to extract the final judgment text. No evidence is presented about overlap between the judgment and the input, even though the public repository could have been used to test this directly. As submitted, the central claim that the benchmark evaluates judicial outcome prediction is unsupported. I considered whether this is merely a disagreement with common practice; it is not, because whether the target text appears in the input is an empirical property of the dataset that can be measured. I credit the authors for releasing dataset, code, and models and for using deterministic generation, but public availability does not by itself establish that the target text is absent from the input. The internal contradiction in Section III.B about the number of training epochs and the circular instruction-generalization design in Section V.A reinforce the primary concern without replacing it. If the overlap test comes back clean, the main comparison could stand; if it does not, the paper would need to be reframed as an extraction or instruction-following benchmark rather than LJP. Therefore the reader's REJECT verdict remains appropriate.","tokens_in":10148,"tokens_out":3155,"duration_ms":33739,"concrete_test":"Select 50 test cases from the released dataset. For each, compute token-level overlap between the gold 'Judgment' field and the 'Facts + Reasons' input (e.g., ROUGE-L, BLEU, and containment of the final operative phrase). If median BLEU or ROUGE-L is high (roughly >0.5 or >0.7, respectively), extraction leakage is confirmed. Then rerun LLaMA-3.2-3B-FT on the same 50 cases with only the facts portion (reasons removed) as input. If BERTScore and ROUGE do not drop materially, the model is not relying on reasoning to produce the judgment, and the reported metrics describe a copying benchmark rather than LJP.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing assumption is that the target judgment is not recoverable from the input. Section III.A says judgments were parsed and organized into a structured format, and Section III.D.1 formalizes the task as f(Instruction, Facts + Reasons) -> Judgment, with the example instruction literally asking the model to 'analyze the reasons and extract the final judgment text.' If the reasons portion of Saudi judgments recites or paraphrases the operative outcome (e.g., 'the court therefore rules that...'), then the high BLEU/ROUGE/BERTScore numbers in Table I mostly measure copying and rewriting, not legal prediction. The paper provides no overlap statistics between the gold judgment and the input, no manual audit of test instances, and no ablation removing reasons. This matters because the abstract's central claim ('fine-tuned smaller models achieve comparable performance to larger models') is only interesting if the models actually predict an outcome from facts and legal reasoning rather than retrieve it from the prompt. The same issue also affects the instruction-generalization analysis in Section V.A: if the task is extraction, improved BERT F1 after fine-tuning reflects better copying behavior, not better legal judgment. This concern is decisive; without ruling it out, Table I cannot be read as evidence for legal judgment prediction.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces a new Arabic Legal Judgment Prediction (LJP) dataset constructed from Saudi commercial court judgments, and benchmarks two open-source LLMs (LLaMA-3.2-3B-Instruct and LLaMA-3.1-8B-Instruct) under zero-shot, one-shot, and LoRA fine-tuning settings. The authors evaluate performance using BLEU, ROUGE, BERTScore, and an LLM-based qualitative scoring, and report that fine-tuned smaller models achieve comparable performance to larger models. They also analyze instruction generalization after fine-tuning on 75 Arabic instructions, and release the dataset, code, and models.","tokens_in":10315,"tokens_out":4281,"duration_ms":41672,"significance":"If the benchmark is valid, the paper makes a useful contribution to Arabic legal NLP: it provides a new domain-specific dataset, releases code and models, and offers a practical comparison of parameter-efficient fine-tuning for a low-resource language. The resource-efficiency claim (fine-tuned 3B model close to 8B model) is practically relevant. However, two load-bearing issues undermine the current interpretation: (1) the task may be extraction rather than prediction because the input includes 'Reasons' and the prompt explicitly asks to 'extract the final judgment text', and (2) the instruction-generalization evaluation uses the same 75 instructions that were used to build the fine-tuning data. Both points need to be resolved before the central claims can be accepted.","major_comments":[{"comment":"The task formalization f(Instruction, Facts + Reasons) -> Judgment uses the example instruction 'analyze the reasons and extract the final judgment text.' Because the dataset is created by parsing court judgments into 'Facts + Reasons' and 'Judgment' components, the gold judgment may be largely recoverable from the 'Reasons' portion. The paper provides no overlap statistics (e.g., ROUGE or BERTScore between the input's Reasons and the gold Judgment), no manual audit of test instances, and no ablation that removes the Reasons component. Without such evidence, the high BLEU/ROUGE/BERTScore values in Table I may reflect copying and rewriting rather than legal outcome prediction, which would invalidate the paper's central claim. Please add quantitative overlap analysis, show representative test examples with the corresponding gold judgments, and run an ablation that uses only the Facts (or a summarized version) as input.","section":"Section III.D.1 and Section III.A"},{"comment":"The instruction-generalization analysis evaluates models on the same 75 instructions that were used to construct the fine-tuning data in Section III.A. The reported improvement in BERT F1 after fine-tuning is therefore expected from memorization of the exact instruction prompts and does not demonstrate generalization to new instructions. The paper must evaluate on held-out instructions (e.g., a subset reserved from the 75, or a fresh set of instructions written specifically for evaluation) to support the 'generalization' and 'human-centric' claims.","section":"Section V.A and Section III.A"},{"comment":"The fine-tuning description is internally inconsistent: it states 'Both models were trained for two epochs on a single A100 GPU' and later 'Training was performed for one epoch using the AdamW optimizer.' The number of training epochs must be clarified, since the resource-efficiency and convergence claims depend on it, and reproducibility requires an unambiguous protocol.","section":"Section III.B"},{"comment":"The main quantitative claim of 'comparable performance' between LLaMA-3.2-3B-FT and LLaMA-3.1-8B-FT rests on point estimates (e.g., BERTScore 0.74 vs. 0.76) without confidence intervals or paired significance tests on the 538 test samples. Given the small test set and the close margins, the paper should report bootstrap confidence intervals or a paired test (e.g., Wilcoxon signed-rank on per-sample BLEU, ROUGE, and BERTScore values) to support the comparability claim.","section":"Section IV and Table I"},{"comment":"The qualitative evaluation uses LLaMA-3.1-8B-Instruct as a judge without any validation against human ratings. The eight-dimensional scores are presented as if they are reliable measures of response quality, but the paper does not report agreement with human annotators (e.g., correlation or percentage agreement on a sample). Please provide such validation or explicitly discuss the limitation and justify the use of the LLM judge for Arabic legal text.","section":"Section IV.A"}],"minor_comments":[{"comment":"There is a typo: 'finetning' should be 'fine-tuning'; additionally, 'LLama' appears in Section I and the caption of Figure 1, and the metric name 'BERT' in Table I should be 'BERTScore' to match the text.","section":"Section V.B"},{"comment":"The reported 'p-value of 0' is not a valid p-value; it should be reported as p < 0.001 or with the actual computed value.","section":"Section V.A"},{"comment":"The training setup mentions a single A100 GPU while inference is described using four NVIDIA A16 GPUs; clarify which hardware was used for each stage and why the configuration differs.","section":"Section III.B and Section III.E"},{"comment":"Figure 2 is described as 'a sample data point' but not explained in the text; include a brief description of the fields and the structure of the dataset sample.","section":"Section III.A"},{"comment":"Reference [17] is a self-citation to the authors' prior work on native vs. non-native prompting; ensure the comparison is presented with appropriate context and not overgeneralized.","section":"References"},{"comment":"The paper should include a statement on data licensing, privacy, and ethical use of the scraped court judgments, including any restrictions on redistribution of the dataset.","section":"Dataset release"}],"recommendation":"major_revision","confidential_remarks":"The paper contributes a useful resource (a new Arabic LJP dataset, code, and model releases) and addresses a practically relevant question about resource-efficient fine-tuning. However, the central task-validity concern (extraction vs. prediction) and the circular instruction-generalization evaluation are serious and must be addressed before publication. I believe these issues are potentially fixable with additional analysis and experiments, so I recommend major revision rather than outright rejection. If the overlap analysis reveals that the gold judgment is largely contained in the input, the paper would need to be reframed as a legal text extraction/generation study, and the conclusion about prediction would need to be substantially revised."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper has a genuinely useful artifact: a new Arabic instruction-following LJP dataset from Saudi commercial court judgments, plus released code and models. Benchmarking LLaMA-3.2-3B and 3.1-8B under zero-shot, one-shot, and LoRA fine-tuning is standard but useful for practitioners. The qualitative evaluation with an LLM judge is a reasonable addition.\n\nThe problem is that the central claim doesn't survive contact with Section III.D.1. The zero-shot instruction is 'based on the facts, analyze the reasons and extract the final judgment text.' That language assumes the judgment text is present in the input. The paper never shows that the gold judgment is not recoverable from the Facts+Reasons input. No overlap statistics, no manual audit, no ablation without reasons. If the judgment text is a substring or a near-rewrite of the reasons, then Table I measures copying and extraction, not outcome prediction. That is a load-bearing flaw: the title and abstract promise prediction, and the results can't be read as evidence for it.\n\nThe instruction-generalization analysis in V.A is also circular: the same 75 instructions used to build the fine-tuning data are the test set, so the post-fine-tuning improvement is expected. And there are smaller issues: an internal contradiction about one vs. two training epochs, no error bars or significance tests on the headline comparison, and vague description of how the dataset was sampled.\n\nI would not trust the current conclusions. However, the dataset itself could be valuable for Arabic legal NLP if the task is redefined or the overlap question is properly examined. A serious referee should see it, but the verdict should be reject unless the authors can show the task is genuinely predictive.\n\nRecommendation: send to peer review, with a strong request for overlap analysis and an ablation.","headline":"A useful Arabic legal dataset is buried under a task-design flaw: the prompt asks models to 'extract' the judgment, so the benchmark likely measures copying, not prediction.","tokens_in":10920,"tokens_out":5556,"would_cite":false,"duration_ms":53645,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 3-billion-parameter LLaMA model fine-tuned on a new Arabic court dataset predicts legal judgments within a few points of an 8-billion-parameter model, at a fraction of the resource cost.","keywords":["legal judgment prediction","Arabic NLP","large language models","LoRA fine-tuning","low-resource languages","instruction following","Saudi commercial court","BERTScore"],"falsifier":"Measure the ROUGE-L score between the 'Reasons' and 'Judgment' parts of the released dataset; if the score is high, say above 0.7, the task reduces to extraction rather than prediction.","tokens_in":9864,"feed_emoji":"⚖️","tokens_out":9412,"duration_ms":72408,"temperature":0.7,"pith_summary":"This paper sets out to show that legal judgment prediction (LJP) in Arabic, a low-resource language, is feasible with open-source language models, provided they are fine-tuned efficiently. The authors built a dataset from Saudi commercial court judgments, split each case into 'Facts + Reasons' and final 'Judgment' text, and crafted 75 Arabic instructions to fine-tune LLaMA-3.2-3B and LLaMA-3.1-8B with LoRA. Across BLEU, ROUGE, and BERTScore, the fine-tuned 3B model lands close to the 8B model (BERTScore 0.74 vs 0.76), and the authors argue that this makes smaller models a resource-efficient option. The paper also reports statistically significant gains from instruction diversity and releases the dataset, code, and models.","feed_headline":"Fine-tuned 3B LLaMA rivals 8B on Arabic legal rulings","feed_subtitle":"LoRA fine-tuning on a new Arabic court dataset cuts the gap to under 3% in BERTScore, cutting compute needs.","key_machinery":"The mechanism that carries the argument is LoRA (Low-Rank Adaptation), a parameter-efficient fine-tuning method that updates only small low-rank matrices instead of all model weights, combined with a custom Arabic instruction-following dataset. The task is formalized as $f(\\text{Instruction}, \\text{Facts} + \\text{Reasons}) \\rightarrow \\text{Judgment}$, and one-shot examples are retrieved with a Sentence-BERT multilingual embedding model. LoRA allows the 3B model to be fine-tuned on a single GPU and to reach performance close to the 8B model, while the instruction-diverse training set is credited with improving generalization across the 75 prompt styles.","core_discovery":"The central claim is that LoRA fine-tuning on a modest Arabic instruction-following dataset narrows the gap between a 3B and an 8B open-source LLM to a few points on legal judgment prediction. On the authors' test set, the fine-tuned LLaMA-3.2-3B reaches BERTScore 0.74, BLEU 0.24, ROUGE-1 0.50, and ROUGE-2 0.39, while the fine-tuned LLaMA-3.1-8B reaches 0.76, 0.26, 0.53, and 0.41, respectively; both far exceed their zero-shot baselines (BERTScore 0.54 and 0.58). The paper interprets this as evidence that efficient parameter adaptation, not raw model size, is the main driver of LJP performance in this setting, and that a diverse set of 75 Arabic instructions yields statistically significant improvements in instruction generalization. It also introduces what it calls the first Arabic instruction-following dataset for LJP, constructed from public Saudi commercial court judgments and made publicly available.","pith_inferences":["If the final judgment text is largely recoverable from the 'Reasons' portion of the input, the reported numbers may measure extractive copying rather than genuine outcome prediction; a Facts-only evaluation would settle this.","A stronger test would recast LJP as classification over a fixed set of outcome classes and measure accuracy on cases where the judgment is truly future information, rather than text overlap.","The LLM-based qualitative scores were produced by the same family of models being evaluated, so the 'Legal Language' and 'Clarity' judgments may partly reflect stylistic similarity to the reference; a human-annotated subset would test this.","The resource-efficiency claim could be quantified in terms of training time, GPU memory, and cost per prediction, allowing practitioners to weigh a 2-3% score gap against compute savings."],"forward_implications":["Fine-tuned 3B models become a practical choice for Arabic legal NLP on modest hardware, since the BERTScore gap to the 8B model is only 0.02.","LoRA fine-tuning beats one-shot in-context learning on every reported metric, suggesting parameter-efficient adaptation is the better deployment strategy for legal documents.","Instruction diversity during fine-tuning produces statistically significant gains, indicating that prompt variation improves a model's ability to follow unseen Arabic legal instructions.","The released dataset, code, and models give other researchers a starting point for Arabic legal NLP and for adapting the pipeline to other low-resource languages.","Because the difference between the 3B and 8B fine-tuned models is under 3% on average, model size is not the dominant factor after fine-tuning on a domain dataset."],"supporting_citations":[{"why":"Supplies the base LLaMA-3.2-3B and LLaMA-3.1-8B models that are benchmarked and fine-tuned.","marker":"[8]"},{"why":"Defines LoRA, the parameter-efficient fine-tuning method whose comparison of 3B vs 8B is the paper's central result.","marker":"[44]"},{"why":"Provides BERTScore, the semantic similarity metric used for the headline 0.74 vs 0.76 comparison.","marker":"[48]"},{"why":"Provides the Sentence-BERT multilingual embedding model used for semantic retrieval of the one-shot example.","marker":"[43]"},{"why":"Defines legal judgment prediction and gives the general evaluation context the paper extends to Arabic.","marker":"[12]"},{"why":"The closest prior work on Arabic LJP with LLMs, which this paper builds on with fine-tuning and an instruction dataset.","marker":"[37]"},{"why":"Shows that non-native instruction language can affect Arabic task performance, motivating the instruction-diversity design.","marker":"[17]"}],"fun_headline_variants":["Small Arabic-tuned LLM closes gap to 8B on court rulings","LoRA-tuned 3B model rivals 8B on Arabic legal prediction","New dataset and fine-tuning shrink LLM size gap for Arabic law","Arabic legal judgment: 3B beats bigger model with LoRA tuning","Under 3% BERTScore gap: efficient Arabic legal AI"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the 'Judgment' text is not already contained in the 'Facts + Reasons' input; if it is, the models are being evaluated on extraction and copying rather than on predicting judicial outcomes.","fun_headline_variants_meta":{"raw":{"variants":["Small Arabic-tuned LLM closes gap to 8B on court rulings","LoRA-tuned 3B model rivals 8B on Arabic legal prediction","New dataset and fine-tuning shrink LLM size gap for Arabic law","Arabic legal judgment: 3B beats bigger model with LoRA tuning","Under 3% BERTScore gap: efficient Arabic legal AI"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000165,"raw_usage":{"total_tokens":1287,"prompt_tokens":1022,"completion_tokens":265,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":638,"completion_tokens_details":{"reasoning_tokens":168}},"tokens_in":638,"tokens_out":265,"duration_ms":2971,"temperature":1.0,"reasoning_tokens":168,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:18:17.743979+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the ROUGE-L score between the 'Reasons' and 'Judgment' parts of the released dataset; if the score is high, say above 0.7, the task reduces to extraction rather than prediction.","supporting_citations":[{"cited_title":"The llama 3 herd of models,","cited_arxiv_id":null,"evidence_quote":"Supplies the base LLaMA-3.2-3B and LLaMA-3.1-8B models that are benchmarked and fine-tuned."},{"cited_title":"A comprehensive evaluation of large language models on legal judgment prediction,","cited_arxiv_id":null,"evidence_quote":"Defines legal judgment prediction and gives the general evaluation context the paper extends to Arabic."},{"cited_title":"Prediction of arabic legal rulings using large language models,","cited_arxiv_id":null,"evidence_quote":"The closest prior work on Arabic LJP with LLMs, which this paper builds on with fine-tuning and an instruction dataset."},{"cited_title":"Native vs Non-Native Language Prompting: A Comparative Analysis","cited_arxiv_id":"2409.07054","evidence_quote":"Shows that non-native instruction language can affect Arabic task performance, motivating the instruction-diversity design."}],"review_version":1}