Pith. sign in

REVIEW 5 major objections 5 minor 18 references

ALKAFI-LLAMA3: Fine-Tuning LLMs for Precise Legal Understanding in Palestine

T0 review · 5 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Fine-tuning a 4-bit quantized 1-billion-parameter language model on roughly 244,000 synthetic Arabic question-answer pairs from official Palestinian texts yields a legal assistant that answers citizens' questions with article citations…

desk verdict A genuinely useful new dataset for Palestinian legal QA sitting behind an evaluation that doesn't yet support the accuracy claim. read the letter →

arxiv 2412.14771 v1 pith:YT3JA7H3 submitted 2024-12-19 cs.CL cs.AIcs.LG

Rabee Qasem , Mohannad Hendi , Banan Tantour This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords Palestinianlawlegalquestionansweringlargelanguagemodelfine-tuningsyntheticdatagenerationlow-resourceNLPArabicdomainquantizedmodelsconsumerGPUtraining
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a small language model, compressed to 4-bit precision and carrying only 1 billion parameters, can be taught enough Palestinian law to answer citizens' legal questions accurately, while running on a single mid-range desktop graphics card. The authors built roughly 244,000 Arabic question-answer pairs by prompting two general-purpose chatbots with 1,277 official Palestinian legal texts, then fine-tuned the small model on those pairs over 10 days on local hardware. In their own tests the model answers yes/no, explanatory, and comparative questions correctly, citing the law and article number in each answer; it stumbles on arithmetic questions and fails to format lists. The payoff, if the claim holds, is a locally sustainable legal-assistance tool for a jurisdiction with scarce legal data, scarce funding, and unreliable cloud access. Whether the model's answers are actually correct law is not independently verified, because the training data itself was generated by chatbots and never checked by legal experts.

What carries the argument

The load-bearing mechanism is synthetic instruction fine-tuning: legal article texts are fed to a general-purpose chatbot with a prompt that demands questions phrased by a layperson and answers phrased by a legal advisor, each answer beginning with the article number and law title. The resulting 243,841 question-answer records become the training set for a 4-bit quantized 1-billion-parameter instruction model, adapted with LoRA (low-rank adaptation, which trains a small set of extra weights rather than the full model) at rank 64 for 10 epochs on a single 8-GB GPU. The design deliberately simulates retrieval-augmented generation, so the model learns to answer only from the legal text placed in its context while citing the article, which is what allows a tiny model to behave like a legal reference tool. A second key element is the inclusion of repealed and amended laws alongside current ones, so the model learns the vocabulary and structure of Palestinian legislation, not just current rules.

What would settle it

Take a random sample of, say, 200 records from the released training dataset and check each answer against the cited article's actual text; if a material share of answers misstate the article or cite an article that does not exist, then the model's knowledge base is corrupted at the source and the accuracy claim fails regardless of loss values. A complementary check is to ask the fine-tuned model fresh calculation questions derived from article formulas with new numbers, and verify whether it computes the result or merely recites the formula.

Watch

Extended reading notes

Core claim

The authors claim that fine-tuning a 4-bit quantized Llama-3.2-1B-Instruct model, a 1-billion-parameter instruction-tuned language model, on roughly 244,000 synthetic Arabic question-answer pairs drawn from 1,277 Palestinian basic-law texts yields a legal assistant that answers common citizen questions with accurate article citations, at a level suitable for real deployment in a resource-constrained setting. The fine-tuned model, which they call ALKAFI-LLAMA3, was trained for 10 epochs with low-rank adapters on a single 8-GB consumer GPU, reaching a training loss of 0.33 and an evaluation loss of 0.31. Their design intentionally mimics retrieval-augmented generation: every training answer is anchored to its law title and article number, and the model is expected to answer from the legal text supplied to it, the way a lawyer answers from the relevant article. In the authors' tests it answers yes/no questions, narrative explanations, and comparative questions correctly, even distinguishing single versus group testimony by citing Article 1735 of the Ottoman Code of Civil Law; its known weak spots are arithmetic questions, where it applies the wrong equation, and list questions, where it fails to format enumerated points.

Load-bearing premise

The whole model learns from question-answer pairs generated by a general-purpose chatbot, not verified by legal experts, so if those generated answers contain wrong or outdated law the model inherits that error; the paper's accuracy claim rests entirely on the generator's output being legally correct.

Editorial extensions

If this is right

  • A Palestinian legal-assistance chatbot could operate on an 8-GB consumer GPU with no cloud dependency, giving citizens 24/7 access to first-line legal guidance.
  • The same pipeline, chatbot-generated question-answer pairs from official texts followed by small-model fine-tuning, can be repeated for other low-resource legal systems, since its only inputs are official documents and access to a general chatbot.
  • Because answers are trained to cite their source article, the model can be paired with a retrieval stage to answer from the full corpus of Palestinian laws rather than from memory alone.
  • The model's reported weaknesses, calculation-based entitlements and list formatting, are the concrete failure modes a follow-up dataset should target, and the paper's own example suggests the remedy is more synthetic examples of those types.
  • The released dataset gives other researchers a starting point for Arabic legal natural-language processing, a field the paper notes is nearly absent in the region.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the synthetic generations were never checked against expert-verified law, the model's plausible answers may quietly reproduce any hallucinated or outdated rule from the generator; auditing a random sample of the released dataset against the cited articles would settle this.
  • The calculation failure suggests the model has learned to recall article text without executing the arithmetic inside it; a cheap, automatically verifiable test is to recombine each article's equation with fresh numbers and check whether the model computes or just recites.
  • If the 1-billion-parameter result holds, the paper's real contribution is a general recipe for low-resource legal AI: pick the smallest model that fits the local GPU, generate synthetic question-answer pairs from official documents, and fine-tune adapters, a route other fragmented jurisdictions and rule-heavy domains could copy.
  • The formatting choice that every answer opens with its article number doubles as a built-in audit mechanism: any answer that cannot name a valid article can be flagged automatically, so a deployable version of this system could self-screen its own guesses.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper describes ALKAFI-LLAMA3, a 4-bit quantized Llama-3.2-1B-Instruct model fine-tuned with LoRA on 243,841 synthetic Arabic question-answer pairs generated from Palestinian legal texts by ChatGPT and Gemma. The authors claim that this resource-efficient model provides accurate and contextually relevant legal guidance for Palestinian users. The evaluation reported in the paper consists of training and validation loss curves and five illustrative query categories (yes/no, narrative, list-based, conditional/comparative, and calculation), with the calculation and list-formatting categories showing failures. The dataset and model are released openly on Hugging Face.

Significance. If the accuracy claim were properly validated, the work would be a useful contribution to low-resource legal AI: it demonstrates a feasible fine-tuning pipeline on a consumer GPU, addresses a real access-to-justice gap, and releases a novel Arabic legal QA dataset. The paper's strength is its reproducible, low-cost methodology and the public release of resources. However, the current evidence does not establish the central claim: loss values do not measure legal correctness, the synthetic data are unvalidated, the evaluation is anecdotal, and the model's known failures on calculation and list-formatting queries contradict the unqualified 'accurate' statement in the abstract. The reported results are best interpreted as a proof-of-concept hypothesis, not a demonstrated accuracy result.

major comments (5)
  1. [§7 (Results)] The only quantitative evidence is training and evaluation loss (Figures 3–5), reaching 0.33 and 0.31 respectively. Loss is a language-modeling metric and does not measure legal correctness; a model can achieve low loss while emitting legally wrong answers or incorrect citations. The paper needs an evaluation protocol with expert-validated answers and quantitative accuracy metrics (e.g., per-category accuracy, citation correctness, human preference) before claiming 'accurate and contextually relevant legal guidance.'
  2. [§5.2 and Table 1] All training knowledge originates from 243,841 ChatGPT/Gemma-generated QA pairs. The prompt in Table 1 instructs the generator to answer as a legal advisor, but nothing ensures that the generated answers are legally accurate, complete, or current. There is no expert validation, no random-sample audit, and no filtering of hallucinated or outdated content. Since this unverified corpus is the sole source of legal knowledge, the accuracy claim rests entirely on an unvalidated assumption. The authors should provide a validation study (e.g., a random sample reviewed by legal experts with inter-annotator agreement) or switch to a verified legal QA source.
  3. [§5.1 (Legal Documents)] The corpus deliberately includes repealed laws, stating this is done 'to enlarge the dataset.' However, the generation prompt in Table 1 does not instruct the generator to mark repealed status or to distinguish current from superseded law, and the fine-tuning procedure does not train the model to flag invalidity. A user asking about a topic covered only by a repealed statute could receive a confidently cited but legally invalid answer. The authors should either exclude repealed texts from training or add explicit temporal/repeal markers and evaluate the model's ability to use them.
  4. [§8 (Experiments)] The experimental evaluation consists of five categories with one example each, and two of the five categories (calculation and list-based answers) are reported as failures. This is anecdotal, not a systematic or quantitative evaluation. The abstract's unqualified claim of 'accurate and contextually relevant legal guidance' is already contradicted by the calculation failure in Figure 8. The authors should (i) soften the claim to match the evidence, (ii) construct a larger expert-validated test set with multiple questions per category, (iii) compare against the base Llama model and a RAG baseline, and (iv) report category-wise success rates rather than single illustrative examples.
  5. [§6.2.3 and §8] The model is explicitly fine-tuned to answer directly from legal articles provided during training, and the evaluation questions are answered from those same articles. This setup tests the model's ability to reproduce the training distribution, not its ability to generalize to novel legal reasoning or to handle queries outside the covered articles. An independent held-out benchmark—e.g., expert-written scenarios based on laws not included in training, or a genuinely held-out set of articles—is needed to support the generalization claim in the conclusion.
minor comments (5)
  1. [Abstract] The abstract uses 'accurate and contextually relevant legal guidance' without qualification, but the body reports failures on calculation and list-formatting queries. The wording should be revised to describe demonstrated capabilities and known limitations.
  2. [§8 (Experimental descriptions)] Several sentences describing the experiments are incomplete or garbled, e.g., 'Graph 7 shows the model's response when we asked it about the period in which the model gave the correct answer for the text and mentioned the full article text.' These passages need rewriting for clarity.
  3. [§6.2.2 (Compute)] The local training setup is a strength of the paper, but reporting wall-clock time per epoch or total GPU-hours would make the cost claim more precise than the generic '10 days.'
  4. [References] The Unsloth model is cited informally as '[unsloth Llama-3.2-1B-Instruct-bnb-4bit]' without a standard bibliographic entry; this should be replaced with a proper citation or URL.
  5. [§5.3 (Dataset Analysis)] The description 'Using the split function' is vague; the authors should state the specific tokenization method and library used for the word and token counts.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation; the legal-accuracy claim is under-supported by unverified synthetic labels, but this is an external validity risk rather than a reduction to the paper's own inputs.

full rationale

The paper's pipeline is legal texts -> ChatGPT/Gemma-generated QA pairs -> LoRA fine-tuning -> qualitative evaluation by a co-author legal expert. No step equates an output to an input by construction. The train/validation split in Sections 5.3 and 7 is a standard disjoint split of the synthetic corpus, and the reported evaluation loss is explicitly a language-modeling loss on that split, not an external legal-accuracy benchmark; hence it is not a fitted parameter renamed as a prediction. The only self-citation (Maree et al., 2024) appears in Section 1 as related work and is not load-bearing. The paper's central weakness is that the synthetic labels are never independently validated by legal experts and the abstract's 'accurate' claim rests on loss curves plus five expert-posed questions, including one admitted failure on calculation queries; this is a correctness and evaluation-validity concern, not circularity. No specific equation or construction in the paper reduces the claimed legal understanding to the training inputs, so no circular step is identified.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper is an empirical ML application with no fitted constants or derived equations. The only 'parameters' are training hyperparameters, which are standard choices rather than fitted values. The core assumptions are about data quality and base model capability.

assumptions (3)
  • domain assumption The synthetic QA pairs generated by ChatGPT and Gemma accurately and faithfully represent the content of the Palestinian legal texts.
    This is the foundation of the training data; Section 5.2 describes the generation process without any human verification of the answers' legal correctness.
  • domain assumption The base Llama-3.2-1B-Instruct model has sufficient Arabic language competence to serve as a substrate for legal fine-tuning.
    Invoked in Section 6.1 without direct evaluation of the base model's Arabic capability in the legal domain.
  • domain assumption Including repealed laws and amendments in the training data improves understanding rather than producing contradictory answers.
    Section 5.1 states this inclusion is intentional, but the model is not evaluated on whether it can distinguish repealed from current law.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ALKAFI-LLAMA3: Fine-Tuning LLMs for Precise Legal Understanding in Palestine." pith.science (2026). https://pith.science/paper/YT3JA7H3

@misc{pith2026241214771,
  author       = {Pith},
  title        = {Pith review of: ALKAFI-LLAMA3: Fine-Tuning LLMs for Precise Legal Understanding in Palestine},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YT3JA7H3}},
  note         = {Machine review of arXiv:2412.14771}
}
read the original abstract

Large Language Models (LLMs) have demonstrated remarkable potential in diverse domains, yet their application in the legal sector, particularly in low-resource contexts, remains limited. This study addresses the challenges of adapting LLMs to the Palestinian legal domain, where political instability, fragmented legal frameworks, and limited AI resources hinder effective machine-learning applications. We present a fine-tuned model based on a quantized version of Llama-3.2-1B-Instruct, trained on a synthetic data set derived from Palestinian legal texts. Using smaller-scale models and strategically generated question-answer pairs, we achieve a cost-effective, locally sustainable solution that provides accurate and contextually relevant legal guidance. Our experiments demonstrate promising performance on various query types, ranging from yes/no questions and narrative explanations to complex legal differentiations, while highlighting areas for improvement, such as handling calculation-based inquiries and structured list formatting. This work provides a pathway for the deployment of AI-driven legal assistance tools tailored to the needs of resource-constrained environments.

Figures

Figures reproduced from arXiv: 2412.14771 by the authors.

Figure 1
Figure 1. Boxplot of the token length with 90th Percentile. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Distribution of the token length across the dataset. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Training loss progression plotted against global steps during the training process. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Boxplot illustrating the distribution of training loss across epochs. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Evaluation loss progression at the end of each epoch. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: The model’s response to a yes/no question [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: The model’s response to a Narrative Question [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: The model’s response to a question regarding calculating entitlements from the Palestinian Labor Law. [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 9
Figure 9. Figure 9: The model’s response to a question requiring a list-based answer. [PITH_FULL_IMAGE:figures/full_fig_p010_9.png]
Figure 10
Figure 10. Figure 10: The Model Explains the Difference Between Two Concepts. [PITH_FULL_IMAGE:figures/full_fig_p010_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

18 extracted references · 4 canonical work pages

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774,

    arXiv

  2. [5]

    Internlm-law: An open source chinese legal large language model

    Zhiwei Fei, Songyang Zhang, Xiaoyu Shen, Dawei Zhu, Xiao Wang, Maosong Cao, Fengzhe Zhou, Yining Li, Wenwei Zhang, Dahua Lin, et al. Internlm-law: An open source chinese legal large language model. arXiv preprint arXiv:2406.14887,

    arXiv

  3. [6]

    Rag vs fine-tuning: Pipelines, tradeoffs, and a case study on agriculture

    Aman Gupta, Anup Shirgaonkar, Angels de Luis Balaguer, Bruno Silva, Daniel Holstein, Dawei Li, Jennifer Marsman, Leonardo O Nunes, Mahsa Rouzbahman, Morris Sharp, et al. Rag vs fine-tuning: Pipelines, tradeoffs, and a case study on agriculture. arXiv preprint arXiv:2401.08406,

    arXiv

  4. [7]

    Zero- and Few-shot Named Entity Recognition and Text Expansion in Medication Prescriptions using ChatGPT

    Natthanaphop Isaradech, Andrea Riedel, Wachiranun Sirikul, Markus Kreuzthaler, and Stefan Schulz. Zero-and few-shot named entity recognition and text expansion in medication prescriptions using chatgpt. arXiv preprint arXiv:2409.17683,

    work page Pith review arXiv

  5. [8]

    Fine-tuning and utilization methods of domain-specific llms

    Cheonsu Jeong. Fine-tuning and utilization methods of domain-specific llms. arXiv preprint arXiv:2401.02981,

    arXiv

  6. [9]

    Mistral 7b

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825,

    arXiv

  7. [10]

    Juru: Legal Brazilian Large Language Model from Reputable Sources

    Roseval Malaquias Junior, Ramon Pires, Roseli Romero, and Rodrigo Nogueira. Juru: Legal brazilian large language model from reputable sources. arXiv preprint arXiv:2403.18140,

    work page Pith review arXiv

  8. [12]

    Arafinnlp 2024: The first arabic financial nlp shared task

    Sanad Malaysha, Mo El-Haj, Saad Ezzini, Mohammed Khalilia, Mustafa Jarrar, Sultan Almujaiwel, Ismail Berrada, and Houda Bouamor. Arafinnlp 2024: The first arabic financial nlp shared task. arXiv preprint arXiv:2407.09818,

    arXiv 2024

Show all 18 references
  1. [13]

    Better call gpt, comparing large language models against lawyers

    Lauren Martin, Nick Whitehouse, Stephanie Yiu, Lizzie Catterson, and Rivindu Perera. Better call gpt, comparing large language models against lawyers. arXiv preprint arXiv:2401.16212,

    arXiv

  2. [15]

    Llama: open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: open and efficient foundation language models. arxiv. arXiv preprint arXiv:2302.13971,

    arXiv

  3. [16]

    Understanding warmup-stable-decay learning rates: A river valley loss landscape perspective

    Kaiyue Wen, Zhiyuan Li, Jason Wang, David Hall, Percy Liang, and Tengyu Ma. Understanding warmup-stable-decay learning rates: A river valley loss landscape perspective. arXiv preprint arXiv:2410.05192,

    arXiv

  4. [17]

    Eda corpus: A large language model dataset for enhanced interaction with openroad

    Bing-Yue Wu, Utsav Sharma, Sai Rahul Dhanvi Kankipati, Ajay Yadav, Bintu Kappil George, Sai Ritish Guntupalli, Austin Rovinski, and Vidya A Chhabria. Eda corpus: A large language model dataset for enhanced interaction with openroad. arXiv preprint arXiv:2405.06676,

    arXiv

  5. [18]

    A continued pretrained llm approach for automatic medical note generation

    Dong Yuan, Eti Rastogi, Gautam Naik, Sree Prasanna Rajagopal, Sagar Goyal, Fen Zhao, Bharath Chintagunta, and Jeff Ward. A continued pretrained llm approach for automatic medical note generation. arXiv preprint arXiv:2403.09057,

    arXiv

  6. [2020]

    Faixid: A framework for enhancing ai explainability of intrusion detection results using data cleaning techniques

    11 A PREPRINT - D ECEMBER 20, 2024 Hong Liu, Chen Zhong, Awny Alnusair, and Sheikh Rabiul Islam. Faixid: A framework for enhancing ai explainability of intrusion detection results using data cleaning techniques. Journal of network and systems management, 29(4):40,

    2024

  7. [2021]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783,

    arXiv

  8. [2023]

    Saullm-7b: A pioneering large language model for law

    Pierre Colombo, Telmo Pessoa Pires, Malik Boudiaf, Dominic Culver, Rui Melo, Caio Corro, Andre FT Martins, Fabrizio Esposito, Vera Lúcia Raposo, Sofia Morgado, et al. Saullm-7b: A pioneering large language model for law. arXiv preprint arXiv:2403.03883,

    arXiv

  9. [2024]

    The falcon series of open language models

    Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, et al. The falcon series of open language models. arXiv preprint arXiv:2311.16867,

    arXiv

  10. [2025]

    Topics, authors, and institutions in large language model research: Trends from 17k arxiv papers

    Rajiv Movva, Sidhika Balachandar, Kenny Peng, Gabriel Agostini, Nikhil Garg, and Emma Pierson. Topics, authors, and institutions in large language model research: Trends from 17k arxiv papers. In Proceedings of the 2024 Conference of the North American Chapter of the Associati...

    2024

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.