Pith. sign in

REVIEW 4 major objections 7 minor 14 references

Algorithm for Automatic Legislative Text Consolidation

T0 review · 4 major / 7 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A 13-billion-parameter open language model, fine-tuned with low-rank adapters, automatically applies French legislative amendments with a 63.2% correctness rate on a live finance bill, rivaling GPT-4.

desk verdict New dataset and live-bill test for generative legislative consolidation, but the '63% success' headline is conditional on processing only half the bill. read the letter →

arxiv 2501.16794 v1 pith:4CUEIIA2 submitted 2025-01-28 cs.CL

Matias Etcheverry , Thibaud Real , Pauline Chavallard This is my paper · ORCID
classification cs.CL
keywords legislativetextconsolidationLoRAQLolegalNLPFrenchfinancebillamendmentapplicationopenlanguagemodelsworderrorrate
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that automatic legislative text consolidation—turning an existing legal article plus a section of amendments into the updated article—can be done by a generative language model, not only by hand-crafted information extraction. Its method fine-tunes a 13-billion-parameter OpenLLaMa model with low-rank adapters (LoRA) on a publicly released dataset of French legal triplets, and runs the result through a pipeline that splits a bill into amendment sections and identifies the articles they target. On a live 2024 French finance bill, the pipeline processed 1,399 modifications in hours, achieving 63.2% correct consolidations on the 49.8% of cases that fit in the model's context window; GPT-4 achieved 61.4% on 91.3% of cases. The authors claim this is the first application of generative models to legislative consolidation, and the practical payoff would be legal databases updated in hours rather than by months of manual lawyer work.

What carries the argument

The consolidation triplet—(existing article, modification section, modified article)—is the unit everything turns on. The model is prompted with the modification section as an instruction and the existing article as input, and is trained to generate only the modified article. Fine-tuning uses QLoRA: the 13B OpenLLaMa weights are frozen and 4-bit quantized, and low-rank trainable matrices are attached to the query, key, and value projections, about 3% of parameters, so training fits on a 16 or 24 GB GPU. The surrounding pipeline uses a regular-expression section splitter to break bill articles into amendment sections and an entity-recognition model to retrieve the legal articles that each section targets. That machinery turns consolidation into a text generation problem rather than a span extraction problem.

What would settle it

Take a random sample of the 1,399 modifications from the PLF 2024, have two independent legal experts consolidate each one from the original bill and existing code without seeing any model output, then compare their texts to both the paper's reference set and the model outputs. If the experts agree with the reference set and the correctness rates stay near 63%, the claim survives; if the current reference set carries GPT-4's choices, the rates change and the claim is weakened.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central discovery is that a modest-size open generative model can perform the consolidation step that earlier systems only approximated with span extraction. Training the model to predict only the final modified article, rather than the full prompt, lowered average word error from 18.6 to 17.0; cleaning the dataset further lowered it to 12.0; and scaling from 3B to 13B parameters with instruction pre-finetuning brought average word error to 5.09 on the original test set, while on a separate harder PLF test set the 13B model reached 20.7, between GPT-3.5's 44.8 and GPT-4's 9.41. In the live bill evaluation, the 13B model's raw outputs were judged correct in 63.2% of the consolidations it could attempt, while GPT-4 was correct in 61.4% of a much larger, harder set. The paper therefore claims a generative consolidation pipeline that is cheap to run, open-source, and close to proprietary frontier performance on the task.

Load-bearing premise

The live-bill result stands on the assumption that the reference consolidations used to score the models—written by human annotators who started from GPT-4's outputs—are complete and correct; if that reference set absorbs GPT-4's mistakes, the reported 63.2% and 61.4% are not true accuracy.

Editorial extensions

If this is right

  • If the 63.2% live-bill result holds, legislative databases can be updated within hours of a bill's amendments being published, rather than after weeks or months of manual consolidation.
  • The recipe—quantized 13B model plus LoRA on a curated triplet dataset—runs on commodity GPUs, so institutions that cannot send sensitive legal texts to closed APIs can keep consolidation in-house.
  • Because the model trains on the triplet format, the same setup can be retrained on other jurisdictions' amendment structures as long as the modification sections are at least semi-structured.
  • The context-window limit is the main coverage bottleneck: on the live bill, the model could attempt only 49.8% of consolidations, so models with longer contexts or section-level decomposition should raise coverage.
  • Dataset quality matters more than model size at the 3B scale: cleaning out no-op and table cases improved average word error from 17.0 to 12.0, suggesting further curation is a direct lever.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: because the live-bill gold standard was produced by human annotators correcting GPT-4 outputs, the 63.2% versus 61.4% comparison is partly anchored to GPT-4's own text; an independent expert-created reference could shift both numbers, so the headline accuracy should be read as provisional.
  • Editorial extension: the error patterns and prompt-length curves suggest a testable recipe for improving coverage—chunk each modification section into independent smaller prompts, since OpenLLaMa's correctness decays as prompt length grows while GPT-4's does not.
  • Editorial extension: the same instruction-style triplet format could be applied to other procedural text updates, such as regulations, contractual amendments, or codified administrative rules, with the main open question being how far performance transfers when the editing instructions are less formulaic.
  • Editorial extension: legal reliability requires treating the model's output as a draft: even 63% correctness means roughly one in three generated consolidations would need lawyer correction, and hallucinated substitutions are plausible enough to require the human verification step the pipeline already includes.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper proposes a generative approach to legislative text consolidation: a 4-bit quantized OpenLLaMa model fine-tuned with LoRA is trained on (existing article, modification section, modified article) triplets and used to produce consolidated legal texts. A CamemBERT span-extraction baseline is compared against the generative method and against GPT-3.5/GPT-4 on a dedicated test set, and the best model is deployed on the live French Projet de Loi de Finance 2024. The reported headline result is a correctness rate above 63% on that difficult bill, with a full pipeline running in a few hours. The dataset is released on HuggingFace.

Significance. If the results hold, the paper makes a useful applied contribution: it defines the consolidation task for modern generative models, releases a public dataset (a real asset for the community), and demonstrates an end-to-end pipeline on a live bill of high practical importance. The comparison with strong proprietary baselines and the use of a human-verified annotation layer are also valuable. The main contribution is empirical rather than methodological, and the central quantitative claim needs to be restated carefully before the paper can be judged.

major comments (4)
  1. [Abstract; §5.2, Table 6] The abstract's claim of a "success rate of more than 63% on a difficult bill" is conditional: Table 6 reports that OpenLLaMa-13b achieved 63.2% correctness only among the 49.8% of the 1,399 modifications that it could actually process. Over all modifications in the bill, the unconditional success rate is approximately 0.498 × 0.632 ≈ 31.5%. The manuscript's own discussion in §5.2 acknowledges that the model "consolidates far fewer samples" than GPT-4, but the abstract and conclusion do not carry that caveat. Since the central claim is a full automated pipeline, this conditional-versus-unconditional distinction must be corrected in the abstract and conclusion.
  2. [§5.2] The live-bill gold standard is produced by human annotators correcting GPT-4 predictions. This makes the GPT-4 comparison partly self-referential: the reference set reflects GPT-4's output conventions and may inherit its errors, which then affects the evaluation of OpenLLaMa as well. The manuscript should specify the annotation protocol in detail: who performed the corrections, how disagreements were resolved, whether the final reference was independently checked against the enacted law, and whether any inter-annotator agreement was measured. Without this information, the 63.2% and 61.4% correctness rates are hard to interpret.
  3. [§4.1, Table 5] The span-extraction baseline is evaluated only on single modifications, as indicated by the footnote in Table 5, whereas the generative models are evaluated on a test set that includes multiple modifications (§4.3). This is not an apples-to-apples comparison and weakens the paper's claim that generative methods outperform span extraction for this task. The authors should either evaluate both approaches on the same subset (including multi-modification cases) or explicitly state this limitation in the comparison and in the conclusion.
  4. [Tables 1–4] The design comparisons for training objective, dataset cleaning, LoRA rank, and model size report only average and median word error with no confidence intervals, significance tests, or multiple runs. For example, the r=16 versus r=64 difference (12.0 vs 11.7 average) and the 3b versus 7b comparison (10.0 vs 13.5 average but 2.0 vs 0.5 median) are presented as supporting design conclusions, yet the observed differences may be within run-to-run noise. Reporting variance, number of runs, or at least per-sample distributions is needed to support these conclusions.
minor comments (7)
  1. [§4 heading] The heading "Approaches & developped methods" contains a typo; "developped" should be "developed".
  2. [§4.2.1] The subsection title "Fune-tuning & Instruction tuning" contains a typo; it should be "Fine-tuning & Instruction tuning".
  3. [Footnotes 1, 2, 5, 6] Several footnotes say "Link to dataset" or "Link to Projet de Loi Finance" without providing actual URLs. Please add the full links for reproducibility.
  4. [§5.2, Table 6] The definition of a "correct consolidation" is not precise. The text says special characters are removed before comparing raw texts, but it is not stated whether correctness is an exact string match after normalization or a looser semantic criterion; please define this explicitly.
  5. [§2.2] The phrase "often succeed when dealing simple tasks, as consolidation would be" is unsupported and at odds with the paper's later claim that consolidation is a highly complex task on a difficult bill; please reconcile or remove this characterization.
  6. [References] The reference to Mazzei et al. (2009) is incomplete (no venue or publication details); please provide the full citation.
  7. [Figure 3] Figure 3 shows word-error distributions by model and modification type, but there is no legend or caption text describing the colors or how error types were assigned; please add this information.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the consolidation model is trained and evaluated on independent official-text triplets, and the reported caveats are reporting issues, not derivation-from-inputs.

full rationale

The paper's claimed derivation chain is a standard supervised sequence-to-sequence setup: dataset triplets (existing article, modification section, modified article) are taken from official Légifrance links, the OpenLLaMa model is LoRA-fine-tuned to generate the modified article conditioned on the other two fields, and performance is measured on held-out test sets (302 triplets, then the live PLF bill) with word error and exact-match correctness. No parameter is fitted to the evaluation metric, and no prediction is defined in terms of the target by construction; the paper also makes no load-bearing self-citation or imported uniqueness claim. The genuine caveats are not circularity. First, the abstract's 'success rate of more than 63%' refers to the 63.2% correctness among the 49.8% of live-bill modifications the model could process, as Section 5.2 and Table 6 state ('it consolidates far fewer samples'); the end-to-end rate over all 1,399 modifications is not reported in the abstract, but the discrepancy is a reporting and interpretation issue. Second, the live-bill reference set was produced by human scrutiny and amendment of GPT-4 outputs ('we scrutinized and amended the predictions made by GPT-4 to produce a third set, representing human annotations'), which is a benchmark-construction caveat for the GPT-4 comparison, but it does not make the authors' model's own outputs equivalent to its training inputs. The central result therefore remains independently derived from the official consolidation data.

Assumptions & free parameters 10 free parameters · 7 assumptions · 0 invented entities

No new physical or mathematical entities are introduced. The paper's results depend on standard ML training hyperparameters (free parameters) and a set of domain assumptions about the structure and evaluation of legal amendment data. The most fragile assumptions are the GPT-4-derived gold standard and the representativeness of the curated single-modification dataset.

free parameters (10)
  • LoRA rank r = 64 (ablated against 16)
    Chosen as a hyperparameter; higher rank slightly improved average word error (12.0 to 11.7) in Section 4.2.4.
  • LoRA alpha = 128 (2*r)
    Set as twice the rank, following common practice; not ablated.
  • Learning rate = 3e-4
    Fixed across experiments (Section 4.2.1).
  • LoRA dropout = 0.05
    Fixed at 5% (Section 4.2.1).
  • Training epochs = 2
    Fixed training duration (Section 4.2.1).
  • 4-bit quantization = True
    Used for QLoRA (Section 4.2.1).
  • Maximum prompt length = 1024 tokens
    Prompts above this were excluded from training (Section 4.2.1).
  • Dataset curation rule = 1784 triplets after removing no-op and table cases
    Curated dataset used for final models; removing these cases changes reported word error from 17.0 to 12.0 (Section 4.2.3).
  • Micro batch size = 4
    Section 4.2.1.
  • Gradient checkpointing interval = 128 samples
    Section 4.2.1.
assumptions (7)
  • domain assumption Modification sections can be reliably split into sub-sections using regular expressions based on the bill's hierarchical structure.
    Used in Section 5.1.1 of the pipeline; if the regular expression splitter fails, modification sections are misaligned.
  • domain assumption All consolidations are expressible as addition, deletion, or substitution of contiguous spans.
    Section 1 defines these three categories; complex modifications (e.g., moving text, conditional rewrites) may not fit.
  • domain assumption The Légifrance links correctly pair each existing article with the modification section and the officially consolidated article.
    Dataset construction (Section 3) relies on this pairing to create training triplets.
  • domain assumption Keeping only articles modified once yields a representative training distribution.
    Section 3 excludes articles with multiple simultaneous modifications, limiting the model to single-modification cases.
  • domain assumption Human-corrected GPT-4 predictions are a valid gold standard for correctness in the live-bill evaluation.
    Section 5.2 creates the reference set by amending GPT-4 output; if the annotators missed errors, the correctness rates are inflated.
  • domain assumption The entity recognition component identifies target articles with sufficient accuracy (82%) for the pipeline evaluation.
    Section 5.1.2 uses an existing system; errors here could drop modifications from evaluation or misattribute them.
  • domain assumption The word error metric is an appropriate proxy for consolidation correctness.
    Used throughout the experiments; word error may not capture legal meaning preservation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Algorithm for Automatic Legislative Text Consolidation." pith.science (2026). https://pith.science/paper/4CUEIIA2

@misc{pith2026250116794,
  author       = {Pith},
  title        = {Pith review of: Algorithm for Automatic Legislative Text Consolidation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4CUEIIA2}},
  note         = {Machine review of arXiv:2501.16794}
}
read the original abstract

This study introduces a method for automating the consolidation process in a legal context, a time-consuming task traditionally performed by legal professionals. We present a generative approach that processes legislative texts to automatically apply amendments. Our method employs light quantized generative model, fine-tuned with LoRA, to generate accurate and reliable amended texts. To the authors knowledge, this is the first time generative models are used on legislative text consolidation. Our dataset is publicly available on HuggingFace1. Experimental results demonstrate a significant improvement in efficiency, offering faster updates to legal documents. A full automated pipeline of legislative text consolidation can be done in a few hours, with a success rate of more than 63% on a difficult bill.

Figures

Figures reproduced from arXiv: 2501.16794 by the authors.

Figure 1
Figure 1. General structure of the PLF towards producing textual content that is inherently construed as a relationship triplet. Hence, Shi and Lin (2019) undertake a notably question answering adaptation of the BERT model to facilitate gener￾ation across a diverse corpus, achieving a remark￾ably good baseline. In more recent times, genera￾tive models appear to exhibit superior performance. Josifoski et al. (2022) introduce t… view at source ↗
Figure 2
Figure 2. Number of modification sections published [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Word error distributions per model per modification type [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Full consolidation pipeline of our algorithm, we executed the consolidation process on the legislative bill using both GPT-4 and our best model, OpenLLaMa-13, generating two sets of predictions. Subsequently, we scruti￾nized and amended the predictions made by GPT-4 to…
Figure 5
Figure 5. Figure 5: The correctness rates against prompt length [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

14 extracted references · 3 canonical work pages

  1. [4]

    Preprint, arXiv:2110.04366

    Towards a unified view of parameter-efficient transfer learning. Preprint, arXiv:2110.04366. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly

    arXiv

  2. [7]

    GenIE: Generative Information Extraction

    Ge- nie: Generative information extraction. Preprint, arXiv:2112.08340. Xiang Lisa Li and Percy Liang

    work page Pith review arXiv

  3. [8]

    Preprint, arXiv:2101.00190

    Prefix- tuning: Optimizing continuous prompts for gener- ation. Preprint, arXiv:2101.00190. Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang

    arXiv

  4. [9]

    Preprint, arXiv:2103.10385

    Gpt understands, too. Preprint, arXiv:2103.10385. Jonathan Mallinson, Aliaksei Severyn, Eric Malmi, and Guillermo Garrido

    arXiv

  5. [10]

    Felix: Flexible Text Editing Through Tagging and Insertion

    Felix: Flexible text editing through tagging and insertion. Preprint, arXiv:2003.10687. Eric Malmi, Sebastian Krause, Sascha Rothe, Daniil Mirylenka, and Aliaksei Severyn

    work page Pith review arXiv 2003

  6. [11]

    Encode, Tag, Realize: High-Precision Text Editing

    Encode, tag, realize: High-precision text editing. Preprint, arXiv:1909.01187. Alessandro Mazzei, Daniele P. Radicioni, and Raffaella Brighi

    work page Pith review arXiv 1909

  7. [12]

    Preprint, arXiv:1910.10683

    Exploring the limits of transfer learning with a unified text-to-text trans- former. Preprint, arXiv:1910.10683. Peng Shi and Jimmy Lin

    arXiv 1910

  8. [13]

    Preprint, arXiv:1904.05255

    Simple bert models for relation extraction and semantic role labeling. Preprint, arXiv:1904.05255. Jason Wei, Maarten Bosma, Vincent Y . Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, An- drew M. Dai, and Quoc V . Le

    arXiv 1904

Show all 14 references
  1. [14]

    Preprint, arXiv:2109.01652

    Finetuned language models are zero-shot learners. Preprint, arXiv:2109.01652

    arXiv

  2. [2019]

    Preprint, arXiv:1902.00751

    Parameter-efficient transfer learning for nlp. Preprint, arXiv:1902.00751. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    1902 arXiv

  3. [2020]

    Preprint, arXiv:2005.14165

    Language models are few-shot learn- ers. Preprint, arXiv:2005.14165. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Al- bert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun ...

    2005 arXiv

  4. [2021]

    Preprint, arXiv:2106.09685

    Lora: Low-rank adaptation of large language models. Preprint, arXiv:2106.09685. Martin Josifoski, Nicola De Cao, Maxime Peyrard, Fabio Petroni, and Robert West

    arXiv

  5. [2022]

    Preprint, arXiv:2210.11416

    Scaling instruction-finetuned language models. Preprint, arXiv:2210.11416. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer

    arXiv

  6. [2023]

    Preprint, arXiv:2305.14314

    Qlora: Efficient finetuning of quantized llms. Preprint, arXiv:2305.14314. Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg- Kirkpatrick, and Graham Neubig

    arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.