Pith. sign in

REVIEW 5 major objections 6 minor 34 references

GerPS-Compare: Comparing NER methods for legal norm analysis

T0 review · 5 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Pith's one-line read XLM-R beats rules and LLMs on legal norm NER

desk verdict A transparent, useful three-way NER benchmark on German legal norms with a plausible ranking, but the LLM baseline is under-tuned and the causal story is speculative; still deserves review. read the letter →

arxiv 2412.02427 v1 pith:NBBKUT7T submitted 2024-12-03 cs.CL cs.AI

classification cs.CLcs.AI
keywords legalNERGermantextspublicadministrationrule-basedsystemdeepdiscriminativemodelslargelanguageheterogeneousclassesspan-basedevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper compares three approaches to named entity recognition on German legal norms governing public administration, using the ten-class GerPS-NER corpus. It finds that deep discriminative models, in particular the XLM-R transformer, clearly outperform both a hand-built rule-based system and a prompted German open-source language model. The authors argue the gap is not an implementation accident but follows from the classes themselves, which mix linguistic categories with administrative ones. A sympathetic reader would take away that for this kind of heterogeneous annotation scheme, fine-tuned discriminative models are the safer baseline, while rules and LLM prompting still help for specific classes.

What carries the argument

The key machinery is the GerPS-NER annotation scheme, whose ten classes deliberately mix linguistically motivated categories such as 'signaling word' and 'condition' with legal-administrative ones such as 'legal grounds for action' and 'data field.' The comparison also rests on a dual evaluation: token-level F1 and span-level Jaccard (intersection-over-union) scores, the latter added because most entities span several tokens. The heterogeneity of the classes is the explanatory mechanism that carries the argument.

What would settle it

Re-run the deep generative approach with few-shot prompts drawn from the GerPS-NER training split, or add dependency-based patterns to the rule-based system; if either then matches or exceeds XLM-R's 0.6455 macro F1, the paper's heterogeneity explanation would be called into question.

Watch

Extended reading notes

Core claim

The paper's central claim is that on GerPS-NER, a corpus of German legal norms with ten classes defined by public administration professionals, deep discriminative models are the best-performing family. XLM-R reaches the highest macro F1-score of 0.6455 and wins in 9 of 10 classes; the rule-based system reaches 0.5082 and the optimistic interpretation of the LLM outputs reaches 0.4488. The authors attribute this result to the semantic and syntactic heterogeneity of the classes, which makes the task unusually hard for both human-written rules and generic LLMs but learnable for discriminative models. Per-class analysis supports the claim: the rule-based system beats XLM-R only on the notoriously difficult data-field class, and span-based Jaccard scores show that token F1 alone can hide poor phrase-boundary detection.

Load-bearing premise

The comparison assumes each implementation faithfully represents its method family, but the LLM was prompted with examples from a smaller, differently annotated dataset and the rule-based system was limited to flat patterns without syntactic parsing, so a better-tuned version of either could shift the ranking.

Editorial extensions

If this is right

  • In 9 of 10 classes, deep discriminative models outperform both the rule-based system and the LLM, with XLM-R as the single best model.
  • The rule-based and LLM approaches perform at similar overall levels, but their strengths are class-specific: the rule-based system wins on the data-field class.
  • The performance gap is driven mainly by class heterogeneity rather than by implementation details of the individual systems.
  • Combining deep discriminative models with rules is a promising route for improving low-scoring classes, and combining rules with LLM prompting could cut development costs.
  • For practical deployment in public administration, a fine-tuned discriminative model is the recommended starting baseline.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The heterogeneity explanation likely generalizes to other domain-specific NER tasks, such as clinical or administrative text, where annotation schemes mix linguistic and domain-specific categories.
  • The LLM's underperformance may be partly an artifact of off-domain prompting; an in-domain few-shot prompt set is an immediate, low-cost experiment the paper leaves open.
  • The dual F1-plus-Jaccard evaluation could become a standard for legal NER, because it distinguishes 'found the entity' from 'found the right span.'
  • A hybrid pipeline that uses the rule-based system to select examples for LLM prompting is testable today and directly addresses the paper's suggested integration.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. GerPS-Compare reports a three-way comparison of NER approaches on GerPS-NER, a German legal-administration corpus with ten classes. The authors evaluate a SpaCy EntityRuler-based rule system, two deep discriminative models (BiLSTM-CRF and XLM-R with FLERT), and LeoLM, a generative LLM, using a prompting scheme from prior work. They report token-level F1 and a span-level Jaccard metric, with macro F1 as the main summary. The main empirical finding is that XLM-R obtains the highest macro F1 (0.6455) and that deep discriminative models generally outperform the rule-based and generative approaches, which perform at similar levels. The paper also interprets this ranking as caused by the semantic and syntactic heterogeneity of the GerPS-NER classes.

Significance. This is a useful, clearly written empirical benchmark for a realistic German legal-administrative NER task. The corpus is close to a real application, the paper reports both token F1 and a span-level Jaccard measure, and the code is promised on Zenodo, which supports reproducibility. The internal consistency between the F1 and Jaccard rankings is a strength: the ordering of approaches is largely monotonic across the two metric families, and the paper is honest about implementation limitations. If the ranking is robust to the implementation concerns raised below, the paper would be a valuable data point for practitioners choosing among rule-based, fine-tuned encoder, and LLM-prompting approaches. The main limitations are that each method family is represented by one particular implementation with acknowledged restrictions, and there is no uncertainty quantification.

major comments (5)
  1. [§4.4, §8] The deep generative implementation is not a representative instance of the LLM method family: the prompts use example annotations from the smaller, differently annotated dataset of Bachinger et al. (2024), and the authors acknowledge in Section 8 that these examples 'may not be representative of the classes in the overall corpus.' Because the headline claim is that deep discriminative models outperform the deep generative approach, this prompt-example mismatch directly weakens the comparison; a tuned prompt set could shift the LeoLM scores. The paper should either re-run the generative approach with prompt examples drawn from the GerPS-NER dev set or explicitly restrict the conclusion to the specific prompt configuration tested.
  2. [§4.4, Table 1] The evaluation of the generative approach is dominated by the choice of consolidation scheme: Table 1 shows macro F1 of 0.4488 under optimistic consolidation versus 0.1067 under pessimistic consolidation. Optimistic consolidation selects, among multiple model predictions, the one that matches the gold standard; this is not a realistic deployment setting and can only overestimate performance. The paper should report the pessimistic result as the primary LLM score, or at least justify why the optimistic oracle is informative for the method comparison.
  3. [§4.3, Table 1] No uncertainty quantification is provided: each model is evaluated once, with no repeated runs, standard deviations, or significance tests. The gaps that drive the ranking—XLM-R 0.6455 vs BiLSTM-CRF 0.6058, rule-based 0.5082 vs LeoLM opt 0.4488—may be within run-to-run variation, particularly for deep models whose training is stochastic. The central ranking claim would be more defensible if the paper reported variance across seeds or at least a significance test (e.g., bootstrap over test sentences).
  4. [§4.2, §6.4, §8] The rule-based system is explicitly limited to flat SpaCy EntityRuler patterns without syntactic information, because of early design decisions (Section 4.2 and Section 8). The paper nevertheless draws a general conclusion that deep discriminative models are 'better equipped' than 'human linguists designing rule-based NER systems' (Section 6.4). A restricted implementation does not license a claim about the rule-based family; a syntax-aware system could plausibly perform better on classes such as Bedingung and Signalwort, as the authors themselves note. I would either soften the family-level conclusion to 'the specific rule-based system implemented here' or add an additional rule-based variant using dependency patterns.
  5. [§6.4] The explanation that class heterogeneity is the 'common cause' of the similar performance of the rule-based and generative approaches is not tested by the experimental design. The compared implementations differ in prompt examples, rule expressivity, and training data, so the similar score levels do not isolate the effect of class definitions. The authors hedge ('arguably'), but the Conclusion repeats the causal claim as a main finding. This should be reframed as a hypothesis, with a proposed test such as comparing the same models on a corpus with more standard, linguistically motivated entity classes.
minor comments (6)
  1. [§5.1] The text says 'The micro F1-scores for the model predictions are presented in Table 1,' but Table 1 reports macro F1-scores; this inconsistency should be corrected.
  2. [Abstract and §5.2] There are several typos, including 'heterogenerity' in the abstract and 'notriously difficult' in Section 5.2; a proofreading pass is needed.
  3. [Figure 2] The legend reads 'F1 ScoreJaccard Score' without a separator; this should be 'F1 Score / Jaccard Score'.
  4. [§4.1] The description of the Jaccard aggregation says 'the arithmetic mean of the Jaccard score values of all file inputs'; the term 'file inputs' is vague and should be defined precisely.
  5. [§3 and References] The code availability is given as 'Anonymous, 2024'; this placeholder should be replaced with the actual author and repository information before publication.
  6. [Table 3] Some class names run into their scores in the table (e.g., 'Recipient of service0.77'), which hurts readability; formatting should be fixed.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is an empirical benchmark; self-cited components (corpus, prompt scheme, architectures) are inputs, not conclusions.

full rationale

The paper makes no formal derivation; its central claim is an observed ranking of macro F1 scores on a held-out test split. The self-citations supply the GerPS-NER corpus (Feddoul et al., 2024), the LLM prompting scheme from Bachinger et al. (2024), and the two discriminative architectures from Erd et al. (2022), but these are experimental inputs, not targets of the comparison. The deep generative prompt is not fitted to GerPS-NER; the paper reports lower performance than the optimistic small-dataset result, and Section 8 explicitly flags that the prompt examples 'may not be representative of the classes in the overall corpus.' The rule-based patterns are developed on the dev split and evaluated on test, which is standard practice. The macro F1 and Jaccard results are externally measurable outputs, not constructed from the compared methods' own definitions. Concerns about prompt representativeness and missing confidence intervals are validity threats, not circular reductions, and no prediction reduces by construction to a fitted parameter or to a self-citation chain.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The central claim is an empirical ranking, so it rests on measurement choices: the macro F1 metric, the hand-crafted rules, the prompt examples, and the hyperparameter selections. The paper does not derive these from theory, and several are acknowledged as limitations.

free parameters (4)
  • LLM prompt examples = Not specified
    Three examples per class from a smaller dataset (Bachinger et al. 2024), not tuned on the GerPS-NER dev set.
  • Rule gazetteers and patterns = N/A
    Hand-crafted by a linguist from gold standard and dev set; content not fully specified in the paper.
  • BiLSTM-CRF hyperparameters = lr=0.05, batch size=16
    Selected by grid search on the dev set; results are from a single run.
  • XLM-R hyperparameters = epochs=30, batch size=1
    Selected by grid search on the dev set; results are from a single run.
assumptions (3)
  • domain assumption GerPS-NER gold annotations are correct and consistent.
    The evaluation treats the corpus labels as ground truth; the paper notes that annotators struggle with some classes (Section 6.2), which could affect scores.
  • domain assumption Macro F1 across ten classes is the appropriate primary metric.
    The paper states all classes are equally important, but this choice affects the ranking of methods.
  • domain assumption The three implementations are representative of their method families.
    See the weakest assumption; limitations are acknowledged in Section 8, so the ranking could change with different implementations.

how reviews work

0 comments
Cite this review

Pith. "Pith review of GerPS-Compare: Comparing NER methods for legal norm analysis." pith.science (2026). https://pith.science/paper/NBBKUT7T

@misc{pith2026241202427,
  author       = {Pith},
  title        = {Pith review of: GerPS-Compare: Comparing NER methods for legal norm analysis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NBBKUT7T}},
  note         = {Machine review of arXiv:2412.02427}
}
read the original abstract

We apply NER to a particular sub-genre of legal texts in German: the genre of legal norms regulating administrative processes in public service administration. The analysis of such texts involves identifying stretches of text that instantiate one of ten classes identified by public service administration professionals. We investigate and compare three methods for performing Named Entity Recognition (NER) to detect these classes: a Rule-based system, deep discriminative models, and a deep generative model. Our results show that Deep Discriminative models outperform both the Rule-based system as well as the Deep Generative model, the latter two roughly performing equally well, outperforming each other in different classes. The main cause for this somewhat surprising result is arguably the fact that the classes used in the analysis are semantically and syntactically heterogeneous, in contrast to the classes used in more standard NER tasks. Deep Discriminative models appear to be better equipped for dealing with this heterogenerity than both generic LLMs and human linguists designing rule-based NER systems.

Figures

Figures reproduced from arXiv: 2412.02427 by the authors.

Figure 1
Figure 1. Overview of our workflow for comparing multiple machine learning approaches [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Evaluation results for the different approaches by class and score type. [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

34 extracted references · 16 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Sherief Abdallah, Khaled Shaalan, and Muhammad Shoaib. 2012. https://doi.org/10.1007/978-3-642-28604-9_26 Integrating Rule-Based System with Classification for Arabic Named Entity Recognition . In David Hutchison, Takeo Kanade, Josef Kittler, Jon M. Kleinberg, Friedemann Mattern, John C. Mitchell, Moni Naor, Oscar Nierstrasz, C. Pandu Rangan, Bernhard Ste...

  4. [4]

    Alan Akbik, Tanja Bergmann, Duncan Blythe, Kashif Rasul, Stefan Schweter, and Roland Vollgraf. 2019. https://doi.org/10.18653/v1/n19-4010 FLAIR: An Easy-to-Use Framework for State-of-the-Art NLP . In NAACL-HLT 2019, pages 54--59. Association for Computational Linguistics

  5. [5]

    Alan Akbik, Duncan Blythe, and Roland Vollgraf. 2018. https://aclanthology.org/C18-1139 Contextual String Embeddings for Sequence Labeling . In COLLING 2018, pages 1638--1649. Association for Computational Linguistics

  6. [6]

    Author Anonymous. 2024. https://doi.org/10.5281/zenodo.13734643 Gerps-compare: Dataset and code

  7. [7]

    Bachinger, Leila Feddoul, Marianne Jana Mauch, and Birgitta K\" o nig-Ries

    Sarah T. Bachinger, Leila Feddoul, Marianne Jana Mauch, and Birgitta K\" o nig-Ries. 2024. https://doi.org/10.1145/3657054.3657277 Extracting legal norm analysis categories from german law texts with large language models . In Proceedings of the 25th Annual International Conference on Digital Government Research, dg.o '24, page 481–493, New York, NY, USA....

  8. [8]

    Dor Bernsohn, Gil Semo, Yaron Vazana, Gila Hayat, Ben Hagag, Joel Niklaus, Rohit Saha, and Kyryl Truskovskyi. 2024. https://aclanthology.org/2024.eacl-long.130 L egal L ens: Leveraging LLM s for legal violation identification in unstructured text . In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguisti...

Show all 34 references
  1. [9]

    Sergei Bogdanov, Alexandre Constantin, Timoth \'e e Bernard, Benoit Crabb \'e , and Etienne Bernard. 2024. https://arxiv.org/abs/2402.15343 NuNER : Entity Recognition Encoder Pre-training via LLM-Annotated Data . Preprint, arXiv:2402.15343

  2. [10]

    Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tom \' a s Mikolov. 2017. https://doi.org/10.1162/tacl\_a\_00051 Enriching Word Vectors with Subword Information . Trans. Assoc. Comput. Linguistics, 5:135--146

  3. [11]

    Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzm \'a n, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. https://doi.org/10.18653/v1/2020.acl-main.747 Unsupervised Cross-lingual Representation Learning ...

  4. [12]

    Yongfu Dai, Duanyu Feng, Jimin Huang, Haochen Jia, Qianqian Xie, Yifang Zhang, Weiguang Han, Wei Tian, and Hao Wang. 2023. https://doi.org/10.48550/arXiv.2310.05620 Laiw: A chinese legal large language models benchmark (a technical report) . arXiv e-prints, pages arXiv--2310

  5. [13]

    Harshil Darji, Jelena Mitrovi \'c , and Michael Granitzer. 2023. https://doi.org/10.5220/0011749400003393 German BERT Model for Legal Named Entity Recognition . In Proceedings of the 15th International Conference on Agents and Artificial Intelligence , pages 723--728

  6. [14]

    Christopher Dozier, Ravikumar Kondadadi, Marc Light, Arun Vachher, Sriharsha Veeramachaneni, and Ramdev Wudali. 2010. https://doi.org/10.1007/978-3-642-12837-0_2 Named Entity Recognition and Resolution in Legal Text . In Enrico Francesconi, Simonetta Montemagni, Wim Peters, an...

  7. [15]

    Robin Erd, Leila Feddoul, Clara Lachenmaier, and Marianne Jana Mauch. 2022. Evaluation of data augmentation for named entity recognition in the german legal domain. In AI4LEGAL/KGSum@ISWC, pages 62--72

  8. [16]

    Leila Feddoul, Sarah T Bachinger, Clara Lachenmaier, Sebastian Apel, Pirmin Karg, Norman Klewer, Denys Forshayt, Robin Erd, and Marianne Mauch. 2024. https://ceur-ws.org/Vol-3697/paper1.pdf Gerps-ner: A dataset for named entity recognition to support public service process cre...

  9. [17]

    Philip John Gorinski, Honghan Wu, Claire Grover, Richard Tobin, Conn Talbot, Heather Whalley, Cathie Sudlow, William Whiteley, and Beatrice Alex. 2019. https://arxiv.org/abs/1903.03985 Named Entity Recognition for Electronic Health Records : A Comparison of Rule-based and Mach...

  10. [18]

    Abhinav Joshi, Shounak Paul, Akshat Sharma, Pawan Goyal, Saptarshi Ghosh, and Ashutosh Modi. 2024. https://aclanthology.org/2024.acl-long.618 IL - TUR : Benchmark for I ndian legal text understanding and reasoning . In Proceedings of the 62nd Annual Meeting of the Association ...

  11. [19]

    Sung Jae Jung, Hajung Kim, and Kyoung Sang Jang. 2024. LLM Based Biological Named Entity Recognition from Scientific Literature . In 2024 IEEE International Conference on Big Data and Smart Computing ( BigComp ) , pages 433--435. IEEE

  12. [20]

    Elena Leitner, Georg Rehm, and Julian Moreno-Schneider . 2019. https://doi.org/10.1007/978-3-030-33220-4_20 Fine- Grained Named Entity Recognition in Legal Documents . In Semantic Systems . The Power of AI and Knowledge Graphs , pages 272--287, Cham. Springer International Publishing

  13. [21]

    Elena Leitner, Georg Rehm, and Julian Moreno-Schneider. 2020. https://aclanthology.org/2020.lrec-1.551 A Dataset of G erman Legal Documents for Named Entity Recognition . In LREC 2020, pages 4478--4485. European Language Resources Association

  14. [22]

    Masoud Monajatipoor, Jiaxin Yang, Joel Stremmel, Melika Emami, Fazlolah Mohaghegh, Mozhdeh Rouhsedaghat, and Kai-Wei Chang. 2024. https://arxiv.org/abs/2404.07376 LLMs in Biomedicine : A study on clinical Named Entity Recognition . Preprint, arXiv:2404.07376

  15. [23]

    Marco Naguib, Xavier Tannier, and Aur \'e lie N \'e v \'e ol. 2024. https://arxiv.org/abs/2402.12801 Few shot clinical entity recognition in three languages: Masked language models outperform LLM prompting . Preprint, arXiv:2402.12801

  16. [24]

    Kalyani Pakhale. 2023. https://arxiv.org/abs/2309.14084 Comprehensive Overview of Named Entity Recognition : Models , Domain-Specific Applications and Challenges . Preprint, arXiv:2309.14084

  17. [25]

    Silvio Peikert, Celia Birle, Jamal Al Qundus, VU Le Duyen Sandra, and Adrian Paschke. 2022. Extracting references from german legal texts using named entity recognition1

  18. [26]

    o rn Pl \

    Bj \"o rn Pl \"u ster. 2023. https://laion.ai/blog/leo-lm/ Leolm: Igniting german-language llm research

  19. [27]

    N Kannaiya Raja, Naol Bakala, and S Suresh. 2019. https://doi.org/10.35940/ijitee.K2047.0981119 NLP : Rule Based Name Entity Recognition . International Journal of Innovative Technology and Exploring Engineering, 8(11):4285--4290

  20. [28]

    Stefan Schweter and Alan Akbik. 2021. https://arxiv.org/abs/2011.06993 Flert: Document-level features for named entity recognition

  21. [29]

    Amir Soleimani, Christof Monz, and Marcel Worring. 2021. https://doi.org/10.18653/v1/2021.eacl-main.106 NLQuAD : A Non-Factoid Long Question Answering Data Set . In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics : Ma...

  22. [30]

    Shuhe Wang, Xiaofei Sun, Xiaoya Li, Rongbin Ouyang, Fei Wu, Tianwei Zhang, Jiwei Li, and Guoyin Wang. 2023. https://doi.org/10.48550/arXiv.2304.10428 GPT-NER : Named Entity Recognition via Large Language Models . Preprint, arXiv:2304.10428

  23. [31]

    Vikas Yadav and Steven Bethard. 2019. https://doi.org/10.48550/arXiv.1910.11470 A Survey on Recent Advances in Named Entity Recognition from Deep Learning models . Preprint, arXiv:1910.11470

  24. [32]

    Junjie Ye, Nuo Xu, Yikun Wang, Jie Zhou, Qi Zhang, Tao Gui, and Xuanjing Huang. 2024. https://arxiv.org/abs/2402.14568 LLM-DA : Data Augmentation via Large Language Models for Few-Shot Named Entity Recognition . Preprint, arXiv:2402.14568

  25. [33]

    Zhen Zhang, Yuhua Zhao, Hang Gao, and Mengting Hu. 2024. https://doi.org/10.1145/3589334.3645414 LinkNER : Linking Local Named Entity Recognition Models to Large Language Models using Uncertainty . In Proceedings of the ACM Web Conference 2024 , pages 4047--4058, Singapore Sin...

  26. [34]

    Jochen Z \" o llner, Konrad Sperfeld, Christoph Wick, and Roger Labahn. 2021. https://doi.org/10.3390/info12110443 Optimizing Small BERT s Trained for German NER . Inf., 12(11):443

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.