REVIEW 4 major objections 4 minor 26 references
Named Entity Recognition in Historical Italian: The Case of Giacomo Leopardi's Zibaldone
T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Fine-tuned GliNER decisively outperforms LLaMa3.1-8B on a new NER benchmark built from Leopardi's Zibaldone.
desk verdict A useful new Italian historical NER benchmark, but the label provenance question needs a straight answer before the headline comparison is fully trusted. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The dataset generation pipeline is the load-bearing instrument: the digital edition encodes entity references as hyperlinks to standard authority records, and the authors scrape 260 notes (pp. 2700-3000) for evaluation and 688 shorter notes from two other page ranges for training, converting the links into span-level annotations with PER, LOC, and WORK types. The model comparison hinges on the GliNER architecture, a bidirectional transformer that learns shared embeddings for text spans and entity-type names, fine-tuned here for four epochs on the Zibaldone training data, and on the instruction-tuned LLaMa3.1-8B used with two Italian prompts (generative re-annotate-the-text and extractive list-of-entities). Evaluation uses exact and fuzzy span matching with class-level and micro/macro averages.
What would settle it
Have independent annotators re-label a random sample of the held-out Zibaldone notes from scratch on the same three entity classes, compare those gold labels to the deposited silver labels to measure agreement, and recompute all four systems' exact and fuzzy F1 against the new gold set; if the fine-tuned GliNER advantage over LLaMa3.1-8B shrinks to statistical parity, the paper's central claim is not supported.
Extended reading notes
Core claim
On the new Zibaldone evaluation set, the fine-tuned GliNER model dominates all four tested configurations on every metric, with micro-averaged F1 of 68.98% (exact) and 75.64% (fuzzy), while the strongest LLaMa3.1-8B configuration, the extractive prompt, reaches only 32.58% exact F1. In per-class results, the fine-tuned model achieves 89.75% exact precision on persons (92% fuzzy) and 81.25% on locations, but drops to 44.50% exact precision on literary works; the work class is hardest for every model, with non-fine-tuned models falling below 20% exact F1. The authors interpret this as evidence that domain-specific fine-tuning, not scale alone, determines NER reliability on historical literary Italian.
Load-bearing premise
The benchmark assumes the hyperlink-based annotations scraped from the digital edition are accurate, complete, and reliable enough to serve as ground truth, even though the deposited dataset is explicitly labeled 'Silver annotations.'
Editorial extensions
If this is right
- Digital edition projects working with historical Italian can get more reliable entity annotations from a small model fine-tuned on their own corpus than from a large foundation model used out of the box.
- The released Zibaldone dataset gives the research community a reusable, publicly available benchmark for NER and later entity linking on nineteenth-century Italian scholarly prose.
- Because the 'work' class is the weakest point for all systems, progress on recognizing bibliographic and literary references (including metonymic references like 'Il Forcellini') is the clearest lever for improving historical NER.
- Even the best model leaves room for error, so the authors' recommended deployment pattern is human-in-the-loop annotation, with automatic predictions reviewed by experts.
Reading between the lines
- Beyond the paper, the comparison is asymmetric: the LLM is used zero-shot while GliNER is fine-tuned, so the result shows the value of task-specific tuning more than an inherent ceiling for 8B models; fine-tuning LLaMa on the same training split could narrow or change the ranking.
- Beyond the paper, the benchmark's entity boundaries come from the digital edition's hyperlinks, which reflect editorial choices; entities that editors chose not to link are invisible to training and evaluation, so the reported difficulty may be optimistic relative to fully gold-annotated corpora.
- Beyond the paper, a cheap validation of the ground truth, such as reporting inter-annotator agreement on a re-annotated sample, would strengthen the benchmark's usefulness for the community.
- Beyond the paper, the same scraping-based labeling recipe could be applied to other hyperlinked digital editions, producing multi-author historical benchmarks and tests of whether the fine-tuned model's advantage transfers across authors.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces a new NER benchmark for historical Italian based on Giacomo Leopardi's Zibaldone, with 2,899 references to persons, locations, and literary works extracted from the DigitalZibaldone digital edition. The authors evaluate four systems: LLaMa3.1-8B-Instruct with a generative and an extractive prompt, a zero-shot GliNER model, and a GliNER model fine-tuned on the new training split. Using exact and fuzzy matching, they report micro-averaged F1 of 68.98% exact and 75.64% fuzzy for fine-tuned GliNER, versus at most 32.58% exact F1 for LLaMa3.1, and conclude that fine-tuned domain-specific NER models are more robust than instruction-tuned LLMs on historical humanistic texts. The paper also discusses per-class results, with the 'work' class emerging as the most difficult, and it releases the dataset on Zenodo and notebooks on GitHub.
Significance. If the benchmark and evaluation are trustworthy, the paper fills a clear gap: there is no public NER benchmark for historical Italian, and the comparison between a 90M-parameter fine-tuned model and an 8B instruction-tuned LLM is practically relevant for digital humanities projects. The release of the dataset, code, and reproducible training setup is a concrete contribution. The main conclusions align with prior work on historical NER in other languages, which also finds that task-specific fine-tuned models outperform zero-shot LLMs. However, the significance is conditional on the quality of the evaluation labels: the Zenodo deposit is titled 'Silver annotations', and no independent gold-standard validation or inter-annotator agreement is reported. If the labels are incomplete, the ranking between the fine-tuned model and the zero-shot LLMs could be an artifact of the annotation pipeline rather than a true difference in NER ability.
major comments (4)
- [§3.1, dataset and reference [19]] The evaluation ground truth is derived from hyperlinks present in DigitalZibaldone, and the cited Zenodo deposit [19] is explicitly titled 'Silver annotations for Entity Disambiguation from Digitalzibaldone'. Silver annotations are generally not validated as gold. Because the same link-derived annotation pipeline produced both the training and test labels, the fine-tuned GliNER is optimized to reproduce the linkers' decisions, including any omissions and boundary choices, while the zero-shot LLaMa model is penalized for finding entities that DigitalZibaldone did not link. The paper reports no inter-annotator agreement, no gold-recall check, and no manual verification of a sample. This is load-bearing for the central claim that fine-tuned NER models are more robust than instruction-tuned LLMs. I ask the authors to (a) clarify the provenance and validation status of the labels, (b) report a manual or independent completeness check on a sample of the evaluation set (e.g., count unlinked mentions that a human annotator would label), and (c) if the labels are indeed silver, either upgrade a test subset to gold or soften the benchmark claim accordingly.
- [§4, Tables 4 and 5] The paper states that the fine-tuned GliNER 'significantly outperforms the others across all metrics', but no significance tests, confidence intervals, or error bars are reported. The evaluation appears to be a single run for each model; for LLMs with stochastic decoding, this is problematic. The authors should report the number of runs, the variance across runs, or bootstrap confidence intervals for the F1 scores, at least for the main micro-averaged comparison in Table 4.
- [§3.1, training and evaluation splits] The training set was filtered to notes with total length <=350 tokens and containing at least one annotated reference (pp. 1000-2001 and 3001-4000), whereas the evaluation set consists of 260 notes from pp. 2700-3000 without mention of the same filtering criteria. This distribution shift could affect both the fine-tuned model and the zero-shot models; for example, the fine-tuned model was not trained on longer, sparsely annotated notes, and LLaMa's zero-shot behavior may also differ on them. The paper should state whether the same filtering was applied to the test set, and if not, report test-set statistics and consider an additional evaluation on a filtered subset to make the comparison fair.
- [§3.2.1, LLaMa prompting setup] The LLaMa3.1 evaluation uses only two hand-written prompts (generative and extractive) with no prompt optimization, no validation set, and no exploration of alternative formulations. This is acknowledged in the paper as a limitation, but it is nevertheless important for the conclusion: the reported gap between LLaMa and fine-tuned GliNER may partly reflect suboptimal prompting rather than an inherent limitation of instruction-tuned LLMs for historical NER. The authors should either test additional prompt variants (e.g., few-shot or different instruction phrasings) or explicitly frame the conclusion as specific to the two prompts used.
minor comments (4)
- [Throughout] There are several typos and stylistic inconsistencies: 'mantain' (Section 3.2.1), '5×10 −6' and '1×10 −5' with odd spacing, and the use of commas as decimal separators in Tables 4 and 5, which is acceptable in Italian but should be made consistent with the journal's style.
- [§4, Table 5] The 'Avg.' column appears to be the unweighted arithmetic mean of the per-class scores. This should be explicitly defined as macro-averaging in the text; if a different aggregation is intended, the caption should say so.
- [§3.1, Table 3] The test set contains only 61 LOC annotations, which makes the per-class LOC results in Table 5 very sensitive to a few instances. The authors should note this low count when interpreting the 'place' class results.
- [§5.1] The discussion of 'Il Forcellini' as an indirect reference is instructive, but it would be strengthened by a quantitative analysis of how often such indirect references occur in the test set, since they may be a major source of the WORK-class errors.
Circularity Check
No significant circularity: the benchmark comparison is a standard supervised train/test split with labels derived externally from DigitalZibaldone's editorial hyperlinks, not from the evaluated models.
full rationale
The paper's central claim—that a fine-tuned GliNER model outperforms instruction-tuned LLaMa3.1 on historical Italian NER—rests on a conventional evaluation. The dataset (Section 3.1) is constructed by scraping DigitalZibaldone's hyperlinks, which were created by domain experts and point to Wikidata/VIAF; these annotations constitute the gold standard. The fine-tuned GliNER model is trained on pages 1000-2001 and 3001-4000, while evaluation uses pages 2700-3000, a disjoint range. No model output is used to generate the labels, and no parameter is fit to the test set. The zero-shot LLaMa and GliNER baselines are evaluated against the same external gold standard. The citation of the authors' own Zenodo dataset [19] is transparent and does not substitute for an independent derivation. The title 'Silver annotations' and the lack of inter-annotator agreement are legitimate concerns about label quality and benchmark validity, but they are not circularity: the model predictions do not feed back into the labels, and the ranking is not forced by any definition or fitted quantity. The paper's results are reproducible from the released dataset and code, so the comparison is self-contained. No load-bearing self-citation, no uniqueness theorem imported from the authors, and no ansatz smuggled in via citation are present.
Assumptions & free parameters
free parameters (2)
- Fine-tuning hyperparameters =
LR 5e-6 (NER), 1e-5 (backbone), 4 epochs, batch 4, weight decay 0.01
- Training note filter threshold =
max 350 tokens, at least one annotation
assumptions (3)
- domain assumption DigitalZibaldone annotations are complete and correct.
- domain assumption Training and evaluation page ranges are disjoint.
- domain assumption LLaMa output generation is deterministic enough for single-run comparison.
Cite this review
Pith. "Pith review of Named Entity Recognition in Historical Italian: The Case of Giacomo Leopardi's Zibaldone." pith.science (2026). https://pith.science/paper/VWNT5Z4O
@misc{pith2026250520113,
author = {Pith},
title = {Pith review of: Named Entity Recognition in Historical Italian: The Case of Giacomo Leopardi's Zibaldone},
year = {2026},
howpublished = {\url{https://pith.science/paper/VWNT5Z4O}},
note = {Machine review of arXiv:2505.20113}
}
read the original abstract
The increased digitization of world's textual heritage poses significant challenges for both computer science and literary studies. Overall, there is an urgent need of computational techniques able to adapt to the challenges of historical texts, such as orthographic and spelling variations, fragmentary structure and digitization errors. The rise of large language models (LLMs) has revolutionized natural language processing, suggesting promising applications for Named Entity Recognition (NER) on historical documents. In spite of this, no thorough evaluation has been proposed for Italian texts. This research tries to fill the gap by proposing a new challenging dataset for entity extraction based on a corpus of 19th century scholarly notes, i.e. Giacomo Leopardi's Zibaldone (1898), containing 2,899 references to people, locations and literary works. This dataset was used to carry out reproducible experiments with both domain-specific BERT-based models and state-of-the-art LLMs such as LLaMa3.1. Results show that instruction-tuned models encounter multiple difficulties handling historical humanistic texts, while fine-tuned NER models offer more robust performance even with challenging entity types such as bibliographic references.
Reference graph
Works this paper leans on
-
[19]
Santini, ZibaldonED: Silver annotations for Entity Disambiguation from Digitalzibaldone, 2024
C. Santini, ZibaldonED: Silver annotations for Entity Disambiguation from Digitalzibaldone, 2024. URL: https://zenodo.org/records/14103094. doi:10.5281/zenodo.14103094
-
[1]
P. Gaitanou, I. Andreou, M.-A. Sicilia, E. Garoufallou, Linked data for libraries: Creating a global knowledge space, a systematic literature review, Journal of Information Science 50 (2024) 204–244. URL: https://doi.org/10.1177/01655515221084645. doi:10.1177/01655515221084645, publisher: SAGE Publications Ltd
-
[2]
T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, D. Amodei, ...
-
[3]
H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, G. Lample, Llama: Open and efficient foundation language models, 2023. URL: https://arxiv.org/abs/2302.13971.arXiv:2302.13971
arXiv 2023
-
[4]
C.-E. González-Gallardo, E. Boros, N. Girdhar, A. Hamdi, J. G. Moreno, A. Doucet, Yes but.. Can ChatGPT Identify Entities in Historical Documents?, 2023 ACM/IEEE Joint Conference on Digital Libraries (JCDL) (2023) 184–189. URL: https://ieeexplore.ieee.org/document/10266291/. doi:10.1109/JCDL57899.2023.00034, conference Name: 2023 ACM/IEEE Joint Conference...
-
[5]
C.-E. González-Gallardo, T. T. H. Hanh, A. Hamdi, A. Doucet, Leveraging Open Large Language Models for Historical Named Entity Recognition, 2024. URL: https://univ-rochelle.hal.science/ hal-04662000
work page 2024
-
[6]
F. Aladağ, The Potential of GPT in Ottoman Studies: Computational Analysis of Evliya Celebi’s Travelogue with NLP and Text Mining and Digital Edition with TEI, CULTURE 5 (2023)
work page 2023
-
[7]
S. Spina, Biscari Epistolography. From Archive to the Website., DigItalia 18 (2023) 245–259. URL: https://digitalia.cultura.gov.it/article/view/3010. doi:10.36181/digitalia-00090, number: 2
Show all 26 references
-
[8]
Stoyanova, Working with the Digital Edition of Giacomo Leopardi’s Zibaldone, magazén 4 (2023) 13
S. Stoyanova, Working with the Digital Edition of Giacomo Leopardi’s Zibaldone, magazén 4 (2023) 13
2023
-
[9]
S. M. Stoyanova, Fragmentary narrative and the formation of pre-digital scholarly hypertextuality: G. Leopardi’s Zibaldone and its hypertext rendition, in: Proceedings of the 3rd Narrative and Hypertext Workshop, NHT ’13, Association for Computing Machinery, New York, NY, USA,...
2013
- [10]
- [11]
-
[12]
Ehrmann, A
M. Ehrmann, A. Hamdi, E. L. Pontes, M. Romanello, A. Doucet, Named entity recognition and classification on historical documents: A survey, arXiv preprint arXiv:2109.11406 (2021)
2021 arXiv
-
[13]
Hamdi, E
A. Hamdi, E. Linhares Pontes, E. Boros, T. T. H. Nguyen, G. Hackl, J. G. Moreno, A. Doucet, A Multilingual Dataset for Named Entity Recognition, Entity Linking and Stance Detection in Historical Newspapers, in: Proceedings of the 44th International ACM SIGIR Conference on Rese...
2021
-
[14]
Ehrmann, M
M. Ehrmann, M. Romanello, A. Flückiger, S. Clematide, Extended Overview of CLEF HIPE 2020: Named Entity Processing on Historical Newspapers, CEUR Workshop Proceedings (2020). URL: http://ceur-ws.org/Vol-2696/paper_255.pdf. doi: 10.5167/uzh-200192, conference Name: Working Note...
2020 doi
-
[15]
Romanello, S
M. Romanello, S. Najem-Meyer, A Named Entity-Annotated Corpus of 19th Century Classical Commentaries., Journal of Open Humanities Data 10 (2024)
2024
- [16]
-
[17]
X. Tang, Q. Su, J. Wang, Z. Deng, CHisIEC: An Information Extraction Corpus for Ancient Chinese History, in: N. Calzolari, M.-Y. Kan, V. Hoste, A. Lenci, S. Sakti, N. Xue (Eds.), Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Reso...
2024
-
[18]
Santini, M
C. Santini, M. A. Tan, O. Bruns, T. Tietz, E. Posthumus, H. Sack, Knowledge extraction for art history: the case of vasari’s the lives of the artists (1568), CEUR Workshop Proceedings 3234 (2022)
2022
-
[20]
Zaratiana, N
U. Zaratiana, N. Tomeh, P. Holat, T. Charnois, GLiNER: Generalist Model for Named Entity Recog- nition using Bidirectional Transformer, in: K. Duh, H. Gomez, S. Bethard (Eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computationa...
2024 doi
-
[21]
Devlin, M.-W
J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, BERT: Pre-training of Deep Bidirectional Transform- ers for Language Understanding, in: J. Burstein, C. Doran, T. Solorio (Eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational...
2019 doi
-
[22]
Z. Chen, B. Liu, Continual learning and catastrophic forgetting, in: Lifelong Machine Learning, Springer, 2018, pp. 55–75
2018
-
[23]
González-Gallardo, E
C.-E. González-Gallardo, E. Boros, E. Giamphy, A. Hamdi, J. G. Moreno, A. Doucet, Injecting Temporal-Aware Knowledge in Historical Named Entity Recognition, in: Advances in Information Retrieval: 45th European Conference on Information Retrieval, ECIR 2023, Dublin, Ireland, Ap...
2023 doi
-
[24]
N. Jain, R. Krestel, Who is mona l.? identifying mentions of artworks in historical archives, in: Digital Libraries for Open Knowledge: 23rd International Conference on Theory and Practice of Digital Libraries, TPDL 2019, Oslo, Norway, September 9-12, 2019, Proceedings 23, Spr...
2019
-
[25]
Vrandečić, M
D. Vrandečić, M. Krötzsch, Wikidata: a free collaborative knowledgebase, Communications of the ACM 57 (2014) 78–85
2014
-
[26]
F. M. Suchanek, M. Alam, T. Bonald, L. Chen, P.-H. Paris, J. Soria, Yago 4.5: A large and clean knowledge base with a rich taxonomy, in: Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2024, pp. 131–140
2024
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.