{"id":"e98e5251-e04f-4c6a-8cd0-fa5b82f702aa","arxiv_id":"2505.20113","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A fine-tuned GliNER model reaches 68.98% exact F1 and 75.64% fuzzy F1 on a new Italian historical NER benchmark, outperforming zero-shot LLaMa3.1-8B and zero-shot GliNER.","lead":"This paper builds a new benchmark from Leopardi's Zibaldone, with 2,899 annotated mentions of people, places, and literary works, and tests four language model setups on it. It finds that a small fine-tuned model is more accurate than a much larger general-purpose chatbot-style model, suggesting that domain-specific training still matters for historical text.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The benchmark ranking may be an artifact of training and evaluating on the same link-derived annotations: if DigitalZibaldone's hyperlinks are incomplete, the fine-tuned GliNER learns the linkers' omissions while zero-shot LLaMa is penalized for finding unlinked mentions.","rationale":"The reader's conditional verdict already asks for exactly this kind of validation. I agree with the reader's weakest assumption, with the additional emphasis that the training and test labels come from the same link-extraction process, so even internally consistent silver labels can bias the comparison. This is a correctness risk rather than a novelty or significance issue. Because the paper's headline result is plausible and consistent with prior work, and because the requested check is feasible, I do not recommend changing the conditional verdict; the paper should be accepted only if the gold-recall check comes out clean, or with the conclusion explicitly restricted to the DigitalZibaldone annotation style.","tokens_in":10876,"tokens_out":6849,"duration_ms":75795,"concrete_test":"Select a random sample of 50 notes from the evaluation set (or about 150 mentions). Have two independent expert annotators, blind to DigitalZibaldone links, mark all person, location, and work mentions following the paper's guidelines; measure inter-annotator agreement and compute the recall of the existing DigitalZibaldone-derived annotations against this adjudicated gold. Then recompute micro-F1 for LLaMa3.1 (generative and extractive) and GliNER (zero-shot and fine-tuned) using the corrected labels. If gold recall is above 95% and the model ranking and F1 gaps in Tables 4 and 5 are unchanged, the concern is dismissed; if substantial unlinked mentions exist, the gap favoring the fine-tuned model will shrink and the central claim needs qualification.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (fine-tuned NER models outperform instruction-tuned LLMs on historical Italian) depends on the evaluation labels being complete and unbiased ground truth, not merely consistent linker outputs. Section 3.1 states that references on DigitalZibaldone are 'encoded as links' and 'annotated by domain experts,' but the cited dataset [19] is titled 'Silver annotations for Entity Disambiguation from Digitalzibaldone.' Silver typically means the labels were not independently validated as gold. No inter-annotator agreement or gold-recall check is reported. Because the same hyperlink-derived annotation pipeline generated both training and test labels, the fine-tuned GliNER is optimized to reproduce the linkers' decisions, including any omissions and boundary choices. LLaMa3.1, with no in-domain training, is evaluated against that same possibly incomplete standard; any valid entity it finds that DigitalZibaldone did not link counts as a false positive. This systematically favors the fine-tuned model and could explain part of the large gap in Tables 4 and 5. The paper's own examples of indirect references such as 'Il Forcellini' and abbreviations show where completeness is most at risk. The concern is not that the labels are certainly wrong, but that the central comparison is not trustworthy until label completeness is established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces a new NER benchmark for historical Italian based on Giacomo Leopardi's Zibaldone, with 2,899 references to persons, locations, and literary works extracted from the DigitalZibaldone digital edition. The authors evaluate four systems: LLaMa3.1-8B-Instruct with a generative and an extractive prompt, a zero-shot GliNER model, and a GliNER model fine-tuned on the new training split. Using exact and fuzzy matching, they report micro-averaged F1 of 68.98% exact and 75.64% fuzzy for fine-tuned GliNER, versus at most 32.58% exact F1 for LLaMa3.1, and conclude that fine-tuned domain-specific NER models are more robust than instruction-tuned LLMs on historical humanistic texts. The paper also discusses per-class results, with the 'work' class emerging as the most difficult, and it releases the dataset on Zenodo and notebooks on GitHub.","tokens_in":11127,"tokens_out":2961,"duration_ms":56856,"significance":"If the benchmark and evaluation are trustworthy, the paper fills a clear gap: there is no public NER benchmark for historical Italian, and the comparison between a 90M-parameter fine-tuned model and an 8B instruction-tuned LLM is practically relevant for digital humanities projects. The release of the dataset, code, and reproducible training setup is a concrete contribution. The main conclusions align with prior work on historical NER in other languages, which also finds that task-specific fine-tuned models outperform zero-shot LLMs. However, the significance is conditional on the quality of the evaluation labels: the Zenodo deposit is titled 'Silver annotations', and no independent gold-standard validation or inter-annotator agreement is reported. If the labels are incomplete, the ranking between the fine-tuned model and the zero-shot LLMs could be an artifact of the annotation pipeline rather than a true difference in NER ability.","major_comments":[{"comment":"The evaluation ground truth is derived from hyperlinks present in DigitalZibaldone, and the cited Zenodo deposit [19] is explicitly titled 'Silver annotations for Entity Disambiguation from Digitalzibaldone'. Silver annotations are generally not validated as gold. Because the same link-derived annotation pipeline produced both the training and test labels, the fine-tuned GliNER is optimized to reproduce the linkers' decisions, including any omissions and boundary choices, while the zero-shot LLaMa model is penalized for finding entities that DigitalZibaldone did not link. The paper reports no inter-annotator agreement, no gold-recall check, and no manual verification of a sample. This is load-bearing for the central claim that fine-tuned NER models are more robust than instruction-tuned LLMs. I ask the authors to (a) clarify the provenance and validation status of the labels, (b) report a manual or independent completeness check on a sample of the evaluation set (e.g., count unlinked mentions that a human annotator would label), and (c) if the labels are indeed silver, either upgrade a test subset to gold or soften the benchmark claim accordingly.","section":"§3.1, dataset and reference [19]"},{"comment":"The paper states that the fine-tuned GliNER 'significantly outperforms the others across all metrics', but no significance tests, confidence intervals, or error bars are reported. The evaluation appears to be a single run for each model; for LLMs with stochastic decoding, this is problematic. The authors should report the number of runs, the variance across runs, or bootstrap confidence intervals for the F1 scores, at least for the main micro-averaged comparison in Table 4.","section":"§4, Tables 4 and 5"},{"comment":"The training set was filtered to notes with total length <=350 tokens and containing at least one annotated reference (pp. 1000-2001 and 3001-4000), whereas the evaluation set consists of 260 notes from pp. 2700-3000 without mention of the same filtering criteria. This distribution shift could affect both the fine-tuned model and the zero-shot models; for example, the fine-tuned model was not trained on longer, sparsely annotated notes, and LLaMa's zero-shot behavior may also differ on them. The paper should state whether the same filtering was applied to the test set, and if not, report test-set statistics and consider an additional evaluation on a filtered subset to make the comparison fair.","section":"§3.1, training and evaluation splits"},{"comment":"The LLaMa3.1 evaluation uses only two hand-written prompts (generative and extractive) with no prompt optimization, no validation set, and no exploration of alternative formulations. This is acknowledged in the paper as a limitation, but it is nevertheless important for the conclusion: the reported gap between LLaMa and fine-tuned GliNER may partly reflect suboptimal prompting rather than an inherent limitation of instruction-tuned LLMs for historical NER. The authors should either test additional prompt variants (e.g., few-shot or different instruction phrasings) or explicitly frame the conclusion as specific to the two prompts used.","section":"§3.2.1, LLaMa prompting setup"}],"minor_comments":[{"comment":"There are several typos and stylistic inconsistencies: 'mantain' (Section 3.2.1), '5×10 −6' and '1×10 −5' with odd spacing, and the use of commas as decimal separators in Tables 4 and 5, which is acceptable in Italian but should be made consistent with the journal's style.","section":"Throughout"},{"comment":"The 'Avg.' column appears to be the unweighted arithmetic mean of the per-class scores. This should be explicitly defined as macro-averaging in the text; if a different aggregation is intended, the caption should say so.","section":"§4, Table 5"},{"comment":"The test set contains only 61 LOC annotations, which makes the per-class LOC results in Table 5 very sensitive to a few instances. The authors should note this low count when interpreting the 'place' class results.","section":"§3.1, Table 3"},{"comment":"The discussion of 'Il Forcellini' as an indirect reference is instructive, but it would be strengthened by a quantitative analysis of how often such indirect references occur in the test set, since they may be a major source of the WORK-class errors.","section":"§5.1"}],"recommendation":"major_revision","confidential_remarks":"This is a solid empirical study with a useful new resource, but the central comparison rests on the reliability of the evaluation labels. The mismatch between the in-text claim of 'domain expert' annotation and the Zenodo title 'Silver annotations' should be resolved before publication. The paper would be suitable for the journal if the authors either validate the test labels on a sample or substantially temper the benchmark claim. I do not see this as a reject, because the concern is fixable within the scope of the manuscript."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The real contribution here is the dataset, ZibaldonED, and the first NER benchmark for historical Italian. That fills a genuine gap, and the authors deserve credit for shipping it: the scraping pipeline is described, the data is on Zenodo, the code is on GitHub, and the per-class results give a clear picture of where the hard cases live. The main comparison, fine-tuned GliNER versus zero-shot LLaMa and zero-shot GliNER, is straightforward and the large gap is consistent with what other groups have found for French, German, Chinese, and Ottoman historical texts. So the empirical pattern is not surprising, but it is now documented for Italian.\n\nThe soft spot is label provenance. The paper calls the annotations expert-made and encoded as links, but the Zenodo deposit cited as [19] is titled 'Silver annotations.' Silver normally means the labels were not independently validated as gold. There is no inter-annotator agreement and no gold-recall check reported. Because the same hyperlink-derived pipeline produced both training and test labels, the fine-tuned model is essentially optimized to reproduce the linkers' decisions, omissions included. LLaMa, with no in-domain training, is then penalized for finding valid mentions that DigitalZibaldone simply did not link. That asymmetry could inflate the reported gap. I do not think the labels are certainly wrong, and the domain experts are real, but the authors need to state explicitly what validation, if any, was done, or relabel the resource as silver and temper the claims accordingly.\n\nSecondary issues are minor to moderate: no error bars or significance tests, only two fixed prompts for LLaMa with no prompt exploration, and the training split was filtered to notes with at most 350 tokens and at least one annotation while the test split appears unfiltered, so the two distributions differ. These are fixable in a revision.\n\nWho should read this: Italian digital humanists and anyone building or evaluating NER tools for scholarly editions. The dataset is the lasting value; the model comparison is useful but secondary.\n\nRecommendation: this deserves peer review, not desk rejection. A serious referee should push on label validation and ask for uncertainty measures, but the benchmark itself is worth publishing.","headline":"A useful new Italian historical NER benchmark, but the label provenance question needs a straight answer before the headline comparison is fully trusted.","tokens_in":11648,"tokens_out":1554,"would_cite":true,"duration_ms":17533,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuned GliNER decisively outperforms LLaMa3.1-8B on a new NER benchmark built from Leopardi's Zibaldone.","keywords":["named entity recognition","historical Italian","Zibaldone","LLM evaluation","GliNER","LLaMa","digital humanities","entity linking"],"falsifier":"Have independent annotators re-label a random sample of the held-out Zibaldone notes from scratch on the same three entity classes, compare those gold labels to the deposited silver labels to measure agreement, and recompute all four systems' exact and fuzzy F1 against the new gold set; if the fine-tuned GliNER advantage over LLaMa3.1-8B shrinks to statistical parity, the paper's central claim is not supported.","tokens_in":10679,"feed_emoji":"📜","tokens_out":6505,"duration_ms":63467,"temperature":0.7,"pith_summary":"This paper builds a new benchmark for named entity recognition (NER) on historical Italian by extracting 2,899 person, location, and literary-work references from the hyperlinked digital edition of Giacomo Leopardi's Zibaldone. It evaluates four systems on this data: LLaMa3.1-8B prompted generatively and extractively, a zero-shot GliNER model, and the same GliNER model fine-tuned on the Zibaldone training split. The central claim is that the fine-tuned 90-million-parameter model substantially outperforms the 8-billion-parameter instruction-tuned model, reaching 68.98% exact and 75.64% fuzzy micro-F1 versus at most 32.58% exact F1 for LLaMa3.1. The authors conclude that instruction-tuned large language models still face serious difficulties with historical humanistic texts, while fine-tuned NER models offer more robust performance for digital-editing workflows.","feed_headline":"Small tuned model beats 8B LLM at Leopardi NER","feed_subtitle":"On a new Zibaldone benchmark, fine-tuned GliNER scores 69% exact F1 while LLaMa tops out at 33%.","key_machinery":"The dataset generation pipeline is the load-bearing instrument: the digital edition encodes entity references as hyperlinks to standard authority records, and the authors scrape 260 notes (pp. 2700-3000) for evaluation and 688 shorter notes from two other page ranges for training, converting the links into span-level annotations with PER, LOC, and WORK types. The model comparison hinges on the GliNER architecture, a bidirectional transformer that learns shared embeddings for text spans and entity-type names, fine-tuned here for four epochs on the Zibaldone training data, and on the instruction-tuned LLaMa3.1-8B used with two Italian prompts (generative re-annotate-the-text and extractive list-of-entities). Evaluation uses exact and fuzzy span matching with class-level and micro/macro averages.","core_discovery":"On the new Zibaldone evaluation set, the fine-tuned GliNER model dominates all four tested configurations on every metric, with micro-averaged F1 of 68.98% (exact) and 75.64% (fuzzy), while the strongest LLaMa3.1-8B configuration, the extractive prompt, reaches only 32.58% exact F1. In per-class results, the fine-tuned model achieves 89.75% exact precision on persons (92% fuzzy) and 81.25% on locations, but drops to 44.50% exact precision on literary works; the work class is hardest for every model, with non-fine-tuned models falling below 20% exact F1. The authors interpret this as evidence that domain-specific fine-tuning, not scale alone, determines NER reliability on historical literary Italian.","pith_inferences":["Beyond the paper, the comparison is asymmetric: the LLM is used zero-shot while GliNER is fine-tuned, so the result shows the value of task-specific tuning more than an inherent ceiling for 8B models; fine-tuning LLaMa on the same training split could narrow or change the ranking.","Beyond the paper, the benchmark's entity boundaries come from the digital edition's hyperlinks, which reflect editorial choices; entities that editors chose not to link are invisible to training and evaluation, so the reported difficulty may be optimistic relative to fully gold-annotated corpora.","Beyond the paper, a cheap validation of the ground truth, such as reporting inter-annotator agreement on a re-annotated sample, would strengthen the benchmark's usefulness for the community.","Beyond the paper, the same scraping-based labeling recipe could be applied to other hyperlinked digital editions, producing multi-author historical benchmarks and tests of whether the fine-tuned model's advantage transfers across authors."],"forward_implications":["Digital edition projects working with historical Italian can get more reliable entity annotations from a small model fine-tuned on their own corpus than from a large foundation model used out of the box.","The released Zibaldone dataset gives the research community a reusable, publicly available benchmark for NER and later entity linking on nineteenth-century Italian scholarly prose.","Because the 'work' class is the weakest point for all systems, progress on recognizing bibliographic and literary references (including metonymic references like 'Il Forcellini') is the clearest lever for improving historical NER.","Even the best model leaves room for error, so the authors' recommended deployment pattern is human-in-the-loop annotation, with automatic predictions reviewed by experts."],"supporting_citations":[{"why":"Supplies the source digital edition that the dataset is scraped from.","marker":"[8]"},{"why":"Deposits the ZibaldonED dataset, labeled 'Silver annotations,' which provides the training and evaluation data.","marker":"[19]"},{"why":"Introduces GliNER, the bidirectional-transformer architecture behind the fine-tuned model.","marker":"[20]"},{"why":"Provides the BERT pretraining that the GliNER backbone is built on.","marker":"[21]"},{"why":"Is a prior study of ChatGPT's entity recognition on historical documents that motivates the evaluation and the observed LLM difficulties.","marker":"[4]"},{"why":"Is a prior evaluation of open-source LLMs on historical NER that similarly finds fine-tuned models ahead.","marker":"[5]"},{"why":"Offers a comparison corpus of nineteenth-century classical commentaries that shares the Zibaldone's abbreviations and scholarly reference habits.","marker":"[15]"},{"why":"Supplies the exact and fuzzy matching evaluation methodology used in the experiments.","marker":"[23]"}],"fun_headline_variants":["Fine-tuned GliNER crushes LLaMa on 19th-century Italian NER","Leopardi's Zibaldone: tuned model beats 8B LLM by 2x F1","Historical Italian NER: fine-tuning trumps scale on new benchmark","New Zibaldone dataset shows LLMs lag on old text NER"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark assumes the hyperlink-based annotations scraped from the digital edition are accurate, complete, and reliable enough to serve as ground truth, even though the deposited dataset is explicitly labeled 'Silver annotations.'","fun_headline_variants_meta":{"raw":{"variants":["Fine-tuned GliNER crushes LLaMa on 19th-century Italian NER","Leopardi's Zibaldone: tuned model beats 8B LLM by 2x F1","Historical Italian NER: fine-tuning trumps scale on new benchmark","New Zibaldone dataset shows LLMs lag on old text NER"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000264,"raw_usage":{"total_tokens":1599,"prompt_tokens":938,"completion_tokens":661,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":554,"completion_tokens_details":{"reasoning_tokens":571}},"tokens_in":554,"tokens_out":661,"duration_ms":7364,"temperature":1.0,"reasoning_tokens":571,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:58:42.436444+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have independent annotators re-label a random sample of the held-out Zibaldone notes from scratch on the same three entity classes, compare those gold labels to the deposited silver labels to measure agreement, and recompute all four systems' exact and fuzzy F1 against the new gold set; if the fine-tuned GliNER advantage over LLaMa3.1-8B shrinks to statistical parity, the paper's central claim is not supported.","supporting_citations":[{"cited_title":"Stoyanova, Working with the Digital Edition of Giacomo Leopardi’s Zibaldone, magazén 4 (2023) 13","cited_arxiv_id":null,"evidence_quote":"Supplies the source digital edition that the dataset is scraped from."},{"cited_title":"Santini, ZibaldonED: Silver annotations for Entity Disambiguation from Digitalzibaldone, 2024","cited_arxiv_id":null,"evidence_quote":"Deposits the ZibaldonED dataset, labeled 'Silver annotations,' which provides the training and evaluation data."},{"cited_title":"Devlin, M.-W","cited_arxiv_id":null,"evidence_quote":"Provides the BERT pretraining that the GliNER backbone is built on."},{"cited_title":"González-Gallardo, E","cited_arxiv_id":null,"evidence_quote":"Is a prior study of ChatGPT's entity recognition on historical documents that motivates the evaluation and the observed LLM difficulties."},{"cited_title":"González-Gallardo, T","cited_arxiv_id":null,"evidence_quote":"Is a prior evaluation of open-source LLMs on historical NER that similarly finds fine-tuned models ahead."},{"cited_title":"Romanello, S","cited_arxiv_id":null,"evidence_quote":"Offers a comparison corpus of nineteenth-century classical commentaries that shares the Zibaldone's abbreviations and scholarly reference habits."}],"review_version":1}