{"id":"fb6e4d4c-e8e4-4951-9b2e-7de9b3689372","arxiv_id":"2506.02589","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"On a 1,000-newsletter sample of Russian cultural event announcements, GPT-4.1 achieves F1=0.94 for person name recognition, slightly ahead of GPT-4o (F1=0.93 with JSON prompting).","lead":"This paper compares how well different named entity recognition models, including OpenAI's GPT-4 and GPT-4.1, find person names in Russian news texts about cultural events. The best model, GPT-4.1, reaches an F1 score of 0.94, suggesting that large language models now outperform older specialized tools on this type of text.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central F1=0.94 claim rests on an undocumented GPT-4.1 run, and the same GPT-4o simple-prompt condition receives contradictory scores in Tables 6 and 7; published numbers are not internally verifiable.","rationale":"The reader identified the gold-standard annotation quality as the weakest assumption, which is a genuine concern because no inter-annotator agreement or error analysis is reported. However, the more immediate and load-bearing problem is that the paper's own quantitative results are internally inconsistent. The central claim is about GPT-4.1, but Section 4.1 does not document that model's evaluation at all, and the only overlapping condition across the two main result tables (GPT-4o simple API) has contradictory precision, recall, and F1 values. This is not a matter of missing error bars or arguable annotation choices; it is a direct conflict in the reported measurements. The published Zenodo dataset and the GitHub notebook are positive evidence of reproducibility intent, and crediting those is appropriate. Yet the inconsistency means that the numbers cannot be trusted until the evaluation is rerun or the raw outputs are released and reconciled. I therefore keep the reader's CONDITIONAL verdict: the central claim may survive after correction, but it is not currently verifiable from the manuscript. If the GPT-4.1 numbers fail to reproduce under independent computation, the verdict should move to REJECT.","tokens_in":11300,"tokens_out":2980,"duration_ms":27329,"concrete_test":"Run the released evaluation notebook (github.com/mary-lev/NER) on the Zenodo-annotated 1,000 records using the exact prompts and API calls for gpt-4o-2024-05-13 simple and json conditions and gpt-4.1-2025-04-14, then recompute precision, recall, and F1. If gpt-4o simple reproduces 0.96/0.86/0.91 (Table 6) rather than 0.85/0.80/0.82 (Table 7), Table 7's other rows—including GPT-4.1—need independent confirmation. If the raw outputs are not in the repository, request them; without them the central claim should be treated as unverified.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's headline result—GPT-4.1 achieves F1=0.94 for both prompt types (§5.3, Table 7, §6.1)—cannot be audited from the manuscript alone. Section 4.1 and Table 5 define the evaluated models and list only gpt-3.5-turbo-0125, gpt-4-turbo-2024-04-09, and gpt-4o-2024-05-13; GPT-4.1-2025-04-14 appears for the first time in the results. No API parameters, prompt text, date of run, or parsing procedure for the GPT-4.1 follow-up is described. More seriously, the one configuration that appears in both summary tables is reported inconsistently: Table 6 gives gpt-4o-2024-05-13 (simple API) P=0.96, R=0.86, F1=0.91, while Table 7 gives the same model/condition P=0.85, R=0.80, F1=0.82—an F1 difference of 0.09. Section 5.5 additionally cites GPT-4 recall as 0.74 while Table 6 reports 0.69. Because all model rankings and the 'paradigm shift' conclusion are derived from these metric tables, the inconsistency makes the central quantitative claim unverifiable as published. The gold-standard annotation quality (the reader's concern) matters, but it is secondary: even a perfect gold standard cannot rescue a results section whose own tables disagree.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents a comparative evaluation of named entity recognition models for extracting PERSON entities from a new Russian cultural-news dataset (SPbLitGuide). It compares several transformer-based models (DeepPavlov, RoBERTa, SpaCy) with GPT-3.5, GPT-4, GPT-4o, and, in a follow-up, GPT-4.1, on 1,000 manually annotated records containing 5,611 person labels. The headline claim is that GPT-4.1 achieves F1=0.94 with both simple and JSON-structured prompts, slightly surpassing GPT-4o's JSON-prompt F1=0.93, and that this removes the need for structured prompting. The paper also reports that GPT-4 has the highest precision (0.99) and that traditional models remain competitive, and it derives practical recommendations from the precision/recall trade-offs.","tokens_in":11681,"tokens_out":6169,"duration_ms":50037,"significance":"If the results can be verified, the paper would make a useful contribution: an original, publicly released benchmark for Russian cultural-news NER, a breadth of model comparison in an under-resourced domain, and the code and data are made available, which aligns with reproducibility recommendations. The observed progression from GPT-3.5 to GPT-4.1 is interesting. However, the internal inconsistencies in the metric tables and the under-documented GPT-4.1 evaluation currently prevent the findings from being used as reliable evidence. The significance of the claimed 'paradigm shift' (Section 6.1) cannot be assessed until the data are made consistent and auditable.","major_comments":[{"comment":"The GPT-4.1-2025-04-14 model, which is the basis of the paper's headline result (F1=0.94), is absent from the experimental setup and model inventory. Section 4.1 and Table 5 list only gpt-3.5-turbo-0125, gpt-4-turbo-2024-04-09, and gpt-4o-2024-05-13; GPT-4.1 appears for the first time in the results (Section 5.3, Table 7). No API parameters, prompt text, run date, or output-parsing procedure are given for this follow-up. Because Sections 5.5 and 6.1 draw their main conclusions from this run, the central quantitative claim is not auditable from the manuscript as submitted.","section":"§4.1, Table 5, §5.3, Table 7, §6.1"},{"comment":"The paper reports conflicting scores for the same model and prompt condition. Table 6 gives gpt-4o-2024-05-13 (simple API) P=0.96, R=0.86, F1=0.91, while Table 7 reports P=0.85, R=0.80, F1=0.82 for the same configuration, an F1 difference of 0.09. In addition, Section 5.5 cites GPT-4 recall as 0.74 while Table 6 reports 0.69; and Section 6.1 attributes an F1 of 0.78 to GPT-3.5, whereas Table 6 gives 0.81. These inconsistencies make the model rankings and the claimed progression from GPT-3.5 to GPT-4.1 impossible to verify as published.","section":"Table 6, Table 7, §5.5, §6.1"},{"comment":"The description of the gold-standard sample is not reproducible. The text says the 1,000 records were 'randomly selected' but also 'based on event date and event description length,' without specifying the selection mechanism or strata. More importantly, no inter-annotator agreement is reported for the manual annotation, even though the paper itself emphasizes the difficulty of the exclusion rules (names in titles, addresses, fictional characters). Since every precision/recall/F1 number depends on this gold standard, the lack of a reliability measure is a load-bearing gap.","section":"§3.2"},{"comment":"No uncertainty quantification or statistical significance testing is provided. The paper interprets small differences in F1 (e.g., 0.01–0.02 between GPT-4.1 simple and JSON; 0.02 between GPT-4o JSON and simple) as meaningful, but without confidence intervals, error bars, or paired tests (for instance, McNemar's test on entity spans), these differences could be noise. Given the inconsistencies already noted, the significance of these observed gaps cannot be evaluated.","section":"§5.1, §5.3"}],"minor_comments":[{"comment":"The manuscript contains several encoding or transcription artifacts (e.g., 'paradigm shiffi,' 'offien,' the garbled author footnote beginning 'envel⌢pe-⌢pen...'), which should be cleaned before publication.","section":"§2.1, §5.5, footnote"},{"comment":"The phrase 'randomly selected' and 'based on event date and event description length' are contradictory; please describe the actual sampling procedure (e.g., stratified random sampling) explicitly.","section":"§3.2"},{"comment":"The model inventory should be updated to include GPT-4.1-2025-04-14, together with the version identifiers and API settings used for all models.","section":"§4.1, Table 5"},{"comment":"The sentence 'one year affier our initial experiments' contains a typo ('affier' for 'after').","section":"§5.3"},{"comment":"The claim of a '21% absolute improvement' from GPT-3.5 to GPT-4.1 should be recalculated with consistent F1 values; based on Table 6, the improvement would be 0.13 (0.94 - 0.81), not 0.21.","section":"§6.1"}],"recommendation":"major_revision","confidential_remarks":"The two main issues—the undocumented GPT-4.1 run and the contradictory F1 scores for the same configuration in Tables 6 and 7—are severe enough that the paper should not be accepted without a thorough revision. The authors should be asked to reconcile all reported numbers, provide full experimental details for the GPT-4.1 evaluation, and ideally release the model outputs and prompts. Once those issues are fixed, the paper could be a solid empirical contribution, but as it stands the central claim is not verifiable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The genuinely new thing here is the SPbLitGuide dataset: 1,000 manually annotated records with 5,611 PERSON mentions from Russian cultural news, published on Zenodo. That is a real resource for a niche but active domain. The paper also does a few things well: the task definition is clear, the discussion of hard cases (names inside titles, addresses, fictional characters) is concrete, and code and data are promised on GitHub and Zenodo. As a practical benchmark comparison, it would be useful to practitioners who need a Russian NER model for similar texts.\n\nThe problem is that the numbers as published do not add up. Table 6 gives gpt-4o-2024-05-13 (simple API) precision 0.96, recall 0.86, F1 0.91; Table 7 gives the same model and condition precision 0.85, recall 0.80, F1 0.82. That is a nine-point F1 gap for the same configuration. Section 5.5 also says GPT-4 recall is 0.74 while Table 6 reports 0.69. Worse, GPT-4.1-2025-04-14 appears for the first time in the results without being listed in the evaluated models section, without API parameters, prompt text, or run date. The headline result (F1=0.94 for both prompts) rests on that undocumented run. These are not minor rounding issues; they make the central quantitative claim unverifiable from the manuscript alone.\n\nThe annotation quality concern is real but secondary. Even a perfect gold standard cannot fix contradictory tables. The sampling description also says the 1,000 records were randomly selected and then says the selection was based on event date and description length—that is not random sampling. No inter-annotator agreement or error bars are reported.\n\nFor whom is this paper? Researchers and practitioners working on Russian NER or cultural heritage text processing. The dataset alone is worth citing; the evaluation results as printed are not. If the authors reconcile the tables, document the GPT-4.1 run, and release raw outputs, this could become a solid benchmark paper. As it stands, it needs major revision before the scores can be trusted.\n\nMy recommendation: send it to peer review—the dataset and the comparative goal are valuable enough to deserve referee time—but the editor should make clear that the numeric inconsistencies are a blocker. Do not desk reject, but do not accept in current form.","headline":"Useful new dataset, but the headline F1 numbers are internally inconsistent and need correction before the evaluation can be trusted.","tokens_in":12109,"tokens_out":1637,"would_cite":false,"duration_ms":16984,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"On a manually annotated sample of Russian cultural-event announcements, GPT-4.1 identifies person names at F1 = 0.94 with either a plain or JSON prompt, the best score among all models tested.","keywords":["Named Entity Recognition","Russian NLP","cultural heritage texts","SPbLitGuide dataset","GPT-4.1","prompt engineering","PERSON entity extraction","benchmark evaluation"],"falsifier":"Have two independent annotators re-label the same 1,000 records under the same rules, keep only spans where both annotators agree, and recompute each model's F1 against that adjudicated standard; if GPT-4.1 no longer reaches 0.94 or no longer beats GPT-4o, the central ranking claim fails.","tokens_in":11046,"feed_emoji":"📰","tokens_out":7534,"duration_ms":68082,"temperature":0.7,"pith_summary":"The paper asks which off-the-shelf NER model can reliably extract real person names from Russian-language announcements of cultural events, where inflection, mixed-script names, pseudonyms, and names embedded in titles make the task harder than standard news NER. To answer it, the author manually annotated a 1,000-record sample of the SPbLitGuide event-announcement corpus with 5,611 PERSON labels and evaluated a set of transformer and LLM systems, including GPT-3.5, GPT-4, GPT-4o, and GPT-4.1. The central result is that GPT-4.1-2025-04-14 reaches F1 = 0.94 for PERSON extraction with both a simple prompt and a JSON-output prompt, matching or beating every other model, while GPT-4o with JSON prompting reaches F1 = 0.93 and the best non-LLM transformer reaches F1 = 0.84. If the benchmark is sound, the newest GPT-4.1 family removes the need for structured-output prompt engineering on this kind of Russian cultural text while improving accuracy.","feed_headline":"GPT-4.1 hits F1 0.94 on Russian cultural-news names","feed_subtitle":"Newest model matches simple and JSON prompting, beating all tested transformers and earlier GPTs on person extraction.","key_machinery":"The load-bearing object is the SPbLitGuide gold-standard sample: 1,000 event announcements randomly selected from 15,012 records spanning 1999–2019, with 5,611 manually annotated PERSON mentions under explicit exclusion rules that forbid labeling names inside organization names, work titles, addresses, or references to fictional characters. The evaluation machinery then normalizes each model's raw output into the same span-label format as the annotations and computes precision, recall, and F1 per model. For the LLMs, the study varies one additional mechanism: the prompt requests either plain text output or structured JSON output, testing whether output-mode constraints change extraction quality.","core_discovery":"The paper's central claim is a measured ranking on the SPbLitGuide sample: among the evaluated systems, GPT-4.1-2025-04-14 achieves the best F1 of 0.94 for PERSON entities, and it does so in both the simple API-prompt mode (precision 0.94, recall 0.93) and the JSON-output mode (precision 0.93, recall 0.95). This is presented as a shift from the earlier result in the same study, where GPT-4o-2024-05-13 needed JSON prompting to reach F1 = 0.93 (precision 0.96, recall 0.90) versus F1 = 0.91 with a simple prompt. The paper also reports that GPT-4-turbo has the highest precision at 0.99, and that the strongest non-LLM baseline, RoBERTa Large NER Russian, reaches F1 = 0.84. The author interprets the convergence of simple and structured prompts as evidence that model improvements, rather than prompt engineering, drove the gain.","pith_inferences":["A testable extension is to re-run the same protocol on a second independently annotated sample; without inter-annotator agreement, the 0.94 F1 may partly reflect how predictable one annotator's rule application is, rather than model quality.","The benchmark's exclusion of fictional characters and names inside titles means the practical NER ceiling depends on downstream goals: a system optimized for this gold standard may over-suppress names that a literary-network analysis actually wants.","If the convergence of simple and JSON prompting replicates on other morphologically rich languages, prompt-engineering effort in production NER pipelines should shift toward entity linking and normalization, where the paper itself identifies the next bottleneck."],"forward_implications":["For Russian cultural-news texts, teams can deploy GPT-4.1-2025-04-14 with a plain prompt and expect the same F1 as with JSON-structured output, cutting token overhead.","Applications that cannot tolerate false positives still have a precision-first option in GPT-4-turbo at 0.99 precision, at the cost of lower recall.","Budget- or CPU-constrained projects can use the Russian RoBERTa configuration or a CPU-oriented Russian pipeline as competitive alternatives to API-based LLMs.","The reported 6% residual error suggests human-in-the-loop review remains necessary if downstream literary-network analysis requires near-complete name coverage."],"supporting_citations":[{"why":"Supplies the SPbLitGuide dataset and the manually annotated PERSON gold standard that all metrics are computed against.","marker":"[6]"},{"why":"Provides the survey of NER evolution that frames the model selection and task definition.","marker":"[3]"},{"why":"Documents prior zero-shot LLM NER results on historical documents, the baseline this study extends and improves on.","marker":"[4]"},{"why":"Motivates prompt-engineering strategies for LLM NER, including the output-format manipulation tested here.","marker":"[5]"},{"why":"Defines the reproducible-evaluation principles the study says it follows.","marker":"[1]"},{"why":"Offers a counterexample where masked language models beat LLM prompting for clinical NER, sharpening the domain-specific comparison.","marker":"[7]"}],"fun_headline_variants":["GPT-4.1 dominates Russian NER with F1 0.94","F1 0.94: GPT-4.1 nails Russian cultural names","GPT-4.1: best yet for Russian news NE extraction","Russian NER: GPT-4.1 hits 0.94 F1 across prompts","GPT-4.1 outperforms all on Russian cultural NER"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the single annotator's 5,611 PERSON labels in the 1,000-record sample correctly apply the study's exclusion rules; if the gold standard mislabels or inconsistently handles names in titles, organizations, addresses, or fictional references, every reported precision, recall, and F1 value—and the model ranking—changes.","fun_headline_variants_meta":{"raw":{"variants":["GPT-4.1 dominates Russian NER with F1 0.94","F1 0.94: GPT-4.1 nails Russian cultural names","GPT-4.1: best yet for Russian news NE extraction","Russian NER: GPT-4.1 hits 0.94 F1 across prompts","GPT-4.1 outperforms all on Russian cultural NER"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000325,"raw_usage":{"total_tokens":1841,"prompt_tokens":985,"completion_tokens":856,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":601,"completion_tokens_details":{"reasoning_tokens":754}},"tokens_in":601,"tokens_out":856,"duration_ms":6945,"temperature":1.0,"reasoning_tokens":754,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:19:59.915192+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have two independent annotators re-label the same 1,000 records under the same rules, keep only spans where both annotators agree, and recompute each model's F1 against that adjudicated standard; if GPT-4.1 no longer reaches 0.94 or no longer beats GPT-4o, the central ranking claim fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the SPbLitGuide dataset and the manually annotated PERSON gold standard that all metrics are computed against."},{"cited_title":"E., Boros, E., Girdhar, N., Hamdi, A., Moreno, J","cited_arxiv_id":null,"evidence_quote":"Documents prior zero-shot LLM NER results on historical documents, the baseline this study extends and improves on."}],"review_version":1}