{"id":"109b8ddd-80e8-44f3-930c-2b6cce09a411","arxiv_id":"2412.20414","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Across a small multilingual Telegram corpus, GPT-4 and XLM-RoBERTa achieve the best geo-entity F1 scores, while SpaCy and mLUKE score zero on Arabic and GeoLM collapses outside English.","lead":"This paper tests six NLP models and large language models, including GPT-4 and XLM-RoBERTa, on finding location names in English, Russian, and Arabic Telegram posts about conflicts. The reported scores suggest GPT-4 and XLM-RoBERTa are strongest, while SpaCy and mLUKE fail on Arabic and GeoLM transfers poorly across languages.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Recall values in Table 1 are not computable from the stated protocol: Section 4.2 verifies only entities the models identified, so false negatives (the denominator of recall) are never established.","rationale":"The reader's verdict correctly identifies the manual verification as underspecified, but the load-bearing issue is sharper than 'noisy labels.' The stated protocol cannot compute recall at all: verifying only model-identified entities yields TP and FP counts, but not FN counts, so the recall denominator is missing. This is an internal methodological gap, not just a reproducibility complaint. It directly affects every F1 value in Table 1 and therefore the paper's main conclusion about which models perform best. I still recommend CONDITIONAL rather than REJECT because a full annotation set may exist and, if released, could validate the numbers. The paper's other issues (no code/data, no confidence intervals, promised accuracy not reported) are secondary but reinforce the conditional verdict. My disagreement with the reader is partial: they frame the risk as annotation noise and consistency, whereas I see a more fundamental problem — the protocol as written cannot produce recall values, independent of how careful the annotators were.","tokens_in":7578,"tokens_out":1998,"duration_ms":22623,"concrete_test":"Obtain or reconstruct a full gold standard: take a random sample of, say, 100 posts per language from the cited Telegram channels, have two independent annotators mark every geo-entity according to a written protocol (including multi-word and contested terms), compute inter-annotator agreement, and recompute precision, recall, and F1 for all six models against this complete annotation. If recall values shift materially (for example, XLM-RoBERTa's English recall drops well below 0.98), then the Table 1 ranking and the paper's central comparison are not supported. If the authors already possess such a gold standard, releasing it and the evaluation script would settle the question directly.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim depends on the precision/recall/F1 values in Table 1, and every recall value requires a complete enumeration of true geo-entities in the corpus. Section 4.2 states only that 'each geographical entity identified by the models was subjected to manual verification.' This labels model predictions as true or false positives, but it provides no procedure for identifying false negatives — the entities present in the text that no model detected. Without a full gold-standard annotation of the evaluated posts, the denominator of recall (TP + FN) is unknowable, so recall and F1 are undefined for every model. This is not merely an omitted detail: it is an internal gap in the evaluation logic. For example, XLM-RoBERTa's reported English recall of 0.98 would mean that nearly all true location mentions appear among its predictions, a fact that cannot be established by checking only its own output. If a full annotation pass was actually performed, it is absent from the paper: no annotation protocol, no annotator count, no inter-annotator agreement, and no handling of the ambiguous multi-word and offensive terms the authors themselves discuss in Section 5. Because the headline finding is that XLM-RoBERTa and GPT-4 achieve the best balanced F1 scores, and F1 is directly built on these recall denominators, the reported ranking is not reproducible from the described method. The paper also promises an 'accuracy' metric in the abstract and Section 4.3 but never reports it, further weakening the quantitative basis.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper compares six NLP models—SpaCy, XLM-RoBERTa, mLUKE, GeoLM, GPT-3.5, and GPT-4—for geo-entity detection in Telegram posts in English, Russian, and Arabic. It reports precision, recall, and F1 scores in Table 1 and concludes that XLM-RoBERTa and GPT-4 achieve the best balanced performance. The authors describe a pipeline that applies each model sequentially and manually verifies the geographic entities identified by the models. The paper does not release its dataset, code, annotation guidelines, or inter-annotator agreement measures.","tokens_in":7801,"tokens_out":3113,"duration_ms":32113,"significance":"If the numerical results were trustworthy, the paper would provide a practically useful comparison for security and crisis-monitoring applications. The selection of models is reasonable, and the focus on multilingual Telegram data is relevant. However, the evaluation protocol as described cannot support the reported recall and F1 values, and the promised accuracy metric is absent. The paper also lacks the reproducibility artifacts (data, code, annotation protocol) that would allow independent verification. These gaps are not cosmetic; they undermine the central comparative claim.","major_comments":[{"comment":"The stated manual verification protocol cannot yield the recall or F1 values in Table 1. Section 4.2 says that 'each geographical entity identified by the models was subjected to manual verification,' which labels model predictions as true or false positives but provides no procedure for identifying false negatives—entities present in the text that no model detected. Without a full gold-standard annotation of the corpus, the denominator of recall (TP + FN) is undefined, so every recall and F1 entry in Table 1 is unsupported. For instance, XLM-RoBERTa's English recall of 0.98 would require knowing that nearly all true location mentions in the corpus appear among its predictions, which cannot be established by checking only its own output. The paper must either supply the complete gold-standard annotation procedure (including annotator count, instructions, and inter-annotator agreement) or restrict its claims to precision, which is the only metric computable from the described protocol.","section":"§4.2, Table 1"},{"comment":"The abstract states that the paper examines performance 'through metrics such as accuracy, precision, recall, and F1 scores,' and Section 4.3 is titled 'Accuracy Assessment,' yet no accuracy values are reported anywhere in the paper, including in Table 1. This is an internal inconsistency. The authors should either report the accuracy metric they promised (with a clear definition, e.g., token-level or entity-level accuracy) or explicitly remove 'accuracy' from the abstract and Section 4.3.","section":"Abstract and §4.3"},{"comment":"The experimental results are not reproducible because the dataset is not described at the level needed to reconstruct it. The paper gives only channel URLs and counts of posts per channel; it does not provide post identifiers, timestamps, or a data availability statement. Moreover, no annotation guidelines, annotator qualifications, or inter-annotator agreement are reported for the manual verification, and the evaluation script is not released. For a comparative benchmark paper, these details are necessary for readers to trust or replicate the reported scores. The authors should make the annotated data and code available, or at minimum provide a detailed and deterministic description of the data collection and annotation process.","section":"§3.6, §4.2, References [11]–[14]"}],"minor_comments":[{"comment":"Section 4.4 states that 'the inclusion of tables or graphs summarizing the evaluation results is planned,' although Table 1 is already included in Section 5. This suggests leftover text from an earlier draft and should be reconciled.","section":"§4.4"},{"comment":"The table is difficult to parse as typeset: the model names and metric values run together, and the footnote 'a' appears detached. Please reformat it as a proper grid with clear column separators.","section":"Table 1"},{"comment":"The description of the SpaCy multi-language model says it is 'presumably' for Arabic; the paper should specify exactly which model is used for Arabic and why no dedicated Arabic SpaCy model is employed.","section":"§3.1"},{"comment":"The prose refers to 'GPT-3.5' and 'GPT-4' while Section 3.5 gives specific API names 'gpt-3.5-turbo-0125' and 'gpt-4-0125-preview.' Please use the exact model identifiers consistently throughout the paper, including in Table 1.","section":"§3.5"},{"comment":"The paper uses 'F-1 score,' 'F1 Score,' and 'F1' inconsistently. Please standardize the terminology.","section":"Global"},{"comment":"The Telegram channel references would be more useful if they included the date of data collection or the specific date range of the retrieved posts, rather than only a general retrieval date.","section":"References [11]–[14]"}],"recommendation":"major_revision","confidential_remarks":"The manuscript currently reads as an extended abstract rather than a complete research article. The core technical problem—the undefined recall denominator—is fixable in principle by performing a full gold-standard annotation of the corpus, but that is a substantial additional effort. If the authors cannot provide such annotations, the paper should be restructured to report only precision and to avoid comparative F1 claims. I also note that the paper does not mention any data-sharing or reproducibility plan, which would likely be expected by the journal."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this one carefully before trusting any number in Table 1. It is a small multilingual benchmark comparing SpaCy, XLM-RoBERTa, mLUKE, GeoLM, GPT-3.5 and GPT-4 on geo-entity detection in English, Russian and Arabic Telegram posts. The empirical scores are new, and the qualitative findings match what I would expect: SpaCy and mLUKE collapse on Arabic, GeoLM is English-centric, GPT-3.5 is mediocre, and XLM-RoBERTa and GPT-4 lead on F1. The model selection is sensible, the data sources are identified, and the authors are honest about practical constraints like the 512-token limit and hard cases such as multi-word and offensive toponyms.\n\nThe soft spot is not minor. The paper's evaluation protocol only manually verifies entities the models identified. That yields true positives and false positives, but never false negatives. Without a complete annotation of every true location mention in the corpus, you cannot compute recall, and every F1 score in Table 1 is built on recall. XLM-RoBERTa's English recall of 0.98 would mean it found nearly all true mentions, which cannot be established by inspecting only its own output. Either a full gold-standard pass was performed and omitted from the write-up, or the reported recall and F1 values are not well-defined. The paper also promises an accuracy metric that never appears, and provides no dataset, code, annotation guidelines, inter-annotator agreement, or error bars. The exact numbers are therefore unverifiable from the text as-is.\n\nThat said, the central ranking is plausible, and the paper is not circular or incoherent in its reasoning. It is simply missing the load-bearing part of its own evaluation. The fix is clear: release the annotated data and code, describe how false negatives were established, and report variance. With that, this could be a useful practical reference for practitioners choosing off-the-shelf models for multilingual monitoring. As written, though, I would not cite the scores.\n\nSend it to serious peer review. The question is legitimate and the paper is worth engaging; a good referee can force the authors to close the evaluation gap. But prepare for major revision.","headline":"A straightforward six-model benchmark on Telegram data whose headline F1 numbers are not actually computable from the method as written, because recall is never grounded in a full gold-standard pass.","tokens_in":8334,"tokens_out":1853,"would_cite":false,"duration_ms":19651,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper reports a six-model benchmark in which XLM-RoBERTa and GPT-4 achieve the best balanced geo-entity F1 across English, Russian, and Arabic, while SpaCy and mLUKE detect no Arabic locations at all.","keywords":["Named Entity Recognition","geospatial entity detection","multilingual NLP","large language models","Telegram data","XLM-RoBERTa","GPT-4","GeoLM"],"falsifier":"Re-verify a random sample of the same Telegram posts using multiple annotators and a written guideline for multi-word and pejorative location names, then recompute F1; if the new labels disagree substantially with the paper's manual verification, the reported rankings and especially the Arabic 0.00 scores would change.","tokens_in":7326,"feed_emoji":"📍","tokens_out":7143,"duration_ms":57915,"temperature":0.7,"pith_summary":"This paper reports a benchmark of six named-entity recognition systems for detecting geographical references in multilingual Telegram posts: SpaCy, XLM-RoBERTa, mLUKE, GeoLM, GPT-3.5, and GPT-4. The central claim is that on posts in English, Russian, and Arabic about ongoing conflicts, XLM-RoBERTa and GPT-4 achieve the best balanced F1 scores, while SpaCy and mLUKE detect no Arabic locations at all and GeoLM is nearly ineffective outside English. The stakes are practical: security monitoring, crisis response, and situational awareness depend on reliably extracting location names from social media in many languages, and model choice should follow from measured cross-lingual coverage.","feed_headline":"XLM-RoBERTa and GPT-4 top multilingual geo-entity F1","feed_subtitle":"On conflict Telegram posts in English, Russian, and Arabic, SpaCy and mLUKE detect zero Arabic locations.","key_machinery":"The evaluation pipeline is the mechanism: a custom Python script normalizes Telegram raw text, runs each model sequentially to extract location mentions, then submits every detected entity to manual verification before computing precision, recall, and F1 per language. The six models are SpaCy with language-specific and multi-language pipelines, XLM-RoBERTa fine-tuned on CoNLL-2003, mLUKE whose token spans come from SpaCy, GeoLM's toponym-recognition model, and GPT-3.5 and GPT-4 accessed through their API with a custom location-only prompt. The pipeline's design means the Arabic 0.00 scores for SpaCy and mLUKE are a direct consequence of SpaCy's generic multi-language model being the only Arabic path.","core_discovery":"Across the curated Telegram corpus (1,054 English posts, 2,406 Russian, 1,065 Arabic), XLM-RoBERTa and GPT-4 tie or lead in every language: F1 of 0.90 and 0.90 in English, 0.91 and 0.90 in Russian, and 0.84 and 0.74 in Arabic. SpaCy and mLUKE both score precision, recall, and F1 of 0.00 on Arabic; GeoLM, an English-centric geospatial language model, drops from 0.69 F1 in English to 0.15 in Russian and 0.00 in Arabic; GPT-3.5 is mediocre everywhere, with its best F1 0.58 in English. The paper attributes SpaCy's Arabic failure to its generic multi-language pipeline, mLUKE's failure to its reliance on SpaCy for span detection, and GeoLM's to its English-only training, and it flags the 512-token limit of mLUKE and GeoLM plus hard cases like multi-word and pejorative location terms as additional causes of errors.","pith_inferences":["The Arabic zero scores are likely an artifact of model selection rather than an inherent limitation: a dedicated Arabic SpaCy model or another Arabic NER would be a direct test of whether SpaCy's ecosystem can handle Arabic.","The reported F1 differences between XLM-RoBERTa and GPT-4 are within a few points; without confidence intervals or significance testing, claims of superiority for either model are not established.","A formal annotation guideline for multi-word and pejorative geographic names would likely change recall values substantially, since the paper itself identifies these as the main error sources.","The same comparison could be extended to a public multilingual NER benchmark to test whether the observed cross-lingual gaps generalize beyond this Telegram corpus."],"forward_implications":["For short multilingual posts, XLM-RoBERTa and GPT-4 are the only evaluated models with usable geo-entity F1 across English, Russian, and Arabic.","Any system depending on SpaCy's generic multi-language pipeline will also fail on Arabic, as mLUKE did.","GeoLM should be treated as an English-only model until it receives cross-lingual adaptation.","The 512-token truncation of mLUKE and GeoLM is a practical constraint even for brief Telegram messages.","GPT-3.5's inconsistent location detection makes it a weak default compared with GPT-4."],"supporting_citations":[{"why":"Supplies the cross-lingual pretraining that underlies the XLM-RoBERTa model used for English and other languages.","marker":"[2]"},{"why":"Defines mLUKE, the entity-aware multilingual model whose token spans are supplied by SpaCy.","marker":"[4]"},{"why":"Introduces GeoLM and its geospatially grounded pretraining, the basis for the toponym-recognition model evaluated.","marker":"[5]"},{"why":"Identifies the exact XLM-RoBERTa NER model (fine-tuned on CoNLL-2003) used in the experiment.","marker":"[7]"},{"why":"Identifies the exact mLUKE model used, aligned to SpaCy character spans.","marker":"[8]"},{"why":"Identifies the exact GeoLM toponym-recognition implementation used.","marker":"[9]"},{"why":"Documents the GPT-3.5 and GPT-4 API models accessed for the LLM comparison.","marker":"[10]"},{"why":"Provides the English Telegram posts from the Intel SlavaZ channel used in the evaluation.","marker":"[11]"},{"why":"Provides the Russian Telegram posts from the DvaMayora channel used in the evaluation.","marker":"[13]"},{"why":"Provides the Arabic Telegram posts from the ALHodeidah Media Center used in the evaluation.","marker":"[14]"}],"fun_headline_variants":["SpaCy and mLUKE score zero on Arabic geo-entities","XLM-RoBERTa and GPT-4 lead multilingual geo-entity F1","On Telegram posts, XLM-RoBERTa and GPT-4 best geo-entity F1","Arabic geo-entity detection: SpaCy and mLUKE get zero","GeoLM's F1 drops from 0.69 in English to zero in Arabic"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the manual verification of every detected entity is accurate and consistent, but the paper gives no annotation protocol, no number of annotators, and no inter-annotator agreement, so noisy gold labels would shift every reported score.","fun_headline_variants_meta":{"raw":{"variants":["SpaCy and mLUKE score zero on Arabic geo-entities","XLM-RoBERTa and GPT-4 lead multilingual geo-entity F1","On Telegram posts, XLM-RoBERTa and GPT-4 best geo-entity F1","Arabic geo-entity detection: SpaCy and mLUKE get zero","GeoLM's F1 drops from 0.69 in English to zero in Arabic"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001178,"raw_usage":{"total_tokens":4877,"prompt_tokens":964,"completion_tokens":3913,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":580,"completion_tokens_details":{"reasoning_tokens":3805}},"tokens_in":580,"tokens_out":3913,"duration_ms":27568,"temperature":1.0,"reasoning_tokens":3805,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T23:21:40.689354+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-verify a random sample of the same Telegram posts using multiple annotators and a written guideline for multi-word and pejorative location names, then recompute F1; if the new labels disagree substantially with the paper's manual verification, the reported rankings and especially the Arabic 0.00 scores would change.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the cross-lingual pretraining that underlies the XLM-RoBERTa model used for English and other languages."},{"cited_title":"(2024, March21)","cited_arxiv_id":null,"evidence_quote":"Identifies the exact mLUKE model used, aligned to SpaCy character spans."},{"cited_title":"(2024, March 21)","cited_arxiv_id":null,"evidence_quote":"Identifies the exact GeoLM toponym-recognition implementation used."},{"cited_title":"(2024, March 21)","cited_arxiv_id":null,"evidence_quote":"Documents the GPT-3.5 and GPT-4 API models accessed for the LLM comparison."},{"cited_title":"(2024, March 21)","cited_arxiv_id":null,"evidence_quote":"Provides the English Telegram posts from the Intel SlavaZ channel used in the evaluation."}],"review_version":1}