REVIEW 3 major objections 5 minor 18 references
ICPR 2024 Competition on Multilingual Claim-Span Identification
T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read This paper presents HECSI, a publicly released benchmark of about 16,000 English and Hindi social media posts with human-marked claim spans, and reports the first competition results showing that token-level claim identification remains…
desk verdict A genuinely new Hindi-English claim-span benchmark worth having, but the paper needs metric, agreement, and table fixes before the leaderboard numbers can be taken at face value. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the HECSI dataset itself: posts with token-level labels marking each minimal claim span, produced by paid human annotators following the definition of a claim as a statement asserted as true. The evaluation machinery treats CSI as binary token classification, scoring each post with Macro-F1 and Jaccard over tokens and averaging over the test set, following the protocol introduced for the earlier CURT task. The dataset's deliberate inclusion of posts without claim spans and posts with multiple claim spans is what makes the task harder than sentence-level claim classification, and it is the mechanism that the competition results use to separate methods.
What would settle it
Re-annotate a random sample of HECSI posts (say 300 per language) with the same guidelines from two or more independent annotators; if pairwise agreement on exact span boundaries is low (for example, token-level F1 below the 74-81% leaderboard range), the benchmark's labels cannot support the reported model rankings.
Extended reading notes
Core claim
The core claim is that HECSI is a workable testbed for multilingual claim-span identification: it combines the CAVES English anti-vaccine tweets and the CONSTRAINT Hindi hostile posts, re-annotated by a contracted firm with instructions to mark every minimal claim span, and split into train, validation, and test sets for the competition. Under token-level Macro-F1 and Jaccard evaluation, the best constrained-track system (JU_NLP, fine-tuned XLM-RoBERTa and multilingual BERT) scored 74.8 Macro-F1 and 54.5 Jaccard on English and 81.7 and 67.1 on Hindi, beating the organizer's multilingual BERT baseline (72.6/50.3 and 74.2/63.6). In the unconstrained multilingual track, the best submitted Macro-F1 was 59.3 (DLRG) and best Jaccard 41.4 (FactFinders), both below the baseline's 74.2 and 57.4, despite the winning teams adding external claim-span data. The paper reads these results as showing that exact claim-span detection remains difficult, that domain transfer from auxiliary datasets did not help, and that the benchmark is useful precisely because it exposes this gap.
Load-bearing premise
The whole benchmark rests on the unmeasured assumption that the hired annotators marked every minimal claim span consistently, since no inter-annotator agreement is reported.
Editorial extensions
If this is right
- HECSI gives the community a public benchmark with labeled minimal claim spans for English and Hindi, so future models can be compared on identical train, validation, and test splits.
- Fine-tuned multilingual encoder models set the current practical bar for CSI; in the constrained tracks they beat instruction-tuned LLMs on token-level Macro-F1.
- Throwing external claim-span datasets into multilingual training did not lift performance above a simple baseline, suggesting that more data alone is not a fix for cross-lingual span identification.
- Because spans are marked at the minimal level, model outputs can double as explanations of why a post is flagged, supporting fact-checking pipelines that need justifications.
- The large gap between English and Hindi constrained performance and the multilingual track indicates that combining languages in one model is currently a bottleneck rather than a benefit.
Reading between the lines
- The paper does not report inter-annotator agreement, so a natural next step is to measure agreement on a held-out sample; if token-level annotator agreement is near or below the reported leaderboard gaps, part of the ranking could be annotation noise.
- HECSI's no-claim and multi-span posts make it possible to decompose errors by post type; such an error analysis could show whether models mostly fail on boundary precision or on whole-span recall.
- A testable extension is to use the English HECSI portion to pretrain or adapt a model and then evaluate zero-shot on the Hindi portion, and vice versa, isolating the cross-lingual transfer signal that the multilingual track's data-pooling approach failed to exploit.
- Since the English source data is vaccine-focused and the Hindi source is hostile-speech-focused, domain and language are confounded in HECSI; future annotation from matched domains could separate language difficulty from content difficulty.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper reports the ICPR 2024 competition on multilingual Claim Span Identification (CSI). The authors introduce HECSI, a dataset of approximately 8,000 English and 8,000 Hindi social media posts with claim spans marked by human annotators, and describe three tracks: constrained English, constrained Hindi, and unconstrained multilingual. The evaluation treats the task as token-level binary classification and reports Macro-F1 and Jaccard scores. The paper presents the organizer baseline (fine-tuned multilingual BERT) and the solutions of nine participating teams, with JU_NLP achieving the best scores in the English and Hindi constrained tracks and FactFinders and DLRG toping the multilingual track. The conclusion states that the task remains challenging because no team significantly beat the baseline, although JU_NLP's Hindi score substantially exceeded it.
Significance. If the HECSI annotations are reliable, the dataset is a useful public resource for claim-span identification in English and Hindi, a task that is more granular than standard claim/non-claim classification. The competition report documents a diverse set of model solutions and provides a reproducible evaluation protocol based on token-level metrics. The public availability of the dataset and the participation of multiple teams are concrete strengths. The significance is currently tempered, however, by the absence of annotation reliability evidence and by a malformed definition of the primary evaluation metric, both of which need to be addressed before the leaderboard numbers can be fully trusted.
major comments (3)
- [Section 3, Evaluation] The printed Macro-F1 formula is malformed. As typeset, 'M-F1 = TP / 2·TP+FP+FN+TN / 2·TN+FP+FN' is not a valid mathematical expression and cannot be used to reproduce the leaderboard results. Since M-F1 is the primary ranking metric in Table 5, the formula must be corrected and the averaging procedure must be spelled out clearly, in particular whether the metric is averaged over the positive/negative classes or over individual posts after computing per-post F1.
- [Section 2, Dataset annotation] No inter-annotator agreement, adjudication procedure, or post-hoc quality check is reported for HECSI. The dataset is annotated entirely by a contracted annotation firm, and the claim that HECSI is a valid benchmark presupposes that annotators consistently identified all and only the minimal claim spans. The dataset contains many posts with multiple claim spans and many with no claim span (Table 3), precisely the cases where boundary and relevance decisions are hardest. Without a reliability measurement, the differences between systems in Table 5 (e.g., JU_NLP's M-F1 of 81.7 vs. the baseline's 74.2 in Hindi) could reflect annotation noise. Please report IAA on a sample and describe the adjudication procedure.
- [Section 5, Conclusion] The conclusion states that 'no team could achieve significantly better results than the baseline,' but no statistical significance testing is reported. In the Hindi track, JU_NLP's M-F1 is 81.7 versus the baseline's 74.2 (Table 5), a 7.5-point gap. To support this claim, the authors should either run significance tests (e.g., bootstrap or approximate randomization over test posts) or soften the wording to remove 'significantly.'
minor comments (5)
- [Table 3] The Multilingual column lists 'Total #posts' as 14,097, but the three rows below it sum to 16,097, and the English and Hindi totals also sum to 16,097. This appears to be a typo and should be corrected.
- [Table captions] The captions for Tables 1 through 5 contain the typo 'T able' instead of 'Table'.
- [Section 3, Evaluation] The text says the metrics are 'averaged over all the posts in the test set,' but the notation of Macro-F1 is ambiguous: it is not clear whether the macro-averaging is over the two classes or over posts. Please clarify the precise computation, ideally with a corrected formula.
- [Table 5] The caption does not explain the meaning of '-' entries; please state that '-' indicates the team did not submit a run for that track.
- [Section 4.2] The text refers to 'the FactFinder team,' but the team name is FactFinders; please use a consistent name.
Circularity Check
No significant circularity; the HECSI benchmark and competition results rest on external human annotations and standard evaluation, not on self-referential construction.
full rationale
The paper's central contribution is a benchmark dataset and the reporting of competition results; there is no derivation chain in which a prediction follows from an input by construction. The English portion of HECSI is built from tweets in the authors' earlier CAVES dataset (Section 2 cites [15] as 'our prior work'), but the paper states that HECSI was then 'annotated by a team of human workers from a reputed annotation firm,' so the claim-span labels are not inherited from CAVES by definition. The organizer baseline is trained on the same provided splits and evaluated on the same test sets as participant runs; this is a standard controlled comparison and does not force any participant's score. Evaluation metrics (Macro-F1 and Jaccard) are standard and are computed from submitted token predictions against fixed gold labels, so no fitted parameter is renamed as a prediction. No uniqueness theorem or ansatz is imported via self-citation. The absence of inter-annotator agreement is a validity concern, but it is an annotation-quality limitation, not a circularity: the labels are external inputs to the evaluation, not derived from the model outputs. The paper is therefore self-contained with respect to circularity.
Assumptions & free parameters
assumptions (3)
- domain assumption Annotators from the contracted firm reliably marked all minimal claim spans in HECSI.
- domain assumption Token-level binary classification is an adequate operationalization of claim-span identification.
- domain assumption Teams followed the constrained-track rules forbidding the use of external training data.
Cite this review
Pith. "Pith review of ICPR 2024 Competition on Multilingual Claim-Span Identification." pith.science (2026). https://pith.science/paper/TY25JUSE
@misc{pith2026241119579,
author = {Pith},
title = {Pith review of: ICPR 2024 Competition on Multilingual Claim-Span Identification},
year = {2026},
howpublished = {\url{https://pith.science/paper/TY25JUSE}},
note = {Machine review of arXiv:2411.19579}
}
read the original abstract
A lot of claims are made in social media posts, which may contain misinformation or fake news. Hence, it is crucial to identify claims as a first step towards claim verification. Given the huge number of social media posts, the task of identifying claims needs to be automated. This competition deals with the task of 'Claim Span Identification' in which, given a text, parts / spans that correspond to claims are to be identified. This task is more challenging than the traditional binary classification of text into claim or not-claim, and requires state-of-the-art methods in Pattern Recognition, Natural Language Processing and Machine Learning. For this competition, we used a newly developed dataset called HECSI containing about 8K posts in English and about 8K posts in Hindi with claim-spans marked by human annotators. This paper gives an overview of the competition, and the solutions developed by the participating teams.
Reference graph
Works this paper leans on
-
[1]
arXiv preprint arXiv:2011.03588 (2020)
Bhardwaj, M., Akhtar, M.S., Ekbal, A., Das, A., Chakraborty, T.: Hostility detec- tion dataset in hindi. arXiv preprint arXiv:2011.03588 (2020)
arXiv 2020
-
[2]
Journal of artificial intelligence research16, 321– 357 (2002)
Chawla, N.V., Bowyer, K.W., Hall, L.O., Kegelmeyer, W.P.: Smote: synthetic mi- nority over-sampling technique. Journal of artificial intelligence research16, 321– 357 (2002)
2002
-
[3]
arXiv preprint arXiv:1911.02116 (2019)
Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., Stoyanov, V.: Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02116 (2019)
arXiv 2019
-
[4]
arXiv preprint arXiv:1810.04805 (2018)
Devlin,J.,Chang,M.W.,Lee,K.,Toutanova,K.:Bert:Pre-trainingofdeepbidirec- tional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)
arXiv 2018
-
[5]
Gupta,S.,Singh,P.,Sundriyal,M.,Akhtar,M.S.,Chakraborty,T.:Lesa:Linguistic encapsulation and semantic amalgamation based generalised claim detection from online content. arXiv preprint arXiv:2101.11891 (2021)
work page Pith review arXiv 2021
-
[6]
arXiv preprint arXiv:2111.09543 (2021)
He, P., Gao, J., Chen, W.: Debertav3: Improving deberta using electra- style pre-training with gradient-disentangled embedding sharing. arXiv preprint arXiv:2111.09543 (2021)
arXiv 2021
-
[7]
arXiv preprint arXiv:2106.09685 (2021)
Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W.: Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 (2021)
arXiv 2021
-
[8]
arXiv preprint arXiv:2310.06825 (2023)
Jiang, A., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D., de las Casas, D., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al.: Mistral 7b (2023). arXiv preprint arXiv:2310.06825 (2023)
arXiv 2023
Show all 18 references
-
[9]
In: Findings of the Association for Computational Linguistics: EMNLP 2020
Kakwani, D., Kunchukuttan, A., Golla, S., Gokul, N., Bhattacharyya, A., Khapra, M.M., Kumar, P.: Indicnlpsuite: Monolingual corpora, evaluation benchmarks and pre-trained multilingual language models for indian languages. In: Findings of the Association for Computational Lingu...
2020
-
[10]
arXiv preprint arXiv:2103.10730 (2021)
Khanuja, S., Bansal, D., Mehtani, S., Khosla, S., Dey, A., Gopalan, B., Margam, D.K., Aggarwal, P., Nagipogu, R.T., Dave, S., et al.: Muril: Multilingual represen- tations for indian languages. arXiv preprint arXiv:2103.10730 (2021)
2021 arXiv
-
[11]
arXiv preprint arXiv:1907.11692 (2019) ICPR 2024 Competition on Multilingual Claim-Span Identification 11
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., Stoyanov, V.: Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 (2019) ICPR 2024 Competition on Multilingual Claim-Span Identification 11
2019 arXiv
-
[12]
Meta AI (2024)
Meta, A.: Introducing meta llama 3: The most capable openly available llm to date. Meta AI (2024)
2024
-
[13]
In: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing
Mittal, S., Sundriyal, M., Nakov, P.: Lost in translation, found in spans: Identifying claims in multilingual social media. In: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. pp. 3887–3902 (2023)
2023
-
[14]
arXiv preprint arXiv:2211.01786 (2022)
Muennighoff, N., Wang, T., Sutawika, L., Roberts, A., Biderman, S., Scao, T.L., Bari, M.S., Shen, S., Yong, Z.X., Schoelkopf, H., et al.: Crosslingual generalization through multitask finetuning. arXiv preprint arXiv:2211.01786 (2022)
2022 arXiv
-
[15]
In: Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval
Poddar, S., Samad, A.M., Mukherjee, R., Ganguly, N., Ghosh, S.: Caves: A dataset to facilitate explainable classification and summarization of concerns towards covid vaccines. In: Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Informa...
2022
-
[16]
arXiv preprint arXiv:1910.01108 (2019)
Sanh, V., Debut, L., Chaumond, J., Wolf, T.: Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108 (2019)
2019 arXiv
-
[17]
In: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing
Sundriyal, M., Kulkarni, A., Pulastya, V., Akhtar, M.S., Chakraborty, T.: Em- powering the fact-checkers! automatic identification of claim spans on twitter. In: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. pp. 7701–7715 (2022)
2022
-
[18]
Cambridge university press (2003)
Toulmin, S.E.: The uses of argument. Cambridge university press (2003)
2003
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.