Pith. sign in

REVIEW 3 major objections 4 minor 11 references

Duluth at SemEval-2025 Task 7: TF-IDF with Optimized Vector Dimensions for Multilingual Fact-Checked Claim Retrieval

T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A tuned TF-IDF retriever reaches 0.69 success@10 across ten languages for multilingual fact-checked claim retrieval, the paper reports.

desk verdict A credible shared-task baseline paper with an unsubstantiated 'optimal' parameter claim and a contradiction in the language-resource narrative. read the letter →

arxiv 2505.12616 v1 pith:46S37PGY submitted 2025-05-19 cs.CL

Shujauddin Syed , Ted Pedersen This is my paper · ORCID
classification cs.CL
keywords TF-IDFmultilingualclaimretrievalfact-checkedsuccess@10vector-sizeoptimizationword-leveltokenizationbaselinelow-resourcelanguages
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that a deliberately tuned TF-IDF retriever remains a competitive baseline for multilingual fact-checked claim retrieval: with word-level tokenization and a vocabulary capped at 15,000 features, it reaches 0.78 average success@10 on the development set and 0.69 on the test set over ten languages. The score matters because the same paper finds this configuration beats several neural embedding baselines in its own comparisons, while still trailing the top system's 0.96. The authors present the result as evidence that optimized statistical retrieval is still useful, especially when compute is limited.

What carries the argument

The load-bearing component is a term frequency-inverse document frequency (TF-IDF) vectorizer whose two knobs are the analyzer and the vocabulary size. The default word analyzer tokenizes each fact check and post into words, and max_features=15,000 keeps only the 15,000 highest-frequency terms, producing sparse vectors; retrieval ranks fact checks by similarity of their vectors to the query vector and returns the top ten. The vocabulary cap carries the argument: raising the cap from the default configuration to 10,000 lifts the development average from 0.5513 to 0.7630, and the further step to 15,000 adds another 0.0127 (to 0.7757), whereas switching to character-level analyzers drops the average to 0.46 or below.

What would settle it

Rerun the released pipeline on the official test split with max_features set to 10,000 and 15,000 and compare average success@10. If 15,000 does not beat 10,000 on the test set, or the reported 0.69 average is not reproducible, the paper's claim that 15,000 is the optimized configuration fails.

Watch

Extended reading notes

Core claim

On the monolingual track of the multilingual fact-checked claim retrieval task, the paper's central discovery is that a sparse TF-IDF vector space restricted to the 15,000 most frequent word tokens retrieves a relevant previously fact-checked claim in the top ten results for 78 percent of development queries and 69 percent of test queries on average across ten languages. Per-language test scores range from 0.452 for English to 0.8495 for Malay, with stronger performance on higher-resource languages and clear losses from character-level tokenization. In the paper's experiments on the development split, this configuration outperforms every neural retriever tested, including multilingual E5 and GTR-T5, which reach 0.6494 and 0.6360 respectively, while the top-ranked system on the same test set reaches 0.9601. The paper reads this as evidence that traditional statistical retrieval can still act as a strong baseline and that its remaining gap to neural systems is largest on lower-resource languages.

Load-bearing premise

The choice of 15,000 features is based on the development-set average, where it beats 10,000 by 0.0127; the paper assumes this development ranking transfers to the test set, although per-language scores move dramatically between the two splits.

Editorial extensions

If this is right

  • A 15K word-level TF-IDF retriever is a usable low-cost baseline for monolingual fact-checked claim retrieval, with a test average of 0.69 across English, Spanish, German, Portuguese, French, Arabic, Malay, Thai, Polish, and Turkish.
  • Raising vocabulary size from 10K to 15K improves development success@10 by about 1.3 percentage points on average, and word-level tokenization is consistently better than character-level for this data.
  • The gap to the top system (0.69 versus 0.96) quantifies how far a tuned statistical retriever is from current neural systems on this task.
  • The results support the paper's practical recommendation that TF-IDF is a reasonable choice when compute or fine-tuning resources are limited, because it outperformed the neural baselines the authors ran without any training.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The development-set margin of 15K over 10K is small (0.0127), while several languages swing sharply between dev and test; a reader should not assume the 15K choice is the true optimum until it is confirmed on test data held out from configuration search.
  • A hybrid retriever that fuses TF-IDF lexical scores with E5 or T5 dense scores could plausibly combine exact-match precision with semantic recall, especially for lower-resource languages where the paper's neural models were weakest; the paper lists this as future work but does not test it.
  • The success@10 metric rewards recall at a shallow rank; under a different evaluation regime such as mean reciprocal rank or precision-oriented ranking, the ordering of TF-IDF versus dense models might change because dense models can rank relevant claims lower.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper describes the University of Minnesota Duluth submission to SemEval-2025 Task 7 (multilingual and crosslingual fact-checked claim retrieval). The system is a TF-IDF retriever with scikit-learn's TfidfVectorizer; the authors experiment with the max_features parameter (10,000 vs. 15,000) and with word/character analyzers, and compare against several neural baselines (XLM-R, FastText, distilBERT, T5, E5) on the development set. The selected configuration, word-level TF-IDF with max_features=15,000, is reported to achieve average success@10 of 0.78 on the development set and 0.69 on the test set across ten languages, ranking 23rd of 28 teams. The paper's broader claim is that a carefully tuned TF-IDF system remains a competitive baseline in limited-compute settings, while acknowledging a large gap to the top neural system (0.96).

Significance. If the reported numbers are taken at their official evaluation value, the paper provides a useful reference point for traditional retrieval methods in a multilingual fact-checking setting. Its strengths are that the headline development and test scores come from the official SemEval evaluation, the system description is concrete enough to reproduce with available code, and the comparison across several TF-IDF configurations and neural baselines is informative for practitioners with constrained resources. The main limitation is that the paper's load-bearing interpretive claims about the optimality of the 15,000-feature configuration and about performance on higher-resource languages are not supported by the evidence as presented. The paper would be acceptable after a revision that either supplies the missing test-set comparison or carefully rephrases the claims to match the reported tables.

major comments (3)
  1. [§5.1, Table 2; §6, Table 3] The central claim that word-level TF-IDF with max_features=15,000 is the best configuration rests on a development-set margin of only 0.0127 average success@10 over the 10,000-feature variant (0.7757 vs. 0.7630). Table 2 shows that this margin is not spread across languages: French, Arabic, Malay, and Thai have identical scores for the 10K and 15K variants, while the improvements come from English, Spanish, German, and Portuguese. Table 3 shows that exactly these languages exhibit large, heterogeneous development-to-test drops (Spanish 0.836 to 0.546, Portuguese 0.828 to 0.558, English 0.613 to 0.452). Since no test-set scores for the 10K configuration are reported, the claim that 15K is the best-performing configuration is not supported; the choice may be a development-set artifact. The authors should either provide a test-set comparison of the 10K and 15K configurations or rephrase the abstract and Section 6 to say that 15K was the best configuration on the development set only.
  2. [Abstract; §6, Table 3] The abstract states that the system 'showed stronger performance on higher-resource languages,' but this is contradicted by the test-set results in Table 3. English, the highest-resource language, has one of the lowest scores (0.452), while Thai, the lowest-resource language in the training distribution, has the highest per-language score (0.8415). Section 6 also says 'TF-IDF performed well on English (0.6130) and Spanish (0.8358),' quoting development-set numbers without labeling them as such; these values obscure the test-set outcomes. This interpretive claim should be corrected or removed.
  3. [§6, Table 1] The discussion of neural methods states that E5 Large and T5 had 'more errors in lower-resource languages (e.g., Malay and Thai),' but Table 1 shows the opposite for E5 on the development set: Thai (0.9048) and Malay (0.7810) are among E5's best per-language scores, and Arabic (0.8205) is also high, while English (0.4351) and German (0.4096) are lower. This statement is not supported by the paper's own table and should be revised or qualified.
minor comments (4)
  1. [§5.1, Table 2] Table 2 labels the character-based rows as 'C-WB' and 'Char' but does not specify which max_features value was used for those rows. Section 6 refers to 'TF-IDF 15K (Char WB Analyzer)' with average 0.4528, but the reader cannot tell from the table whether this is a 15K configuration or a separate row. Please add the parameter values directly to the row labels.
  2. [§6] The term 'success@K' appears once in the prose ('the best success@K score of 0.78') but the metric throughout the rest of the paper is success@10. Please use the consistent name.
  3. [§7] The sentence 'Future work could incorporate the reviewers' recommendation of a hybrid approach...' refers to unspecified reviewers. In a standalone system description, this should be removed or rephrased so that the suggestion is attributed to the authors' own analysis rather than to anonymous review comments.
  4. [Table 3] The table header 'Unseen Test Set Scores' is clear, but the caption does not note that the dev-set numbers in Table 2 and the test-set numbers in Table 3 are not directly comparable because the test set includes two surprise languages (Polish and Turkish) and different amounts of data per language. A one-sentence caveat in the caption would help readers interpret the dev-to-test shifts.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found; the system description reports empirically measured dev/test scores with no derivation that reduces to its inputs.

full rationale

This is a system-description paper, not a derivation. The reported results are direct evaluations of a TF-IDF retriever on the task's development and test splits. The choice of max_features=15,000 and word-level tokenization was made from development-set comparisons (Tables 1 and 2), and the test-set scores in Table 3 are independent in the sense that the paper states only the task configuration file, not the test labels, was used for the final system ('For the final system, only task.json, which is a task configuration file was used to ensure that the test data remained unseen'). No fitted parameter is renamed as a prediction, no equation is derived from an equivalent input, and no load-bearing claim rests on a self-citation: the only citations are standard external references for TF-IDF, multilingual models, and the shared task itself. The mild methodological weakness that the development average is reported for the configuration selected because of that same development average is a standard model-selection consideration, not circularity, because the held-out test average provides an independent check. Therefore the circularity score is 0.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The system makes no theoretical contribution; its operating assumptions are standard: TF-IDF cosine similarity is a usable relevance signal, the task's success@10 harness is valid, and the two tuned parameters (max_features=15,000, analyzer='word') were selected on the development set. No new entities are invented.

free parameters (2)
  • max_features (vocabulary size) = 15,000
    Selected by a development-set scan; 15K surpassed 10K by 0.0127 average success@10 (Table 2), and the paper assumes this choice transfers to the test set.
  • analyzer = word
    Chosen by development-set comparison against 'char' and 'char_wb' analyzers (Section 5.1, Table 2).
assumptions (3)
  • domain assumption TF-IDF vector similarity is an adequate relevance signal for claim retrieval in all ten languages
    The whole system rests on bag-of-words cosine similarity ranking fact checks, with no semantic modeling attempted (Section 4.2).
  • domain assumption The task's success@10 metric and the official evaluation harness measure the intended retrieval quality
    The authors rely entirely on the organizers' evaluation rather than any independent evaluation (Sections 2 and 6).
  • domain assumption scikit-learn TfidfVectorizer defaults (token pattern, preprocessing, IDF smoothing) behave as standard
    The pipeline inherits library defaults without modification, and these choices are not tested (Section 4.2).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Duluth at SemEval-2025 Task 7: TF-IDF with Optimized Vector Dimensions for Multilingual Fact-Checked Claim Retrieval." pith.science (2026). https://pith.science/paper/46S37PGY

@misc{pith2026250512616,
  author       = {Pith},
  title        = {Pith review of: Duluth at SemEval-2025 Task 7: TF-IDF with Optimized Vector Dimensions for Multilingual Fact-Checked Claim Retrieval},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/46S37PGY}},
  note         = {Machine review of arXiv:2505.12616}
}
read the original abstract

This paper presents the Duluth approach to the SemEval-2025 Task 7 on Multilingual and Crosslingual Fact-Checked Claim Retrieval. We implemented a TF-IDF-based retrieval system with experimentation on vector dimensions and tokenization strategies. Our best-performing configuration used word-level tokenization with a vocabulary size of 15,000 features, achieving an average success@10 score of 0.78 on the development set and 0.69 on the test set across ten languages. Our system showed stronger performance on higher-resource languages but still lagged significantly behind the top-ranked system, which achieved 0.96 average success@10. Our findings suggest that though advanced neural architectures are increasingly dominant in multilingual retrieval tasks, properly optimized traditional methods like TF-IDF remain competitive baselines, especially in limited compute resource scenarios.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

11 extracted references · 5 linked inside Pith

  1. [1]

    Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzm \'a n, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. https://doi.org/10.18653/v1/2020.acl-main.747 Unsupervised cross-lingual representation learning at scale . In Proceedings of the 58th Annual Meeting of the Association for Comp...

    doi:10.18653/v1/2020.acl-main.747 2020

  2. [2]

    Khapra, Anoop Kunchukuttan, and Pratyush Kumar

    Sumanth Doddapaneni, Gowtham Ramesh, Mitesh M. Khapra, Anoop Kunchukuttan, and Pratyush Kumar. 2021. https://arxiv.org/abs/2107.00676 A primer on pretrained multilingual language models . arXiv preprint. Preprint, arXiv:2107.00676

    arXiv 2021

  3. [3]

    Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. 2018. Learning word vectors for 157 languages. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2018)

    2018

  4. [4]

    Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, H \'e rve J \'e gou, and Tomas Mikolov. 2016. Fasttext.zip: Compressing text classification models. arXiv preprint arXiv:1612.03651

    arXiv 2016

  5. [5]

    Zhao, Yi Luan, Keith B

    Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernández Ábrego, Ji Ma, Vincent Y. Zhao, Yi Luan, Keith B. Hall, Ming-Wei Chang, and Yinfei Yang. 2021. https://arxiv.org/abs/2112.07899 Large dual encoders are generalizable retrievers . Preprint, arXiv:2112.07899

    arXiv 2021

  6. [6]

    Qiwei Peng, Robert Moro, Michal Gregor, Ivan Srba, Simon Ostermann, Marian Simko, Juraj Podroužek, Matúš Mesarčík, Jaroslav Kopčan, and Anders Søgaard. 2025. Semeval-2025 task 7: Multilingual and crosslingual fact-checked claim retrieval. In Proceedings of the 19th International Workshop on Semantic Evaluation, SemEval 2025, Vienna, Austria

    2025

  7. [7]

    Matúš Pikuliak, Ivan Srba, Robert Moro, Timo Hromadka, Timotej Smoleň, Martin Melišek, Ivan Vykopal, Jakub Simko, Juraj Podroužek, and Maria Bielikova. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.1027 Multilingual previously fact-checked claim retrieval . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, page...

    doi:10.18653/v1/2023.emnlp-main.1027 2023

  8. [8]

    Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2020. https://arxiv.org/abs/1910.01108 Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter . Preprint, arXiv:1910.01108

    arXiv 2020

Show all 11 references
  1. [9]

    Shaden Shaar, Nikolay Babulkov, Giovanni Da San Martino, and Preslav Nakov. 2020. https://doi.org/10.18653/v1/2020.acl-main.332 That is a known lie: Detecting previously fact-checked claims . In Proceedings of the 58th Annual Meeting of the Association for Computational Lingui...

    2020 doi

  2. [10]

    Karen Sparck Jones. 1972. A statistical interpretation of term specificity and its application in retrieval. Journal of documentation, 28(1):11--21

    1972

  3. [11]

    Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024. Multilingual e5 text embeddings: A technical report. arXiv preprint arXiv:2402.05672

    2024 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.