{"id":"86402558-6a63-4079-91d2-259079163f6b","arxiv_id":"2505.12616","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":2.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Word-level TF-IDF with a 15,000-term vocabulary reaches 0.69 average success@10 on the SemEval-2025 Task 7 multilingual test set, far behind the top system's 0.96.","lead":"A shared task report showing that a tuned TF-IDF retrieval system scores 0.69 success@10 on a ten-language fact-checked claim retrieval test set, ranking 23rd of 28 teams. It documents that a cheap statistical baseline remains competitive when compute is limited, though 0.27 below the top neural system.","discovery_kind":"incremental","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported optimality of the 15K TF-IDF configuration rests entirely on a 0.0127 dev-set average margin over 10K; the paper never shows a test-set comparison, and the languages that drive the margin are exactly those with the largest dev-to-test drops.","rationale":"The paper is a shared-task system description; its main factual claim is the configuration and its scores. The reader's strongest_claim is the Abstract/Section 6 statement, and the load-bearing condition is that choosing 15K over 10K improves the reported result. Table 2 shows the improvement is real on dev but only 0.0127 on average, and only four languages contribute while the other four are identical. Because no test-set ablation is reported, the only way to know whether 15K was actually best on test is to run it. The dev-to-test instability in Table 3 is severe and concentrated in the languages that create the dev margin: Spanish and Portuguese both drop by roughly 0.28, and the surprise languages were not present in dev. This is not an internal inconsistency; it is an unsupported generalization from dev to test. It does not invalidate the reported official scores, which match between the abstract and Table 3 and come from the task leaderboard, so the verdict should remain CONDITIONAL rather than moving to REJECT: the score is credible, but the optimality claim needs a test-set check. The GitHub code makes the proposed ablation feasible. I do not see a stronger objection; the other issues, such as the claim about stronger performance on higher-resource languages, are interpretive and less load-bearing.","tokens_in":6736,"tokens_out":4359,"duration_ms":46196,"concrete_test":"Using the authors' released repository, reproduce the 10K and 15K TF-IDF configurations exactly as described in Sections 4.2 and 5.1, run both on the official test posts against the full fact-check corpus, and score with the task's success@10 script. Compare per-language and average scores, and report 10K vs 15K on Polish and Turkish specifically. If average(10K) is greater than or equal to average(15K), or the difference is within the 0.0127 dev margin, the claim that 15K is the best configuration is not supported by the evidence.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the best configuration is word-level TF-IDF with max_features=15,000 (Abstract and Section 6), backed by dev averages in Table 2: 15K avg 0.7757 vs 10K avg 0.7630. That 0.0127 margin is not spread across languages: in Table 2, eng, spa, deu, and por improve (por +0.0331, spa +0.0293, deu +0.0241, eng +0.0147), while fra, ara, msa, and tha are identical between 10K and 15K. The paper provides no test-set scores for the 10K configuration, so selecting 15K assumes the dev ranking transfers. Table 3 shows large, heterogeneous dev-to-test movement for exactly the languages that favored 15K: Spanish falls from 0.836 to 0.546 and Portuguese from 0.828 to 0.558, while Thai drops from 0.881 to 0.842. Because the two surprise test languages (Polish, Turkish) were absent from dev, the test average cannot validate the dev-based choice. Thus the Abstract and Section 6 statement that the best-performing configuration used 15,000 features and achieved 0.78/0.69 is credible as a reported score, but the claim that 15K is optimal relative to 10K is load-bearing and unsupported without a test-set comparison.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes the University of Minnesota Duluth submission to SemEval-2025 Task 7 (multilingual and crosslingual fact-checked claim retrieval). The system is a TF-IDF retriever with scikit-learn's TfidfVectorizer; the authors experiment with the max_features parameter (10,000 vs. 15,000) and with word/character analyzers, and compare against several neural baselines (XLM-R, FastText, distilBERT, T5, E5) on the development set. The selected configuration, word-level TF-IDF with max_features=15,000, is reported to achieve average success@10 of 0.78 on the development set and 0.69 on the test set across ten languages, ranking 23rd of 28 teams. The paper's broader claim is that a carefully tuned TF-IDF system remains a competitive baseline in limited-compute settings, while acknowledging a large gap to the top neural system (0.96).","tokens_in":6937,"tokens_out":3724,"duration_ms":39791,"significance":"If the reported numbers are taken at their official evaluation value, the paper provides a useful reference point for traditional retrieval methods in a multilingual fact-checking setting. Its strengths are that the headline development and test scores come from the official SemEval evaluation, the system description is concrete enough to reproduce with available code, and the comparison across several TF-IDF configurations and neural baselines is informative for practitioners with constrained resources. The main limitation is that the paper's load-bearing interpretive claims about the optimality of the 15,000-feature configuration and about performance on higher-resource languages are not supported by the evidence as presented. The paper would be acceptable after a revision that either supplies the missing test-set comparison or carefully rephrases the claims to match the reported tables.","major_comments":[{"comment":"The central claim that word-level TF-IDF with max_features=15,000 is the best configuration rests on a development-set margin of only 0.0127 average success@10 over the 10,000-feature variant (0.7757 vs. 0.7630). Table 2 shows that this margin is not spread across languages: French, Arabic, Malay, and Thai have identical scores for the 10K and 15K variants, while the improvements come from English, Spanish, German, and Portuguese. Table 3 shows that exactly these languages exhibit large, heterogeneous development-to-test drops (Spanish 0.836 to 0.546, Portuguese 0.828 to 0.558, English 0.613 to 0.452). Since no test-set scores for the 10K configuration are reported, the claim that 15K is the best-performing configuration is not supported; the choice may be a development-set artifact. The authors should either provide a test-set comparison of the 10K and 15K configurations or rephrase the abstract and Section 6 to say that 15K was the best configuration on the development set only.","section":"§5.1, Table 2; §6, Table 3"},{"comment":"The abstract states that the system 'showed stronger performance on higher-resource languages,' but this is contradicted by the test-set results in Table 3. English, the highest-resource language, has one of the lowest scores (0.452), while Thai, the lowest-resource language in the training distribution, has the highest per-language score (0.8415). Section 6 also says 'TF-IDF performed well on English (0.6130) and Spanish (0.8358),' quoting development-set numbers without labeling them as such; these values obscure the test-set outcomes. This interpretive claim should be corrected or removed.","section":"Abstract; §6, Table 3"},{"comment":"The discussion of neural methods states that E5 Large and T5 had 'more errors in lower-resource languages (e.g., Malay and Thai),' but Table 1 shows the opposite for E5 on the development set: Thai (0.9048) and Malay (0.7810) are among E5's best per-language scores, and Arabic (0.8205) is also high, while English (0.4351) and German (0.4096) are lower. This statement is not supported by the paper's own table and should be revised or qualified.","section":"§6, Table 1"}],"minor_comments":[{"comment":"Table 2 labels the character-based rows as 'C-WB' and 'Char' but does not specify which max_features value was used for those rows. Section 6 refers to 'TF-IDF 15K (Char WB Analyzer)' with average 0.4528, but the reader cannot tell from the table whether this is a 15K configuration or a separate row. Please add the parameter values directly to the row labels.","section":"§5.1, Table 2"},{"comment":"The term 'success@K' appears once in the prose ('the best success@K score of 0.78') but the metric throughout the rest of the paper is success@10. Please use the consistent name.","section":"§6"},{"comment":"The sentence 'Future work could incorporate the reviewers' recommendation of a hybrid approach...' refers to unspecified reviewers. In a standalone system description, this should be removed or rephrased so that the suggestion is attributed to the authors' own analysis rather than to anonymous review comments.","section":"§7"},{"comment":"The table header 'Unseen Test Set Scores' is clear, but the caption does not note that the dev-set numbers in Table 2 and the test-set numbers in Table 3 are not directly comparable because the test set includes two surprise languages (Polish and Turkish) and different amounts of data per language. A one-sentence caveat in the caption would help readers interpret the dev-to-test shifts.","section":"Table 3"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Straight to it: this is a standard shared-task system description—TF-IDF with a two-parameter scan over max_features and analyzer, evaluated on the SemEval-2025 Task 7 dev and test sets. The headline numbers (0.78 dev, 0.69 test S@10) are credible because they come from the official leaderboard and are consistent between the abstract and Table 3. The paper also compares several neural baselines on dev, and the finding that a tuned TF-IDF outscores them in this low-resource setting is a useful data point for people who need a cheap baseline.\n\nThat is the extent of the novelty. No new technique, no new dataset, no surprising result—and the paper doesn't claim otherwise. It is an honest system description, with code and official scores.\n\nThe soft spots are in the interpretation, not the measurements. The stress-test note is right: the claim that 15,000 features is the best configuration rests on a 0.0127 dev average margin over 10K, and the paper never gives test-set scores for the 10K configuration. The dev-to-test movement in Table 3 is large and heterogeneous—Spanish drops from 0.836 to 0.546, Portuguese from 0.828 to 0.558—and those are exactly the languages where the 15K margin over 10K was largest. So the 'optimal' label is load-bearing and unsupported. The authors should either provide 10K test scores or rephrase to 'we selected 15K based on dev performance.'\n\nThere's also a contradiction about language resources. The abstract says the system performed stronger on higher-resource languages, but the test scores show English (the highest-resource language) at 0.452, while French and Arabic are higher. The intro says the opposite—'performed better on French and Arabic, but struggled with English and Spanish.' That inconsistency needs fixing.\n\nAnd the 'remain competitive baselines' framing is a bit generous given the 0.69 average versus the top system's 0.96. It's a reasonable lightweight baseline, but not a competitive one.\n\nBottom line: this paper is appropriate for a shared-task venue. It should go to peer review, but with a request for revision to address the 10K issue and the language-resource contradiction. A serious editor shouldn't desk-reject it; the base is solid, the analysis is sloppy in places.","headline":"A credible shared-task baseline paper with an unsubstantiated 'optimal' parameter claim and a contradiction in the language-resource narrative.","tokens_in":7549,"tokens_out":3137,"would_cite":false,"duration_ms":30420,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A tuned TF-IDF retriever reaches 0.69 success@10 across ten languages for multilingual fact-checked claim retrieval, the paper reports.","keywords":["TF-IDF","multilingual claim retrieval","fact-checked claim retrieval","success@10","vector-size optimization","word-level tokenization","baseline retrieval","low-resource languages"],"falsifier":"Rerun the released pipeline on the official test split with max_features set to 10,000 and 15,000 and compare average success@10. If 15,000 does not beat 10,000 on the test set, or the reported 0.69 average is not reproducible, the paper's claim that 15,000 is the optimized configuration fails.","tokens_in":6448,"feed_emoji":"🌐","tokens_out":7787,"duration_ms":72914,"temperature":0.7,"pith_summary":"This paper argues that a deliberately tuned TF-IDF retriever remains a competitive baseline for multilingual fact-checked claim retrieval: with word-level tokenization and a vocabulary capped at 15,000 features, it reaches 0.78 average success@10 on the development set and 0.69 on the test set over ten languages. The score matters because the same paper finds this configuration beats several neural embedding baselines in its own comparisons, while still trailing the top system's 0.96. The authors present the result as evidence that optimized statistical retrieval is still useful, especially when compute is limited.","feed_headline":"Tuned TF-IDF hits 0.69 success@10 across ten languages","feed_subtitle":"Cheap word statistics still compete with heavy neural models on multilingual fact-check retrieval.","key_machinery":"The load-bearing component is a term frequency-inverse document frequency (TF-IDF) vectorizer whose two knobs are the analyzer and the vocabulary size. The default word analyzer tokenizes each fact check and post into words, and max_features=15,000 keeps only the 15,000 highest-frequency terms, producing sparse vectors; retrieval ranks fact checks by similarity of their vectors to the query vector and returns the top ten. The vocabulary cap carries the argument: raising the cap from the default configuration to 10,000 lifts the development average from 0.5513 to 0.7630, and the further step to 15,000 adds another 0.0127 (to 0.7757), whereas switching to character-level analyzers drops the average to 0.46 or below.","core_discovery":"On the monolingual track of the multilingual fact-checked claim retrieval task, the paper's central discovery is that a sparse TF-IDF vector space restricted to the 15,000 most frequent word tokens retrieves a relevant previously fact-checked claim in the top ten results for 78 percent of development queries and 69 percent of test queries on average across ten languages. Per-language test scores range from 0.452 for English to 0.8495 for Malay, with stronger performance on higher-resource languages and clear losses from character-level tokenization. In the paper's experiments on the development split, this configuration outperforms every neural retriever tested, including multilingual E5 and GTR-T5, which reach 0.6494 and 0.6360 respectively, while the top-ranked system on the same test set reaches 0.9601. The paper reads this as evidence that traditional statistical retrieval can still act as a strong baseline and that its remaining gap to neural systems is largest on lower-resource languages.","pith_inferences":["The development-set margin of 15K over 10K is small (0.0127), while several languages swing sharply between dev and test; a reader should not assume the 15K choice is the true optimum until it is confirmed on test data held out from configuration search.","A hybrid retriever that fuses TF-IDF lexical scores with E5 or T5 dense scores could plausibly combine exact-match precision with semantic recall, especially for lower-resource languages where the paper's neural models were weakest; the paper lists this as future work but does not test it.","The success@10 metric rewards recall at a shallow rank; under a different evaluation regime such as mean reciprocal rank or precision-oriented ranking, the ordering of TF-IDF versus dense models might change because dense models can rank relevant claims lower."],"forward_implications":["A 15K word-level TF-IDF retriever is a usable low-cost baseline for monolingual fact-checked claim retrieval, with a test average of 0.69 across English, Spanish, German, Portuguese, French, Arabic, Malay, Thai, Polish, and Turkish.","Raising vocabulary size from 10K to 15K improves development success@10 by about 1.3 percentage points on average, and word-level tokenization is consistently better than character-level for this data.","The gap to the top system (0.69 versus 0.96) quantifies how far a tuned statistical retriever is from current neural systems on this task.","The results support the paper's practical recommendation that TF-IDF is a reasonable choice when compute or fine-tuning resources are limited, because it outperformed the neural baselines the authors ran without any training."],"supporting_citations":[{"why":"Defines the TF-IDF term-weighting scheme that the whole retrieval pipeline is built on.","marker":"Sparck Jones, 1972"},{"why":"Defines the shared task, the data splits, and the success@10 metric used for evaluation.","marker":"Peng et al., 2025"},{"why":"Provides the original multilingual corpus of social media posts and previously fact-checked claims that the task augments.","marker":"Pikuliak et al., 2023"},{"why":"Supplies the XLM-RoBERTa model used as one of the neural retrieval baselines in the comparison.","marker":"Conneau et al., 2020"},{"why":"Supplies the Multilingual E5 dense retriever, the strongest neural comparison on the development set.","marker":"Wang et al., 2024"},{"why":"Supplies the GTR-T5 dual-encoder retriever, another neural baseline that TF-IDF 15K outperforms in the authors' experiments.","marker":"Ni et al., 2021"}],"fun_headline_variants":["Sparse TF-IDF beats neural baselines on multilingual dev","Word-level TF-IDF tops neural baselines on dev set","15k-token TF-IDF rivals neural models at low cost","Optimized TF-IDF: strong multilingual retrieval without deep learning","Cheap TF-IDF edges out neural baselines on dev"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The choice of 15,000 features is based on the development-set average, where it beats 10,000 by 0.0127; the paper assumes this development ranking transfers to the test set, although per-language scores move dramatically between the two splits.","fun_headline_variants_meta":{"raw":{"variants":["Sparse TF-IDF beats neural baselines on multilingual dev","Word-level TF-IDF tops neural baselines on dev set","15k-token TF-IDF rivals neural models at low cost","Optimized TF-IDF: strong multilingual retrieval without deep learning","Cheap TF-IDF edges out neural baselines on dev"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001077,"raw_usage":{"total_tokens":4486,"prompt_tokens":906,"completion_tokens":3580,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":522,"completion_tokens_details":{"reasoning_tokens":3493}},"tokens_in":522,"tokens_out":3580,"duration_ms":24645,"temperature":1.0,"reasoning_tokens":3493,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:30:00.644420+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the released pipeline on the official test split with max_features set to 10,000 and 15,000 and compare average success@10. If 15,000 does not beat 10,000 on the test set, or the reported 0.69 average is not reproducible, the paper's claim that 15,000 is the optimized configuration fails.","supporting_citations":[],"review_version":1}