{"id":"ad63df06-81f4-4bc4-b6cc-862c4f9a4332","arxiv_id":"2412.00539","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"TextClass Benchmark applies a continuous Elo and Meta-Elo rating to LLMs for social-science text classification; the first snapshot covers toxicity detection in Chinese, English, German, and Russian.","lead":"A new project ranks AI text classifiers on social-science tasks using a chess-style Elo rating and publishes its first results on toxicity detection in four languages. The first snapshot finds large language differences in model quality, but one of its headline claims is contradicted by its own figure.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Cross-language test sets are not shown to be comparable, so the Chinese F1 gap and GPT-4o's Meta-Elo lead may be dataset artifacts.","rationale":"The reader's weakest_assumption points to exactly this dataset comparability issue, and I agree it is the primary risk. The first-cycle leaderboard is internally coherent as a within-language ranking, but the cross-language claims—the language hierarchy and the Meta-Elo winner—require comparable test sets. Nothing in Section 3 establishes comparability: the sources differ, the label balance is only described as 'balanced,' and no item-difficulty statistics are reported. Since the language weights in Equation 6 are explicit policy choices, they are less dangerous by themselves; they can be updated. The empirical question of test-set comparability, however, is not addressed and would change the interpretation of the results if it failed. A difficulty-calibration baseline is the cleanest way to test this. I would keep the CONDITIONAL verdict: the benchmark infrastructure is useful and the within-language Elo rankings are plausible, but the headline cross-language conclusion needs the calibration evidence before it can be accepted as stated.","tokens_in":8573,"tokens_out":9475,"duration_ms":93002,"concrete_test":"Fine-tune XLM-RoBERTa-large (or mBERT) on each language's training split with identical hyperparameters and evaluate on the corresponding test split. If the Chinese baseline F1 is also far below English (gap comparable to the LLM gap), the test sets are intrinsically harder and the cross-language hierarchy in Section 4 is a dataset artifact; if the baseline obtains near-English F1, the LLM gap reflects a genuine language-capability difference. As a secondary check, recompute Meta-Elo with equal language weights and with weights derived from the baseline difficulty gaps; GPT-4o should remain top-1 under both for the 'best model' claim to be robust.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing assumption is in Section 3: a 'balanced sample of 5,000 observations per country' is drawn from different source corpora (Jigsaw/Unitary for English, DeTox/GemEval for German, unspecified sources for Chinese and Russian), and the paper then treats the resulting F1 differences as language capability. Section 4 reports average F1 of 0.952 (English) versus 0.346 (Chinese) and a 'hierarchy by language.' Because GPT-4o's top Meta-Elo position is driven by the highest F1 in Chinese (0.751) and Russian (0.952), any dataset-level difference in item difficulty, label noise, or topic composition across the four test sets changes the headline. The hand-set language weights in Equation 6 (1.3, 1.1, 1.4) are a second-order manifestation: they encode the same unverified difficulty assumption into the ranking. Absent evidence that the sets are equated (same label balance, similar ambiguity, comparable annotation quality), 'best model across leaderboards' is not a well-defined comparison.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes TextClass Benchmark, an ongoing Elo-based benchmark for LLMs and BERT-family models on text classification tasks relevant to social sciences. It introduces a tailored Elo rating system based on pairwise F1 comparisons and a Meta-Elo aggregation that weights task complexity, language data scarcity, normalized absolute performance, and cycle count. The first cycle covers toxicity/incivility classification in Chinese, English, German, and Russian for 24 models with 96 evaluations, reporting per-language accuracy, precision, recall, F1, Elo, and Meta-Elo. The headline results are a language hierarchy (English and Russian highest, Chinese markedly lowest), GPT-4o as the best model in Meta-Elo, and a reported quadratic relationship between weighted F1 and Meta-Elo with R2 = 0.981.","tokens_in":8844,"tokens_out":5643,"duration_ms":147252,"significance":"The infrastructure is timely and potentially useful: a transparent, continuously updated leaderboard with zero-temperature settings, fixed test sets, and public code and data could help social scientists choose models for text annotation. The explicit formulas for Elo and Meta-Elo make the method reproducible. However, the current snapshot's central comparative claims are not yet supported: the cross-lingual test sets are not shown to be comparable, the reported R2 = 0.981 is partly an artifact of F1 appearing on both sides of the relationship, and one stated finding is contradicted by the paper's own Figure 1. These issues are substantial but fixable with additional analysis and careful reframing.","major_comments":[{"comment":"Section 4 states that 'another interesting finding is that all LLMs outperform more classical transformer approaches such as Perspective API,' but Figure 1 shows Perspective 0.80 (ZH) and Perspective 0.80 (DE) placed above GPT-4o (ZH, F1 = 0.751) and above many open-weight LLMs in the Chinese and German blocks. Because Perspective API is included in the leaderboard and outperforms the flagship LLM in Chinese and German, the sentence as written is false. Please restrict the claim to the languages where it holds or report per-language comparisons; this matters because the abstract and introduction position the benchmark as demonstrating LLM superiority over older tools.","section":"Section 4, Figure 1"},{"comment":"The high R2 between weighted F1 and Meta-Elo is expected by construction. Equation (6) defines the Meta-Elo weight w_j to include w_F1, the normalized F1-Score, and the x-axis 'weighted F1-Score' is computed by 'emulating the abovementioned process' (Section 4). Thus F1 contributes to both variables, so R2 = 0.981 cannot be read as independent validation that Meta-Elo summarizes absolute performance. Report the correlation between F1 and a version of Meta-Elo computed without the w_F1 weight, or with out-of-sample weights, and avoid presenting the quadratic fit as evidence of validity.","section":"Section 4, Figure 2; Equations (5)-(6)"},{"comment":"The four test sets are drawn from different source corpora (e.g., Jigsaw/Unitary for English, DeTox and GemEval for German; Chinese and Russian sources are not cited) and are described only as a 'balanced sample of 5,000 observations per country.' The paper then interprets average F1 differences (0.952 English vs. 0.346 Chinese) as a 'hierarchy by language' and uses hand-set language weights (Section 2.2) that presuppose that Chinese and Russian are intrinsically harder. Without evidence that the sets are equated in label distribution, annotation quality, and item difficulty, the cross-language ranking is not well-defined. At minimum, provide per-language label balance, validation statistics, and dataset descriptions; otherwise present each language as a separate leaderboard and drop the cross-lingual hierarchy claim.","section":"Section 3 and Section 2.2"},{"comment":"The ranking depends on several free parameters (K = 40, draw threshold 0.05, language weights 1.3/1.1/1.4, cycle weight 1 + log(cycle + 1)) whose values are asserted without justification or sensitivity analysis. Since the top Meta-Elo position is driven by GPT-4o's Chinese and Russian F1, it is important to show that the conclusion is robust to reasonable alternatives, e.g., K = 20 or 60, draw thresholds of 0.03/0.07, and equal language weights. The paper should also report uncertainty (e.g., bootstrap confidence intervals for F1 and Elo), as many models are separated by less than the 0.05 draw threshold.","section":"Section 2.1-2.2"}],"minor_comments":[{"comment":"The heading 'A venues' should be 'Avenues,' and the text uses 'wights' where 'weights' is intended.","section":"Section 5"},{"comment":"The text says '5,000 observations per country,' but the data are per language; use 'per language' to avoid confusion.","section":"Section 3"},{"comment":"The summation symbol in Equation (5) is typeset as 'nX'; please replace with proper LaTeX summation notation.","section":"Equation (5)"},{"comment":"The Chinese and Russian data sources are not identified; provide citations or URLs for these subsets, as the reproducibility of the benchmark depends on them.","section":"Section 3"},{"comment":"The Perspective API entries are labeled with thresholds (0.80, 0.70, 0.60, 0.55) that are not explained in the text or caption; define what these numbers represent.","section":"Figure 1"},{"comment":"The planned weights for Arabic, Hindi, and Spanish are listed but not used in this cycle; clarify that these are prospective values rather than results from the current snapshot.","section":"Footnote 1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is better framed as an infrastructure/benchmark proposal than as a definitive comparison of LLM quality across languages. The author may want to separate the per-language leaderboards from the Meta-Elo aggregation and to avoid claiming cross-lingual fairness until data equivalence is demonstrated. The self-citations to the author's previous benchmark paper are appropriate, but the manuscript should be clear that the current snapshot is a preliminary evaluation rather than a settled comparative verdict."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a genuinely useful benchmarking effort for social scientists, with open materials and a sensible plan for continuous Elo ratings, but the first snapshot overclaims. The stated finding that all LLMs beat Perspective API is false against the paper's own Figure 1, and the cross-language ranking (especially the Chinese gap) rests on test sets from different sources with no evidence of comparable difficulty.\n\nWhat's actually new: not the Elo adaptation itself — that exists in LMArena-style leaderboards — but the application to social-science text classification, the Meta-Elo weighting, and the first concrete measurements of 24 models across Chinese, English, German, and Russian on toxicity classification. The authors ship code and data, and the discussion section shows they know the language weights are provisional. That is real credit.\n\nSoft spots: the Perspective claim is a factual error. The cross-language comparison is the bigger problem: the four test sets come from different corpora (Jigsaw/Unitary for English, DeTox/GemEval for German, unspecified for Chinese and Russian), and the paper gives no evidence they are equally hard. So the 'hierarchy by language' and GPT-4o's top Meta-Elo position may be dataset artifacts. The hand-set language weights (Chinese 1.3, German 1.1, Russian 1.4) encode the same unverified assumption. Also, the R2 = 0.981 between Meta-Elo and weighted F1 is partly by construction, since normalized F1 is one of the weights in Meta-Elo — not an independent finding. Minor: no confidence intervals and no stability checks across runs.\n\nWho this is for: social scientists doing text annotation who need a practical model-selection tool. It deserves a serious referee. The infrastructure is important and the flaws are correctable. I'd recommend the authors fix the Perspective claim, add comparability evidence for the test sets, and reframe the Meta-Elo–F1 correlation as a sanity check rather than a result. With those changes, this becomes a solid contribution.","headline":"Useful benchmark infrastructure for social-science LLM evaluation, but the first snapshot overclaims: the Perspective claim is wrong and the cross-language ranking is not yet supported by the data.","tokens_in":9343,"tokens_out":3260,"would_cite":false,"duration_ms":42204,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A continuous Elo rating system ranks GPT-4o as the best of 24 models on multilingual toxicity classification.","keywords":["large language models","Elo rating","text classification","toxicity detection","multilingual evaluation","benchmarking","Meta-Elo","computational social science"],"falsifier":"Build a Chinese toxicity test set matched to the English set in source type, label balance, annotation guidelines and difficulty, then rerun the same models; if GPT-4o's Chinese F1 rises near its English F1, the reported language hierarchy is an artifact of test-set difficulty rather than model capability.","tokens_in":8340,"feed_emoji":"🏆","tokens_out":6403,"duration_ms":55410,"temperature":0.7,"pith_summary":"The paper establishes a continuous benchmarking protocol, TextClass Benchmark, that rates large language models on text classification tasks using a chess-style Elo system, then presents its first cycle on toxicity detection in four languages. The author argues that Elo ratings updated over repeated cycles give a fair, dynamic comparison of closed and open models, and that a weighted Meta-Elo summarises the domain-specific leaderboards. In the first snapshot, GPT-4o is the top model overall, open models such as Qwen 2.5 are close, and every tested LLM beats the older Perspective API. A strong quadratic fit between Meta-Elo and weighted F1 (R2 = 0.981) supports treating Meta-Elo as a compressed ranking that tracks absolute performance. The results also reveal a large gap by language, with models doing well in English and Russian but far worse in Chinese, a gap the paper attributes to language difficulty.","feed_headline":"GPT-4o tops multilingual toxicity ranking of 24 LLMs","feed_subtitle":"Continuous Elo scores across four languages put GPT-4o first and expose a steep gap in Chinese.","key_machinery":"The load-bearing mechanism is the adapted Elo rating system: models are paired in round-robin matches, the winner is decided by F1-score if the margin exceeds 0.05, ratings update with K = 40 from a 1,500 baseline, and a Meta-Elo combines the language-specific Elo scores using weights for task complexity, language data scarcity, absolute F1 and number of cycles. The same pairwise logic turns raw accuracy, precision, recall and F1 into comparable relative ratings that can be updated as new models and test sets are added.","core_discovery":"The central claim is that the TextClass Benchmark's first cycle provides a fair, cross-lingual snapshot of 24 models on binary toxicity classification, with GPT-4o (2024-11-20) as the best model across leaderboards. Using zero-shot prompting at temperature zero on balanced samples of 5,000 observations per language, the author reports that models reach average F1 of 0.952 in English, 0.910 in Russian, 0.814 in German and only 0.346 in Chinese; GPT-4o posts the top Chinese score (0.751) and the top Russian score (0.952), while Nous Hermes 2 Mixtral leads English (0.977) and Hermes 3 70B leads German (0.848). All evaluated LLMs outperform the Perspective API. The paper further claims that Meta-Elo, a weighted combination of language-specific Elo scores, is a valid compressed measure of relative performance because it relates to weighted F1 through a quadratic curve with R2 = 0.981.","pith_inferences":["The paper's four test sets come from different source corpora, so the headline language gap may partly reflect dataset difficulty rather than model ability; an equivalent-difficulty test set in Chinese would settle this.","The hand-set language weights (Chinese 1.3, Russian 1.4, German 1.1) embed a judgment about scarcity and complexity; changing those weights could reorder the Meta-Elo ranking.","The same Elo-plus-Meta-Elo protocol could be transplanted to other social-science classification tasks such as policy agenda coding or misinformation detection, giving a general-purpose leaderboard for the discipline.","If later cycles show the quadratic Meta-Elo versus F1 relationship degrading, that divergence would itself be diagnostic of when relative Elo ratings stop reflecting absolute performance."],"forward_implications":["A researcher selecting a model for incivility classification in these four languages gets a clear default: GPT-4o currently leads the combined Meta-Elo ranking.","Meta-Elo can be read as a single number that tracks absolute performance, so future cycles can compare models added at different times on a common scale.","The wide gap between English/Russian and Chinese performance means a model's multilingual claims should be checked per language before deployment.","Because every tested LLM beat Perspective API, the benchmark suggests classical toxicity-classification tools have been overtaken for this zero-shot task.","The continuous design will make model drift and obsolescence measurable: a model's rating now has a temporal trajectory, not just a one-off accuracy."],"supporting_citations":[{"why":"Supplies the TextDetox shared-task framework and multilingual toxicity data used to assemble the first-cycle datasets.","marker":"(Dementieva et al., 2024)"},{"why":"Provides the DeTox German dataset used as a source of German toxicity observations.","marker":"(Demus et al., 2022)"},{"why":"Provides the GermEval shared-task data used for German toxicity evaluation.","marker":"(Risch et al., 2021)"},{"why":"Provides the Detoxify/Jigsaw English toxicity data and the Perspective API evaluated as a baseline.","marker":"(Hanu and Unitary, 2020)"},{"why":"Supplies the prompt strategy and toxicity/incivility classification setup that the benchmark adopts in zero-shot form.","marker":"(González-Bustamante, 2024)"}],"fun_headline_variants":["GPT-4o tops new multilingual Elo for 24 LLMs","ChatGPT-4o leads social science text Elo rollout","Toxicity Elo: GPT-4o dominates across four languages","TextClass Elo rates LLMs; GPT-4o ranks first","Multilingual Elo: GPT-4o best of 24 LLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the four balanced test sets are comparable in difficulty, so that cross-language differences in F1 reflect model language ability rather than dataset-specific properties.","fun_headline_variants_meta":{"raw":{"variants":["GPT-4o tops new multilingual Elo for 24 LLMs","ChatGPT-4o leads social science text Elo rollout","Toxicity Elo: GPT-4o dominates across four languages","TextClass Elo rates LLMs; GPT-4o ranks first","Multilingual Elo: GPT-4o best of 24 LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000246,"raw_usage":{"total_tokens":1537,"prompt_tokens":942,"completion_tokens":595,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":558,"completion_tokens_details":{"reasoning_tokens":500}},"tokens_in":558,"tokens_out":595,"duration_ms":5571,"temperature":1.0,"reasoning_tokens":500,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T05:14:52.939662+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a Chinese toxicity test set matched to the English set in source type, label balance, annotation guidelines and difficulty, then rerun the same models; if GPT-4o's Chinese F1 rises near its English F1, the reported language hierarchy is an artifact of test-set difficulty rather than model capability.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the TextDetox shared-task framework and multilingual toxicity data used to assemble the first-cycle datasets."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the GermEval shared-task data used for German toxicity evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Detoxify/Jigsaw English toxicity data and the Perspective API evaluated as a baseline."}],"review_version":1}