{"id":"513f4bd2-b3aa-44e2-89e6-efcea511f1af","arxiv_id":"2504.16921","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A community-run benchmark for Iberian languages shows that LLMs underperform on industry-relevant NLP tasks and on Basque and Galician relative to fundamental tasks and other Iberian languages.","lead":"IberBench is a new benchmark for testing large language models on Spanish, Portuguese, Catalan, Basque, Galician, and related varieties, built from 101 existing datasets. It evaluates 23 models and finds they do worse on industry-style tasks than on academic ones, and especially poorly on Basque and Galician.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Aggregation confound in 4.2.1/4.2.3: Basque scores are dominated by a few hard single-language datasets, not by a robust cross-task estimate of language difficulty.","rationale":"The reader's weakest_assumption identifies the same aggregation imbalance as the central risk, and the paper's own text substantiates it: Section 3.3 states that the task-language table is sparse, that Mental Health and Intent Classification have data in a single language, and that 'LLMs might appear weaker or stronger in cross-lingual and cross-task comparisons depending on how well they perform on overrepresented or underrepresented combinations.' Since the conclusion's first two findings are aggregate claims, the imbalanced averaging is precisely what would have to be true for those claims to be valid. The reader's CONDITIONAL verdict is appropriate: the benchmark resource itself is valuable and largely reproducible in infrastructure, but the headline empirical insights should not be treated as settled until the aggregation confound is tested and either controlled or explicitly reframed. I agree with the reader's assessment, with the added emphasis that a matched-dataset/same-task-category recomputation is the single most feasible and informative check, and that until such a check is reported the correctness risk remains medium rather than low.","tokens_in":34078,"tokens_out":1780,"duration_ms":16435,"concrete_test":"Recompute Figures 7 and 8 restricted to task categories present in at least two of Basque, Galician, and Spanish (e.g., MGT Detection, Textual Entailment, and Question Answering). If the Basque/Galician deficit shrinks or disappears on this matched subset, the language-difficulty findings are dataset-coverage artifacts. Also rerun Figure 6 with equal weight per dataset (instead of per sample) to test whether the industry-vs-fundamental gap is driven by the differing composition of the two relevance classes.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The headline findings (i) and (ii) rest on averaging scores across an unbalanced (task, language) matrix, and the paper itself documents the imbalance in Table 4 and Section 3.3, including the admission that the task-language matrix is sparse and that overrepresented combinations can distort cross-lingual and cross-task comparisons. The reader flagged this; the concern is load-bearing because the two most prominent empirical claims are exactly those aggregates. Concretely, the Basque entry is driven by FMTODeu Intent, BHTC Topic, EusProficiency, EusTrivia, QNLI, and MGT tasks, several of which are also among the benchmark's hardest categories (Figure 5), so a low Basque mean may partly reflect task selection, not intrinsic language difficulty. Galician's 12.8k samples come almost entirely from fundamental tasks from Proxecto Nós plus MGT, so its mean mixes a different task distribution than Spanish's 165.2k samples spanning many industry tasks. The paper never reports per-language performance stratified by shared task categories, and never quantifies how much of the language gap survives when comparing only datasets that exist in both languages, or when weighting each (task, language) cell equally rather than by samples. Without such a controlled comparison, the 'Galician and Basque present greater challenges' claim is a correlation with data availability, not an established model-capability result. The same issue affects the industry-vs-fundamental gap, since the two relevance classes draw from different dataset sources with different label counts, prompt formats, and metric types (Section 3.4), so the aggregate comparison in Figure 6 may reflect benchmark construction rather than model capability alone. The concern is not that the benchmark is useless; it is that the strongest empirical claims inherit a confound that the paper explicitly acknowledges but does not resolve.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"IberBench is a multi-language benchmark for evaluating LLMs on Iberian languages (Spanish, Portuguese, Catalan, Basque, Galician, English, and several Spanish varieties), integrating 101 datasets from shared tasks (IberLEF, IberEval, TASS, PAN) and existing benchmarks, organized into 22 task categories split into 'fundamental' and 'industry-relevant' groups. The paper describes a complete evaluation infrastructure: dataset normalization, private hosting, a leaderboard UI, an organization committee, and a modified lm-evaluation-harness that supports sequence labeling, incremental evaluation, and on-premise execution. It reports zero-shot evaluations of 23 LLMs (0.1B–14B parameters) using Macro-F1, ROUGE-1, and seqeval F1, and claims (i) LLMs underperform on industry-relevant tasks relative to fundamental ones, (ii) Galician and Basque are harder than other Iberian languages, (iii) several tasks (lexical borrowing, intent, MGT detection) are close to random, and (iv) in other tasks LLMs beat random but lag shared-task systems. The empirical analysis relies heavily on aggregate means over a highly imbalanced (task, language) matrix, which is documented in Table 4 and acknowledged in Section 3.3.","tokens_in":34415,"tokens_out":7468,"duration_ms":66830,"significance":"The paper delivers a genuinely useful resource: a large, curated, standardized collection of Iberian-language evaluation datasets, many previously scattered or difficult to access, with a reproducible evaluation pipeline and public leaderboard infrastructure. The release of prompts, YAML configs, normalization code, and a caching mechanism is a concrete strength, as is the custom annotation scheme that enables sequence-labeling evaluation with LLMs. The empirical insights, if robust, would be valuable for practitioners choosing models for Iberian languages. However, the headline aggregate conclusions (especially the language-difficulty ranking and the fundamental-vs-industry gap) are not yet supported by the analysis as presented, because the aggregation is confounded by the sparse and imbalanced task-language matrix. The benchmark contribution itself is solid; the analysis needs additional controls or, failing that, appropriately hedged claims.","major_comments":[{"comment":"The claim that \"Galician and Basque present greater challenges than other languages\" rests on unadjusted averages over a highly imbalanced task-language matrix. Table 4 shows that Basque's 42.6k samples are concentrated in difficult fundamental tasks (EusTrivia, EusProficiency, QNLI, BHTC, FMTODeu) and that Galician's 12.8k samples are almost entirely fundamental tasks from Proxecto Nós plus MGT detection, whereas Spanish's 165.2k samples span many more categories. Since the averages in Figures 7 and 8 are not stratified by task category and are not restricted to datasets that exist in multiple languages, the observed language gap could be driven by which datasets happen to be available in each language rather than by intrinsic language difficulty. The manuscript itself acknowledges this risk in Section 3.3 (\"LLMs might appear weaker or stronger in cross-lingual and cross-task comparisons depending on how well they perform on overrepresented or underrepresented combinations\"). The authors should add a controlled comparison—e.g., per-category language means, equal-weighting of (task, language) cells, or a matched-dataset analysis—and temper the claim if the gap does not survive that control.","section":"Section 4.2.3, Figures 7–8, Table 4"},{"comment":"The headline finding that LLMs perform worse on industry-relevant than fundamental tasks is subject to the same aggregation issue. Industry datasets are predominantly Spanish (139.6k of 247.1k industry samples) and include notoriously hard categories (MGT, intent, stance, author profiling), while fundamental datasets include easier categories (Commonsense Reasoning, Question Answering) and are more dispersed across languages. A direct comparison of the two relevance groups without task or language controls conflates task difficulty with the relevance label. The authors should either match fundamental/industry pairs within language and task family, or reframe the conclusion as a claim about these specific collections, explicitly citing the per-category medians in Figure 5 rather than the aggregate in Figure 6.","section":"Section 4.2.1, Figure 6"},{"comment":"The comparison with shared-task results is not apples-to-apples: LLMs are evaluated zero-shot on a single prompt, while the \"best published models\" are fine-tuned on task-specific training data, and the averaging protocol for \"best published results\" is unspecified (which submissions, which metrics, how many datasets per category). The reported numbers, e.g., 72.67% vs. 84.26% for humor detection, are presented without adjustment for this asymmetry. While the text mentions the zero-shot caveat, the abstract and conclusion repeat the gap without it. The authors should provide a per-dataset comparison using the same metric and test split, and state the fine-tuning/zero-shot asymmetry explicitly where the comparison is summarized.","section":"Section 4.2.2, paragraph \"Shared task participants still lead...\""},{"comment":"Figure 5 shows that the random baseline outperforms every LLM on MGT detection and attribution. The paper reports this without analysis, yet it contradicts the summary claim (iii) that these tasks are \"close to random\"—they are actually worse than random, which often signals a methodological artifact such as a systematic label bias in likelihood scoring or a prompt-induced majority-class response. The authors should investigate the cause (e.g., report the distribution of predicted labels, check the tokenization of label strings, verify that Macro-F1 is computed identically for the random baseline and the models) and either explain the phenomenon or soften the claim in the abstract and conclusion.","section":"Section 4.2.2, Figure 5 (MGT Detection and Attribution)"}],"minor_comments":[{"comment":"The product notation in Equation (1) appears corrupted in the current rendering (\"|y|Y j=0\"); it should read ∏_{j=1}^{|y|} pLLM(y_j | s_i, y_0.99 with p<1×10−11\" is surprisingly strong given only 23 models and the presence of tied ranks in some language plots. Please report the exact statistics for each pair, state how ties are handled in Kendall's τ, and list the number of models included per language (since not all models have results for all languages).","section":"Section 4.2.3, paragraph on ranking consistency"},{"comment":"The heatmaps are difficult to read because the numeric annotations are small and the color contrast between adjacent ranks is weak. Consider using a discrete color scale with larger font, or a table with a small multiples layout, so that the rankings are legible in print.","section":"Appendix B, Figures B.10 and B.11"},{"comment":"The mojibake example (\"café\" recovered from \"café\") contains a visible OCR/encoding artifact in the printed text (\"caf ˜A©\"); please check the encoding of this example in the final PDF.","section":"Section 3.3, dataset preparation paragraph"}],"recommendation":"major_revision","confidential_remarks":"The benchmark resource itself is a strong contribution and the infrastructure is well executed. My main concern is that the empirical claims—the central selling points of the paper—are not yet robustly established because of the uncontrolled task-language imbalance. I would like to see (1) a matched or stratified analysis for the language-difficulty claim, (2) a similar control or a revised formulation for the fundamental-vs-industry gap, and (3) a transparent protocol for the shared-task comparison. The paper should also either explain the below-random MGT results or adjust the 'close to random' claim. These are fixable within a revision, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know about IberBench: it is a useful resource and a less useful set of headline findings. The resource is real—101 datasets, 22 categories, the normalization of 58 workshop datasets into one lm-evaluation-harness pipeline, plus a committee-governed leaderboard. That is a genuine service to the multilingual NLP community, especially for Spanish and Portuguese industry adopters. The evaluations of 23 models are transparent: standard metrics, open prompts, and an honest limitations section.\n\nThe soft spots are where the paper makes its most prominent claims. The headline results—Galician and Basque are hardest; LLMs are worse on industry tasks—are averages over a task-language matrix that is very sparse and heavily imbalanced. The paper admits this in Section 3.3: “LLMs might appear weaker or stronger in cross-lingual and cross-task comparisons depending on how well they perform on overrepresented or underrepresented combinations.” Then it averages anyway. Basque’s mean is driven by a handful of hard single-language datasets (FMTODeu, BHTC Topic, EusTrivia, QNLI). Galician’s 12.8k samples are mostly fundamental tasks plus MGT. So “language difficulty” is entangled with task selection. The paper never reports per-language performance restricted to datasets that exist in more than one language, nor cell-weighted averages. Similarly, the industry-vs-fundamental gap mixes source, prompt format, label count, and metric, so Figure 6 may be measuring benchmark construction as much as model capability.\n\nNone of this kills the benchmark. The task-category level results in Figure 5 are more granular and more believable. But the two headline claims in the abstract should be treated as provisional correlations, not established findings. Also: no error bars or significance tests—standard for leaderboards, but worth naming—and only one sequence-labeling dataset, so the sequence-labeling conclusions are thin. The comparison to shared-task systems is zero-shot vs fine-tuned, which they acknowledge.\n\nWho’s this for? Anyone building or choosing LLMs for Iberian languages, and the Iberian NLP community that wants a consolidated evaluation harness. I’d bring it to reading group, and I’d cite it when describing the resource. It deserves a serious referee; the right revision would add a robustness analysis that controls for the imbalance—overlapping language comparisons, cell weighting, variance estimates—and soften the abstract accordingly.","headline":"IberBench is a genuinely useful benchmark resource whose headline empirical claims are confounded by its own acknowledged task-language imbalance; the resource deserves publication, the findings deserve caution.","tokens_in":34976,"tokens_out":2712,"would_cite":true,"duration_ms":26165,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"IberBench's evaluation of 23 LLMs on 101 Iberian-language datasets finds that industry-relevant tasks are hardest, with Basque and Galician trailing.","keywords":["LLM evaluation","Iberian languages","multilingual benchmark","industry-relevant NLP","Basque","Galician","machine-generated text detection","zero-shot evaluation"],"falsifier":"Recompute the four headline findings after reweighting the datasets so each language and each task category contributes equal weight; if the industry-versus-fundamental gap or the Basque/Galician difficulty shrinks to noise, those findings are artifacts of the benchmark's imbalanced coverage rather than stable model properties.","tokens_in":33856,"feed_emoji":"📊","tokens_out":6414,"duration_ms":56114,"temperature":0.7,"pith_summary":"Large language models are usually tested on English-centric benchmarks that emphasize reading comprehension and general knowledge, so their practical value for languages like Spanish, Catalan, Basque, and Galician is largely unknown. This paper introduces IberBench, a benchmark that assembles 101 existing datasets across 22 task categories covering six languages and five Spanish varieties, and evaluates 23 open models from 100 million to 14 billion parameters. Its central claim is that current LLMs are measurably weaker at industry-relevant tasks such as sentiment analysis, intent classification, toxicity detection, and machine-generated-text detection than at fundamental language tasks, and that the gap is especially wide for Basque and Galician. The benchmark also reports that on several tasks, including Basque intent classification and machine-text detection, top models barely beat random guessing, and that zero-shot LLMs trail the best fine-tuned shared-task systems on tasks like humor and fake-news detection. If these findings hold, they imply that LLM leaderboards built on fundamental tasks alone overstate readiness for real-world deployment in Iberian languages.","feed_headline":"Iberian benchmark: LLMs lag most on industry tasks","feed_subtitle":"23 models, 101 datasets: Basque and Galician are hardest, and machine-text detection is near random.","key_machinery":"The load-bearing object is IberBench's measurement pipeline, built on a standardized evaluation harness extended by the authors. Each dataset is normalized to a common format; classification tasks are scored by the model's likelihood of the correct label using Macro-F1, summarization by ROUGE-1, and the sequence-labeling task by chunk-level F1 on a custom tag-wrapping annotation schema. All models are evaluated zero-shot with one prompt per task, and each task is anchored by a uniform random baseline. The analytical core is the aggregation of these per-dataset scores into averages over task categories, relevance classes, and languages, which is what generates the paper's comparative findings.","core_discovery":"The paper's central claim is that a broad, community-extensible benchmark for Iberian languages reveals a consistent capability profile across 23 open LLMs. IberBench assembles 101 datasets, most from evaluation campaigns and the rest from recent LLM benchmarks, spanning 22 task categories and covering Spanish, Portuguese, Catalan, Basque, Galician, English, and five Spanish varieties. Evaluated zero-shot, the models score higher on fundamental tasks such as reading comprehension, question answering, commonsense reasoning, and textual entailment than on industry-relevant tasks such as sentiment, toxicity, stance, author profiling, summarization, and intent classification. The four headline findings are: industry-relevant tasks trail fundamental ones; Galician and Basque are the hardest languages; lexical borrowing chunking, Basque intent classification, and machine-generated-text detection sit near random; and on sentiment, humor, and fake-news detection LLMs beat random but fall below the best published shared-task systems. The paper frames the gap as evidence that existing fundamental-only leaderboards overstate the practical usefulness of LLMs for industrial NLP in these languages.","pith_inferences":["Beyond the paper: the imbalanced language-task matrix means aggregate findings should be re-tested on a matched subset, for instance only tasks that exist in both Spanish and Basque, before guiding resource decisions.","Beyond the paper: a few-shot variant of the same benchmark would directly test whether the industry-versus-fundamental gap shrinks when models are given examples, which the zero-shot design likely exaggerates.","Beyond the paper: the near-random machine-text-detection results offer a built-in contamination check; a sudden jump in future submissions could signal that test data leaked into training.","Beyond the paper: the Portuguese-tuned model's strong Galician transfer suggests deliberately pairing under-resourced languages with close relatives is a cheaper path than building new data from scratch for Galician."],"forward_implications":["Leaderboards that test only fundamental skills will keep overstating how ready models are for industry uses such as moderation, profiling, and summarization in Iberian languages.","Galician and Basque need deliberate resource building: generic multilingual scaling alone leaves most models at or near random in these languages.","Tasks near random, including lexical borrowing detection, Basque intent classification, and machine-generated-text detection, define the current reliability frontier and should be the focus of task-specific work.","Zero-shot evaluation understates models relative to fine-tuned systems; the gap to shared-task results is the available headroom for prompting or adaptation.","Because model rankings are nearly identical across languages, a model selected on Spanish data will likely rank similarly in Catalan and Portuguese, simplifying deployment choices."],"supporting_citations":[{"why":"Supplies the majority of the workshop-sourced industry-relevant datasets, grounding the benchmark's industrial task coverage.","marker":"[14]"},{"why":"Contributes the fundamental-task datasets that anchor the comparison between fundamental and industry-relevant performance.","marker":"[11]"},{"why":"Provides the Basque evaluation suite and a Basque-adapted model, underpinning the finding that Basque remains hardest.","marker":"[18]"},{"why":"Forms the evaluation backbone that the authors extend, making the zero-shot and sequence-labeling measurements reproducible.","marker":"[19]"},{"why":"Serves as the closest parallel Iberian benchmark, defining the gap IberBench fills with industry tasks and extensibility.","marker":"[12]"},{"why":"Adds multilingual reading comprehension, supporting the fundamental-task side of the benchmark's design.","marker":"[44]"}],"fun_headline_variants":["IberBench: LLMs lag on industry tasks, near random on some","Basque and Galician hardest for LLMs in new Iberian benchmark","Iberian LLM study: industry tasks flop, some near random","Community Iberian benchmark exposes LLM blind spots"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline comparisons assume that averaging scores over 101 datasets is a fair basis for ranking tasks and languages even though Spanish supplies about 60% of the samples and some task categories exist in only one language.","fun_headline_variants_meta":{"raw":{"variants":["IberBench: LLMs lag on industry tasks, near random on some","Basque and Galician hardest for LLMs in new Iberian benchmark","Iberian LLM study: industry tasks flop, some near random","Community Iberian benchmark exposes LLM blind spots"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000336,"raw_usage":{"total_tokens":1914,"prompt_tokens":1053,"completion_tokens":861,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":669,"completion_tokens_details":{"reasoning_tokens":785}},"tokens_in":669,"tokens_out":861,"duration_ms":7781,"temperature":1.0,"reasoning_tokens":785,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:51:50.881536+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the four headline findings after reweighting the datasets so each language and each task category contributes equal weight; if the industry-versus-fundamental gap or the Basque/Galician difficulty shrinks to noise, those findings are artifacts of the benchmark's imbalanced coverage rather than stable model properties.","supporting_citations":[{"cited_title":"Bandarkar, D","cited_arxiv_id":null,"evidence_quote":"Adds multilingual reading comprehension, supporting the fundamental-task side of the benchmark's design."}],"review_version":1}