Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

Evalita-LLM: Benchmarking Large Language Models on Italian

T0 review · 3 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A new benchmark evaluates Italian LLMs on native tasks with multiple prompts.

desk verdict A useful, reproducible Italian benchmark whose robustness numbers are partly self-selected; worth reviewing and citing. read the letter →

arxiv 2502.02289 v1 pith:ZH6L2A2H submitted 2025-02-04 cs.CL

Bernardo Magnini , Roberto Zanoli , Michele Resta , Martin Cimmino , Paolo Albano , Marco Madeddu , Viviana Patti This is my paper · ORCID
classification cs.CL
keywords Evalita-LLMItalianLLMbenchmarklargelanguagemodelsmulti-promptevaluationgenerativetaskspromptvalidationnative
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Evalita-LLM is a benchmark for evaluating large language models on Italian, built from existing native-Italian datasets rather than translated English benchmarks. The paper claims that this design avoids translation artifacts and cultural bias, and that scoring every task under several prompt templates gives a fairer picture of model ability than single-prompt evaluation. The authors also propose an iterative development method in which candidate tasks and prompts are validated on a small set of anonymized instruction-tuned models. If the benchmark is sound, it gives the Italian LLM community a public, reproducible evaluation suite that measures both peak performance and prompt robustness.

What carries the argument

The central machinery is the iterative validation methodology: candidate Italian datasets are adapted to a common evaluation harness, multiple prompt templates are designed for each task following explicit guidelines (Italian language, no role preambles, minimal verbosity, named input type), and both tasks and prompts are tested on a set of dev LLMs. The task is kept only if it is neither too easy nor too hard for those models, and a prompt is kept only if it is understood and produces differentiated results. The scoring framework adds a Combined Performance Score ($CPS$) that multiplies a model's best prompt performance ($MaxP$) by a saturation score ($1 - (MaxP - AvgP)$), so a model ranks well only if it is both strong and insensitive to prompt variation.

What would settle it

Run the benchmark on a set of deliberately diverse models, including a smaller non-instruction model, a much larger instruction-tuned model, and a model trained primarily on English; if task scores or prompt variability shift dramatically relative to the dev-phase results, the task selection and prompt choices would be shown to depend on the dev model population.

Watch

Extended reading notes

Core claim

The paper's central claim is that a methodologically well-founded benchmark for Italian LLMs can be built by combining three properties: all tasks are native Italian, the suite includes generative tasks alongside multiple-choice ones, and every task is evaluated under multiple prompt templates. The authors select ten tasks from Evalita and other Italian resources, adapt them to the lm-evaluation-harness format, define six prompt templates for multiple-choice tasks and four for generative tasks, and score them with metrics for maximum performance, average performance, and a combined performance score that penalizes prompt sensitivity. They report development-phase results from six anonymized 7B-9B instruction-tuned models, showing that tasks are challenging yet solvable and that different prompts capture different model characteristics. The paper argues that this constitutes a reference benchmark for Italian LLM evaluation that avoids the quality, content, and stylistic problems of translated benchmarks.

Load-bearing premise

The benchmark's validity rests on the assumption that the six anonymized dev LLMs, a small set of mid-size instruction-tuned open models, are representative enough of the LLMs that will later be evaluated that tasks and prompts validated on them will work for the broader population of Italian LLMs.

Editorial extensions

If this is right

  • Italian LLM providers can use Evalita-LLM to compare models on native language tasks without worrying that translations or cultural content skewed the results.
  • A model's reported score becomes more trustworthy because it must perform well across several prompts, not just one hand-picked template.
  • The generative tasks (lexical substitution, NER, relation extraction, summarization) push evaluation toward natural interaction, revealing which models can follow output-format instructions.
  • The development-phase scores give early, anonymized evidence about how current open Italian LLMs behave, including the difficulty of conditioning output format in generative tasks.
  • The benchmark's public release on Hugging Face makes the suite usable by others for reproducible evaluation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The anonymized dev models are all roughly 7B-9B and instruction-tuned, so the selected tasks may be skewed toward what mid-size instruction-tuned models can do; larger or non-instruction models might behave differently, and the benchmark may under-represent very hard or very easy tasks.
  • The multi-prompt design could be reused for other low-resource languages by following the same adaptation pipeline, provided equivalent native datasets exist.
  • A testable extension would be to compare Evalita-LLM scores against the same tasks rendered via translated prompts, to quantify how much of the performance gap is due to language and culture rather than task difficulty.
  • The CPS metric could be adopted as a general standard for any benchmark wanting to penalize prompt sensitivity, not just Italian-language ones.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper presents Evalita-LLM, a benchmark for evaluating large language models on Italian, built from ten existing native-Italian datasets. Six tasks are cast as multiple-choice and four as generative (generate-until), each with several Italian prompt templates. The authors describe an iterative development methodology in which candidate tasks and prompts are validated on six anonymized 7B-9B instruction-tuned LLMs, report development-phase results using MaxP/AvgP/CPS metrics adapted from Mizrahi et al., and release the datasets on Hugging Face. The central claim is that the benchmark is methodologically well-founded for Italian LLM evaluation.

Significance. If the methodological claims held, the contribution would be a useful public resource: native-language evaluation tasks, multiple prompts per task, inclusion of generative tasks, and detailed per-task documentation with baselines. The use of lm-evaluation-harness and public Hugging Face datasets are strengths for reproducibility, and the explicit comparison of acc and acc_norm for multiple-choice scoring is a careful detail. However, the benchmark's validity hinges on the task and prompt selection procedure, and that procedure is currently underspecified and partly circular; the significance is therefore conditional on the issues below being resolved.

major comments (3)
  1. [Section 2, Section 6.1, Section 6.5] The iterative selection procedure uses the same six anonymized dev LLMs on which all reported dev-phase statistics are computed. Section 2 states that prompts 'not well understood by any of the assessed LLMs' are discarded and tasks 'too complex for the dev LLMs' are excluded, while Section 6.5 uses MaxP/AvgP/CPS values from those same models to conclude that the final prompt set captures different characteristics. This is a selection-on-the-evaluation-set problem: the reported multi-prompt robustness is partly a consequence of having removed low-performing prompts and tasks, not a property of the tasks themselves. The 'minimum criteria' promised in Section 2 are never quantified in Section 6, making the selection rule unfalsifiable. Please specify the criteria, report the number and identity of rejected tasks and prompts, and either validate the final benchmark on a held-out set of models or explicitly qualify the scope of the validity claims.
  2. [Appendix A (Tables A7, A8, A10), Section 6.4] Several generative tasks that were selected under the stated 'not too complex' criterion show zero or near-zero zero-shot performance for some dev LLMs: Lexical Substitution (Table A7) gives 0.00 for LLM-1 and LLM-3 on both prompts, Summarization (Table A8) gives 0.00 for LLM-1 on both prompts and an unexplained N/A for LLM-1 few-shot, and Relation Extraction (Table A10) gives 0.00 for LLM-1 on both prompts. If these tasks satisfied the minimum criteria, the criteria must be very close to zero; if they did not, the selection rule was not applied consistently. This matters because Section 6.5 uses the same tables to claim that the generative prompts capture useful signal. Please reconcile the selection rule with these results or provide per-task minimum thresholds.
  3. [Section 4.3, Tables A7-A10] For the four generative tasks the reported experiments use only two of the four generative templates (p7/p9 for Lexical Substitution and Summarization, p8/p10 for NER and Relation Extraction), even though Section 4.3 presents four templates. The paper does not say whether the other two templates were tried and rejected, and if so on what evidence. The MaxP/AvgP/CPS numbers for generative tasks are therefore computed over two prompts only, which weakens the 'all tasks evaluated against multiple prompts' claim for exactly the tasks where prompt sensitivity is emphasized. Please state the intended final prompt set per task and justify the reduction.
minor comments (6)
  1. [Table 14, Section 6.2] The caption and text call the Textual Entailment results 'zero-shot F1', but Section 3.2 states that the TE evaluation metric is accuracy; the column header and narrative should be aligned.
  2. [Section 6.3] There are typos ('yje', 'featured implemented') and the discussion of byte-length normalization implicitly assumes English byte counts; for Italian, accented characters are multi-byte in UTF-8, so the normalization's effect on accented answers should be acknowledged.
  3. [Section 4.2, Section 4.3, Section 6.2] The text uses 'affermative' for 'affirmative' and 'for instances' for 'for instance', and says LLM-4 scores 63.10 with p6 in Sentiment Analysis while Table 15 reports 63.08; please harmonize.
  4. [Section 3.5] The dataset splits label the trial file 'multichoice v1 test.jsonl' and the test file 'multichoice v1 dev.jsonl', which is confusing for users; rename the files or add a note explaining the convention.
  5. [Appendix A] The appendix mixes 'LLM 1' and 'LLM-1' naming, and Table A8 contains an unexplained N/A for LLM-1 few-shot Summarization; add a footnote explaining the missing value.
  6. [Table 1, References] Table 1 has spacing issues ('F AQ', 'T ask') and Reference [18] has a typo in the title ('NotSo' instead of 'Not So').

Circularity Check

1 steps flagged · score 4.0 of 10

Partial circularity: tasks and prompts are selected on the six dev LLMs and then validated on those same six dev LLMs, so the dev-phase confirmation is partly self-imposed.

  1. self definitional [Section 2 (incremental methodology) and Section 6.2 (dev results, Textual Entailment)]
    "Section 2: 'a task that reveals itself too complex for the dev LLMs is not selected to be included in the benchmark.' Section 6 (Objective 1): 'a good task ... should be enough challenging for a set of selected LLMs, without being too difficult.' Section 6.2: 'These data confirm that the task is well understood by the dev LLMs while still being challenging.'"

    The inclusion rule for tasks is the dev LLMs' ability to handle them, and the validation criterion is the same ability. Section 2 removes tasks that are 'too complex for the dev LLMs'; Section 6 defines a good task as one that is 'enough challenging ... without being too difficult' for 'a set of selected LLMs.' Reporting that the six dev LLMs outperform baselines and that the task is 'well understood' by them is a restatement of the selection rule rather than independent confirmation. The same loop operates on prompts: prompts 'not well understood by any of the assessed LLMs' are discarded, and Section 6 later uses prompt-score spreads on the same models to 'empirically confirm' the prompt selection.

full rationale

No equation in the paper reduces to its own input: MaxP, AvgP, and CPS are explicit aggregations of the same per-prompt scores rather than predictions drawn from fitted parameters. The self-citations (CLinkaRT, E3C, and other Evalita overview papers by the same authors) are used as dataset provenance, not as load-bearing uniqueness or validity arguments. The substantive circularity is the selection-validation loop: Section 2 iteratively refines tasks and prompts using the six anonymized dev LLMs, excluding anything those models do not understand, and Section 6 then reports surviving tasks and prompts on those same six dev LLMs as evidence that the tasks are 'well understood' and the prompts 'capture different characteristics.' That makes the dev-phase validation partly self-confirming and, because the 'minimum criteria' are never quantified, unfalsifiable. This is a genuine validity concern for the claim of a 'methodologically well-founded benchmark.' However, the benchmark's independent content - native Italian datasets, six multiple-choice and four generative prompt templates, and transparent metrics - does not reduce to the dev-LLM selection, and the paper is explicit that these are development-phase results rather than held-out predictions. External validity beyond the 7B-9B instruction-tuned cluster is a correctness/limitation risk, not an additional circularity. Score 4 reflects one partial selection-validation loop rather than a fully forced derivation.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim rests on one design choice (task and prompt selection on anonymized dev LLMs), one representativeness assumption (dev pool generalizes to target models), two measurement assumptions (metrics and multi-prompt scoring capture ability), and three hand-set choices (selection thresholds, four generative prompts, 100-instance summarization sample). No new theoretical entities are postulated; the benchmark is an assembly of existing datasets with new prompt and scoring layers.

free parameters (3)
  • Task and prompt selection thresholds = not reported
    Section 2 states tasks and prompts are selected when "some minimum criteria" are achieved, but never quantifies those criteria; the thresholds are tuned on dev-LLM performance and determine the final benchmark composition.
  • Number of generative prompt templates = 4
    Section 4.3 restricts generative tasks to four templates "because of the high intensive computation required"; this resource-driven choice affects every generative-task score in the paper.
  • Summarization evaluation subset = 100 random Fanpage instances
    Section 3.10: evaluation uses a random selection of 100 instances; the random seed and selection are not reported, so summarization results depend on an unreported draw.
assumptions (5)
  • domain assumption Six anonymized 7B-9B instruction-tuned LLMs pretrained on Italian are representative of the models Evalita-LLM will evaluate.
    Section 6.1 selects the dev pool and uses it to validate every task and prompt; if the pool is not representative, the validation does not transfer to other models.
  • ad hoc to paper A task that is too hard for the dev LLMs is unsuitable for the benchmark.
    Section 2: "a task that reveals itself too complex for the dev LLMs is not selected to be included in the benchmark." This is a design choice, not an externally validated principle.
  • domain assumption Native Italian content removes translation and cultural bias and therefore makes evaluation fairer.
    Section 1 argues translated benchmarks introduce noise and cultural bias; the benchmark assumes native tasks solve these issues without measuring the residual bias directly.
  • domain assumption Accuracy, F1 and ROUGE, plus the proposed CPS, adequately capture model performance.
    Sections 3 and 5 choose these metrics; the paper notes that generative metrics are approximate but proceeds with them.
  • ad hoc to paper The lm-evaluation-harness few-shot prompt replication is an acceptable choice for few-shot tasks.
    Section 4.4: the authors found no clear difference between two few-shot formats and "decided to use the original implementation"; this choice affects few-shot Evalita-LLM scores.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Evalita-LLM: Benchmarking Large Language Models on Italian." pith.science (2026). https://pith.science/paper/ZH6L2A2H

@misc{pith2026250202289,
  author       = {Pith},
  title        = {Pith review of: Evalita-LLM: Benchmarking Large Language Models on Italian},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZH6L2A2H}},
  note         = {Machine review of arXiv:2502.02289}
}
read the original abstract

We describe Evalita-LLM, a new benchmark designed to evaluate Large Language Models (LLMs) on Italian tasks. The distinguishing and innovative features of Evalita-LLM are the following: (i) all tasks are native Italian, avoiding issues of translating from Italian and potential cultural biases; (ii) in addition to well established multiple-choice tasks, the benchmark includes generative tasks, enabling more natural interaction with LLMs; (iii) all tasks are evaluated against multiple prompts, this way mitigating the model sensitivity to specific prompts and allowing a fairer and objective evaluation. We propose an iterative methodology, where candidate tasks and candidate prompts are validated against a set of LLMs used for development. We report experimental results from the benchmark's development phase, and provide performance statistics for several state-of-the-art LLMs.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Easy to Complete, Hard to Choose: Investigating LLM Performance on the ProverbIT Benchmark

    cs.CL 2026-08 conditional novelty 5.0 of 10

    ProverbIT shows that large language models can complete Italian proverbs but often fail to select 'none of the above' when the exact ending is absent, revealing a gap between memorized knowledge and discriminative reasoning.

Reference graph

Works this paper leans on

22 extracted references · 16 canonical work pages · cited by 1 Pith paper

  1. [1]

    Polignano, M., Basile, P., Semeraro, G.: Advanced Natural-based interaction for the ITAlian language: LLaMAntino-3-ANITA (2024)

    work page 2024

  2. [2]

    In: Proceedings of the Tenth Italian Conference on Computational Linguistics (CLiC-it 2024) (2024)

    Moroni, L., Conia, S., Martelli, F., Navigli, R., et al.: Ita-bench: Towards a more comprehensive evaluation for italian llms. In: Proceedings of the Tenth Italian Conference on Computational Linguistics (CLiC-it 2024) (2024)

    work page 2024

  3. [3]

    Proceedings of the International Conference on Learning Representations (ICLR) (2021)

    Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., Stein- hardt, J.: Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR) (2021)

    work page 2021

  4. [4]

    : Calamita: Challenge the abilities of language models in italian

    Attanasio, G., Basile, P., Borazio, F., Croce, D., Francis, M., Gili, J., Musacchio, E., Nissim, M., Patti, V., Rinaldi, M., et al. : Calamita: Challenge the abilities of language models in italian. In: Proceedings of the 10th Italian Conference on Computational Linguistics (CLiC-it 2024), Pisa, Italy (2024)

    work page 2024

  5. [5]

    https://arxiv.org/abs/2405.14782

    Biderman, S., Schoelkopf, H., Sutawika, L., Gao, L., Tow, J., Abbasi, B., Aji, A.F., Ammanamanchi, P.S., Black, S., Clive, J., DiPofi, A., Etxaniz, J., Fattori, B., Forde, J.Z., Foster, C., Hsu, J., Jaiswal, M., Lee, W.Y., Li, H., Lovering, C., Muennighoff, N., Pavlick, E., Phang, J., Skowron, A., Tan, S., Tang, X., Wang, K.A., Winata, G.I., Yvon, F., Zou...

    arXiv 2024

  6. [6]

    EV ALITA (2023)

    Cassotti, P., Siciliani, L., Passaro, L.C., Gatto, M., Basile, P., et al.: Wic-ita at evalita2023: Overview of the evalita2023 word-in-context for italian task. EV ALITA (2023)

    work page 2023

  7. [7]

    In: Proceedings of EV ALITA 2009 (2009)

    Bos, J., Zanzotto, F.M., Pennacchiotti, M.: Textual entailment at evalita 2009. In: Proceedings of EV ALITA 2009 (2009). http://evalita.fbk.eu/reports/Textual Entailment/TE ORGANIZERS.pdf

    work page 2009

  8. [8]

    : Overview of the evalita 2016 sentiment polarity classification task

    Barbieri, F., Basile, V., Croce, D., Nissim, M., Novielli, N., Patti, V., et al. : Overview of the evalita 2016 sentiment polarity classification task. In: CEUR Workshop Proceedings, vol. 1749 (2016). CEUR-WS

    work page 2016

Show all 22 references
  1. [9]

    In: of the Final Workshop 7 December 2016, Naples, p

    Caputo, A., Gemmis, M., Lops, P., Lovecchio, F., Manzari, V., Spa, A.P.A.: Overview of the evalita 2016 question answering for frequently asked questions (qa4faq) task. In: of the Final Workshop 7 December 2016, Naples, p. 124 (2016)

    2016

  2. [10]

    In: CLiC-it (2023)

    Casola, S., Labruna, T., Lavelli, A., Magnini, B., et al.: Testing chatgpt for stabil- ity and reasoning: A case study using italian medical specialty tests. In: CLiC-it (2023)

    2023

  3. [11]

    In: Proceedings of 40 EV ALITA Workshop, 11th Congress of Italian Association for Artificial Intelli- gence, Reggio Emilia, Italy (2009)

    Toral, A.: The lexical substitution task at evalita 2009. In: Proceedings of 40 EV ALITA Workshop, 11th Congress of Italian Association for Artificial Intelli- gence, Reggio Emilia, Italy (2009)

    2009

  4. [12]

    : Nermud at evalita 2023: overview of the named-entities recognition on multi-domain documents task

    Palmero Aprosio, A., Paccosi, T., et al. : Nermud at evalita 2023: overview of the named-entities recognition on multi-domain documents task. In: CEUR WORKSHOP PROCEEDINGS, vol. 3473 (2023). CEUR

    2023

  5. [13]

    EV ALITA (2023)

    Altuna, B., Karunakaran, G., Lavelli, A., Magnini, B., Speranza, M., Zanoli, R.: Clinkart at evalita 2023: Overview of the task on linking a lab result to its test event in the clinical domain. EV ALITA (2023)

    2023

  6. [14]

    Proceedings of the Seventh Italian Conference on Computational Linguistics CLiC-it 2020 (2020)

    Magnini, B., Altuna, B., Lavelli, A., Speranza, M., Zanoli, R.: The e3c project: Collection and annotation of a multilingual corpus of clinical cases. Proceedings of the Seventh Italian Conference on Computational Linguistics CLiC-it 2020 (2020)

    2020

  7. [15]

    Information 13(5) (2022) https://doi

    Landro, N., Gallo, I., La Grassa, R., Federici, E.: Two new datasets for italian- language abstractive text summarization. Information 13(5) (2022) https://doi. org/10.3390/info13050228

    2022 doi

  8. [16]

    https://arxiv.org/abs/2107.13586

    Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., Neubig, G.: Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing (2021). https://arxiv.org/abs/2107.13586

    2021 arXiv

  9. [17]

    In: Toutanova, K., Rumshisky, A., Zettlemoyer, L., Hakkani-Tur, D., Beltagy, I., Bethard, S., Cotterell, R., Chakraborty, T., Zhou, Y

    Qin, G., Eisner, J.: Learning how to ask: Querying LMs with mixtures of soft prompts. In: Toutanova, K., Rumshisky, A., Zettlemoyer, L., Hakkani-Tur, D., Beltagy, I., Bethard, S., Cotterell, R., Chakraborty, T., Zhou, Y. (eds.) Proceed- ings of the 2021 Conference of the North...

    2021

  10. [18]

    https: //arxiv.org/abs/2412.12173

    Sane, S., McLean, A.: A NotSo Simple Way to Beat Simple Bench (2024). https: //arxiv.org/abs/2412.12173

    2024 arXiv

  11. [19]

    https://arxiv

    Mizrahi, M., Kaplan, G., Malkin, D., Dror, R., Shahaf, D., Stanovsky, G.: State of What Art? A Call for Multi-Prompt LLM Evaluation (2024). https://arxiv. org/abs/2401.00595

    2024 arXiv

  12. [20]

    https://arxiv.org/abs/2405.17202

    Polo, F.M., Xu, R., Weber, L., Silva, M., Bhardwaj, O., Choshen, L., Oliveira, A.F.M., Sun, Y., Yurochkin, M.: Efficient multi-prompt evaluation of LLMs (2024). https://arxiv.org/abs/2405.17202

    2024 arXiv

  13. [21]

    In: Al-Onaizan, Y., Bansal, M., Chen, Y.-N

    Marchisio, K., Ko, W.-Y., Berard, A., Dehaze, T., Ruder, S.: Understanding and mitigating language confusion in LLMs. In: Al-Onaizan, Y., Bansal, M., Chen, Y.-N. (eds.) Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 6653–6677. Assoc...

    2024 doi

  14. [22]

    arXiv preprint arXiv:2303.18223 (2023) 42

    Zhao, W.X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al.: A survey of large language models. arXiv preprint arXiv:2303.18223 (2023) 42

    2023 arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.