Pith. sign in

REVIEW 3 major objections 4 minor 29 references

Climate-Eval: A Comprehensive Benchmark for NLP Tasks Related to Climate Change

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read ClimateEval is a standardized evaluation suite of 25 climate-related NLP tasks that shows open-source LLMs gain from few-shot examples on most tasks while still lagging on fine-grained climate claim and entity classification.

desk verdict A useful climate-NLP benchmark and genuinely new Guardian corpus, but the unexplained zero scores in Table 2 must be fixed before the evaluation claims hold. read the letter →

arxiv 2505.18653 v1 pith:4XH5HMS5 submitted 2025-05-24 cs.CL

Murathan Kurfal{i} , Shorouq Zahra , Joakim Nivre , Gabriele Messori This is my paper · ORCID
classification cs.CL
keywords climatechangeNLPbenchmarklargelanguagemodelstextclassificationquestionansweringinformationextractionnewsfew-shotevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces ClimateEval, a benchmark that folds 13 existing climate-related NLP datasets plus one new news dataset into 25 standardized tasks for evaluating large language models. The authors' goal is to give the field a common yardstick for measuring how well LLMs handle climate discourse, from stance detection and claim verification to question answering and entity extraction. To demonstrate the benchmark, they run open-source models from 2B to 70B parameters in zero-shot and five-shot settings. Their results show that few-shot examples usually help, but many climate-specific tasks, especially fine-grained claim and entity classification, remain hard for current open models.

What carries the argument

The central object is the evaluation suite itself: 25 tasks defined on 13 datasets, each with a standard prompt and metric, all wired into the LM Evaluation Harness so that a single command reproduces the evaluation. The new Guardian corpus is curated by mapping the newspaper's tag taxonomy to ten categories and keeping only articles whose tags fall into a single category. Metrics are macro-F1 for classification tasks, accuracy for multiple-choice QA, and span-and-type exact-match F1 for the named-entity recognition task.

What would settle it

Take a random sample of articles from the Guardian Climate News Corpus, have human annotators assign the ten categories, and measure agreement with the tag-derived labels; if agreement is low, the dataset does not measure what it claims.

Watch

Extended reading notes

Core claim

ClimateEval is a standardized evaluation suite of 25 tasks built from 13 datasets, adding a new Guardian Climate News Corpus for ten-way topic classification of news titles and bodies. Benchmarked open-source LLMs improve on most tasks when given five demonstrations, but gains shrink as label counts rise, and tasks requiring external knowledge, such as fact verification and multiple-choice QA, show little or no few-shot benefit. Domain-adapted ClimateGPT outperforms its base Llama-2 models by only a small margin on average, with inconsistent gains across tasks.

Load-bearing premise

The new Guardian news dataset gets its labels from the Guardian's own article tags rather than from human annotation, so those labels are accurate only if the tags reliably indicate the intended categories.

Editorial extensions

If this is right

  • Any future open or closed LLM can be scored against the same 25 task definitions, making model comparisons in climate NLP directly reproducible.
  • Few-shot prompting is confirmed as a cheap win for most classification tasks, so baseline evaluations should report both zero-shot and five-shot numbers.
  • Tasks with many labels or domain-specific taxonomies, such as Exeter sub-claims, SciDCC, and climate NER, resist few-shot help, pointing to retrieval or fine-tuning as the next lever.
  • ClimateGPT's modest and uneven gains over Llama-2 suggest domain-adapted pretraining helps only on selected tasks, not uniformly.
  • The Guardian Title versus Body comparison shows titles carry nearly as much topic information as full articles for this corpus.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the Guardian corpus keeps publication dates, researchers could use it to study how the salience of climate categories shifts over 1998-2024, something the paper does not analyze.
  • The authors' filtering rule that removes articles carrying tags from several categories means the dataset measures single-topic news articles; a multi-label version built from the same tag pool would test a different, arguably more realistic, setting.
  • The small few-shot gains on high-label-count tasks suggest that in-context examples alone do not teach label taxonomies, so future benchmark results could separate a model's classification skill from its ability to follow prompt formats.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper presents ClimateEval, a benchmark that aggregates 13 climate-related NLP datasets, including a newly scraped Guardian Climate News Corpus, into 25 tasks implemented in the LM Evaluation Harness. The authors evaluate nine open-source LLMs (2B to 70B parameters) in zero-shot and five-shot settings, reporting macro-F1 (with exact-match accuracy for multiple-choice QA and span/type F1 for NER), and use the results to analyze few-shot improvements, task difficulty, and the effect of in-domain training by comparing ClimateGPT with its Llama-2 base models. The central claim is that ClimateEval provides a standardized, reproducible evaluation suite for assessing LLM strengths and limitations in the climate domain.

Significance. If the pipeline artifacts and dataset-validation gaps are fixed, this is a genuinely useful community resource. The paper's strengths are its breadth of tasks, the public GitHub/HuggingFace release, one-line evaluation through a widely used harness, explicit model versions, label-distribution visualizations, and a carbon-emissions estimate for the experiments. The work is empirical throughout and does not rest on fitted parameters or circular derivations. However, the unexplained exact-zero F1 scores in Table 2 and the undocumented Net-Zero split directly affect the reliability of the reported model comparison, and the Guardian labels need validation; all three are fixable within the manuscript's scope.

major comments (3)
  1. [Table 2 and Section 4] The exact 0.00 macro-F1 entries in Table 2 for Guardian Body (e.g., Llama-3.3-70B in both settings and Mistral-24B in 5-shot) and Guardian Title (e.g., Llama-2-7B 5-shot, Qwen-2.5-7B 5-shot, and Llama-3.3-70B 5-shot) are not credible under the log-likelihood classification protocol described in Section 4. With roughly 4,018 test instances and 10 classes, a constant-class predictor alone would yield a nonzero macro-F1 for the class it selects, so these zeros almost certainly reflect label mismatches, dropped test instances, or generation/parsing failures. The paper does not explain these values, yet it averages them into Table 3 and uses them in the task-level analysis of Section 4.2. This is a load-bearing validity issue for the central evaluation claim, and the affected configurations should be rerun and corrected, or explicitly excluded with sensitivity analysis.
  2. [Table 1 and Section 3.1] Net-Zero Reduction is listed in Table 1 with 3,441 training instances and no development or test split, but Table 2 reports evaluation scores for this task. Neither Section 3.1 nor the appendix states which split is used for evaluation or how it was derived. For a benchmark whose stated purpose is standardized and reproducible evaluation, this missing split documentation prevents other researchers from reproducing or extending the comparison; the authors should specify the split source, add it to Table 1, and confirm that it is identical across all model runs.
  3. [Section 3.1 (Guardian Climate News Corpus)] The labels in the new Guardian Climate News Corpus are derived from the Guardian's tag taxonomy rather than from direct human annotation: the authors curate tag lists per category, scrape articles carrying those tags, and delete any article whose tags fall into more than one defined category. There is no human validation of the resulting 40,173 labels, no inter-annotator agreement, and no analysis of how the multi-tag filtering rule affects the label distribution or task difficulty. Since this dataset is one of the paper's two main contributions and the paper draws conclusions about Guardian Title and Body tasks, the authors should provide at least a sample-based validation of the tag-to-category mapping and a robustness check of the filtering rule.
minor comments (4)
  1. [Section 3.1] The text says the corpus has nine climate-related categories but then lists only eight; correct the count or add the missing category.
  2. [Section 4.1] The claim that few-shot prompting consistently improves performance is contradicted by Table 3, where Guardian Body decreases by 0.01 and PIRA with Context is unchanged; please qualify the claim.
  3. [Throughout] The benchmark name is written inconsistently as 'ClimateEval' and 'Climate-Eval', and Section 3.2 contains a typo ('Y AML'); the prose in Section 1 also says 'in unprecedented in breadth'.
  4. [Appendix C] The normalized label distributions are helpful, but the paper should also report raw class counts for the Guardian and Net-Zero splits, because macro-F1 results and their interpretation depend on class frequency.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: ClimateEval is an empirical benchmark assembly and model evaluation with no derivation chain that reduces to its own inputs.

full rationale

The paper makes no theoretical derivation or fitted-parameter claim. Its contributions are (1) a new Guardian Climate News Corpus scraped using the Guardian's own tag taxonomy, (2) a unification of existing climate NLP datasets into 25 tasks, and (3) an empirical evaluation of open-source LLMs. None of these involves an equation in which an output is defined in terms of the claimed prediction, nor is any fitted parameter renamed as a prediction. The only self-citations (e.g., Li et al. 2024) appear in the introduction as examples of prior LLM applications, not as load-bearing justification for the benchmark's design or results. The new dataset's labels are derived from the Guardian's editorial tags rather than fresh human annotation; this is a data-construction and validity concern (the tag-to-category mapping may be imperfect, and filtering multi-topic articles may bias the sample), but it is not circular reasoning, because the paper does not use the benchmark's own outputs or the evaluated models to define those labels. Similarly, the unexplained 0.00 macro-F1 entries in Table 2 for Guardian Title/Body are an evaluation-pipeline or reporting concern, not a circularity concern: they indicate possible label mismatches or preprocessing failures, but they do not make the benchmark's conclusions equivalent to its inputs. The paper is self-contained as an empirical benchmark paper, and no circular step can be exhibited from its text.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The benchmark uses no fitted parameters and introduces no theoretical entities. The load-bearing premises are about data quality (the tag-derived labels of the new Guardian corpus) and the validity of the authors' task reformulations, both of which are disclosed in the paper but not externally validated.

assumptions (3)
  • domain assumption The Guardian newspaper's tag taxonomy is a reliable indicator of an article's true topic, and the manually curated tag-to-category mapping is accurate.
    The new dataset's labels are not human-annotated but derived from publisher tags; the paper reports no validation or agreement study for the mapping. This assumption underlies the validity of all results on the Guardian Title and Body tasks.
  • ad hoc to paper Articles whose tags span multiple defined categories can be removed without biasing the topic distribution.
    The overlap-exclusion rule (Section 3.1) is introduced specifically to enforce single-label classes; it may remove a non-random subset of articles (e.g., those covering climate policy impacts), affecting the label distribution and downstream benchmark difficulty.
  • domain assumption The task reformulations (e.g., Climate-FEVER as pairwise claim-evidence entailment, PIRA as 5-way MCQ) preserve the original datasets' intended evaluation.
    The paper defines several tasks differently from their source papers; the choice to convert Climate-FEVER to 7,675 pairs or to split PIRA with/without context is reasonable but not proven harmless to comparability.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Climate-Eval: A Comprehensive Benchmark for NLP Tasks Related to Climate Change." pith.science (2026). https://pith.science/paper/4XH5HMS5

@misc{pith2026250518653,
  author       = {Pith},
  title        = {Pith review of: Climate-Eval: A Comprehensive Benchmark for NLP Tasks Related to Climate Change},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4XH5HMS5}},
  note         = {Machine review of arXiv:2505.18653}
}
read the original abstract

Climate-Eval is a comprehensive benchmark designed to evaluate natural language processing models across a broad range of tasks related to climate change. Climate-Eval aggregates existing datasets along with a newly developed news classification dataset, created specifically for this release. This results in a benchmark of 25 tasks based on 13 datasets, covering key aspects of climate discourse, including text classification, question answering, and information extraction. Our benchmark provides a standardized evaluation suite for systematically assessing the performance of large language models (LLMs) on these tasks. Additionally, we conduct an extensive evaluation of open-source LLMs (ranging from 2B to 70B parameters) in both zero-shot and few-shot settings, analyzing their strengths and limitations in the domain of climate change.

Figures

Figures reproduced from arXiv: 2505.18653 by the authors.

Figure 1
Figure 1. The distribution of articles by year in the [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Average model performance across all tasks [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. The normalized distribution of labels in each task across all sets (train, development, and test). [PITH_FULL_IMAGE:figures/full_fig_p014_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

29 extracted references · 14 canonical work pages

  1. [1]

    Little, Elizabeth Fancher, Irina Gerasimov, Armin Mehrabian, Lauren Sanders, Sylvain V

    Bishwaranjan Bhattacharjee, Aashka Trivedi, Masayasu Muraoka, Muthukumaran Ramasubramanian, Takuma Udagawa, Iksha Gurung, Nishan Pantha, Rong Zhang, Bharath Dandala, Rahul Ramachandran, Manil Maskey, Kaylin Bugbee, Michael M. Little, Elizabeth Fancher, Irina Gerasimov, Armin Mehrabian, Lauren Sanders, Sylvain V. Costes, Sergi Blanco-Cuaresma, Kelly Lockha...

    2024

  2. [2]

    Julia Bingler, Mathias Kraus, Markus Leippold, and Nicolas Webersinke. 2023. H ow C heap T alk in C limate D isclosures R elates to C limate I nitiatives, C orporate E missions, and R eputation R isk. Working paper, Available at SSRN 3998435

    work page 2023

  3. [3]

    Coan, Constantine Boussalis, John Cook, and Mirjam O

    Travis G. Coan, Constantine Boussalis, John Cook, and Mirjam O. Nanko. 2021. https://doi.org/10.1038/s41598-021-01714-4 Computer-assisted classification of contrarian claims about climate change . Scientific Reports, 11(1):1--12

    work page doi:10.1038/s41598-021-01714-4 2021

  4. [4]

    Thomas Diggelmann, Jordan Boyd-Graber, Jannis Bulian, Massimiliano Ciaramita, and Markus Leippold. 2020. CLIMATE \-- FEVER : A D ataset for V erification of R eal- W orld C limate C laims. arXiv preprint arXiv:2012.00614

    arXiv 2020

  5. [5]

    May El Barachi , Manar AlKhatib, Sujith Mathew, and Farhad Oroumchian. 2021. https://doi.org/10.1016/j.jclepro.2021.127820 A novel sentiment analysis framework for monitoring the evolving public opinion in real-time: C ase study on climate change . Journal of Cleaner Production, 312:127820

    work page arXiv 2021

  6. [6]

    Michael Fore, Simranjit Singh, Chaehong Lee, Amritanshu Pandey, Antonios Anastasopoulos, and Dimitrios Stamoulis. 2024. https://doi.org/10.18653/v1/2024.climatenlp-1.14 U nlearning C limate M isinformation in L arge L anguage M odels . In Proceedings of the 1st Workshop on Natural Language Processing Meets Climate Change (ClimateNLP 2024), pages 178--192,...

    doi:10.18653/v1/2024.climatenlp-1.14 2024

  7. [7]

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2024. https...

    doi:10.5281/zenodo.12608602 2024

  8. [8]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava S...

    arXiv 2024

Show all 29 references
  1. [9]

    Graham, F.Q

    Rodney Michael Kinney, Chloe Anastasiades, Russell Authur, Iz Beltagy, Jonathan Bragg, Alexandra Buraczynski, Isabel Cachola, Stefan Candra, Yoganand Chandrasekhar, Arman Cohan, Miles Crawford, Doug Downey, Jason Dunkelberger, Oren Etzioni, Rob Evans, Sergey Feldman, Joseph Go...

    2023 arXiv

  2. [10]

    Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. 2019. https://arxiv.org/abs/1910.09700 Q uantifying the C arbon E missions of M achine L earning . Preprint, arXiv:1910.09700

    2019 arXiv

  3. [11]

    Jing Li, Aixin Sun, Jianglei Han, and Chenliang Li. 2020. A S urvey on D eep L earning for N amed E ntity R ecognition. IEEE transactions on knowledge and data engineering, 34(1):50--70

    2020

  4. [12]

    Ni Li, Shorouq Zahra, Mariana Brito, Clare Flynn, Olof G \"o rnerup, Koffi Worou, Murathan Kurfal , Chanjuan Meng, Wim Thiery, Jakob Zscheischler, et al. 2024. U sing LLM s to B uild a D atabase of C limate E xtreme I mpacts. In Proceedings of the 1st Workshop on Natural Langu...

    2024

  5. [13]

    Chen Ling, Xujiang Zhao, Jiaying Lu, Chengyuan Deng, Can Zheng, Junxiang Wang, Tanmoy Chowdhury, Yun Li, Hejie Cui, Tianjiao Zhao, Amit Panalkar, Wei Cheng, Haoyu Wang, Yanchi Liu, Zhengzhang Chen, Haifeng Chen, Chris White, Quanquan Gu, Carl Yang, and Liang Zhao. 2023. https:...

    2023 doi arXiv

  6. [14]

    Mariana Madruga de Brito, Jan Sodoge, Heidi Kreibich, and Christian Kuhlicke. 2025. Comprehensive assessment of flood socioeconomic impacts through text-mining. Water Resources Research, 61(1):e2024WR037813

    2025

  7. [15]

    Prakamya Mishra and Rohan Mittal. 2021. https://www.climatechange.ai/papers/icml2021/76 Neural\- NERE : N eural N amed E ntity R elationship E xtraction for E nd-to- E nd C limate C hange K nowledge G raph C onstruction . In ICML 2021 Workshop on Tackling Climate Change with M...

    2021

  8. [16]

    Paulo Pirozelli, Marcos M Jos \'e , Igor Silveira, Fl \'a vio Nakasato, Sarajane M Peres, Anarosa AF Brand \ a o, Anna HR Costa, and Fabio G Cozman. 2024. Benchmarks for pir \'a 2.0, a R eading C omprehension D ataset about the O cean, the B razilian C oast, and C limate C han...

    2024

  9. [17]

    Tobias Schimanski, Julia Bingler, Mathias Kraus, Camilla Hyslop, and Markus Leippold. 2023. Climate BERT \-- N et Z ero: D etecting and A ssessing N et Z ero and R eduction T argets. In The 2023 Conference on Empirical Methods in Natural Language Processing

    2023

  10. [18]

    Daniel Spokoyny, Tanmay Laud, Tom Corringham, and Taylor Berg-Kirkpatrick. 2023. https://arxiv.org/abs/2301.04253 T owards A nswering C limate Q uestionnaires from U nstructured C limate R eports . Preprint, arXiv:2301.04253

    2023 arXiv

  11. [19]

    Dominik Stammbach, Nicolas Webersinke, Julia Anna Bingler, Mathias Kraus, and Markus Leippold. 2023. https://arxiv.org/abs/2209.00507 E nvironmental C laim D etection . arXiv preprint arXiv:2209.00507v4, arXiv:2209.00507

    2023 arXiv

  12. [20]

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan...

    2024 arXiv

  13. [21]

    David Thulke, Yingbo Gao, Petrus Pelser, Rein Brune, Rricha Jalota, Floris Fok, Michael Ramos, Ian van Wyk, Abdallah Nasir, Hayden Goldstein, Taylor Tragemann, Katie Nguyen, Ariana Fowler, Andrew Stanco, Jon Gabriel, Jordan Taylor, Dean Moro, Evgenii Tsymbalov, Juliette de Waa...

    2024 arXiv

  14. [22]

    Apoorva Upadhyaya, Marco Fisichella, and Wolfgang Nejdl. 2023. A M ulti-task M odel for S entiment A ided S tance D etection of C limate C hange T weets. In Proceedings of the international AAAI conference on web and social media, volume 17, pages 854--865

    2023

  15. [23]

    Roopal Vaid, Kartikey Pant, and Manish Shrivastava. 2022. T owards F ine-grained C lassification of C limate C hange related S ocial M edia T ext. In Proceedings of the 60th annual meeting of the association for computational linguistics: Student research workshop, pages 434--443

    2022

  16. [24]

    Francesco S Varini, Jordan Boyd-Graber, Massimiliano Ciaramita, and Markus Leippold. 2021. https://arxiv.org/abs/2012.00483 C lima T ext: A D ataset for C limate C hange T opic D etection

    2021 arXiv

  17. [25]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Me...

    2025 arXiv

  18. [26]

    Francisco Zanartu, Yulia Otmakhova, John Cook, and Lea Frermann. 2024. https://doi.org/10.18653/v1/2024.climatenlp-1.4 G enerative D ebunking of C limate M isinformation . In Proceedings of the 1st Workshop on Natural Language Processing Meets Climate Change (ClimateNLP 2024),...

    2024 doi

  19. [27]

    Haiqi Zhang, Zhengyuan Zhu, Zeyu Zhang, Jacob Devasier, and Chengkai Li. 2024. G ranular A nalysis of S ocial M edia U sers’ T ruthfulness S tances T oward C limate C hange F actual C laims. In Proceedings of the 1st Workshop on Natural Language Processing Meets Climate Change...

    2024

  20. [28]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  21. [29]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.