REVIEW 3 major objections 6 minor 31 references
Nano-ESG: Extracting Corporate Sustainability Information from News Articles
T0 review · 3 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read From 850,000 news articles, a GPT-4o pipeline produces an open ESG dataset whose summaries and labels largely match expert judgment.
desk verdict Useful open ESG dataset for German DAX firms, but the accuracy evaluation only samples articles the model already kept, so the headline numbers are conditional precision, not end-to-end accuracy. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the multi-stage filter design that spends cheap processing on most articles and reserves the expensive LLM for the survivors. Paragraph and summary deduplication use a bilingual embedding model with a cosine-similarity threshold of 0.8 to drop near-duplicates within a week; the entity filter uses a zero-shot named-entity recognizer to discard articles where a company keyword names something other than an organization; the relevance filter uses GPT-3.5-turbo (2023 data) or GPT-4o-mini (2024 data) with a chain-of-thought style prompt that asks for a first relevance guess, an explanation, a summary, and a second judgment on direct ESG relevance; and the final model GPT-4o-2024-05-13 generates the summary, relevance score, sentiment, aspect, and keywords. The evaluation machinery is an expert annotation study: five sustainability experts label 123 summaries and 367 sentiment/aspect instances, with an inter-annotator agreement statistic (kappa) quantifying reliability and majority vote defining ground truth.
What would settle it
Run the same expert annotation on a random sample of articles that the relevance filters rejected; if a substantial share turn out to be ESG-relevant, the reported 95.9%, 79.9%, and 78.5% accuracies overstate the pipeline's real-world precision.
Extended reading notes
Core claim
On its own terms, the paper claims that Nano-ESG demonstrates a viable, fully automatic route from raw news to structured corporate sustainability intelligence. Starting from roughly 850,000 crawled articles, the pipeline keeps 51,087 that directly address ESG issues for one of the covered DAX companies; for each, GPT-4o writes a summary in German (with an English translation), assigns a relevance score from 1 to 10, a sentiment (positive, neutral, or negative), and an ESG aspect (Environmental, Social, or Governance). The authors validate a stratified sample: all five expert annotators judged 95.9% of 38 commonly seen summaries as correct, and over the larger evaluation set the LLM's sentiment and aspect labels matched the expert majority in 79.9% and 78.5% of cases. They further find that the LLM's relevance score correlates with expert judgment of irrelevance, and that topic modeling over the summaries surfaces time-stamped ESG events such as the forced-labor reports concerning Volkswagen. The conclusion is that the dataset is accurate enough for use in research and monitoring, and that the pipeline transfers to other domains.
Load-bearing premise
The accuracy figures are computed only on articles the pipeline's relevance filter already kept, and those figures treat the five experts' labels as ground truth, so recall of relevant articles and label reliability are not measured.
Editorial extensions
If this is right
- Nano-ESG is the first open dataset with a time series of ESG sentiments and aspects for German DAX companies, so researchers can connect individual ESG events to dates and external factors.
- The LLM-assigned relevance score appears meaningful, since expert-judged irrelevance drops as the score rises, so users can filter articles by a relevance threshold.
- The release of summaries, labels, and keywords enables training smaller specialized models, which the authors propose as a way to cut costs for continuous monitoring.
- The pipeline is described as transferable to other companies, countries, and domains beyond sustainability.
- Topic modeling on the summaries can identify company-specific ESG opportunities and risks with timestamps, as demonstrated for Volkswagen's forced-labor reports.
Reading between the lines
- Editorial: The evaluation measures conditional precision, not recall; the true number of ESG-relevant articles among the 840,000 could be larger than 51,087, so downstream users should treat the dataset's counts as lower bounds on relevant coverage.
- Editorial: The expert agreement on ESG aspect labels is low (kappa 0.427), which means the 78.5% aspect accuracy depends heavily on the chosen label-aggregation rule; a different aggregation could move the number substantially.
- Editorial: Because the final stage relies on GPT-4o at per-token cost, scaling to more companies or countries will likely require distilling the labels into cheaper specialized models; the paper proposes this but does not measure the distillation gap.
- Editorial: Joining Nano-ESG's sentiment time series with stock-price or rating data would test whether news-derived ESG sentiment predicts market movements, a validation the paper does not run.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents Nano-ESG, a dataset of more than 840,000 news articles for German DAX companies, reduced to 51,087 relevant articles through a pipeline of paragraph deduplication, GLiNER-based entity filtering, GPT-3.5/4o-mini relevance filtering, summary deduplication, and a final GPT-4o determination step. For each surviving article, the dataset provides a company-specific ESG summary, an ESG sentiment label, an ESG aspect label, a relevance score, and a set of keywords. The authors report a human evaluation by five sustainability experts in which GPT-4o summaries are judged correct in 95.9% of cases, sentiment labels match expert majority labels in 79.9% of cases, and aspect labels match in 78.5% of cases. The paper also analyzes temporal trends in ESG aspects, the informativeness of the GPT-4o relevance score, and a BERTopic-based topic analysis for Volkswagen.
Significance. If the claims hold, Nano-ESG is a useful, openly released resource for studying corporate sustainability in the German market, with a time-series dimension that prior ESG news datasets largely lack. The pipeline is described with enough detail to be reproduced, and the decision to release both the dataset and the code is a clear strength. The human evaluation is a genuine attempt at quality assessment with multiple domain experts, which is more than many dataset papers provide. The main limitation is that the evaluation only measures precision on the subset of articles that survived the full filtering pipeline; recall is not measured, so the central accuracy claims are conditional and potentially misleading. The moderate annotator agreement on aspects (Fleiss' kappa = 0.427) further complicates the interpretation of the aspect accuracy figure.
major comments (3)
- [Section 5.1, Section 5.3] All evaluation samples are drawn from articles that already passed the final GPT-4o relevance filter, as described in Section 5.1 (random sampling and k-means sampling applied to the GPT-4o output). Consequently, the reported accuracy figures of 95.9%, 79.9%, and 78.5% are conditional precision values on the surviving set, not end-to-end accuracy estimates. Recall is never measured, so the dataset could be missing a large fraction of ESG-relevant articles. The paper should either sample and evaluate articles discarded at each pipeline stage (paragraph deduplication, GLiNER filtering, GPT-3.5/4o-mini filtering, summary deduplication) to estimate recall, or explicitly reframe all accuracy claims as precision on the filtered set.
- [Section 5.3 (Aspects)] Fleiss' kappa of 0.427 for aspect annotations indicates only moderate agreement among the five sustainability experts, which weakens the reliability of the majority-vote ground truth used to compute the 78.5% aspect accuracy. With this level of disagreement, a low model-annotator match could reflect annotation ambiguity rather than model error. The paper should report per-aspect accuracy, restrict the evaluation to samples with stronger annotator consensus (e.g., at least 4 of 5 agreement), or otherwise characterize how the result changes under alternative ground-truth definitions.
- [Section 5.2] The computation of the 95.9% summary correctness figure is ambiguous. The text states that 35 of 38 consensus-evaluated samples were unanimously judged correct, which would be 92.1%, while 95.9% appears to correspond to 118 out of 123 samples. The paper should state the exact denominator, explain how many annotators evaluated each of the 123 samples, and provide confidence intervals given the small sample size. The current wording makes it impossible to verify the headline number.
minor comments (6)
- [Section 5.2] The phrase "Looking at all samples" is ambiguous; clarify whether it refers to the 38-sample subset or the full 123-sample set, and report both values.
- [Section 5.3] The sentiment simplification rules (neutral combined with positive/negative, etc.) are described after the kappa result is presented; move this description before the kappa calculation so the reader can interpret it properly.
- [Section 6.2] The relevance-score analysis is based on very small per-score samples (e.g., only 1 of 50 annotated articles with score 9 was deemed irrelevant by an annotator), so the conclusion that "the LLM is generally able to determine to which degree an article is relevant" is stronger than the evidence supports; temper the claim or present the analysis as exploratory.
- [Figure 1] The percentages in the middle and right panels of Figure 1 are small and difficult to read; consider presenting the reduction ratios in a table as well.
- [Section 3.4] The sentence "we ask it to first generate a first answer" is redundant; rephrase to avoid the repeated "first."
- [Abstract and Conclusion] The abstract and conclusion state that the LLM-produced answers are "accurate" without qualifying that this is measured only on articles that passed the relevance filter; add the qualifier to avoid overstating the claim.
Circularity Check
No circularity: the accuracy claims rest on independent human annotations, not on quantities defined by the model itself.
full rationale
The paper's central claims concern the quality of GPT-4o-generated summaries, sentiments, and aspects in the Nano-ESG dataset. These claims are evaluated against external human annotations from five sustainability experts (Sections 5.2 and 5.3), not against any quantity derived from the model being evaluated. There is no fitted parameter that is later renamed as a prediction: the cosine-similarity threshold of 0.8, the weekly deduplication window, and the k-means sampling choice are preprocessing and sampling decisions, not parameters fit to the evaluation labels. The paper does not invoke any load-bearing self-citation or uniqueness theorem; its related-work citations provide background, not the evidence for the reported accuracy. The main validity caveat is that evaluation samples are drawn only from articles that survived the relevance filter, so the reported percentages estimate conditional precision rather than end-to-end recall; this is a real external-validity limitation, but not circularity, because the ground-truth labels are independent of the model's outputs. The paper's own Section 7 limitations further acknowledge model errors, but again do not reveal a definitional equivalence between inputs and outputs. The derivation chain is therefore self-contained with respect to circularity, and no circular step can be exhibited.
Assumptions & free parameters
free parameters (4)
- Cosine similarity threshold for deduplication =
0.8
- Deduplication time window =
7 days
- Number of k-means clusters for evaluation sampling =
3
- Sentiment simplification rules for annotator labels =
neutral+positive -> positive; neutral+negative -> negative; positive+negative -> neutral
assumptions (5)
- domain assumption News articles are a valid, real-time source for assessing corporate sustainability practices.
- domain assumption The per-company keyword list is sufficiently comprehensive to capture ESG-relevant news coverage.
- domain assumption GPT-4o's summaries preserve the information needed for correct ESG aspect and sentiment classification.
- domain assumption The five experts' annotations are the ground truth for ESG sentiment and aspect.
- domain assumption Cosine similarity between sentence embeddings is a valid proxy for semantic duplication.
Cite this review
Pith. "Pith review of Nano-ESG: Extracting Corporate Sustainability Information from News Articles." pith.science (2026). https://pith.science/paper/VW4XCNGK
@misc{pith2026241215093,
author = {Pith},
title = {Pith review of: Nano-ESG: Extracting Corporate Sustainability Information from News Articles},
year = {2026},
howpublished = {\url{https://pith.science/paper/VW4XCNGK}},
note = {Machine review of arXiv:2412.15093}
}
read the original abstract
Determining the sustainability impact of companies is a highly complex subject which has garnered more and more attention over the past few years. Today, investors largely rely on sustainability-ratings from established rating-providers in order to analyze how responsibly a company acts. However, those ratings have recently been criticized for being hard to understand and nearly impossible to reproduce. An independent way to find out about the sustainability practices of companies lies in the rich landscape of news article data. In this paper, we explore a different approach to identify key opportunities and challenges of companies in the sustainability domain. We present a novel dataset of more than 840,000 news articles which were gathered for major German companies between January 2023 and September 2024. By applying a mixture of Natural Language Processing techniques, we first identify relevant articles, before summarizing them and extracting their sustainability-related sentiment and aspect using Large Language Models (LLMs). Furthermore, we conduct an evaluation of the obtained data and determine that the LLM-produced answers are accurate. We release both datasets at https://github.com/Bailefan/Nano-ESG.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[1]
SSRN Electronic Journal (2019)
Berg, F., Kölbel, J.F., Rigobon, R.: Aggregate Confusion: The Divergence of ESG Ratings. SSRN Electronic Journal (2019). https://doi.org/10.2139/ ssrn.3438533
work page 2019
-
[2]
Chen, C.C., Tseng, Y.M., Kang, J., Lhuissier, A., Day, M.Y., Tu, T.T., Chen, H.H.: Multi-Lingual ESG Issue Identification. In: Proceedings of the Fifth Workshop on Financial Technology and Natural Language Processing (FinNLP) and the Second Multimodal AI For Financial Forecasting (Muf- fin). pp. 111–115. Macao (8 2023), https://aclanthology.org/2023.finnlp-1. 11
work page 2023
-
[3]
Proceed- ings of the Sixth Workshop on Financial Technology and Natural Language Processing pp
Chen, C.C., Tseng, Y.M., Kang, J., Lhuissier, A., Seki, Y., Day, M.Y., Tu, T.T., Chen, H.H.: Multi-Lingual ESG Impact Type Identification. Proceed- ings of the Sixth Workshop on Financial Technology and Natural Language Processing pp. 46–50 (2023). https://doi.org/10.18653/v1/2023.finnlp-2.6
-
[4]
Chen, C.C., Tseng, Y.M., Kang, J., Lhuissier, A., Seki, Y., Lee, H., Day, M.Y., Tu, T.T., Chen, H.H.: Multi-Lingual ESG Impact Duration Infer- ence. Proceedings of the Joint Workshop of the 7th Financial Technology and Natural Language Processing, the 5th Knowledge Discovery from Un- structured Data in Financial Services, and the 4th Workshop on Economics...
work page 2024
-
[5]
Chiang, W.L., Zheng, L., Sheng, Y., Angelopoulos, A.N., Li, T., Li, D., Zhang, H., Zhu, B., Jordan, M., Gonzalez, J.E., Stoica, I.: Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference. arXiv (2024). https://doi.org/10.48550/arxiv.2403.04132
-
[6]
Clément, A., Robinot, E., Trespeuch, L.: The use of ESG scores in academic literature: a systematic literature review. Journal of Enterprising Commu- nities: People and Places in the Global Economyahead-of-print(ahead- of-print) (2023). https://doi.org/10.1108/jec-10-2022-0147
-
[7]
Ding, B., Qin, C., Liu, L., Chia, Y.K., Li, B., Joty, S., Bing, L.: Is GPT-3 a Good Data Annotator? Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) pp. 11173–11195 (2023). https://doi.org/10.18653/v1/2023.acl-long.626
-
[8]
Schmalenbach Jour- nal of Business Research76(2), 197–244 (2024)
Dorfleitner, G., Zhang, R.: ESG News Sentiment and Stock Price Re- actions: A Comprehensive Investigation via BERT. Schmalenbach Jour- nal of Business Research76(2), 197–244 (2024). https://doi.org/10.1007/ s41471-024-00185-3
work page 2024
Show all 31 references
-
[9]
2023 IEEE International Conference on Big Data (BigData)00, 2823–2830 (2023)
Fischbach, J., Adam, M., Dzhagatspanyan, V., Mendez, D., Frattini, J., Kosenkov, O., Elahidoost, P.: Automatic ESG Assessment of Companies by Mining and Evaluating Media Coverage Data: NLP Approach and Tool. 2023 IEEE International Conference on Big Data (BigData)00, 2823–2830...
2023
-
[10]
Psychological Bulletin 76(5), 378–382 (1971)
Fleiss, J.L.: Measuring nominal scale agreement among many raters. Psychological Bulletin 76(5), 378–382 (1971). https://doi.org/10.1037/ h0031619
1971
- [11]
-
[12]
arXiv (2024)
He, Z., Huang, C.Y., Ding, C.K.C., Rohatgi, S., Huang, T.H.K.: If in a Crowdsourced Data Annotation Pipeline, a GPT-4. arXiv (2024). https: //doi.org/10.1145/3613904.3642834
2024
- [13]
-
[14]
Sustainable Production and Consumption25, 123–135 (8 2020)
Islam, T., Islam, R., Pitafi, A.H., Xiaobei, L., Rehmani, M., Irfan, M., Mubarak, M.S.: The impact of corporate social responsibility on customer loyalty: The mediating role of corporate reputation, customer satisfaction, and trust. Sustainable Production and Consumption25, 12...
2020 doi
-
[15]
arXiv (2024)
Mohr, I., Krimmel, M., Sturua, S., Akram, M.K., Koukounas, A., Günther, M., Mastrapas, G., Ravishankar, V., Martínez, J.F., Wang, F., Liu, Q., Yu, Z., Fu, J., Ognawala, S., Guzman, S., Wang, B., Werk, M., Wang, N., Xiao, H.: Multi-Task Contrastive Learning for 8192-Token Bilin...
-
[16]
IEEE Access12, 71876–71900 (2024)
Nasution, A.H., Onan, A.: ChatGPT Label: Comparing the Quality of Human-Generated and LLM-Generated Annotations in Low-Resource Lan- guage NLP Tasks. IEEE Access12, 71876–71900 (2024). https://doi.org/ 10.1109/access.2024.3402809
2024
-
[17]
Environment, Development and Sustainability26(2), 3045–3079 (2024)
Oliveira, U.R.d., Menezes, R.P., Fernandes, V.A.: A systematic literature review on corporate sustainability: contributions, barriers, innovations and future possibilities. Environment, Development and Sustainability26(2), 3045–3079 (2024). https://doi.org/10.1007/s10668-023-02933-7
2024 doi
-
[18]
arXiv (2023)
OpenAI: GPT-4 Technical Report. arXiv (2023)
2023
-
[19]
arXiv (2022)
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C.L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., Lowe, R.: Training language models to follo...
- [20]
-
[21]
Journal of Business Ethics171(3), 583–597 (2021)
Salmones, M.d.M.G.d.l., Herrero, A., Martínez, P.: Determinants of Elec- tronic Word-of-Mouth on Social Networking Sites About Negative News on CSR. Journal of Business Ethics171(3), 583–597 (2021). https://doi.org/ 10.1007/s10551-020-04466-9
2021 doi
-
[22]
arXiv (2024)
Tan, Z., Beigi, A., Wang, S., Guo, R., Bhattacharjee, A., Jiang, B., Karami, M., Li, J., Cheng, L., Liu, H.: Large Language Models for Data Annotation: A Survey. arXiv (2024). https://doi.org/10.48550/arxiv.2402.13446 Nano-ESG: Extracting ESG Information from News Data 15
-
[23]
Team, L.: The Llama 3 Herd of Models (7 2024), https://ai.meta.com/ research/publications/the-llama-3-herd-of-models/
2024
-
[24]
arXiv (2023)
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., Lample, G.: LLaMA: Open and Efficient Foundation Language Models. arXiv (2023). https://doi.org/10.48550/arxiv.2...
-
[25]
arXiv (2023)
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C.C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N.,...
- [26]
-
[27]
PLoS ONE15(9), e0237073 (2020)
Vliet, L.v., Törnberg, P., Uitermark, J.: The Twitter parliamentarian database: Analyzing Twitter politics across 26 countries. PLoS ONE15(9), e0237073 (2020). https://doi.org/10.1371/journal.pone.0237073
2020 doi
- [28]
- [29]
-
[30]
Practice
Zumente,I.,Bistrova,J.:ESGImportanceforLong-TermShareholderValue Creation: Literature vs. Practice. Journal of Open Innovation: Technol- ogy, Market, and Complexity7(2), 127 (2021). https://doi.org/10.3390/ joitmc7020127
2021
-
[31]
https://doi.org/10.3390/ su13168940
Zumente, I., L¯ ace, N.: ESG Rating—Necessity for the Investor or the Company? Sustainability 13(16), 8940 (2021). https://doi.org/10.3390/ su13168940
2021
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.