{"id":"0115c933-c2d3-492f-ba55-017686bebbca","arxiv_id":"2506.15208","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"On a balanced benchmark built from the OSDG community dataset, fine-tuned LLaMa-2 13B achieves the highest macro F1 (92.4%), and small models such as Flan-T5-base (220M) reach 90.5%, close to fine-tuned GPT-3.5 (91.4%).","lead":"This paper compares 18 large language models on classifying text into the UN Sustainable Development Goals, using zero-shot, few-shot, and fine-tuning approaches. It finds that smaller open models, especially when fine-tuned, can match or beat much larger commercial models like GPT-3.5.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'Other' class is drawn from online news while the 16 SDG classes come from UN reports and policy documents, so macro-F1 may reflect genre discrimination rather than SDG relevance; the paper reports no source-stratified or per-class analysis to rule this out.","rationale":"The paper's headline is about SDG identification, not about separating UN reports from online news. The construction of the seventeenth class in Section III-A introduces a distribution shift between 'Other' and the 16 SDG classes. A model could achieve high macro-F1 by learning this source cue, especially in the fine-tuning setting where a task-specific layer can pick up superficial regularities. This threatens the validity of the benchmark that all model comparisons rely on, including the top F1 scores in Table 7. The reader's conditional verdict already flags this; my stress test agrees and makes the required control explicit. I also note the internal contradiction about LLaMa-2-70b ZSL inclusion (Section IV vs Table 3) and the absence of significance tests, but those are secondary to the dataset confound. If the matched-distribution control passes, the central ranking is much more credible; if it fails, the abstract overclaims. Hence the verdict remains conditional pending this check.","tokens_in":24406,"tokens_out":8590,"duration_ms":91111,"concrete_test":"Build a control test set in which 'Other' texts are sampled from the same source distribution as the SDG classes—e.g., UN reports and policy documents that do not map to any SDG—and, symmetrically, include SDG-relevant news articles among the positive classes. Re-run fine-tuned LLaMa-2-13B and GPT-3.5-turbo on this matched-distribution test. If macro-F1 drops by more than about 2 points relative to the original test, or if the LLaMa-2 vs GPT-3.5 ordering changes, the original benchmark's 'Other' class is a genre confound and the headline claim needs qualification.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section III-A builds the 'Other' class from 400 texts sourced from websites providing online news, manually verified as non-SDG, while the 16 SDG classes come from the OSDG Community Dataset (UN reports, policy documents, publication abstracts). A classifier can therefore separate 'Other' from SDG classes using document source and genre features instead of SDG relevance, and the reported macro-F1—including LLaMa-2-13B's 92.4 vs GPT-3.5's 91.4—may be inflated by this cue. The manual verification is a single-step, non-independent screen; no inter-annotator agreement or source-matched negative control is reported. Because the abstract's claim is about identifying SDGs in text, the benchmark must show that the 'Other' category behaves like SDG-irrelevant text from the same distribution as the positive classes. Without per-class F1 or a matched-distribution test, the central comparison is not yet established as a measure of SDG relevance.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a comparative evaluation of 18 language models for single-label multi-class classification of text into 16 SDG classes plus an 'Other' class. Using a benchmark extracted from the OSDG Community Dataset (October 2023), with balanced train/validation/test splits (280/60/60 examples per class), the authors evaluate zero-shot, few-shot (3/5/7 examples with three selection strategies: random, semantic similarity, and semantic similarity across different classes), and fine-tuning. They report macro-averaged precision, recall, accuracy, and F1. Main empirical findings: fine-tuned LLaMa-2-13B achieves F1 92.4%, GPT-3.5-turbo FT 91.4%, Flan-T5-XXL 90.9%; semantic similarity-based example selection outperforms random and diverse-class selection; quantized models remain competitive. The paper concludes that smaller open models can match or exceed larger proprietary models when fine-tuned on high-quality data.","tokens_in":24661,"tokens_out":5686,"duration_ms":54183,"significance":"If the benchmark is valid, this study provides practically useful, falsifiable evidence about model and adaptation-technique choices for SDG text classification. Its strengths include the use of a public volunteer-labelled dataset, public prompt templates, coverage of 18 models across three adaptation families, explicit hyperparameters for fine-tuning, and consistent internal tables. The main caveat is that the negative class ('Other') is constructed from a different document distribution than the 16 SDG classes, which must be addressed before the F1 comparisons can be interpreted as measuring SDG-relevance discrimination rather than genre or source discrimination.","major_comments":[{"comment":"The construction of the 'Other' class is a load-bearing issue for the benchmark claim. The 16 SDG classes are drawn from UN reports, policy documents, and publication abstracts, while the 400 'Other' texts are sourced from online news websites and manually verified as non-SDG. With a balanced 17-class test set, macro-F1 weights 'Other' equally with each SDG class, and a classifier can separate 'Other' from the positive classes using source/genre cues rather than SDG relevance. The paper reports no per-class F1 and no source-stratified analysis that would rule out this shortcut. I recommend reporting per-class F1 (especially for 'Other'), and ideally adding a matched-distribution negative control, e.g., non-SDG excerpts from the same UN/policy/publication document sources, or at least an explicit quantitative analysis of the genre cue. Without this, the central comparison is not yet established as a measure of SDG-identification performance.","section":"Section III-A"},{"comment":"All reported results appear to come from single runs, with no confidence intervals, standard deviations, or significance tests. Topline differences used for ranking (LLaMa-2-13B 92.4 vs GPT-3.5 91.4 vs Flan-T5-XXL 90.9) are within one or two points and could easily be within run-to-run or test-set noise. I recommend adding repeated runs with multiple seeds for open models and reporting mean and standard deviation; for API models, statistical tests over the test set (e.g., McNemar's test or bootstrap confidence intervals) would substantially strengthen the ranking claims.","section":"Section V / Tables 3-7"},{"comment":"There is a direct contradiction about which models were evaluated in zero-shot. The text states: 'Due to resource limitations, we conducted ZSL experiments on all models except LLaMa-2-70b-chat-hf', yet Table 2 and Table 3 both include Llama-2-70b-chat-hf with ZSL results. This is almost certainly a typo (the intended exception is probably fine-tuning), but it must be corrected, as the current wording undermines the reproducibility of the experimental protocol.","section":"Section IV"}],"minor_comments":[{"comment":"The sentence reporting 'an average increase in the F1 score of 25.7% points compared to the ZSL' is unclear because not all fine-tuned models have a corresponding ZSL result; please specify exactly which subset of models the average is over.","section":"Section V"},{"comment":"The caption reads 'ZSL focused on the SEM. SIMILARITY method', but the FS-SS columns are few-shot results; the caption should say 'FSL with semantic similarity'.","section":"Table 8"},{"comment":"Reference [2] appears to share the same URL as reference [1]; please verify and correct the URL for reference [2].","section":"References"},{"comment":"There is a typo: 'Zephir' in the fine-tuning hyperparameter description should be 'Zephyr'.","section":"Section IV"},{"comment":"The text describes BERT as one of the '18 language models' but BERT is an encoder-only model and not typically considered an LLM; please clarify the terminology or describe BERT as a baseline model.","section":"Section III-B"}],"recommendation":"major_revision","confidential_remarks":"The 'Other'-class distribution mismatch is likely to be a central point of contention in review; the authors should be pushed to add per-class F1 and a matched-distribution negative control before resubmission. Additionally, only the prompts are public; releasing fine-tuning code or the exact splits would materially strengthen reproducibility. The manuscript is otherwise a reasonable comparative study that could become acceptable after addressing the benchmark-validity and statistical-reliability concerns."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a genuinely useful comparative evaluation, and the headline result—fine-tuned open/small models (LLaMa-2 13B at 92.4, Flan-T5-base at 90.5) matching or beating fine-tuned GPT-3.5 (91.4)—is supported by the tables. But the benchmark's 'Other' class is drawn from a different text distribution than the 16 SDG classes, and the paper never shows per-class or source-stratified results. Read the absolute F1 numbers with caution.\n\nWhat's new: a balanced 17-class benchmark derived from OSDG-CD (6400 UN/report/policy excerpts plus 400 news 'Other' texts), and a wide landscape of 18 LLMs across zero-shot, few-shot with three selection strategies, fine-tuning, and 4-bit quantization. I don't know of another SDG classification paper covering this much ground. The tables are legible, the methods are described in enough detail to reproduce, and the prompts are posted.\n\nSoft spots, in rough order of severity:\n\n1. The stress-test concern is valid. The 'Other' class is online news; the 16 positive classes are mostly UN reports, policy documents, and publication abstracts. A model can separate by genre rather than by SDG relevance. The authors manually verified the 400 texts, but there is no inter-annotator agreement and no source-matched negative control. Per-class F1 is not reported, so we can't see whether the gap between LLaMa-2 13B and GPT-3.5 is driven by 'Other' or by real SDG distinctions. This doesn't invalidate the relative ranking within this benchmark, but it does mean the benchmark isn't yet established as measuring SDG relevance.\n\n2. No error bars, no significance tests, single runs. Several reported differences (e.g., Flan-T5-large vs base at 90.5 vs 90.5) are likely noise.\n\n3. Internal contradiction: Section IV says ZSL was not run on LLaMa-2-70b due to resource limits, but Table 3 lists it.\n\n4. The abstract says 'optimized through prompt engineering' but the top results come from fine-tuning, which is a different adaptation method. The conclusion says this more accurately.\n\nThe citation pattern is fine; several self-citations are to the authors' prior work on related topics, which is normal. The benchmark construction and split are not released, only prompts; that should be fixed.\n\nBottom line: this is a solid empirical paper worth sending to a serious referee. The referee should ask for confidence intervals, per-class results, and a matched-distribution 'Other' set. If you're building an SDG classifier, the open-model fine-tuning results are encouraging, but treat the absolute numbers as upper bounds until the benchmark issue is addressed.","headline":"Useful wide comparison of LLMs for SDG classification, with a real benchmark flaw in the 'Other' class that should be fixed before the absolute numbers are trusted.","tokens_in":25195,"tokens_out":3392,"would_cite":true,"duration_ms":33668,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A fine-tuned 13-billion-parameter LLaMa-2 model classifies UN Sustainable Development Goal texts at 92.4% F1, slightly beating fine-tuned GPT-3.5; smaller models can match larger ones when prompts and fine-tuning are chosen well.","keywords":["large language models","text classification","sustainable development goals","few-shot learning","zero-shot learning","fine-tuning","prompt engineering","model quantization"],"falsifier":"Reserve a test subset where 'Other' texts are drawn from the same UN report and policy-document sources as the SDG classes (manually verified to mention none of the 16 goals); if fine-tuned LLaMa-2 13B's F1 on that subset falls well below its overall 92.4%, the benchmark rewards a genre shortcut. A cleaner test would be a two-by-two design crossing document genre with SDG relevance and measuring whether models classify relevance within each genre.","tokens_in":24235,"feed_emoji":"🌍","tokens_out":7749,"duration_ms":70107,"temperature":0.7,"pith_summary":"The paper asks whether small open-source language models, guided by well-chosen prompts and fine-tuning, can match or beat much larger proprietary models at assigning short texts to the UN Sustainable Development Goals. It builds a balanced 17-class benchmark from volunteer-validated excerpts in the OSDG Community Dataset and evaluates 18 models under zero-shot learning, few-shot learning, and fine-tuning. Its headline finding is that fine-tuned LLaMa-2 13B reaches 92.4% F1, ahead of fine-tuned GPT-3.5-turbo at 91.4%, while few-shot learning with semantically similar exemplars adds roughly 20 F1 points over random selection. The paper concludes that task adaptation, not raw parameter count, drives performance, making accurate SDG classification feasible with relatively small open-weight models.","feed_headline":"Small fine-tuned models beat GPT on SDG text tagging","feed_subtitle":"A 13B LLaMa-2 hits 92.4% F1; semantic few-shot selection rivals far larger LLMs.","key_machinery":"The benchmark itself is the load-bearing object: 6,800 excerpts, 400 per class, covering SDGs 1–16 (from the OSDG Community Dataset, filtered to high volunteer agreement) plus an 'Other' class built from 400 online news articles, split 4,760/1,020/1,020 for training, validation, and test. Around it, the paper varies only the adaptation machinery: zero-shot prompts; few-shot prompts with 3, 5, or 7 exemplars chosen randomly, by cosine similarity, or by cosine similarity with a distinct-class constraint; and fine-tuning via LoRA (low-rank adaptation) or QLoRA (quantized LoRA) for 4-bit models. The mechanism that carries the argument is the interaction between selection strategy and model: semantic-similarity exemplars give the largest few-shot gains, and LoRA fine-tuning is what pushes a 13B open model past a larger proprietary one.","core_discovery":"The paper's central claim is that for this task the adaptation technique matters more than model scale. In zero-shot settings, the GPT models lead (GPT-3.5 at 81.1%, GPT-4 at 80.3%), but fine-tuning reverses the order: LLaMa-2 13B reaches 92.4%, GPT-3.5-turbo 91.4%, and Flan-T5-XXL 90.7%. In few-shot settings, selecting exemplars by cosine semantic similarity outperforms random selection by 19.3–21.5 F1 points and outperforms similarity-with-different-classes, with LLaMa-3 8B peaking at 3 examples, GPT-4 at 5, and Mixtral-8x7B at 7. Fine-tuning improves every model by an average of 25.7 percentage points over zero-shot, and 4-bit quantized variants such as LLaMa-2 13B (85.7%) and Zephyr 7B (86.4%) remain competitive.","pith_inferences":["If the 'Other' class were drawn from UN-style reports and policy documents instead of online news, classification could get harder; part of the reported accuracy may come from distinguishing news-like from report-like text rather than SDG relevance.","The semantic-similarity result generalizes as a practical recipe: for multi-class text tasks, retrieval-based exemplar selection should be the default few-shot setup, not random sampling.","Because GPT-3.5 edged GPT-4 in zero-shot, the paper implies that prompt configuration matters as much as model generation; testing systematic prompt search across newer models would clarify when scale helps.","The benchmark's single-label design leaves multi-label SDG overlap unexamined, even though real documents often address several goals at once; a natural extension is to evaluate on texts with multiple valid labels."],"forward_implications":["Fine-tuned open-weight models such as LLaMa-2 13B can serve as accurate SDG classifiers, reaching about 92% F1 on a balanced 17-class benchmark.","When fine-tuning is unavailable, few-shot prompts should be built by retrieving semantically similar training excerpts; random exemplars give essentially zero benefit over zero-shot.","Quantized 4-bit models (LLaMa-2 13B 4-bit at 85.7%, Zephyr 7B 4-bit at 86.4%) allow near-competitive SDG classification without large GPU clusters.","Parameter count does not determine rank: BERT at 110M parameters beats LLaMa-3 8B when both are fine-tuned, and Flan-T5-base at 220M reaches 90.5%.","The optimal number of few-shot exemplars is model-specific, so deployment should tune the example count rather than assume more examples are always better."],"supporting_citations":[{"why":"Supplies the volunteer-validated OSDG Community Dataset from which the benchmark's 16 SDG classes are drawn.","marker":"[12]"},{"why":"Defines LLaMa-2 chat models, including the 13B variant that achieves the best fine-tuned F1.","marker":"[45]"},{"why":"Introduces the Flan-T5 model family, whose fine-tuned variants reach near-top performance.","marker":"[47]"},{"why":"Provides QLoRA, used to produce the 4-bit quantized models tested in low-resource settings.","marker":"[42]"},{"why":"Documents GPT-4, the proprietary baseline that few-shot smaller models match or exceed.","marker":"[44]"},{"why":"Presents Mistral 7B, a small open model that outperforms LLaMa-3 8B after fine-tuning.","marker":"[49]"},{"why":"Describes Zephyr 7B, whose 4-bit fine-tuned version is among the top quantized results.","marker":"[51]"},{"why":"Introduces Phi-3 small language models, the weakest fine-tuned performers in the comparison.","marker":"[52]"},{"why":"Supplies BERT, the encoder-only baseline that beats LLaMa-3 8B when fine-tuned.","marker":"[14]"},{"why":"Corroborates that small models fine-tuned on high-quality data can match much larger models.","marker":"[54]"}],"fun_headline_variants":["Fine-tuning flips the LLM leaderboard for SDG tagging","13B LLaMa tops GPT-4 with fine-tuning on SDGs","Semantic few-shot boosts SDG classification by 21 F1","Fine-tuning beats scale: 13B model edges GPT on SDGs","From 81% to 92%: Fine-tuning wins over zero-shot for SDGs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's 'Other' class comes from online news articles while the 16 SDG classes come from UN reports and policy documents, so the models may be separating text by genre rather than by SDG relevance.","fun_headline_variants_meta":{"raw":{"variants":["Fine-tuning flips the LLM leaderboard for SDG tagging","13B LLaMa tops GPT-4 with fine-tuning on SDGs","Semantic few-shot boosts SDG classification by 21 F1","Fine-tuning beats scale: 13B model edges GPT on SDGs","From 81% to 92%: Fine-tuning wins over zero-shot for SDGs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000694,"raw_usage":{"total_tokens":3144,"prompt_tokens":958,"completion_tokens":2186,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":574,"completion_tokens_details":{"reasoning_tokens":2087}},"tokens_in":574,"tokens_out":2186,"duration_ms":14694,"temperature":1.0,"reasoning_tokens":2087,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:40:01.910904+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Reserve a test subset where 'Other' texts are drawn from the same UN report and policy-document sources as the SDG classes (manually verified to mention none of the 16 goals); if fine-tuned LLaMa-2 13B's F1 on that subset falls well below its overall 92.4%, the benchmark rewards a genre shortcut. A cleaner test would be a two-by-two design crossing document genre with SDG relevance and measuring whether models classify relevance within each genre.","supporting_citations":[{"cited_title":"OSDG -- Open-Source Approach to Classify Text Data by UN Sustainable Development Goals (SDGs)","cited_arxiv_id":"2005.14569","evidence_quote":"Supplies the volunteer-validated OSDG Community Dataset from which the benchmark's 16 SDG classes are drawn."},{"cited_title":"Dettmers, A","cited_arxiv_id":null,"evidence_quote":"Provides QLoRA, used to produce the 4-bit quantized models tested in low-resource settings."},{"cited_title":"Devlin, M.-W","cited_arxiv_id":null,"evidence_quote":"Supplies BERT, the encoder-only baseline that beats LLaMa-3 8B when fine-tuned."}],"review_version":1}