{"id":"48d399ff-ed3f-42d5-9e65-32be7aff967f","arxiv_id":"2607.17952","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"In TCFD climate-disclosure classification, the strongest in-source adaptation strategies (similarity retrieval, LoRA fine-tuning) lose most of their advantage under source shift, while definitions and random few-shot examples transfer more reliably.","lead":"This paper tests whether three common LLM adaptation tricks — richer definitions, in-context examples, and LoRA fine-tuning — still work when a climate-disclosure classifier is evaluated on a different document source than it was adapted on. Across 11 models it finds that the strategies that win on the original source often lose their advantage under source shift; simpler choices like random few-shot examples and granularity-matched definitions are the more reliable cross-sou","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Cross-source claims rest on an untested assumption that TCFD-CR and TCFD-EX labels are interchangeably reliable; without label-alignment validation, the strategy ranking may reflect annotation artifacts rather than transfer.","rationale":"The reader's weakest assumption was the single source-target pair. I agree that this is a limitation, but the deeper load-bearing issue is that the target corpus (TCFD-EX) is not a naturally occurring source and the two corpora's labels are not validated as interchangeable. If the label semantics differ across corpora, the central comparison is compromised even for the single pair, because the 'cross-source' difference may capture annotation disagreement rather than a shift in disclosure style. This concern is especially pressing given that the paper itself cites Calamai et al. (2025) reporting pervasive annotation issues in climate-NLP benchmarks, yet it does not test the labels of its own datasets. The proposed concrete test—human annotation and kappa on a sample—would settle whether the labels are reliable enough to support the claims. I do not think this changes the overall verdict: the paper remains valuable as a transparent, carefully controlled study of one shift pair, and the CONDITIONAL verdict is appropriate. Secondary concerns, such as the random few-shot baseline being a single fixed draw without seed variance, are less load-bearing because the per-model pattern is consistent and the aligned comparison in Figure 9 supports the same direction.","tokens_in":16269,"tokens_out":13105,"duration_ms":111727,"concrete_test":"Take a random sample of 100–200 instances from TCFD-CR and TCFD-EX. Have two independent annotators experienced in TCFD label each text using the same official TCFD pillar definitions, then compute Cohen's kappa between the existing labels and the majority human label separately for each corpus. Also run a label-alignment check: for a subset of TCFD-EX instances, prompt a strong LLM with each corpus's original label description; if the label distribution shifts across corpora, the label semantics differ. If kappa is below about 0.7 or label distributions differ beyond sampling noise, the cross-source gains in Table 1 and Figure 5 may be artifacts of annotation rather than transfer.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim—that in-source strength does not imply cross-source transfer and that definitions plus simple prompts are safer defaults—depends on the two corpora sharing a common TCFD label space with interchangeable labels (§3.1). TCFD-CR (Bingler et al., 2022) and TCFD-EX (Qiu, 2023) were annotated independently, and the paper shows no evidence that the four-pillar labels mean the same thing in both. The related work it cites (Calamai et al., 2025) reports pervasive annotation issues in climate-NLP benchmarks, making this a live risk rather than a pedantic one. Unless the label semantics are aligned, the cross-source gains attributed to 'definitions' and the ranking of strategies could reflect annotation differences between the two corpora rather than properties of source shift. For example, if TCFD-EX labels were created by matching texts to the TCFD's Recommended Disclosures, then the D2 definition prompt may be 'matching the target' in the trivial sense of the target's construction, not in a way that transfers to press releases or earnings calls. The Limitations section acknowledges the single-pair design but not the label-equivalence assumption, so the most load-bearing premise is untested.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reframes TCFD climate-disclosure classification as a cross-source adaptation problem. It uses two corpora that share the four-pillar TCFD label space but differ in source: TCFD-CR (annual/sustainability reports) as the adaptation source and TCFD-EX (TCFD example disclosures) as the held-out cross-source target. The authors compare three adaptation strategies—definition prompts (D0–D3), in-context examples (random, TF-IDF, and dense retrieval), and LoRA fine-tuning—across eleven prompted LLMs (with LoRA on four open-weight bases). They report that all strategies yield positive average cross-source gains, but in-source strength does not transfer uniformly: similarity retrieval and LoRA gain most in-source yet lose most of their advantage cross-source, random few-shot retains its smaller advantage more reliably, and definitions transfer most consistently, with performance depending on the match between definition granularity and target text. The paper concludes that 'simpler is often safer' under source shift and offers practitioner-oriented defaults.","tokens_in":16481,"tokens_out":7169,"duration_ms":66463,"significance":"If the results are taken at face value, the paper makes a useful and non-obvious contribution: it documents a concrete setting where in-source adaptation performance is a misleading guide to cross-source deployment, and it does so with a carefully controlled design. Strengths include a shared adaptation pool for all strategies, an aligned k=8/n=2 comparison that isolates the selection rule (Figure 9), three-seed LoRA means, per-model transparency throughout the appendix tables, a clear definition staircase, and a public code release. The paper also measures gains against each strategy's own zero-shot baseline, avoiding the most common circularity in strategy comparisons. However, the external validity of the headline claim rests on a single source-target pair and on an unvalidated assumption that the two corpora's TCFD labels are interchangeable; the limited evidence directly supporting 'definitions transfer' is confounded by the way TCFD-EX appears to be constructed from the same recommended-disclosure wording as the D2 prompt.","major_comments":[{"comment":"The central cross-source comparison assumes the four-pillar labels of TCFD-CR and TCFD-EX are interchangeable, but the paper provides no annotation-agreement or label-alignment evidence for either corpus. The cited reproducibility study (Calamai et al., 2025) reports pervasive annotation issues in climate-NLP benchmarks, so this is a live risk rather than a pedantic one. If the two corpora were annotated under different interpretations of 'Strategy' or 'Risk Management', the strategy ranking could reflect annotation artifacts rather than transfer. A concrete test would be to have annotators label a random sample from each corpus under the other corpus's label protocol and measure agreement, or at minimum to report the original annotation guidelines and a label-conditional error analysis. This assumption is load-bearing for the abstract's cross-source claim.","section":"§3.1, Figures 2–3; Limitations"},{"comment":"The claim that definitions transfer most consistently is partly confounded by the construction of the target corpus. TCFD-EX is derived from the TCFD's example-disclosures list, and the D2 prompt inserts the 11 Recommended Disclosures. The paper's own case study (Figure 12, Case B) shows a TCFD-EX Strategy instance that is a near-verbatim paraphrase of Recommended Disclosure c. The apparent D2 advantage on TCFD-EX may therefore reflect lexical/template overlap between the prompt and the target corpus's construction, not a general property of semantic definitions under source shift. To support the general claim, the authors should evaluate on a target source not built from the same recommended-disclosure wording (e.g., press releases or earnings calls) or, at a minimum, show that the result survives after excluding near-verbatim examples.","section":"§5.2, Appendix B, Figure 12"},{"comment":"The broad conclusion that 'when the source changes, simpler is often safer' rests entirely on a single source-target pair with one shift direction: long, discursive annual-report paragraphs to short, canonical TCFD example disclosures. The Limitations section honestly acknowledges the single pair, but the abstract and conclusion do not carry that caveat. The deployment sources named in the Introduction—press releases and earnings calls—could plausibly produce a different strategy ranking. Without additional target sources or a clearer re-scoping of the claims to the studied pair, the external validity of the headline finding remains unsupported.","section":"§5.1, Limitations, Conclusion"}],"minor_comments":[{"comment":"The main retrieval table selects (k, n) as the per-model 'best-on-CR' configuration using the in-source test set, not a validation split. This can inflate in-source gains and make the cross-source drop look larger. The aligned k=8, n=2 comparison in Figure 9 supports the same qualitative conclusion, so this is not a fatal flaw, but the authors should either use validation-based selection or acknowledge the optimistic bias in Table 1.","section":"§5.3, Table 1, Figure 8"},{"comment":"The text emphasizes that similarity retrieval 'barely improves' cross-source on average, but Table 7 shows that the best TF-IDF configuration reaches the highest TCFD-EX macro-F1 of any strategy (0.713, Qwen3.5-27B), above the best definition prompt (0.707). The average statements should be accompanied by per-model ranges so readers do not conclude retrieval is uniformly poor cross-source.","section":"§5.1, Table 7"},{"comment":"Most reported means lack confidence intervals or model-level variance bars; only Figure 9 provides a bootstrap CI. Since the headline claims are about average behavior across models, reporting CIs for the mean gains (or at least per-model dot plots) would make the strength of the claims easier to assess.","section":"Figures 5–6 and Tables 1/4/5"},{"comment":"The statement 'all strategies bring positive cross-source gains on average' is true for the mean, but Figure 5 and Table 1 show several per-model points at or below zero for random and similarity few-shot. The authors should make the per-model dispersion more prominent in the prose to avoid overstating the robustness of the average.","section":"Abstract and §5.1"}],"recommendation":"major_revision","confidential_remarks":"This is a careful measurement paper with a strong experimental core: the shared adaptation pool, the aligned selection-rule comparison, three-seed LoRA, and per-model transparency are all good practices. My main reservation is that the central claim about definitions transferring is partly confounded by TCFD-EX being derived from the same recommended-disclosure wording as the D2 prompt, and the label-equivalence assumption is untested. I would encourage the editor to invite a revision that adds a label-alignment check and a non-TCFD-derived target source (or clearly re-scopes the conclusions), rather than rejecting."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"I read this one as an empirical measurement paper, not a derivation, and it's a good example of the genre. The setup is unusually clean: a single adaptation pool, eleven models, three strategies, and an explicit in-source vs. cross-source contrast. The headline result — that the strongest in-source strategies (similarity retrieval, LoRA) lose most of their advantage under source shift, while random few-shot and definition-based prompts transfer more reliably — is supported by the reported data within the tested scope. That's a genuinely useful finding for practitioners building climate-disclosure classifiers, and the authors share code and resources, which helps.\n\nWhat's new here is the systematic comparison itself. I'm not aware of another source-shift study for TCFD classification that sweeps definitions, examples, and fine-tuning across this many models with shared pools and a controlled definition staircase (D0–D3). The per-model tables and the aligned comparison in Figure 9 show real discipline.\n\nThe main soft spot is that everything rests on one source-target pair, and the limitations section concedes this but understates the consequence: the two corpora were annotated independently, with no evidence that their four-pillar labels are interchangeable. If TCFD-EX was built from the TCFD's own example disclosures, then the D2 definition prompt may align with how the target labels were constructed, inflating the 'definitions transfer' result. Given that the paper cites Calamai et al. 2025 on annotation issues in climate-NLP benchmarks, this is a live risk, not a pedantic one. I'd want a label-consistency check or at least a frank acknowledgment that the cross-source ranking is conditional on label equivalence.\n\nOther caveats are minor: the 'granularity matches target text' explanation is post hoc and rests on two illustrative cases, though the aggregate direction is clear; the headline retrieval numbers use best-on-CR tuning while the random baseline is fixed, though Figure 9 addresses that; and headline magnitudes lack error bars. None of these break the core claim for the studied pair.\n\nFor a reader who works on climate disclosure NLP or on evaluating adaptation under distribution shift, this is worth a careful read. It's not a general law yet — it's one strong, well-documented case. I'd send it to reviewers, with the label-equivalence issue as the main requested addition.","headline":"Strong, clean empirical comparison of LLM adaptation under source shift for TCFD classification; claims hold for the tested pair but the label-equivalence premise is untested.","tokens_in":17066,"tokens_out":3379,"would_cite":true,"duration_ms":28910,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"For TCFD climate-disclosure classification, the adaptation strategies that win on the source corpus—similarity-based retrieval and LoRA fine-tuning—lose most of their advantage when the disclosure source changes, while random few-shot examp","keywords":["source shift","domain adaptation","climate disclosure classification","TCFD","in-context learning","few-shot selection","LoRA fine-tuning","definition prompting"],"falsifier":"Take a third disclosure source that shares the same four TCFD labels—for example, press releases or earnings-call transcripts—and run the same three strategies with the same adaptation pool. If similarity retrieval or LoRA fine-tuning matches or beats random few-shot and definitions on that target, the claimed in-source/cross-source rank reversal is specific to the CR-to-EX pair rather than a general property of source shift. A second check would be to relabel a sample of both corpora by human experts: if the four labels are not applied consistently across the two sources, the transfer compari","tokens_in":16059,"feed_emoji":"📊","tokens_out":6348,"duration_ms":54250,"temperature":0.7,"pith_summary":"The paper asks which LLM adaptation strategies survive a change in disclosure source for TCFD climate-disclosure classification, where the same four labels appear in annual reports and in official short example disclosures. It studies eleven open- and closed-source LLMs with a fixed adaptation pool, comparing definitions, in-context examples selected randomly or by similarity, and LoRA fine-tuning. It finds that all strategies help on average cross-source, but the strongest in-source strategies do not transfer: similarity retrieval gains +8.0 points in macro-F1 on the source but only +0.5 on the target, and LoRA gains +14.1 but retains only +3.7, while random few-shot holds most of its smaller gain and definitions gain even more cross-source than in-source. The paper's proposed mechanism is that what transfers is semantic guidance about what the TCFD categories mean, and what is lost is source-specific decision patterns and target-to-source textual similarity; consequently, simpler is often safer under source shift.","feed_headline":"In-source winners lose under source shift","feed_subtitle":"Climate-disclosure classification: random few-shot and matching definitions transfer; retrieval and fine-tuning don't.","key_machinery":"The central device is a three-way controlled comparison: three adaptation strategies (definitions, examples, fine-tuning) injected at different points—prompt semantics, in-context selection, and parameters—evaluated on both an in-source held-out set and a cross-source target set drawn from the same shared label space but a different disclosure source. Two analytical instruments carry the argument: the source-binding plane, which plots each strategy's in-source gain against its cross-source gain and exposes which advantages sit above or below the perfect-transfer diagonal; and the definition staircase (D0 label names, D1 pillar descriptions, D2 the 11 Recommended Disclosures, D3 27 criteria),","core_discovery":"The paper's central claim is that in-source adaptation strength is not a reliable guide to cross-source reliability in this task. Using a controlled design that shares one adaptation pool between in-source and cross-source evaluation, it shows that strategies which fit the source corpus well—similarity-based example retrieval and LoRA fine-tuning—are the ones that lose the most when the source changes. Conversely, randomly selected few-shot examples, a weaker in-source baseline, retain their advantage more reliably, and definitions transfer most consistently. The paper attributes this to a distinction: semantic guidance that helps the model grasp what the four TCFD pillars mean transfers, wh","pith_inferences":["Because the cross-source evidence is a single pair (annual reports adapted, official examples evaluated), the 'simpler is safer' generalization is tentative until tested on other shift directions like press releases or earnings calls; the proxy A-distance shows the pair is nontrivial, but it does not guarantee representativeness.","The definition-granularity matching result suggests a practical deployment heuristic: estimate the target text's granularity (e.g., length and lexical alignment with TCFD wording) and choose the definition level accordingly; this could be automated and evaluated.","The weights-plus-prompt finding implies that archived fine-tuned classifiers are incompletely specified without their inference prompts; cross-source evaluations that freeze a training prompt may understate achievable transfer.","If the random-beats-retrieval pattern holds beyond this pair, it would challenge the widespread practice of tuning retrievers in-domain for few-shot classification in regulated disclosure analysis, where the cost of a wrong label is high."],"forward_implications":["A strategy's in-source gain should not be used as its deployment value: the strategies that win most on the source corpus (retrieval, LoRA) overestimate their cross-source usefulness, so practitioners should not select by source performance alone.","Randomly selected few-shot examples are a stronger default than a tuned similarity retriever for cross-source deployment, because the retriever's assumption that textual similarity approximates label similarity weakens under shift.","Definitions should be matched in granularity to the target text rather than maximally detailed; for the TCFD setting, the official 11 Recommended Disclosures help most on the short canonical examples.","Fine-tuned models should be deployed with a definition prompt at inference time; swapping in D1 definitions recovers part of the cross-source loss without retraining, and the recovery is larger on stronger base models."],"fun_headline_variants":["Source shift flips adaptation winners in climate disclosure","Random few-shot examples transfer better than fine-tuning","In-source champs fail cross-source in climate disclosures","Simpler adaptation strategies survive source shift better","Retrieval and fine-tuning lose advantage when source changes"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The comparison stands on one source-target pair—TCFD-CR annual-report paragraphs to TCFD-EX official examples—and on the assumption that the two corpora's shared four labels are interchangeably reliable; if that label reliability or that pair is not representative, the 'simpler is safer' ranking may not generalize.","fun_headline_variants_meta":{"raw":{"variants":["Source shift flips adaptation winners in climate disclosure","Random few-shot examples transfer better than fine-tuning","In-source champs fail cross-source in climate disclosures","Simpler adaptation strategies survive source shift better","Retrieval and fine-tuning lose advantage when source changes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000975,"raw_usage":{"total_tokens":3957,"prompt_tokens":702,"completion_tokens":3255,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":446,"completion_tokens_details":{"reasoning_tokens":3196}},"tokens_in":446,"tokens_out":3255,"duration_ms":22087,"temperature":1.0,"reasoning_tokens":3196,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T16:32:02.356072+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a third disclosure source that shares the same four TCFD labels—for example, press releases or earnings-call transcripts—and run the same three strategies with the same adaptation pool. If similarity retrieval or LoRA fine-tuning matches or beats random few-shot and definitions on that target, the claimed in-source/cross-source rank reversal is specific to the CR-to-EX pair rather than a general property of source shift. A second check would be to relabel a sample of both corpora by human experts: if the four labels are not applied consistently across the two sources, the transfer compari","supporting_citations":[],"review_version":1}