{"id":"304f2088-6cf0-455d-b0e3-ffc206604ce2","arxiv_id":"2412.10906","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"Small fine-tuned models on SusGen-30K are reported to nearly match GPT-4 on financial and ESG tasks, with a new TCFD-Bench benchmark, though the comparison is biased.","lead":"A team fine-tuned 7-8B open-source LLMs on a new 30,000-sample finance and ESG dataset and claims they land within 2% of GPT-4 on financial NLP tasks. The paper also introduces a TCFD-report benchmark, but the evaluation method makes the headline comparison hard to accept.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 3 suggests SusGen-30K includes the exact FinBen test splits used in Table 1 (flare-fomc, flare-tatqa, flare-edtsum-test, flare-causal20-sc-test, flare-finer-ord), so the '0.02 below GPT-4' comparison may be measuring memorization rather than generalization.","rationale":"The reader rejected the paper because TCFD_QA appears in the training data although Section 4.2 calls sustainability report generation a non-adapted task. My stress-test agrees that there is a leakage problem, but the more decisive form of it is broader: Table 3 indicates that multiple FinBen test splits for the adapted benchmarks were themselves used to build SusGen-30K. The paper's own numbers make this visible: for many flare-* rows the 'Test' count equals the 'Final' count, and the final counts were explicitly said to compose the training set. This contaminates the comparison against GPT-4 in Table 1, which is the only quantitative support for the central claim of near-GPT-4 performance at 7-8B scale. I therefore agree with the REJECT verdict and would not change it. At the same time, the dataset and benchmark may still be useful resources if the contamination is removed and the evaluation is rerun on held-out splits; the paper's limitation statements about automated metrics and narrow TCFD coverage are honest and should be retained. The single most load-bearing issue remains the apparent inclusion of evaluation test splits in training, because it undermines the only empirical route to the headline claim.","tokens_in":21286,"tokens_out":6015,"duration_ms":53870,"concrete_test":"Download the released SusGen repository and the FinBen flare-* evaluation files used in Table 1; compute SHA-256 hashes of the normalized instruction/input/output text of every SusGen-30K training example and every flare-* test example. If any test example from FiQASA, FOMC, MultiFin, MLESG, FINER-ORD, SC, TATQA, or EDTSUM appears in the training set, the scores in Table 1 measure contamination and the near-GPT-4 average is invalid. If the training data cannot be inspected, retrain SusGen-GPT excluding all Table 3 rows whose Final count equals the Test count and check whether TATQA 0.80, SC 0.96, and EDTSUM Rouge-1 0.27 persist.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that small 7-8B models match GPT-4 on financial and ESG NLP (Section 4.3, Abstract). The evidence for that claim is Table 1, but the dataset construction table in Appendix C.1 appears to put the evaluation test splits into the training set. Table 3 is described as 'the list of datasets and associated splits used to build the dataset,' and for the FinBen (flare-*) rows the Final counts equal the Test counts: flare-fomc Test 496, Final 496; flare-tatqa Test 1,668, Final 1,668; flare-edtsum-test Test 2,000, Final 2,000; flare-causal20-sc-test Test 8,628, Final 8,628; flare-finer-ord Test 1,075, Final 1,075; flare-mlesg-en Test 300, Final 300; flare-fiqasa Test 235, Final 235. These are exactly the benchmarks evaluated in Table 1 (FOMC, TATQA, EDTSUM, SC, FINER-ORD, MLESG, FiQASA). If those test examples were used for supervised fine-tuning, the scores on the six adapted tasks are not measurements of generalization; they are measurements of how well the model reproduces training examples. The TCFD_QA overlap identified by the reader is real, but it touches only the new TCFD-Bench, which has no GPT-4 comparison. The flare-* leakage is more load-bearing because it contaminates the very Table 1 scores that produce the 'only 0.02 below GPT-4' headline. This is not a disagreement with consensus; it is an internal inconsistency between the paper's own data table and its evaluation claim. If the authors intended the 'Test' column to mean something other than 'used in SusGen-30K,' the paper must say so explicitly and show the actual training split. As written, the natural and most damaging reading is that evaluation test sets were included in training.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SusGen-30K, a category-balanced instruction dataset for financial and ESG NLP, and TCFD-Bench, a benchmark for TCFD-style sustainability report generation. The authors fine-tune Mistral-7B and Llama-3-8B variants with QLoRA and report that their SusGen-GPT models outperform open baselines and trail GPT-4 by only about 0.02 in an aggregate score across six adapted and two off-the-shelf tasks. The main experimental evidence is Table 1 (FinBen-derived financial tasks) and Table 2 (TCFD-Bench vs. ChatReport).","tokens_in":21690,"tokens_out":5702,"duration_ms":43756,"significance":"Data-centric fine-tuning of small open models for finance and ESG is a worthwhile goal, and the paper's dataset construction pipeline and proposed benchmark address a real gap. If the comparisons were valid, a 7-8B model within 2% of GPT-4 would be practically significant. However, the headline result is not currently supported because the training set appears to include the evaluation test splits, and several comparison choices (imported GPT-4 numbers, zero-shot baselines, unweighted metric averaging) make the claimed gap difficult to interpret.","major_comments":[{"comment":"Table 3, rows for flare-fomc, flare-tatqa, flare-edtsum-test, flare-causal20-sc-test, flare-finer-ord, flare-mlesg-en, and flare-fiqasa: the Final counts equal the Test counts (e.g., flare-fomc Test 496 Final 496; flare-tatqa Test 1,668 Final 1,668; flare-edtsum-test Test 2,000 Final 2,000; flare-causal20-sc-test Test 8,628 Final 8,628; flare-finer-ord Test 1,075 Final 1,075; flare-mlesg-en Test 300 Final 300; flare-fiqasa Test 235 Final 235). These are the same datasets evaluated in Table 1, so the reported SusGen-GPT scores on those tasks reflect reproduction of training examples rather than generalization to held-out data. This invalidates the central claim in the Abstract and Section 4.3 that SusGen-GPT is only 0.02 below GPT-4.","section":"Table 3 and Table 1"},{"comment":"Section 4.2 states that Text Summarization and Sustainability Report Generation are non-adapted tasks because the training set 'does not explicitly include them.' This is contradicted by Table 3, which lists flare-edtsum-test under SUM with Final 2,000 and TCFD_QA under ESG-QA with Final 1,669. Additionally, TCFD-Bench is constructed from 14 TCFD-format ESG reports, and TCFD_QA comprises question-answer pairs extracted from that type of report; if the TCFD-Bench test samples come from the same source reports, the Table 2 results are in-domain rather than off-the-shelf. The authors should clarify the exact overlap and rerun the evaluation with a truly held-out benchmark.","section":"Section 4.2 and Table 3"},{"comment":"The 'average score' used for the 2% gap is an unweighted mean over heterogeneous metric rows (F1, MicroF1, EntityF1, Exact Match, Rouge-1, BERTScore) in Table 1, with no task-level weighting, confidence intervals, or significance tests. The GPT-4 and Gemini numbers are imported from FinBen (footnote *), not re-evaluated under the same prompt template, decoding parameters, or evaluation harness, and the comparison is between fine-tuned SusGen-GPT and zero-shot baselines. These factors make the headline difference not a well-defined quantity; an apples-to-apples evaluation with identical prompts and settings is needed.","section":"Section 4.3 and Figure 1"},{"comment":"The TCFD-Bench evaluation includes only SusGen-GPT and ChatReport. No GPT-4 or Gemini results are reported on TCFD-Bench, so the abstract's claim that SusGen-GPT surpasses all models except GPT-4 is not supported for sustainability report generation. The authors should run the same closed-source baselines on TCFD-Bench or qualify the claim.","section":"Section 4.3 and Table 2"}],"minor_comments":[{"comment":"Figure 1 lacks a description of how the 'average score' is computed; please add the formula and per-task metric definitions to the caption.","section":"Figure 1"},{"comment":"Section 4.4 contains a typo: 'Instantce' should be 'instance'.","section":"Section 4.4"},{"comment":"The caption of Table 1 says 'zero-shot and few-shot performance,' but SusGen-GPT is fine-tuned; clarify the inference setting for every model.","section":"Table 1 caption"},{"comment":"Appendix C.1 would benefit from a column explaining the relationship between Train, Test, and Final, especially for rows where Train is marked '✗' but Test and Final are nonzero.","section":"Appendix C.1"},{"comment":"The Limitations section acknowledges the lack of expert evaluation and the restricted TCFD template diversity; these caveats should be more explicitly reflected in the conclusions about report generation quality.","section":"Limitations"}],"recommendation":"reject","confidential_remarks":"Before any resubmission, the authors should audit the released dataset and code to confirm whether the exact FinBen test splits listed in Table 3 were used in supervised fine-tuning. If they were, the central comparison in Table 1 needs to be redone without them, and the paper's contribution should be reframed accordingly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The headline claim that a 7-8B model lands within 0.02 of GPT-4 on financial and ESG NLP does not hold up, because the paper's own Table 3 indicates that the exact FinBen test splits used for evaluation were folded into the training set. That makes the comparison a memorization check, not a generalization test.\n\nWhat is genuinely useful: SusGen-30K is a carefully assembled, category-balanced instruction set covering finance and ESG tasks, and TCFD-Bench is a new benchmark for TCFD-format report generation. The data construction pipeline, with translation, anonymization, augmentation, and synthesis, is detailed and reproducible, and the authors openly release code and checkpoints. The ablation study across dataset sizes is a good practice.\n\nThe soft spots are not minor. Besides the leakage, the average score is an unweighted mean of F1, EM, ROUGE, and BERTScore across heterogeneous tasks, which is not a meaningful aggregate. GPT-4 and Gemini numbers are imported from FinBen rather than rerun under the same prompt and decoding settings, so the comparison is not controlled. All fine-tuned SusGen models are compared against zero-shot baselines, which conflates the gains from fine-tuning with model quality. The paper calls summarization and SRG 'non-adapted' but Table 3 lists flare-edtsum-test and TCFD_QA in the training mix, directly contradicting Section 4.2. The TCFD-Bench overlap that the reader noted is real too.\n\nThe right path: the authors need to redo the evaluation on held-out splits that are verifiably absent from training, rerun the closed baselines in the same harness, and report per-metric results without a single aggregate. If they do that, the dataset and benchmark could be useful resources. As it stands, the central empirical claim is circular.\n\nWould I send this to peer review? No. A desk reject is appropriate because the load-bearing number is undermined by the paper's own supplementary table. It might be worth a conversation with the authors about resubmission after cleaning the evaluation.","headline":"The near-GPT-4 claim collapses because the paper's own Table 3 shows the evaluation test sets were in the training data.","tokens_in":22312,"tokens_out":3582,"would_cite":false,"duration_ms":29780,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 7-8B parameter model fine-tuned on a balanced 30K instruction dataset matches GPT-4's average performance on eight financial and ESG NLP tasks, including generating TCFD-format sustainability reports, despite GPT-4's 1,700B parameters.","keywords":["financial NLP","ESG","sustainability report generation","TCFD-Bench","SusGen-30K","QLoRA fine-tuning","retrieval-augmented generation","large language models"],"falsifier":"Check the overlap between TCFD-Bench and the TCFD_QA samples in SusGen-30K: if any company report, question, or reference answer appears in both, then the benchmark is contaminated and the reported report-generation scores do not demonstrate generalization. A concrete test is to compute the string overlap or report-identity overlap between the two sets; a nonzero overlap would falsify the off-the-shelf evaluation claim.","tokens_in":21095,"feed_emoji":"📈","tokens_out":7140,"duration_ms":52826,"temperature":0.7,"pith_summary":"The paper claims that a compact, open-source language model fine-tuned on a carefully curated instruction dataset can reach near-GPT-4 performance across financial and ESG natural language processing tasks. The authors introduce SusGen-30K, a category-balanced dataset covering seven financial tasks plus ESG question answering, and TCFD-Bench, a benchmark for evaluating sustainability report generation. Their trained suite of 7-8B parameter models, SusGen-GPT, attains an average score only 0.02 below GPT-4 on eight tasks while using roughly two hundred times fewer parameters. If true, this means that a data-centric fine-tuning approach can substitute for massive model scale in specialized financial and sustainability applications.","feed_headline":"7B model trails GPT-4 by 0.02 on finance and ESG","feed_subtitle":"Fine-tuned on a balanced 30K dataset, SusGen-GPT reaches near-frontier scores in sustainability report generation.","key_machinery":"SusGen-30K is a category-balanced instruction dataset built from public financial datasets and TCFD-format annual reports, processed through a pipeline of translation, reformatting, anonymization, instruction augmentation, and synthetic data generation. TCFD-Bench is a benchmark constructed from 14 ESG reports in TCFD format, with contexts generated by GPT-4o and question-answer pairs extracted manually and diversified with Mistral 7B. QLoRA fine-tuning adapts base models (Mistral-v0.3-7B and LLaMA-3-8B) to SusGen-30K, and a RAG module retrieves relevant annual-report chunks for report generation. Together these components carry the argument that a small model plus good data can rival a much larger model.","core_discovery":"The central discovery is that data curation and task balancing can close most of the gap between small open models and frontier proprietary models in finance and ESG. SusGen-GPT, fine-tuned with QLoRA on SusGen-30K, outperforms all open-source baselines and trails GPT-4 by only 2 points on average across six adapted tasks and two off-the-shelf tasks (text summarization and sustainability report generation). On the proposed TCFD-Bench, SusGen-GPT beats ChatReport on ROUGE-L, BERTScore, and METEOR, though ChatReport leads on BLEU-1. The paper presents this as evidence that a data-centric approach delivers efficient, high-performance specialized LLMs.","pith_inferences":["The 'off-the-shelf' distinction is weakened by the TCFD_QA overlap; a clean evaluation would require a report-generation test set built from reports never seen during training, and the current numbers should be read with that caveat.","The average 0.02 gap to GPT-4 aggregates heterogeneous metrics across eight tasks, hiding large task-level deficits (e.g., GPT-4 scores 0.83 EntityF1 on NER vs. 0.35 for SusGen-GPT); parity is not uniform.","A more direct test of the paper's headline claim would be to evaluate GPT-4 on TCFD-Bench; currently the report-generation comparison uses ChatReport, a weaker baseline, so the claim that the model matches GPT-4 on report generation is not directly demonstrated.","The RAG component's contribution could be isolated by ablating retrieval; without that ablation, it is unclear how much of the TCFD-Bench score comes from retrieval versus the fine-tuned generator."],"forward_implications":["If the central claim holds, organizations can fine-tune 7-8B open models to near-frontier performance on financial and ESG tasks with modest GPU resources, avoiding dependence on closed APIs.","TCFD-Bench gives the community a standardized way to evaluate sustainability report generation, so future models can be compared directly on the same reports and metrics.","The data-centric recipe of category balancing, instruction augmentation, and synthetic data generation can be transferred to other specialized, low-resource domains.","The RAG-integrated SusGen system suggests a practical architecture for grounding report generation in unstructured annual reports, reducing hallucination risk."],"supporting_citations":[{"why":"Supplies the GPT-4 and Gemini baseline scores used in Table 1 for the seven financial tasks.","marker":"(Xie and et. al., 2024)"},{"why":"Provides the ChatReport baseline that SusGen-GPT is compared against on TCFD-Bench.","marker":"(Ni et al., 2023)"},{"why":"Provides the QLoRA method used to fine-tune SusGen-GPT on SusGen-30K.","marker":"(Dettmers et al., 2024)"},{"why":"Supplies many of the underlying FinGPT datasets that make up the training portion of SusGen-30K.","marker":"(Wang et al., 2023b)"},{"why":"Motivates the data scaling and balancing method used to construct SusGen-30K.","marker":"(Li et al., 2024)"},{"why":"Provides the Alpaca prompt template and general instruction data mixed into SusGen-30K.","marker":"(Taori et al., 2023)"}],"fun_headline_variants":["7B model trails GPT-4 by only 2% on finance and ESG","Data-centric 7B LLM nearly matches GPT-4 on ESG tasks","SusGen-GPT: 7B parameters, 2% behind GPT-4 in finance","Small open model beats baselines, close to GPT-4 on ESG","How a 7B model closes the gap to GPT-4 in finance and ESG"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim that sustainability report generation is an off-the-shelf, non-adapted task is contradicted by Table 3, which lists TCFD_QA as part of the ESG-QA training data in SusGen-30K; if TCFD-Bench is drawn from the same 14 TCFD reports that produced TCFD_QA, the report-generation evaluation is not held out and the measured performance reflects memorization rather than generalization.","fun_headline_variants_meta":{"raw":{"variants":["7B model trails GPT-4 by only 2% on finance and ESG","Data-centric 7B LLM nearly matches GPT-4 on ESG tasks","SusGen-GPT: 7B parameters, 2% behind GPT-4 in finance","Small open model beats baselines, close to GPT-4 on ESG","How a 7B model closes the gap to GPT-4 in finance and ESG"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000189,"raw_usage":{"total_tokens":1299,"prompt_tokens":869,"completion_tokens":430,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":485,"completion_tokens_details":{"reasoning_tokens":323}},"tokens_in":485,"tokens_out":430,"duration_ms":4200,"temperature":1.0,"reasoning_tokens":323,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:29:14.747956+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Check the overlap between TCFD-Bench and the TCFD_QA samples in SusGen-30K: if any company report, question, or reference answer appears in both, then the benchmark is contaminated and the reported report-generation scores do not demonstrate generalization. A concrete test is to compute the string overlap or report-identity overlap between the two sets; a nonzero overlap would falsify the off-the-shelf evaluation claim.","supporting_citations":[],"review_version":1}