{"id":"9d66170f-7ad3-4efb-b8a4-8aab8d0ae0ea","arxiv_id":"2505.23538","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A shared-task system report in which a multitask DeBERTa-v3 model with attention pooling and test-time augmentation scores 0.5268 on SemEval-2025 Task 6, a 0.4 percent relative gain over the baseline.","lead":"This paper describes three NLP models submitted to the SemEval-2025 PromiseEval task, which checks whether corporate ESG statements contain promises, whether evidence supports them, how clear they are, and when they should be verified. The best system, a multitask DeBERTa-v3 model, scored 0.5268 on the private test set, narrowly beating the 0.5227 baseline.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 0.5268 leaderboard score is a hybrid of Model 3 (Tasks 1-2) and Model 2 (Tasks 3-4), so the abstract's attribution of the gain to attention pooling and multi-objective learning is unsupported by the reported number.","rationale":"The reader's weakest_assumption correctly identifies that Model 3 is not compared to the baseline or to Models 1 and 2 under controlled conditions, with differences in backbone, split, loss, sequence length, and training schedule. My stress-test sharpens this: the headline number is not even a pure Model 3 score. Section 6 explicitly describes a hybrid submission, so the private leaderboard score conflates Model 3 on Tasks 1-2 with Model 2 on Tasks 3-4. This makes the abstract's causal claim ('our work highlights the effectiveness of linguistic feature extraction, attention pooling, and multi-objective learning') unsupported by the reported aggregate metric. The paper is honest about the hybrid composition and about the need for ablations, so this is not a question of integrity; it is a question of whether the central claim is demonstrated. A conditional acceptance requiring either softened attribution or a controlled decomposition remains the appropriate disposition. The mismatched Prabhakaran citation is a mechanical flaw and not load-bearing.","tokens_in":6408,"tokens_out":3428,"duration_ms":31041,"concrete_test":"On the held-out test set used for the leaderboard (or, if unavailable, a stratified re-split of the 400 training instances), recompute the private score for three systems: (a) the exact hybrid submission (Model 3 for Tasks 1-2 plus Model 2 for Tasks 3-4); (b) a matched control with a single-task DeBERTa-v3-large trained on the same 90/10 split for Tasks 1-2 plus Model 2 for Tasks 3-4; and (c) Model 2 alone for all four tasks. If (a) does not exceed both (b) and (c) by more than the seed-to-seed variance estimated from at least three runs, then the 0.0041 gain cannot be attributed to attention pooling or multi-objective learning. Report per-task scores and bootstrap 95% confidence intervals.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on the private leaderboard score of 0.5268 in Table 1, but that score is not produced by the Combined Subtask Model described in Section 4.3. Section 6 states: 'Since the Combined Model only worked on Tasks 1 and 2... we incorporated Task 1 and 2 predictions from our Combined Model while using our Feature-Enhanced Model for Tasks 3 and 4.' Thus the submitted system is a hybrid: Tasks 1 and 2 come from the DeBERTa-v3-large multi-task model with attention pooling, while Tasks 3 and 4 come from the ESG-BERT Feature-Enhanced Model. The reported gain over the Kaggle baseline (0.5268 vs 0.5227) therefore cannot be attributed to attention pooling or multi-objective learning without a controlled decomposition. The gain could come from Model 2's contributions on Tasks 3 and 4, from Model 3's predictions on Tasks 1 and 2, from the different backbone and 90/10 split, or from run-to-run noise. No per-subtask scores, ablations, repeated runs, or significance tests are reported, and the paper's own conclusion explicitly defers 'a systematic ablation study' to future work. This is an acknowledged missing support, but the abstract presents the attribution as established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript reports three model architectures for SemEval-2025 Task 6 (PromiseEval) on the English portion of the ML-Promise dataset: (i) an ESG-BERT base model with per-subtask classifier heads; (ii) an ESG-BERT model with hand-crafted linguistic feature tags prepended to the input; and (iii) a DeBERTa-v3-large multitask model for Subtasks 1 and 2 using attention pooling, document metadata tokens, a 0.6/0.4 multi-objective loss, and test-time augmentation. The authors report private leaderboard scores of 0.4994, 0.5094, and 0.5268 for these models, with the best score surpassing the provided Kaggle baseline of 0.5227. The abstract credits linguistic feature extraction, attention pooling, and multi-objective learning for the improvement.","tokens_in":6630,"tokens_out":4668,"duration_ms":38950,"significance":"The paper's main strength is that the evaluation uses an external held-out leaderboard, so the reported numbers are not fitted to the test set, and the authors are candid about the limitations of their study, including the absence of a systematic ablation. If the attribution were backed by controlled comparisons, the result would be a useful data point for promise-verification system design. As reported, however, the central attribution claim is not established: the best score comes from a hybrid submission rather than from the combined model alone, and multiple architectural and training choices are confounded. The paper is a reasonable system-description contribution, but it requires a careful revision to correct the attribution and to provide per-subtask or ablation evidence.","major_comments":[{"comment":"The abstract states that 'our combined subtask approach achieving a leaderboard score of 0.5268', but Section 6 explicitly says that the Combined Model only worked on Tasks 1 and 2 and that the final submission used Model 3 for those tasks and the Feature-Enhanced Model for Tasks 3 and 4. The 0.5268 private score therefore belongs to a hybrid system, and the comparison with the 0.5227 baseline cannot be used to support claims about the combined model's components in isolation.","section":"Abstract; §6"},{"comment":"The sentence 'We attribute this improvement to three factors: (1) multitask learning benefits from shared representations... (2) attention pooling... (3) test-time augmentation' is an attribution, not a demonstrated effect. Model 3 differs from Models 1 and 2 and from the baseline in the backbone (DeBERTa-v3-large vs ESG-BERT), the train/validation split (90/10 in §5.2 vs 4-fold 75/25 in §5.1), the sequence length (256 tokens), the loss weighting (Eq. 5), gradient accumulation, cosine schedule, and threshold calibration. No ablation isolates any of the named factors, and Section 7 defers a systematic ablation study to future work, which acknowledges the missing support.","section":"§6"},{"comment":"The paper reports only aggregate leaderboard scores and never reports per-subtask results or the official metric definition. Since the submitted system is a hybrid across subtasks, per-subtask breakdowns are necessary to determine whether the gain over the baseline comes from Model 3 on Tasks 1 and 2, from Model 2 on Tasks 3 and 4, or from run-to-run noise. The 0.0041 absolute difference between 0.5268 and 0.5227 is small, and no repeated runs, confidence intervals, or significance tests are provided.","section":"Table 1; §5"}],"minor_comments":[{"comment":"Equation (2) is typeset incorrectly; it should read r = \\sum_{i=1}^{n} \\alpha_i h_i, with an explicit summation symbol.","section":"Eq. (2)"},{"comment":"The phrase '1.74% absolute improvement over the baseline' is inconsistent with Table 1: 0.5268 - 0.5227 = 0.0041, which is 0.41 percentage points, not 1.74 percentage points.","section":"§6"},{"comment":"The 'Kaggle Baseline' is not described; the paper should state what this baseline is and how its predictions were obtained.","section":"§1, Table 1"},{"comment":"The GitHub URL in Footnote 1 is cut off after 'SemEval-2025-Task6'; please provide a complete, working URL so that the claimed code release is verifiable.","section":"Footnote 1"},{"comment":"The attention pooling parameters W_attn in Eq. (1) are not described; specifying their initialization and whether the attention layer is shared between the two task heads would improve reproducibility.","section":"§4.3"}],"recommendation":"major_revision","confidential_remarks":"This is a system-description paper whose main value is the external leaderboard result and the transparent discussion of design choices. The central attribution problem is fixable by rewording the abstract and adding per-subtask or ablation evidence; rejection is not warranted, but the current abstract overstates what the experiments show."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this if you want a textbook case of why a private leaderboard number does not back an attribution claim. The paper is a SemEval-2025 Task 6 system description that reports 0.5268 private score versus 0.5227 baseline. That number is real and externally verified, and the code is on GitHub. The paper is also honest in places: it admits the gains are modest, discusses negative transfer, and explicitly defers a systematic ablation to future work. That honesty counts for something.\n\nThe strongest contribution is the empirical data point plus the clear description of three architectures. Model 2's linguistic feature prepending is simple but concrete, and the idea of tailoring features per subtask is sensible. The attention pooling and multi-objective setup in Model 3 are standard but competently assembled. For a shared-task paper, this is passable engineering work.\n\nThe soft spot is load-bearing. The abstract says 'our combined subtask approach' achieved 0.5268 and highlights attention pooling and multi-objective learning as effective. But Section 6 says the Combined Model only worked on Tasks 1 and 2; Tasks 3 and 4 came from the Feature-Enhanced Model. So the submitted system is a hybrid, and the reported gain cannot be attributed to any single component. Model 3 also differs from Models 1 and 2 in backbone (DeBERTa-v3-large vs ESG-BERT), data split (90/10 vs 75/25), loss, sequence length, and training schedule. No ablations or significance tests are provided. The attribution is therefore unsupported, not necessarily wrong. The paper should either soften the conclusion or run controlled comparisons. Also, the Prabhakaran et al. (2016) citation is wrong: it is about scientific topics, not commitment language in political discourse. Minor, but sloppy.\n\nWho is this for? Someone working on ESG promise verification or on shared-task system descriptions as evidence. It deserves a serious referee because the leaderboard result is real and the code is available, but the referee should push for a clearer separation of the hybrid scoring and for weaker claims in the abstract. I would engage with it as a data point, not as a proof of what works.","headline":"A real external leaderboard score, but the abstract credits components that the submitted hybrid system doesn't isolate.","tokens_in":7271,"tokens_out":1429,"would_cite":false,"duration_ms":14523,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A combined subtask model with attention pooling and multi-objective training beats the baseline for verifying corporate ESG promises.","keywords":["promise verification","ESG reports","multitask learning","attention pooling","linguistic features","test-time augmentation","class imbalance"],"falsifier":"Re-run both the combined model and the baseline with the same backbone, the same train/validation split, and multiple random seeds, then compare their scores on the private test set; if the combined model's edge is no larger than the standard deviation across seeds, the claimed improvement is not distinguishable from noise. A sharper test is to ablate attention pooling, metadata prepending, and test-time augmentation one at a time with everything else held fixed.","tokens_in":6130,"feed_emoji":"🌱","tokens_out":6854,"duration_ms":58264,"temperature":0.7,"pith_summary":"The paper reports a modest but consistent improvement in automatically verifying whether corporate ESG statements really contain promises and evidence. Working from the English portion of the ML-Promise dataset, it compares three architectures: a frozen ESG-BERT with task-specific heads, that model augmented with hand-crafted linguistic tags, and a combined subtask model that shares a DeBERTa-v3-large encoder between promise identification and evidence assessment. The combined model reaches 0.5268 on the private test leaderboard of SemEval-2025 Task 6, above the provided baseline of 0.5227 and above the paper's own feature-enhanced model at 0.5094. The paper attributes the gain to attention pooling, document-metadata prepending, multi-objective learning, and test-time augmentation, while noting the gain is small relative to the added complexity.","feed_headline":"Multitask model beats baseline at verifying corporate promises","feed_subtitle":"Combined attention-pooling model scores 0.5268 on the private PromiseEval leaderboard, ahead of 0.5227.","key_machinery":"The carrying mechanism is the combined subtask model: a DeBERTa-v3-large encoder with two task-specific classifier heads sharing all transformer layers. Three additions do the work. First, attention pooling replaces the [CLS] token by computing $\\alpha_i = \\operatorname{softmax}(W_{\\text{attn}} h_i)$ over hidden states $h_i$ and forming $r = \\sum_{i=1}^{n} \\alpha_i h_i$. Second, the raw text is enriched as $x_{\\text{enriched}} = \\text{[PAGE}_p\\text{] [ESG REPORT]} + x_{\\text{raw}}$, prepending a page marker and report-type tag. Third, the training objective is a weighted sum $L = 0.6\\,L_{\\text{promise}} + 0.4\\,L_{\\text{evidence}}$, with focal loss, gradient accumulation, and test-time augmentation over three forward passes whose probabilities are averaged and thresholded at 0.5.","core_discovery":"The central claim is that promise verification in ESG reports improves when promise identification and supporting-evidence assessment are trained jointly on a shared transformer, with attention-based pooling and metadata-enriched inputs, rather than as separate classification tasks. The paper reports private leaderboard scores of 0.4994 for the ESG-BERT base model, 0.5094 for the feature-enhanced model, and 0.5268 for the combined subtask model, against a Kaggle baseline of 0.5227. Its final system therefore uses the combined model for subtasks 1 and 2 and the feature-enhanced model for clarity and timing subtasks. The paper also finds that prepended linguistic features (promise words, sentiment, metric counts, vague/specific terms, date spans) improve the ESG-BERT models only slightly, which it attributes to redundancy with domain-specific pre-training and to the small 400-instance training set.","pith_inferences":["Because the combined model differs from the baseline in several ways at once (backbone, data split, loss, sequence length, training schedule), the 0.0041 gain should be read as an association, not a proven effect of any single component; a controlled ablation is the natural sequel.","The same architecture could be extended to all four subtasks in one multi-objective model, since the paper itself notes that negative transfer between promise and evidence detection may have limited the gain; whether adding clarity and timing helps or hurts is an open question.","The metadata markers are a cheap testable variable: replacing the real page number with a shuffled or constant tag would show whether the model exploits document position or merely a constant prefix.","The result is specific to the English split; applying the recipe to the multilingual part of ML-Promise would test whether the hand-crafted linguistic tags and document-type markers transfer across languages."],"forward_implications":["If the gain is real, joint training on promise identification and evidence assessment transfers knowledge between the two subtasks, since the combined model beats two separately trained ESG-BERT models.","Attention pooling over token hidden states is a viable alternative to the [CLS] representation on this small imbalanced dataset.","Prepending document metadata (page number, report type) can help a transformer locate promise-relevant context, a cheap addition that requires no extra labels.","Test-time augmentation with probability averaging and a calibrated threshold is a practical variance-reduction tool for a 400-instance training set.","The final recipe is modular: use the strongest model per subtask group and assemble predictions, which is how the paper obtains its leaderboard score."],"supporting_citations":[{"why":"Defines SemEval-2025 Task 6 and the leaderboard against which the paper's scores are measured.","marker":"Chen et al., 2025"},{"why":"Introduces the ML-Promise dataset, the English portion of which provides the 400 training instances.","marker":"Seki et al., 2024"},{"why":"Provides DeBERTa-v3-large, the backbone of the combined subtask model.","marker":"He et al., 2021"},{"why":"Provides ESG-BERT, the domain-specific backbone of Models 1 and 2.","marker":"Mukherjee and Pothireddi, 2021"},{"why":"Supplies the test-time augmentation strategy used in the combined model's inference.","marker":"Shanmugam et al., 2021"},{"why":"Underlies the attention-pooling mechanism for sequence representation.","marker":"Yang et al., 2016"},{"why":"Supports the multi-task learning principle that motivates joint promise and evidence training.","marker":"Liu et al., 2019"}],"fun_headline_variants":["Joint training improves verification of corporate ESG promises","Attention pooling lifts promise verification above baseline","Multi-architecture model beats baseline on promise verification","Combined subtask model edges baseline on ESG promise verification"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's central claim rests on the assumption that the combined model's 0.0041-point lead over the baseline on the private leaderboard comes from its design choices, rather than from the different backbone, the different 90-10 data split, the multi-objective loss, the shorter sequence length, or run-to-run variation.","fun_headline_variants_meta":{"raw":{"variants":["Joint training improves verification of corporate ESG promises","Attention pooling lifts promise verification above baseline","Multi-architecture model beats baseline on promise verification","Combined subtask model edges baseline on ESG promise verification"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000176,"raw_usage":{"total_tokens":1275,"prompt_tokens":919,"completion_tokens":356,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":535,"completion_tokens_details":{"reasoning_tokens":298}},"tokens_in":535,"tokens_out":356,"duration_ms":4029,"temperature":1.0,"reasoning_tokens":298,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:43:56.837670+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run both the combined model and the baseline with the same backbone, the same train/validation split, and multiple random seeds, then compare their scores on the private test set; if the combined model's edge is no larger than the standard deviation across seeds, the claimed improvement is not distinguishable from noise. A sharper test is to ablate attention pooling, metadata prepending, and test-time augmentation one at a time with everything else held fixed.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines SemEval-2025 Task 6 and the leaderboard against which the paper's scores are measured."},{"cited_title":"ML-Promise: A Multilingual Dataset for Corporate Promise Verification","cited_arxiv_id":"2411.04473","evidence_quote":"Introduces the ML-Promise dataset, the English portion of which provides the 400 training instances."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides ESG-BERT, the domain-specific backbone of Models 1 and 2."}],"review_version":1}