{"id":"af65a8e7-6a73-4fd5-888a-2014592f6569","arxiv_id":"2505.22867","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A three-step prompting system, fine-tuned with LoRA and synthetic data, ranked first on English narrative classification at SemEval-2025 Task 10.","lead":"This paper describes a system that sorts news articles into a two-level narrative taxonomy using three sequential prompts to a fine-tuned language model. The system won first place on the English test set of SemEval-2025 Task 10, pointing to a practical approach for monitoring misinformation narratives.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The hierarchy claim is confounded: Table 1 has no flat fine-tuned LLaMA-3.2 baseline, so the H3Prompt gain may come from LoRA and synthetic data rather than three-step prompting.","rationale":"Read in good faith, the paper is a SemEval system description whose strongest empirical result is externally anchored by the leaderboard, and the released code plus the w/o-synthetic ablations are useful evidence. The weakness is that the named method, H3Prompt, bundles three interventions: LoRA fine-tuning, synthetic data augmentation, and hierarchical prompting. The Section 4 discussion over-attributes the outcome to structure. The 'B' row is a binary per-label variant, not a flat control, so the confound remains unresolved. This matches the Reader's weakest assumption, with the added observation that the reported STD values are large relative to the between-method gaps, meaning the magnitude of the advantage is also statistically unquantified. The CONDITIONAL verdict is therefore appropriate: the authors should add a flat ablation and seed-wise error bars before the hierarchy contribution is accepted. This is a methodological gap, not an integrity concern, and it does not undermine the externally verified leaderboard result.","tokens_in":7728,"tokens_out":4669,"duration_ms":50253,"concrete_test":"Add a matched flat baseline: fine-tune LLaMA-3.2-3B-Instruct with the same LoRA hyperparameters, the same 10,220 training articles, and the same taxonomy labels and explanations, but use one prompt that returns all main narratives and sub-narratives in a single pass, without the category or main-narrative decomposition. Run 5 seeds and report 95% confidence intervals for development fine samples F1 alongside Table 1. If the flat model's interval overlaps the H3Prompt ensemble's 0.516, the hierarchy is not the source of the gain; if it lies below, the hierarchy claim is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central contribution is the hierarchical three-step prompting mechanism (Section 2.3), and Section 4 states that the results 'demonstrate the effectiveness of structured classification.' That claim requires the hierarchy itself, rather than LoRA fine-tuning or the 8,129 synthetic articles, to be the cause of the improvement. Table 1 cannot establish this. The only fine-tuned LLaMA-3.2 row outside the H3Prompt block is 'LLaMA-3.2-3B-Instruct (B)', which is a per-sub-narrative binary configuration, not a single-pass flat classifier sharing the same prompt template, label space, and training data. The zero-shot LLaMA rows are not a matched control for fine-tuning. Consequently, the reported differences (0.482 vs 0.428 fine samples F1; 0.577 vs 0.562 coarse macro F1) are consistent with a LoRA/synthetic-data effect rather than a hierarchy effect. The reported STD column is also large (0.390 for the best H3Prompt fine score), and no confidence intervals or seed-wise variances are reported, so even the size of the advantage is not quantified. The leaderboard rank is real evidence for the whole pipeline, but it does not isolate the contribution of the hierarchy.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper describes GateNLP's submission to SemEval-2025 Task 10 Subtask 2, a multilingual narrative and sub-narrative classification task. The proposed method, H3Prompt, is a hierarchical three-step prompting procedure: a fine-tuned LLaMA-3.2-3B-Instruct model first classifies an article into one of two domains (Ukraine-Russia War or Climate Change), then selects main narratives, and finally assigns sub-narratives per selected main narrative. Training uses 2,091 annotated articles (non-English texts machine-translated to English) plus 8,129 synthetic articles generated by Vicuna-7B. LoRA is used for fine-tuning. On the English development set, the best single model obtains 0.577 macro F1 for coarse narratives and 0.482 samples F1 for fine sub-narratives; a union-based ensemble improves these to 0.623 and 0.516. The system achieved first place on the English test set among 28 teams, with reported ranks of 1st for Portuguese and Russian, 3rd for Bulgarian, and 5th for Hindi.","tokens_in":7991,"tokens_out":3471,"duration_ms":35322,"significance":"If the methodological claims are supported, the paper is a solid system description with meaningful external validation: the official SemEval leaderboard confirms that the full pipeline is competitive, and the authors supply code, hyperparameter ranges, and a synthetic-data component that is explicitly ablated. The development-set results also show a clear benefit from synthetic data (0.392 to 0.482 fine samples F1 for the H3Prompt model). However, the paper's stated contribution is the hierarchical prompting mechanism itself, and that contribution is not isolated by the presented experiments. The absence of a matched flat fine-tuned LLaMA baseline and the lack of confidence intervals leave the core attribution claim unverified. The leaderboard result validates the whole pipeline, not the specific effect of the hierarchy.","major_comments":[{"comment":"The central claim that the three-step hierarchy is responsible for the performance improvement is not isolated by the experiments. Table 1 contains no flat fine-tuned LLaMA-3.2-3B-Instruct baseline trained on the same label space, the same prompt template, and the same data in a single pass. The row 'LLaMA-3.2-3B-Instruct (B)' is a binary per-sub-narrative configuration, while the other LLaMA rows are zero-shot, so the reported differences (0.482 vs. 0.428 fine samples F1; 0.577 vs. 0.562 coarse macro F1) are consistent with gains from LoRA fine-tuning and the 8,129 synthetic articles rather than from the three-step structure. The authors should add a matched flat multi-label LLaMA-3.2 LoRA baseline to Table 1, or otherwise the hierarchy effect remains confounded.","section":"Table 1 and Section 2.3"},{"comment":"The reported standard deviations are large (e.g., 0.390 for the best H3Prompt fine score and 0.390 for the H3Prompt with synthetic data), yet the paper does not specify how many runs or seeds produced these values, and no confidence intervals or paired significance tests are reported. Without this information, the development-set differences that support the hierarchy claim, including the 0.482 vs. 0.428 fine-score comparison, may be within run-to-run noise. The authors should report per-seed results, the number of seeds, and ideally a paired test or confidence interval for the key comparisons.","section":"Table 1, STD column"},{"comment":"The statement that binary classification models show 'a slight decrease in performance compared to hierarchical prompting models, reinforcing the importance of a structured three-step classification approach' is not supported by Table 1 as presented. The binary LLaMA model is not a flat multi-label classifier but a per-sub-narrative binary setup, and the binary RoBERTa model uses a different architecture and training procedure. These comparisons do not control for the hierarchy variable. A matched flat classifier sharing the H3Prompt prompt template, label set, LoRA configuration, and training data is required before making this claim.","section":"Section 4, binary model comparison"}],"minor_comments":[{"comment":"There are several language errors: 'emerging as a important task' should be 'an important task', 'policy markers' should be 'policymakers', and 'We used an Vicuna LLM' should be 'a Vicuna LLM'.","section":"Abstract and Section 1"},{"comment":"The paper motivates the method as multilingual, but Table 1 reports English development results only, and no per-language development scores or per-language test scores are given. Reporting the development F1 for Bulgarian, Hindi, Portuguese, and Russian would strengthen the multilingual claim and clarify where machine translation helps or hurts.","section":"Section 3 and Table 1"},{"comment":"The hyperparameter tuning description gives ranges and final values but does not state the selection criterion or whether the development set was used for early stopping or model selection. This is worth clarifying for reproducibility.","section":"Section 2.2"},{"comment":"In Step 3, the pseudocode checks 'if n_s ∈ N_s' and otherwise assigns '(n_m, Other)', but the earlier prose says 'If no suitable sub-narrative is found, \"Other\" is returned.' The pseudocode could be aligned more explicitly with the prompt behavior to avoid ambiguity about whether 'Other' is a sub-narrative label or a rejection.","section":"Algorithm 1"}],"recommendation":"major_revision","confidential_remarks":"This is a competent and useful system paper with real leaderboard validation, but the scientific claim about the hierarchy needs a matched ablation before it can be accepted. The missing flat fine-tuned LLaMA baseline is a straightforward experiment and should be feasible for the authors to add. I would not reject the paper; the contribution is potentially sound, but the current evidence is incomplete."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you need to know: this is a competent SemEval system description. The H3Prompt pipeline—three-step hierarchical prompting, LoRA fine-tuning of LLaMA-3.2-3B, synthetic data augmentation—took first on the English test set for Subtask 2, with code and prompts on GitHub. That's a real, externally validated result, and the paper is honest about its components.\n\nWhat it does well: the method section is clear and reproducible. They give the exact prompts, hyperparameter bounds, data sizes, and they include ablations that actually matter—trained vs. zero-shot, with and without synthetic data, ensemble variants, and a binary classification configuration. The dev-set comparisons are internally consistent, and the test leaderboard adds an external check. For a shared task paper, this is above average.\n\nThe soft spot is exactly what the stress-test note says: the central claim—that the hierarchy itself is what helps—is not isolated. Table 1 has no flat fine-tuned LLaMA-3.2 baseline that shares the same prompt template and label space but skips the three-step decomposition. The closest trained LLaMA row is the binary per-sub-narrative configuration, which is a different learning setup. The zero-shot LLaMA rows are not matched controls for fine-tuning. So the observed gains over those baselines could come mostly from LoRA and the 8,129 synthetic articles, not from the hierarchical decomposition. The synthetic-data ablation (0.392 to 0.482 fine F1) shows that component alone gives a 23% relative jump, which makes it hard to tell how much the hierarchy contributes on top.\n\nAlso, they report a single run per configuration, with the STD column describing sample-level variance, not seed-wise variance. No confidence intervals. Given the best fine F1 is 0.482 with STD 0.390, the advantage over the binary baseline (0.428) is not clearly quantified. This is a moderate issue, not a fatal one: the test leaderboard rank still stands as evidence for the whole pipeline, and the paper never claims the hierarchy was independently ablated.\n\nMinor things: hyperparameters are tuned on the dev set, which is normal for shared tasks. The use of Vicuna to generate synthetic disinformation-flavored articles is reasonable, and they cite the relevant work. The references look appropriate and not self-inflated.\n\nWho this is for: anyone working on narrative classification or multilingual document classification in fact-checking contexts. It's also a useful template for LLM-based hierarchical classification. A serious referee should engage with it; the main missing experiment is a flat LoRA-fine-tuned LLaMA baseline, and I'd want either that or a softened claim about the hierarchy. But that's a revision, not a rejection. Recommend accept with revisions.","headline":"Solid shared-task system paper with a real leaderboard result, but the hierarchy claim is not isolated from fine-tuning and synthetic data effects.","tokens_in":8494,"tokens_out":2018,"would_cite":true,"duration_ms":19610,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning LLaMA-3.2 with a three-step prompt hierarchy—domain, main narrative, sub-narrative—and synthetic data wins the English track of SemEval-2025 Task 10.","keywords":["hierarchical prompting","narrative classification","multilingual NLP","LoRA fine-tuning","synthetic data augmentation","SemEval-2025 Task 10","LLaMA-3"],"falsifier":"Train a flat, non-hierarchical LLaMA-3.2-3B-Instruct model with the same LoRA hyperparameters, translated data, synthetic data, and prompt style (one step that outputs all labels), and compare it on the same English development set; if the flat model matches or exceeds the 0.623/0.516 F1 scores, the hierarchy is not the driver of the result.","tokens_in":7542,"feed_emoji":"🏆","tokens_out":4726,"duration_ms":38703,"temperature":0.7,"pith_summary":"The paper claims that a structured, hierarchical prompting scheme—decide the domain first, then the main narrative, then the sub-narrative—makes a fine-tuned language model substantially better at classifying news articles into a two-level narrative taxonomy than flat or zero-shot alternatives. The specific system, H3Prompt, fine-tunes LLaMA-3.2-3B-Instruct with LoRA on translated training data plus synthetically generated articles, and then applies the three-step prompt at inference time. On the English development set it reaches 0.623 macro F1 for main narratives and 0.516 samples F1 for sub-narratives, and the submitted run placed first on the English test set among 28 teams. The broader point the authors want to establish is that decomposing a fine-grained classification task along its label hierarchy is a practical way to get strong multilingual performance without task-specific architectures.","feed_headline":"Three-step prompting wins English narrative classification","feed_subtitle":"LLaMA-3.2 with LoRA and synthetic data tops 27 other systems on SemEval-2025 Task 10.","key_machinery":"The central object is the three-step prompt hierarchy, implemented as Algorithm 1: a category check, then a main-narrative lookup using the taxonomy with explanations, then a sub-narrative lookup restricted to the chosen main narratives. The prompts ask the model to return labels as a hash-separated string, with 'Other' as an explicit fallback, so each step is a constrained generation. The hierarchy is carried by a LoRA-fine-tuned LLaMA-3.2-3B-Instruct model, and its training data are augmented by synthetic articles generated with Vicuna-7B; predictions from three models trained on different data subsets are combined by union aggregation. The authors attribute the performance gain to the structured decomposition and the synthetic-data augmentation, with the hierarchy filtering the label space at each step.","core_discovery":"On the paper's own terms, the discovery is that hierarchical three-step prompting (H3Prompt) is an effective method for multilingual narrative classification: first classify each article into 'Ukraine-Russia War', 'Climate Change', or 'Other'; then, given that category, identify the main narrative(s); then, for each main narrative, assign sub-narrative(s). Each step uses a prompt that includes the relevant part of the taxonomy with explanations, and returns hash-separated labels. The system fine-tunes LLaMA-3.2-3B-Instruct with LoRA on a mix of human-annotated articles (machine-translated to English) and 8,129 synthetically generated articles, and a union-based bagging ensemble of three such models gives the best results. The authors report that this approach outperformed all baselines they tested, including zero-shot LLMs, and achieved first place on the English test set of SemEval-2025 Task 10 Subtask 2.","pith_inferences":["The paper does not include a flat fine-tuned LLaMA-3.2 baseline on the same training data, so the hierarchy's specific contribution over plain fine-tuning remains untested; a flat LoRA-tuned LLaMA-3.2 with the same data might perform as well or better.","The reliance on English translation for all non-English input means the model never sees the original language; a system that prompts in the source language or uses multilingual LLaMA might capture language-specific framing that translation flattens.","The taxonomy explanations were generated by ChatGPT and manually verified; the method's transferability to new domains depends on the effort of writing those explanations, which is a hidden cost not reflected in the reported numbers.","Since synthetic articles are generated per sub-narrative, the augmentation may bias the model toward the generator's own writing style; evaluating on a held-out set of real articles from different outlets would test whether the gain generalizes or is an artifact."],"forward_implications":["If H3Prompt's decomposition is the cause of the gain, then the same three-step pattern should transfer to any two-level multi-label taxonomy, not just narratives.","Because all non-English test articles are machine-translated to English before inference, the pipeline's multilingual success depends on translation quality, so improving translation should improve non-English scores.","Union-based ensembling improved recall-oriented fine F1 at some cost to precision, suggesting that for tasks where missing a narrative is costly, this aggregation is the right default.","Fine-grained narrative classification at this level may be good enough for fact-checking triage, letting analysts prioritize articles rather than read everything.","The reported 23% improvement in fine F1 from synthetic data suggests data augmentation is at least as important as the prompting hierarchy, which would change where future efforts should focus."],"supporting_citations":[{"why":"Defines SemEval-2025 Task 10, the dataset, the evaluation metrics, and the leaderboard; without it there is no benchmark.","marker":"Piskorski et al., 2025"},{"why":"Provides the two-level taxonomy of main narratives and sub-narratives that the three-step prompts are built around.","marker":"Stefanovitch et al., 2025"},{"why":"Introduces LoRA, the parameter-efficient fine-tuning method used to adapt LLaMA with low-rank weight updates.","marker":"Hu et al., 2021"},{"why":"Presents LLaMA-3.2-3B-Instruct, the base model that is fine-tuned in all H3Prompt experiments.","marker":"Dubey et al., 2024"},{"why":"Supplies the m2m100_418M model used to translate all non-English training and test articles into English.","marker":"Fan et al., 2021"},{"why":"Provides Vicuna-7b, the LLM used to generate the 8,129 synthetic articles that augment the training data.","marker":"Zheng et al., 2023"},{"why":"Introduces bagging, the ensemble technique used to combine three H3Prompt models via union aggregation.","marker":"Breiman, 1996"}],"fun_headline_variants":["Three-step prompt wins English narrative task","Prompt hierarchy tops multilingual narrative classification","LLM prompt chain lands first in English narratives","Stepwise LLM prompting secures top SemEval slot"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the three-step hierarchy itself is what improves accuracy; the paper never compares against a flat fine-tuned LLaMA-3.2 model trained on the same data, so the gain could come entirely from fine-tuning and synthetic data.","fun_headline_variants_meta":{"raw":{"variants":["Three-step prompt wins English narrative task","Prompt hierarchy tops multilingual narrative classification","LLM prompt chain lands first in English narratives","Stepwise LLM prompting secures top SemEval slot"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000193,"raw_usage":{"total_tokens":1344,"prompt_tokens":935,"completion_tokens":409,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":551,"completion_tokens_details":{"reasoning_tokens":352}},"tokens_in":551,"tokens_out":409,"duration_ms":5561,"temperature":1.0,"reasoning_tokens":352,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:57:44.185757+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a flat, non-hierarchical LLaMA-3.2-3B-Instruct model with the same LoRA hyperparameters, translated data, synthetic data, and prompt style (one step that outputs all labels), and compare it on the same English development set; if the flat model matches or exceeds the 0.623/0.516 F1 scores, the hierarchy is not the driver of the result.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces bagging, the ensemble technique used to combine three H3Prompt models via union aggregation."}],"review_version":1}