{"id":"92eefbb3-b7fe-4743-ae72-03887ad82bc8","arxiv_id":"2508.17926","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A single LoRA fine-tuned Llama-3.1-8B-Instruct model trained jointly on eight argument-mining tasks across 19 datasets matches or beats task-specific models, and merged models offer a cheaper compromise.","lead":"This paper unifies 19 argument-mining datasets into a single benchmark and fine-tunes Llama-3.1-8B on eight argumentation tasks, comparing task-specific, multi-task, and model-merging training strategies. A generalist might read it to see whether one open LLM can take on many related NLP classification tasks without losing accuracy.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'state-of-the-art' claim is not anchored to published benchmarks: evaluation uses custom class-balanced 800-instance test subsets (§3.3), so the headline ACC/AR/SD/AQ numbers are not comparable to existing SOTA.","rationale":"The strongest claim has two parts: an internal part (multi-task fine-tuning outperforms task-specific models on most tasks) and an external part (state-of-the-art results on ACC, AR, SD, AQ). The internal part is reasonably supported by Table 2. The load-bearing weakness is the external part: because the test sets are custom class-balanced subsets created in §3.3, the reported numbers are not comparable to published results on the original benchmark splits, and the paper provides no published baseline comparison. The reader's weakest assumption concerned annotation-schema divergence; that is a real generalization risk, but the internal transfer finding would survive even if labels differ across corpora, whereas the SOTA claim fails immediately once the test bed is not the literature's. The reader's rationale already lists custom test splits as issue (2), so my primary concern overlaps partially with the reader's view. I would keep the verdict CONDITIONAL rather than moving to REJECT because the internal findings are credible and reproducible artifacts are released; the required change is to substantiate or remove the SOTA claim, which is exactly the condition for acceptance.","tokens_in":23424,"tokens_out":3052,"duration_ms":31528,"concrete_test":"Re-run the multi-task fine-tuned model (same LoRA configuration) on the official test splits of the corpora defining the four claimed SOTA tasks: Microtext parts 1+2 and Persuasive Essays for ACC and AR, the IAM stance test split for SD, and the standard Dagstuhl-15512 ArgQuality split for AQ. Compute F1 on the original label sets and compare directly with the published results for those datasets. If the paper cannot report such numbers, or if they fall below published SOTA, the 'state-of-the-art' wording in §4.3 and §5 must be replaced by 'strong performance on our custom balanced test sets'.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.3 claims state-of-the-art results on ACC (90.74%), AR (88.33%), SD (95.75%), and AQ (69.80%). For that claim to be true, these numbers must be comparable to published systems on the original tasks. The evaluation setup in §3.3 instead randomly splits each corpus 60/20/20 and then samples class-balanced subsets of 4000/800/800 instances, preserving dataset proportions. These are not the official or standard evaluation splits used by the original dataset papers, and class-balancing changes the label distribution, so macro-F1 on these subsets is not directly comparable to published SOTA numbers computed on full or original test sets. Table 2 compares only against zero-shot, few-shot, DeBERTa, and the authors' own task-specific models; no comparison to the original dataset results or prior argument-mining systems is reported. The internal comparison between multi-task and task-specific fine-tuning is largely supported by Table 2, but the external 'state-of-the-art' claim collapses if the test bed is not shared with the literature. A secondary issue is that all numbers come from a single run without error bars, which further weakens the claim of a new SOTA.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces AMELIA, a family of Llama-3.1-8B-Instruct models for eight argument-mining classification tasks built by converting 19 existing argument-mining datasets into a unified JSONL format. It compares zero-shot and few-shot prompting, task-specific LoRA fine-tuning, joint multi-task fine-tuning, and model merging with DARE and DELLA. The authors report that task-specific fine-tuning improves over baselines, that multi-task fine-tuning preserves or improves per-task performance, and that the DELLA II merge provides a competitive resource-efficient alternative. They further claim state-of-the-art results on argument component classification, argument relation classification, stance detection, and argument quality assessment.","tokens_in":23633,"tokens_out":5517,"duration_ms":60680,"significance":"The paper's main positive contributions are the publicly released converted datasets and model collection, the systematic internal comparison of fine-tuning strategies on a common benchmark, and the exploration of model merging for multi-task argument mining. If the claims are read as internal comparisons on the proposed benchmark, the paper provides useful empirical evidence that multi-task fine-tuning does not degrade performance and that merging is a viable cheap alternative. The paper is open about its resources and describes the merge configurations in enough detail to be reproducible. However, the external state-of-the-art claim is not anchored to shared evaluation splits, and the label-unification step is insufficiently validated; both issues must be addressed before the headline claims can be accepted.","major_comments":[{"comment":"The 'state-of-the-art' claim is not supported because the evaluation protocol is not shared with the literature. The test sets used here are random 60/20/20 splits of each corpus, further sampled to class-balanced 800-instance subsets, rather than the official test splits or label distributions of the original datasets. Macro-F1 measured on these modified subsets is not comparable to previously published numbers on ACC, AR, SD, or AQ. Table 2 compares only against zero-shot, few-shot, DeBERTa, and the authors' own task-specific models, not against published argument-mining systems on the original tasks. Please either remove or reword the state-of-the-art statements, or evaluate on the original test splits and compare against published numbers.","section":"Sections 3.3 and 4.3"},{"comment":"The unified dataset conversion assumes that labels such as claim, premise, support, attack, and evidence are semantically equivalent across the 19 source corpora, but the annotation guidelines differ substantially. For example, Microtext relation types 'reb', 'sup', 'und', and 'add' are collapsed into the generic support/attack/no-relation scheme without an explicit mapping, and stance labels from FEVER, IBM Claim-polarity, and IAM are treated as one label set despite different annotation conventions. No analysis of label consistency, per-dataset performance, or annotation-schema divergence is provided. This matters because the reported multi-task transfer gains could reflect conflicting or overlapping label conventions rather than shared argumentative structure. Please add per-dataset results and a label-alignment analysis.","section":"Sections 3.1 and 3.2"},{"comment":"All reported numbers come from a single run, so no variance estimates or significance tests support the claims of 'significant' improvement or the differences between merge configurations. In addition, the best merged model (DELLA II) is selected by test-set mean F1 among eight configurations, so the reported 'Merged Model' performance is optimistic relative to a fair model-selection procedure. Please provide multiple seeds with confidence intervals and use a separate validation split for selecting merge hyperparameters before reporting test performance.","section":"Tables 2 and 4, Section 4.2"}],"minor_comments":[{"comment":"The task set is written as '{ADUC, CD, ED, ARC, ET, SD, FD, AQ}', but the paper elsewhere uses ACC, AR, and FD Single/FD Multi; please make the abbreviations consistent.","section":"Section 3.3"},{"comment":"Example 8 contains the typo 'Ouput', and the fallacy label list includes 'false dillema' instead of 'false dilemma'; also 'ComARg' appears inconsistently as 'ComArg'.","section":"Section 3.2"},{"comment":"The FD Multi evaluation metric is under-specified: the text says a prediction is correct if it belongs to the set of ground-truth fallacies, but it does not define how precision is computed when multiple predictions are produced for one instance. Please state the exact multi-label precision/recall formulas.","section":"Section 3.2 and Table 5"},{"comment":"The DeBERTa baseline is not described in enough detail: it is unclear whether it was fine-tuned on the same training subsets, with which hyperparameters, or on the original tasks. Please specify this to support the comparison.","section":"Section 4.1"},{"comment":"Table 5 has formatting artifacts such as 'STM erged' instead of 'ST Merged' and many rows with run-together values; please format the table so that F1, precision, and recall are readable.","section":"Appendix A"}],"recommendation":"major_revision","confidential_remarks":"The paper's empirical core is a useful resource contribution, and the internal comparisons are largely plausible. The main blocker is the unsupported state-of-the-art claim combined with non-standard evaluation splits; I would not recommend rejection if the authors are willing to reframe the claims and add the requested validation. There is no evident circularity in the core evaluation, though the merge-hyperparameter selection on the test set is a related concern."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nShort version: this paper gives the argument-mining crowd a genuinely useful resource—a unified 19-dataset, 8-task benchmark—and a clean internal comparison of task-specific LoRA fine-tuning, multi-task fine-tuning, and DARE/DELLA merging on Llama-3.1-8B-Instruct. The internal results are credible: fine-tuning crushes zero/few-shot, multi-task fine-tuning matches or beats task-specific models on almost every task, and DELLA II is a viable lightweight alternative. If you work in this area, the dataset release alone is worth a look.\n\nThe soft spots are real but not fatal to the core empirical story. The headline 'state-of-the-art' claim in Section 4.3 is not backed by comparisons to prior published numbers on original test splits. The test sets are custom class-balanced 800-sample subsets, so macro-F1 here is not directly comparable to numbers in the original dataset papers. That doesn't undermine the internal comparison—multi-task vs task-specific vs merging all share the same splits—but it does mean the SOTA sentence should be dropped or heavily qualified.\n\nSecond, the unification of 19 datasets (Section 3.1) assumes 'claim', 'premise', 'attack', 'support' mean the same thing across corpora. They don't, always. Microtext's 'reb' is collapsed into attack, and 'add' presumably into support, but other datasets have their own quirks. The paper does not measure annotation-schema divergence. This matters for the multi-task transfer claim: if labels conflict, the observed gains may not generalize to the original tasks. It's a legitimate caveat, not a fatal flaw.\n\nThird, all numbers come from a single run with no error bars, and the best merging configuration was selected on the test set. Both are minor in context, but they make the precise reported percentages less load-bearing than the broad pattern.\n\nThe citation pattern is fine; self-citations are background, not load-bearing. No circularity.\n\nWho is this for? Anyone doing empirical argument mining, especially people who want a common benchmark for eight tasks or who care about multi-task LLM training strategies. It deserves a serious referee—the resource and the internal comparison are solid—but the authors should be pushed to either anchor the SOTA claims to shared evaluation protocols or remove the word 'state-of-the-art.' I'd read a revised version with more careful external comparisons.\n\nRecommendation: send to peer review, conditional on the authors fixing the comparability issue.","headline":"Useful unified benchmark and a credible internal comparison of fine-tuning vs merging for argument mining, but the 'state-of-the-art' claim is not supported by the evaluation setup.","tokens_in":24202,"tokens_out":2731,"would_cite":false,"duration_ms":24928,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single Llama-3.1-8B-Instruct model fine-tuned jointly on eight argument-mining tasks built from 19 converted datasets matches or beats every task-specific specialist and sets the paper's top scores on four tasks.","keywords":["argument mining","multi-task fine-tuning","model merging","large language models","Liara","Llama-3.1-8B-Instruct","argument quality assessment","fallacy detection"],"falsifier":"Evaluate the multi-task model on each source dataset's original annotation scheme and native test split instead of the unified conversion — for example, on Microtext's original relation types ('reb', 'sup', 'und', 'add') rather than the collapsed support/attack classes. If the model's scores on the original tasks track the task-specific fine-tuned models, the transfer claim stands; if they drop, the unified benchmark inflated the result.","tokens_in":23151,"feed_emoji":"🧠","tokens_out":16328,"duration_ms":147768,"temperature":0.7,"pith_summary":"Argument mining is normally tackled task by task: one model finds claims, another labels relations, a third scores quality. This paper asks whether one large language model can do all of it, and builds the test bed to find out: it converts 19 existing annotated corpora into one common format, defines eight argument-mining tasks, and fine-tunes Llama-3.1-8B-Instruct on each task, on all tasks at once, and through model merging. Its central claim is that joint training works: the multi-task model matches or beats every specialist (coming within about one and a half points on the tasks where it is not best) and posts the paper's top scores on argument component classification, relation classification, stance detection, and argument quality. If that holds, the practical payoff is that a single model, trained once, can serve most of the argument-mining pipeline, and the merging route makes a cheaper multi-task model available when joint training is too costly.","feed_headline":"One Llama model matches or beats eight specialist argument miners","feed_subtitle":"Fine-tuned on 19 merged datasets, the one model matches every specialist and sets the top score on four tasks.","key_machinery":"Three pieces carry the argument. First, the unified dataset: nineteen corpora with different formats and label inventories are converted into one JSONL schema with a shared label set — for instance Microtext's four relation types ('reb', 'sup', 'und', 'add') are collapsed into support, attack, or no relation — and per-class balanced sampling gives each task 4,000 training, 800 validation, and 800 test instances while preserving each corpus's share. Second, a fixed prompt template (task description, explicit label list, and a <|ANSWER|> delimiter) is used for fine-tuning and inference, so all tasks share one input/output interface, with LoRA (low-rank adaptation, a cheap partial fine-tuning method, rank 16) as the training engine. Third, for the merging route, the paper combines the eight specialists' parameter differences, or task vectors, using the DARE and DELLA methods; DELLA prunes each task vector with probabilities biased toward large-magnitude parameters and then rescales the survivors, the mechanism credited with keeping the hard tasks (fallacies, quality, evidence type) alive in the merged model.","core_discovery":"The paper's central claim, stated on its own terms: training one Llama-3.1-8B-Instruct model simultaneously on all eight argument-mining tasks, using 4,000 balanced samples per task drawn from nineteen unified datasets, produces a model that is the best overall performer in its experiments. It reaches 90.74% F1 on argument component classification, 88.33% on argument relation classification, 95.75% on stance detection, and 69.80% on argument quality assessment — the top scores in the paper — and stays within at most about one and a half points of the task-specific fine-tuned models on the other four tasks. The authors read this as evidence that the tasks transfer into each other: structural siblings such as component, claim, evidence-type, and stance tasks share representations, and joint training creates no measurable conflict. They also show that merging the eight specialists with the DELLA method yields a competitive multi-task model (63.64% mean F1) without any joint training, a compromise that trails the jointly trained model mainly on the hardest tasks.","pith_inferences":["The load-bearing unmeasured factor is label equivalence: the paper collapses each corpus's scheme into one shared vocabulary without measuring how far the original definitions diverge; keeping the full relation inventory distinct (e.g., Microtext's 'reb' vs 'und') would directly test whether the collapsed support/attack labels hide distinctions the benchmark cannot see.","The sampling design makes some datasets dominate a task's training mix — IAM Claim alone supplies 1,659 of the 2,000 claim samples in claim detection — so the reported gains may be driven by one or two large corpora; per-dataset reporting would show which corpora actually transfer.","A cheap follow-up the paper's modularity claim invites: fine-tune a ninth task's LoRA adapter, merge it into the existing collection, and measure whether the merged model retains the eight old tasks while gaining the ninth; this would test the promised incremental-update property directly.","Because task-specific fine-tuning with only 4,000 samples already lifts the model far above its zero-shot baseline on every task, the results hint that strong base models plus small targeted corpora may suffice to adapt an LLM to a new argumentation task — relevant to the many argument-mining domains with little annotated data."],"forward_implications":["One jointly fine-tuned Llama-3.1-8B-Instruct can stand in for eight separate argument-mining models: on no task does it fall more than about a point and a half behind the specialist, and on four tasks it is the paper's best.","Because joint training shows no degradation, adding further argumentation tasks to the same model is plausible without a dedicated specialist; the measured transfer suggests shared structure among classification-style argument-mining tasks.","Merging with difficulty-aware hyperparameters (higher density and weight for hard tasks) gives a multi-task model at a fraction of the training cost, so teams with limited compute or scarce task data have a workable route to a single multi-task system.","The quantized GGUF versions keep most of the full-precision performance (94.50% vs 95.75% on stance detection), so the multi-task model can run on modest local hardware.","The released unified dataset makes the eight tasks and the conversion rules that produced them reproducible, so other models can be compared on the same balanced splits."],"supporting_citations":[{"why":"Identifies the base model, Llama 3, that every fine-tuned and merged model in the paper starts from.","marker":"[24]"},{"why":"Supplies LoRA, the low-rank fine-tuning method used with rank 16 for every task-specific and multi-task model.","marker":"[37]"},{"why":"Provides the Dagstuhl-15512 ArgQuality dataset on which the argument quality task and its 69.80% top score are measured.","marker":"[40]"},{"why":"Provides MAFALDA, one of the two fallacy-detection datasets behind the FD task and its evaluation.","marker":"[44]"},{"why":"Provides Microtext part 1, the dataset used for ACC and AR tasks and the running conversion example in Section 3.1.","marker":"[66]"},{"why":"Defines the DeBERTa baseline whose per-task scores the fine-tuned models must beat.","marker":"[70]"},{"why":"Supplies the open-source merging toolkit used to run the DARE and DELLA experiments.","marker":"[71]"},{"why":"Defines DARE, the baseline merging method whose pruning-and-rescaling the DELLA variants refine.","marker":"[74]"},{"why":"Defines DELLA merging, the method behind the paper's best merged model, DELLA II.","marker":"[75]"}],"fun_headline_variants":["One Llama model tops four argument-mining benchmarks","Multi-task Llama matches or beats eight specialist miners","Single model, 19 datasets: Llama wins argument mining","Llama multitask model sets top scores on four tasks","One Llama to rule argument mining: matches specialists"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole result rests on the assumption that the same label — 'claim', 'premise', 'support', 'attack' — denotes the same thing in all 19 datasets, so that pouring them into one label scheme trains the model on consistent signals rather than on conflicting definitions of the same word.","fun_headline_variants_meta":{"raw":{"variants":["One Llama model tops four argument-mining benchmarks","Multi-task Llama matches or beats eight specialist miners","Single model, 19 datasets: Llama wins argument mining","Llama multitask model sets top scores on four tasks","One Llama to rule argument mining: matches specialists"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000278,"raw_usage":{"total_tokens":1652,"prompt_tokens":942,"completion_tokens":710,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":558,"completion_tokens_details":{"reasoning_tokens":631}},"tokens_in":558,"tokens_out":710,"duration_ms":7769,"temperature":1.0,"reasoning_tokens":631,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:59:05.242544+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Evaluate the multi-task model on each source dataset's original annotation scheme and native test split instead of the unified conversion — for example, on Microtext's original relation types ('reb', 'sup', 'und', 'add') rather than the collapsed support/attack classes. If the model's scores on the original tasks track the task-specific fine-tuned models, the transfer claim stands; if they drop, the unified benchmark inflated the result.","supporting_citations":[{"cited_title":"Computational argumentation quality assessment in natural language","cited_arxiv_id":null,"evidence_quote":"Provides the Dagstuhl-15512 ArgQuality dataset on which the argument quality task and its 69.80% top score are measured."},{"cited_title":"MAFALDA: A Benchmark and Comprehensive Study of Fallacy Detection and Classification","cited_arxiv_id":"2311.09761","evidence_quote":"Provides MAFALDA, one of the two fallacy-detection datasets behind the FD task and its evaluation."},{"cited_title":"An annotated corpus of argumentative microtexts","cited_arxiv_id":null,"evidence_quote":"Provides Microtext part 1, the dataset used for ACC and AR tasks and the running conversion example in Section 3.1."},{"cited_title":"Arcee’s MergeKit: A toolkit for merging large language models","cited_arxiv_id":null,"evidence_quote":"Supplies the open-source merging toolkit used to run the DARE and DELLA experiments."}],"review_version":2}