{"id":"760ab231-e2a9-4d4a-994b-e5473716454d","arxiv_id":"2504.21685","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"Applying prompt tuning and POS tagger features to health mention classification yields small F1 improvements over plain fine-tuning, but the paper does not compare with actual state-of-the-art systems.","lead":"This paper tests parameter-efficient tuning and part-of-speech information for classifying health mentions in social media, reporting F1 gains on three datasets. A generalist might read it to see whether small, cheap models can replace full fine-tuning for health surveillance, but the experiments lack a true state-of-the-art comparison.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Self-made 70/30 splits, not the standard evaluation protocols, support every 'beats SOTA' claim; without split-matched reruns the headline F1 comparisons are unverifiable.","rationale":"The reader's weakest assumption is the right one. The paper's strongest claim is comparative ('compared to state-of-the-art methods'), and every such comparison requires a shared evaluation protocol. Section 4.1 explicitly says the authors created random 70/30 splits because providers 'did not supply them'; however, the cited SOTA systems (WESPAD on PHM2017, HMCNET on RHMD, multi-view active learning on Illness) were evaluated with their own splits or protocols. Table 5 contains no external SOTA row, only the authors' own BERT/RoBERTa fine-tuning baselines, so the F1 values cannot be checked against the published literature. Adding POS can help in some internal rows, but the gains are inconsistent (e.g., Intermediate POS with prefix-tuning on Illness drops from 94.5 to 88.6), and no confidence intervals are reported. The concern is not that PEFT is a bad idea; it is that the paper's headline evidence is not anchored to the comparison it claims. A reproducible rerun on official splits, or removal of the SOTA claim, would settle the matter. I therefore keep the reader's REJECT verdict.","tokens_in":9214,"tokens_out":3524,"duration_ms":35974,"concrete_test":"Take the best Table 5 configurations (e.g., prompt-tuning on PHM2017=91.5, intermediate-POS with prompt-tuning on RHMD=82.2, prompt-tuning on Illness=95.5) and rerun them on the original released splits for PHM2017, RHMD, and Illness, using the exact preprocessing and F1-micro scoring of each cited SOTA paper; then compare against the published numbers in those papers. If the F1 deltas change sign or shrink within noise, the abstract's 'state-of-the-art' claim is not supported. If no official split exists, the authors must adopt the evaluation protocol used by the SOTA papers or withdraw the SOTA claim. The authors should also release split indices and code so the comparison can be reproduced.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that PEFT with POS beats state-of-the-art HMC systems on PHM2017, RHMD, and Illness (abstract). The load-bearing condition is that the reported F1 scores are measured under the same evaluation protocol as the SOTA numbers they are compared against. Section 4.1 states: 'We created our own splits because the dataset providers did not supply them... random 70%/30% split.' No evidence is given that these splits match the train/test partitions used by Karisani and Agichtein (2018a), Naseem et al. (2022), or Karisani et al. (2022). Table 5 compares only to in-house 'Fine-tuning' baselines (90/80/91), not to any previously published SOTA system. Consequently, none of the 'outperforms SOTA' statements are checkable: a 91.5 or 95.5 F1 on a self-made split is not comparable to a published number obtained on an official split, and no error bars or significance tests are reported. This is not a minor caveat; it is the entire evidence for the headline result.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies parameter-efficient fine-tuning (PEFT) techniques, including prompt tuning, soft prompting, prefix tuning, and LoRA, for health mention classification on three datasets: PHM2017, RHMD, and Illness. It also explores two ways of incorporating part-of-speech information: intermediate-task fine-tuning on POS and representation fusion with a POS-tuned model. The main empirical result is that prompt-tuning reaches the highest F1-micro scores on all three datasets (91.5, 80.9, and 95.5), outperforming the in-house BERT/RoBERTa fine-tuning baselines by small to moderate margins. The paper claims in the abstract and conclusion that these results outperform state-of-the-art methods, but no previously published state-of-the-art system is included in the comparison table, and the evaluation uses custom random 70/30 splits rather than the original protocols used in prior work.","tokens_in":9475,"tokens_out":5042,"duration_ms":49385,"significance":"If validated under the standard evaluation protocols of the prior literature, the paper would provide a practically useful result: prompt tuning combined with POS information can beat full fine-tuning for health mention classification while updating far fewer parameters. The internal experiment design is coherent in that all variants share the same optimizer, scheduling, and cross-validation setup, and the exploration of two POS incorporation strategies and modified soft prompting is a reasonable empirical contribution. However, the central state-of-the-art claim is unsupported as presented because Table 5 contains no external prior system and the custom splits may not be comparable to the splits used in the papers cited as state of the art. The paper does not provide code or split files, so the reported numbers are not independently reproducible at present.","major_comments":[{"comment":"The abstract states that PEFT techniques 'significantly improves performance in terms of F1-score compared to state-of-the-art methods across all three datasets,' but Table 5 lists only in-house fine-tuning baselines (BERT and RoBERTa). No prior published HMC system from the Related Work section appears in the comparison. Section 4.1 further states: 'We created our own splits because the dataset providers did not supply them... random 70%/30% split.' Since the authors do not show that their random splits match the train/test partitions used by Karisani and Agichtein (2018a), Naseem et al. (2022), or Karisani et al. (2022), every 'outperforms state-of-the-art' claim in the abstract and conclusion is unverifiable as written. Please rerun the experiments on the original evaluation protocols of the cited datasets, or explicitly reimplement prior systems on the same splits, and compare against those numbers; otherwise the state-of-the-art claim must be removed or replaced with a claim about outperforming the in-house baselines.","section":"Section 4.1, Table 5, Abstract"},{"comment":"No variance estimates, confidence intervals, or significance tests are reported. Several design conclusions rest on differences of 0.1 to 0.5 F1 points, e.g., Table 2 reports 95.4 vs 95.5 for two verbalizers on the Illness dataset and Table 3 reports 80.8 vs 80.9 on RHMD. Such small gaps are within typical run-to-run noise for transformer fine-tuning, so the verbalizer and prompt selections are not yet supported without repeated runs or statistical testing.","section":"Section 4.1, Tables 2-4"},{"comment":"The paper claims that incorporating grammar-aware language models improves performance 'by a magnitude ranging from 1% to 3%.' This is not consistent with the numbers in Table 5: prompt tuning over fine-tuning gives gains of about 1.5 points on PHM2017, 0.9 points on RHMD, and 4.5 points on Illness. Please specify exactly which comparison supports the 1%-3% range and cite the corresponding table rows.","section":"Section 5, contribution 2"},{"comment":"The domain-adaptation experiments are incompletely reported. Table 5 has missing entries ('-') for the PHM2017 dataset under 'pre-training then fine-tuning' and 'pre-training then prefix-tuning,' and the text does not state how much pre-training data is used or whether the two auxiliary datasets are concatenated. Consequently, the Section 4.2 claim that multiple-source domain adaptation achieves a 'performance drop of around 0.5%' cannot be verified from the presented results.","section":"Section 3.2.2, Table 5"}],"minor_comments":[{"comment":"The dataset name is written as 'RHDM' in the abstract but as 'RHMD' in Section 3.1, Table 1, and elsewhere in the paper; please use one consistent spelling.","section":"Abstract and Section 3.1"},{"comment":"There are several typos: 'a a new dataset' in Section 2, 'the a lack' in Section 3.2, 'NonNon-Health Mentions' in Table 1, and 'gives an insights' in Section 4.2. These should be corrected.","section":"Section 2, Section 3.2, Table 1"},{"comment":"The abbreviation 'LoRa' should be 'LoRA' for consistency with the cited work by Hu et al. (2021).","section":"Sections 3.2.3 and 6"},{"comment":"The abstract calls the models 'biomedical natural language methods (NLP),' but the experiments use general-domain BERT and RoBERTa, which are not biomedical models. Please clarify the terminology or use biomedical PLMs such as BioBERT if that is intended.","section":"Abstract and Section 3.2"},{"comment":"References Karisani and Agichtein (2018a) and (2018b) appear to be the same paper with the same title but different venue formatting; please merge or clearly distinguish them.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern is valid: the paper's headline claim rests on custom 70/30 splits and on a comparison table that contains no external state-of-the-art system. This is a load-bearing problem, but it is fixable within the manuscript's scope by (i) rerunning on the original splits or reimplementing prior systems on the same splits and (ii) either adding the external numbers to Table 5 or tempering the claims to what the evidence shows. The internal prompt-tuning-versus-fine-tuning result is plausible and worth pursuing, but the current version should not be accepted without that verification."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a workmanlike empirical comparison of PEFT variants for health mention classification, and the only result it actually establishes is that prompt tuning beats simple fine-tuning by 1.5 to 4.5 micro-F1 on three public datasets. The headline claim of beating state-of-the-art is not supported by the evidence in the paper.\n\nWhat's new and good: the combination of prompt, prefix, soft-prompt (including the wrapped variant), LoRA, and POS-augmented variants across PHM2017, RHMD, and Illness is new in this specific form. The soft-prompt modifications—virtual tokens before and after the text, and virtual tokens plus a hard prompt—are simple and worth a look. The paper reports extensive experiments, uses micro-F1 consistently, and is honest that the gains are modest. It also reports negative results: several POS configurations underperform the baseline, which is more than many papers do.\n\nWhere it falls down: all \"outperforms SOTA\" statements in the abstract and conclusion rest on Table 5, which compares only to in-house fine-tuning baselines (90/80/91). No previously published system appears in the table. Worse, Section 4.1 says the authors created their own random 70/30 splits because the dataset providers did not supply them, with no evidence those splits match the official partitions used by Karisani et al., Naseem et al., or Karisani, Karisani, and Xiong. A 95.5 F1 on a self-made split is not comparable to a published number obtained on a standard split. No error bars, no significance tests, and no code or splits are provided. The discussion highlights the gains while downplaying the configurations that underperform. On the stress-test note: it holds up—the self-made splits are the load-bearing problem, not a minor caveat.\n\nConclusion: this is a useful internal-comparison preprint, not a verifiable state-of-the-art claim. It deserves serious referee attention only if the authors are willing to add external baselines, release splits and code, and rerun the comparisons on standard evaluation protocols. As it stands, the central claim fails, but the underlying work is not worthless. The reader's take is fair.","headline":"Useful internal PEFT comparison that overclaims state-of-the-art: no external baselines, self-made splits, and no code make the headline impossible to verify.","tokens_in":9998,"tokens_out":2233,"would_cite":false,"duration_ms":23348,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that parameter-efficient tuning, especially prompt tuning combined with part-of-speech information, improves health mention classification across three social media datasets while updating far fewer parameters than full…","keywords":["health mention classification","parameter-efficient fine-tuning","prompt tuning","soft prompting","prefix tuning","LoRA","part-of-speech tagging","social media public health"],"falsifier":"A reader could settle the central claim by taking the best configurations, prompt tuning on PHM2017 and Illness and intermediate POS with prompt tuning on RHMD, and running them on the standard or original splits used by the prior published systems, then comparing F1 to the numbers those systems reported; if the scores fall back to or below the prior results, the claim that PEFT outperforms state-of-the-art health mention classification would not survive.","tokens_in":8984,"feed_emoji":"🩺","tokens_out":7292,"duration_ms":66705,"temperature":0.7,"pith_summary":"This paper sets out to show that health mention classification on social media can be improved not by scaling up models but by tuning only a small set of parameters and by giving the model explicit part-of-speech information. The authors compare prompt tuning, soft prompting, prefix tuning, and LoRA-style adapters against full fine-tuning of BERT and RoBERTa on the PHM2017, RHMD, and Illness datasets. Their central result is that prompt tuning, in some cases combined with POS information, achieves the highest F1 scores on all three datasets while updating far fewer parameters than full fine-tuning. They further claim these results beat prior published systems for health mention classification. If correct, this would make public health monitoring from Twitter and Reddit cheaper and easier to deploy, because one small generic model could serve many tasks.","feed_headline":"Prompt tuning beats full fine-tuning on health mention classification","feed_subtitle":"Adding part-of-speech cues gives top F1 scores of 91.5, 82.2, and 95.5 across three datasets.","key_machinery":"The central mechanism is the reformulation of classification as masked language modeling through prompt templates: the text is wrapped in a prompt such as \"What is the health mention class? [MASK].\" and the model's prediction for the [MASK] token is mapped through a verbalizer to a label. This is combined with parameter-efficient tuning techniques, prompt tuning, soft prompting with learnable virtual tokens, prefix/P-tuning V2, and LoRA, that keep the pretrained transformer frozen except for small trainable components. A second mechanism is POS injection, either by intermediate-task fine-tuning, first training on POS tagging and then on HMC, or by representation fusion, concatenating representations from the original model and a POS-tuned model before classification. These pieces let the model exploit grammatical dependencies that often decide whether a disease word refers to a real personal health condition.","core_discovery":"The paper's core claim is that parameter-efficient tuning with prompt-based inputs is a superior strategy for health mention classification than conventional full fine-tuning. In the reported experiments, prompt tuning reaches an F1 of 91.5 on PHM2017, intermediate POS with prompt tuning reaches 82.2 on RHMD, and prompt tuning reaches 95.5 on Illness, all above the fine-tuning baselines of 90, 79-80, and 91 respectively. The paper also claims that injecting part-of-speech knowledge, either by first fine-tuning on a POS task or by fusing POS-tuned representations, lifts performance by roughly 1 to 3 percent, and that two modifications of soft prompting, placing virtual tokens around the text and combining soft tokens with a hard prompt, stabilize training and improve on plain soft prompting. These findings are presented as evidence that health mention classification can be done accurately without updating large language model bodies.","pith_inferences":["If the reported gains hold under the original evaluation splits of PHM2017, RHMD, and Illness, then combining prompt tuning with POS cues could become a cheap default for other noisy social-media classification tasks, such as detecting misinformation or adverse drug reactions.","The paper compares against prior published systems under its own random 70/30 splits; a stronger test would be a shared-benchmark evaluation where every method sees the same test folds, which would reveal whether the improvement is real or split-dependent.","The success of wrapping soft prompts around the text suggests that the position of virtual tokens matters, and a natural extension is to test whether the same two modifications help on other sentence-level classification tasks beyond health mentions."],"forward_implications":["On the paper's reported numbers, prompt tuning is the strongest single technique, with F1 of 91.5 on PHM2017, 80.9 on RHMD, and 95.5 on Illness, beating full fine-tuning on every dataset.","Adding POS information in the best configuration raises F1 by about 1 to 3 percent, and on RHMD the best result, 82.2, comes from intermediate POS combined with prompt tuning.","Wrapping text with soft virtual tokens and mixing them with a hard prompt improves plain soft prompting by 2 percent or more on RoBERTa-based models.","Cross-dataset masked-language-model pre-training performs within about 0.5 percent of fully in-domain training, suggesting domain shift can be handled without large task-specific pretraining corpora.","Because the trainable parameters are only a small set of prompts, prefixes, or adapters, one pretrained model can serve many health-classification tasks with low memory and training cost."],"supporting_citations":[{"why":"Supplies the PHM2017 dataset of health-related tweets and the four-class health mention task that the paper evaluates on.","marker":"Karisani and Agichtein (2018a)"},{"why":"Supplies the Illness dataset with positive and negative health mention labels used as a third benchmark.","marker":"Karisani et al. (2022)"},{"why":"Introduces the RHMD Reddit dataset and the HMCNET approach, which the paper uses as a comparison point.","marker":"Naseem et al. (2022)"},{"why":"Representative prior HMC work emphasizing literal meanings and emotion; provides context and baselines the paper positions itself against.","marker":"Aduragba et al. (2023)"},{"why":"Gives the BERT-large and RoBERTa-large contrastive adversarial training baseline that the paper seeks to improve on.","marker":"Khan et al. (2022)"},{"why":"Defines prompt tuning, the core PEFT method the paper applies and modifies.","marker":"Lester et al. (2021)"},{"why":"Defines prefix-tuning, one of the PEFT variants the paper evaluates.","marker":"Li and Liang (2021)"},{"why":"Defines P-tuning v2, the layer-wise prompt tuning method the paper combines with LoRA.","marker":"Liu et al. (2021)"},{"why":"Defines LoRA, the low-rank adaptation used in the prefix-tuning-with-LoRA configuration.","marker":"Hu et al. (2021)"},{"why":"Supplies the intermediate-task fine-tuning rationale used for the POS-then-HMC approach.","marker":"Chang and Lu (2021)"}],"fun_headline_variants":["Prompt tuning beats full fine-tuning for health mentions","POS cues boost prompt tuning in health classification","Efficient tuning wins for health mention classification","Prompt tuning + POS info lifts HMC F1 scores","POS-aware prompt tuning tops health mention classifiers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the authors' self-made random 70/30 splits are interchangeable with the evaluation protocols used by the earlier systems they claim to beat, since no official splits are used.","fun_headline_variants_meta":{"raw":{"variants":["Prompt tuning beats full fine-tuning for health mentions","POS cues boost prompt tuning in health classification","Efficient tuning wins for health mention classification","Prompt tuning + POS info lifts HMC F1 scores","POS-aware prompt tuning tops health mention classifiers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000682,"raw_usage":{"total_tokens":3104,"prompt_tokens":958,"completion_tokens":2146,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":574,"completion_tokens_details":{"reasoning_tokens":2076}},"tokens_in":574,"tokens_out":2146,"duration_ms":14955,"temperature":1.0,"reasoning_tokens":2076,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:56:25.289167+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A reader could settle the central claim by taking the best configurations, prompt tuning on PHM2017 and Illness and intermediate POS with prompt tuning on RHMD, and running them on the standard or original splits used by the prior published systems, then comparing F1 to the numbers those systems reported; if the scores fall back to or below the prior results, the claim that PEFT outperforms state-of-the-art health mention classification would not survive.","supporting_citations":[{"cited_title":"Identification of disease or symp- tom terms in reddit to improve health mention clas- sification","cited_arxiv_id":null,"evidence_quote":"Introduces the RHMD Reddit dataset and the HMCNET approach, which the paper uses as a comparison point."},{"cited_title":"Improving health mention classification through emphasising literal mean- ings: A study towards diversity and generalisation for public health surveillance","cited_arxiv_id":null,"evidence_quote":"Representative prior HMC work emphasizing literal meanings and emotion; provides context and baselines the paper positions itself against."}],"review_version":1}