{"id":"b0fa7f76-0c53-4f9b-a9c9-353aa5caf7c8","arxiv_id":"2506.01961","paper_version":1,"verdict":"REJECT","confidence":"LOW","novelty_score":2.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"A prompt-augmented BioBERT-MRC model is claimed to improve medical NER, but the reported gains are small and the paper provides insufficient detail to verify the method or results.","lead":"This paper proposes a prompt-augmented machine reading comprehension model for medical named entity recognition, using BioBERT with hand-designed prompts. It reports small F1 gains over some baselines on two medical datasets, but the paper lacks the experimental detail and reproducibility needed to support its claims.","discovery_kind":"incremental","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed 0.86 F1 gain over MRCNER cannot be verified because the paper does not report any numerical result tables or error bars; all comparative claims refer to figures with no visible data.","rationale":"The reader's weakest assumption (unfair or unreproducible baseline comparison) is a legitimate possibility, but the more fundamental issue is that the experiments are not verifiable at all: no numeric tables or error bars appear in the text, figures lack readable values, and no code is supplied. This moves the paper from 'reject on unverifiable evidence' to 'could be acceptable if transparency requirements are met', which is a conditional rather than a firm rejection, given the low confidence of the original verdict. Because the reader's concern about the MRCNER comparison is one specific manifestation of the broader unverifiability, I partially agree with the reader's weakest assumption.","tokens_in":7067,"tokens_out":1023,"duration_ms":8931,"concrete_test":"Request the released model and evaluation script plus the exact GENIA/CADEC splits, then independently rerun the MRCNER baseline and the Prompt-bioMRC model with identical hyperparameters and seeds, reporting mean and standard deviation over at least five runs; the central claim holds only if the 0.86 F1 gap (and the claimed 2-point precision gain on CADEC) is reproduced outside the margin of error.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The strongest claim (Prompt-bioMRC beats MRCNER by 0.86 F1 on GENIA and improves precision over BERT-large on CADEC) rests entirely on Figures 4 and 5, neither of which contains readable numeric values in the manuscript text. The method section (§3.1–3.2) describes a standard MRC span-extraction setup with BioBERT plus concatenated prompts, but it omits template strings, prompt length m, span-loss details, and exact evaluation protocol. No code, no data splits, no checkpoint, no error bars, and no standard deviations are provided. The paper's only concrete table (Table 1) lists generic BERT hyperparameters that are not enough to reproduce even the baseline, so the central experimental assertion is uncheckable from the manuscript alone. This lack of verifiable evidence is load-bearing because the entire contribution is 'we add prompts and get +0.86 F1'.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes Prompt-bioMRC, a medical named entity recognition model that concatenates hand-designed hard prompts and learned soft prompts with BioBERT inside a machine reading comprehension (MRC) span-extraction framework. The paper describes a preprocessing pipeline that converts NER annotations into (PROMPT, ANCHOR, CONTEXT) triplets and then uses BioBERT to extract entity spans conditioned on the prompt and the original text. Experiments are reported on the CADEC and GENIA datasets, with claims that BioBERT improves precision over BERT-large by more than two percentage points on CADEC and that Prompt-bioMRC improves F1 by 0.86 percentage points over the MRCNER baseline on GENIA. The results are presented only through figures, without numerical tables, error bars, or significance tests, and the method section omits equations, prompt template examples, and implementation details. The conclusion frames the contribution as modest and acknowledges that the soft-prompt exploration is preliminary and that hard templates are dataset-specific and manually constructed.","tokens_in":7218,"tokens_out":4176,"duration_ms":41694,"significance":"If the reported gains were reproducible, the paper would provide modest evidence that prompt-based MRC with a medical pretrained language model improves NER on biomedical benchmarks. However, the significance as presented is low because the central comparison is unverifiable: all quantitative results live in unreadable figures, no code, data splits, trained checkpoints, or statistical significance tests are supplied, and the fairness of the MRCNER baseline comparison is not established. The contribution reduces to an assertion that adding hand-designed prompts to an existing MRC framework helps, which is not falsifiable from the manuscript as written. The paper ships no reproducible assets, machine-checked proofs, or parameter-free derivations that would offset the missing experimental evidence.","major_comments":[{"comment":"The central claim that 'in contrast to the baseline MRCNER model, our approach demonstrates an enhanced F1 score by 0.86 percentage points' cannot be checked, because no numerical result table, error bar, standard deviation, or significance test is reported anywhere in the manuscript. The same applies to the stated 64.85% precision for BioBERT and the improvement of more than two percentage points over BERT-large, which appear in prose without a reproducible data source. Provide for every dataset and every model: precision, recall, and F1 with standard deviations across at least three seeds, the number of runs, and the exact evaluation split. Without this, the improvement over MRCNER is an unsupported assertion rather than an experimental result.","section":"§4, Figures 4 and 5"},{"comment":"The method description is insufficient to reproduce the model. The text does not specify the hard-template strings, the length or initialization of the soft-prompt vectors, the span start and end probability equations, the loss function, or the exact sequence construction around the [CLS] and [SEP] markers. Figure 2 is a generic diagram without parameter counts or tensor shapes. Because the only claimed contribution is the prompt integration, omitting the prompt design is load-bearing and leaves open the possibility that differences from the MRCNER baseline arise from unspecified implementation choices rather than from prompting.","section":"§3.1–3.2"},{"comment":"The experimental setup is underreported. Table 1 lists only generic BERT hyperparameters such as hidden size, attention heads, and number of layers; it omits learning rate, optimizer, batch size, warmup, random seeds, and the exact training and evaluation data splits for CADEC and GENIA. The paper also does not state whether the MRCNER baseline was reimplemented under identical conditions or taken from a prior publication, so the reported 0.86 F1 gain could be an artifact of differing evaluation protocols rather than an effect of prompting. State the provenance of each baseline number and confirm identical preprocessing, hyperparameters, and splits, or report newly reimplemented baseline results. In addition, because the hard prompts are manually designed, the comparison should address how baseline models were given an equivalent amount of prompt or hyperparameter tuning.","section":"§3.3 and Table 1"}],"minor_comments":[{"comment":"The section titled 'Tips for learning' is unrelated to prompt learning and cites works on remote patient monitoring and neural architecture search; this section should either be rewritten to cover actual prompt-based NER literature or removed.","section":"§2.2"},{"comment":"The comparative results appear only as image placeholders with no legible numeric labels in the manuscript text; replace them with high-resolution figures and, more importantly, with tables listing exact numeric values.","section":"Figures 4 and 5"},{"comment":"The manuscript contains numerous grammatical errors and typos, including 'Experimental desig' in §3.3 and inconsistent capitalization such as 'Prompt bioMRC' versus 'Prompt-bioMRC'; the text should be edited by a native speaker before resubmission.","section":"Throughout"},{"comment":"The conclusion admits that soft-prompt exploration is preliminary and that hard templates require dataset-specific manual construction; this concession should appear in the abstract and introduction, which currently claim that the approach consistently surpasses traditional models across diverse datasets.","section":"§5, Conclusion"}],"recommendation":"reject","confidential_remarks":"The manuscript is far below the standard expected for a refereed journal: the experimental core is unverifiable, the method section omits the essential prompt-design mechanism, and substantial parts of the related work are off-topic. The missing numerical evidence is not a local presentation issue because the entire contribution is an empirical claim of improved F1. A publishable version would require new experiments with numerical tables, error bars, a fully specified method, and a careful baseline comparison, which goes beyond a standard revision. I would not recommend major revision unless the venue explicitly accepts preliminary position papers."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this paper is a re-implementation of prompt-based MRC for medical NER. The reported gain over MRCNER on GENIA is 0.86 F1, but the numbers appear only in figures and the text gives no tables, error bars, or significance tests. I cannot verify the central claim from the manuscript, and neither can a referee.\n\nTo be fair, the basic idea is sane: use BioBERT in an MRC span-extraction setup and prepend hand-designed prompts per entity type. That is a reasonable thing to try, and the paper evaluates on two standard medical NER datasets (GENIA and CADEC). But the combination is already established in the cited literature (Du et al. 2022, MRCNER), and the paper adds no new mechanism, formalism, or reusable design. The only concrete numbers mentioned in the text—64.85% precision on CADEC and a 0.86 F1 improvement on GENIA—are single-sentence summaries with no error bars. The method section (§3) omits the actual prompt templates, prompt lengths, span-loss details, and evaluation protocol, so reproduction from the text alone is impossible.\n\nThe problems go beyond missing details. Section 2.2 is titled 'Tips for learning' and opens with 'Learning is a lifelong journey'; it then cites papers on Internet of Medical Things, neural architecture search, and few-shot learning, none of which are about prompt learning. Section 2.3 lumps MRC for NER together with MRC-TransUNet for medical image segmentation. That incoherence suggests the related work was assembled by keyword rather than by reading. The paper itself calls its contribution 'modest,' which is accurate but undercuts the strong claims in the abstract and conclusion.\n\nNone of this is a fatal flaw in the research question—prompt engineering for medical NER is worth studying—but as presented, the evidence is not there. There is no code, no data splits, no checkpoints, and no way to tell whether the 0.86 F1 gain is a real effect of the prompts or an artifact of differing baselines. The stress-test note is right: the central assertion is uncheckable.\n\nI would not send this to peer review in its current form. If the authors supply code, data, and a proper table with standard deviations, there might be a small empirical result worth a workshop paper. As it stands, desk reject.","headline":"A prompt-based MRC for medical NER that claims a small F1 gain but provides no usable numbers, no code, and a related-work section that reads as if assembled by keyword search; not ready for review.","tokens_in":7754,"tokens_out":2496,"would_cite":false,"duration_ms":21741,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adding hand-designed and learned prompts to BioBERT in an MRC span-extraction setup improves medical named-entity recognition over the MRCNER baseline on GENIA and over BERT-large precision on CADEC.","keywords":["named entity recognition","medical NER","prompt learning","BioBERT","machine reading comprehension","hard template","soft prompt","intelligent consultation system"],"falsifier":"Run the same data split, hyperparameters, and evaluation script for MRCNER and Prompt-bioMRC, then replace the prompt with a generic filler and re-measure. If the F1 gap over MRCNER disappears or the prompt ablation does not lower F1, the improvement is not caused by prompt engineering.","tokens_in":6852,"feed_emoji":"🩺","tokens_out":8438,"duration_ms":77424,"temperature":0.7,"pith_summary":"The paper sets out to show that prompt engineering improves medical named entity recognition (NER) when combined with a biomedical language model in a machine-reading-comprehension (MRC) setup. Instead of labeling tokens directly, the model is asked a label-specific question—formed by a hand-written hard template or a learned soft prompt—and answers by predicting the start and end of the entity span. On the GENIA dataset the resulting Prompt-bioMRC model reports an F1 score 0.86 points above the MRCNER baseline; on CADEC, BioBERT reaches a precision of 64.85%, more than two points above BERT-large. The claimed payoff is practical: more accurate extraction of diseases, drugs, and other clinical entities supports automated consultation and diagnosis systems.","feed_headline":"Prompted BioBERT beats its baseline by 0.86 F1 on medical NER","feed_subtitle":"Recasting entity recognition as question answering with label prompts helps on GENIA and CADEC medical texts.","key_machinery":"The load-bearing object is the Prompt-bioMRC model: an MRC-style span extractor that concatenates a label-specific prompt with the input text and uses BioBERT—a BERT model pretrained on biomedical literature—to encode the combined string, then predicts the start and end positions of the entity span. The prompt acts as a per-entity-type query, converting a single NER model into a set of question-answering tasks; soft prompts are embeddings learned during fine-tuning rather than fixed natural-language strings. This machinery matters because it replaces the usual token-level labeling head with a span-prediction objective and because it lets the entity-type knowledge enter through the input text, which is what the paper credits for the accuracy gains.","core_discovery":"On its own terms, the paper's central claim is that recasting medical NER as prompted machine reading comprehension improves accuracy over conventional sequence-labeling or span-classification baselines. For each entity type, the model constructs a triplet (PROMPT, ANCHOR_START, ANCHOR_END, X) in which the prompt names the entity type; BioBERT encodes the prompt and the text together, and the model locates the answer span by predicting its start and end indices. Hard templates are designed manually per dataset, while the soft prompt is a learned embedding added during fine-tuning, and both feed the same BioBERT-based MRC core. The reported consequence is that Prompt-bioMRC surpasses the MRCNER baseline by 0.86 F1 points on GENIA and, with BioBERT as the encoder, raises precision by more than two points over BERT-large on CADEC.","pith_inferences":["The span-extraction core is label-agnostic, so if the GENIA result holds, the same prompt scaffolding could in principle be carried to untested entity types such as genes, chemicals, or procedures; the paper does not run that test.","Because the reported edge over MRCNER is under one F1 point, run-to-run seed variance could matter; repeated runs would tell whether prompt engineering or noise explains the gap.","A natural untested extension is to swap the BioBERT encoder for a larger or generative model, where prompts carry more of the task specification and the prompt-MRC gain might be bigger."],"forward_implications":["On GENIA, Prompt-bioMRC reports an F1 gain of 0.86 points over MRCNER and roughly five points over the Hyper-Graph model, placing the prompted MRC approach ahead of several earlier medical NER systems reported in the paper.","On CADEC, using BioBERT inside the framework yields 64.85% precision, more than two points above BERT-large, meaning domain-specific pretraining continues to contribute even after prompting is added.","Because entity types enter through prompts, a new entity category can be added by designing or learning a new prompt instead of retraining a separate classification layer.","Hard templates and soft prompts both operate on the same BioBERT-MRC core, so the model supports a modular route from fully manual to learned prompt construction."],"supporting_citations":[{"why":"Supplies the MRC-based span-extraction formulation of NER that Prompt-bioMRC adapts to medical texts.","marker":"[8]"},{"why":"Provides a medical few-shot NER method that prompt-style learning is compared against and positioned relative to.","marker":"[6]"},{"why":"Establishes the text-to-text and prompt paradigm that motivates using hard and soft prompts rather than task-specific heads.","marker":"[9]"},{"why":"Gives a medical BERT hybrid baseline that frames the comparison against existing pretrained-model NER systems.","marker":"[3]"},{"why":"Supplies a deep-neural-network medical NER baseline for the experimental comparisons.","marker":"[4]"}],"fun_headline_variants":["Prompt-bioMRC: medical NER as QA beats baseline by 0.86 F1","BioBERT with prompt learning: +0.86 F1 on GENIA medical NER","Hard and soft prompts turn BioBERT into medical NER QA","Asking BioBERT to locate spans: medical NER gets +0.86 F1","Prompt-bioMRC: QA-style prompts for medical entity recognition"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claimed 0.86-point F1 gain over MRCNER assumes the comparison used identical training data, hyperparameters, and evaluation, so that only the added prompts differ; the paper gives no details confirming this.","fun_headline_variants_meta":{"raw":{"variants":["Prompt-bioMRC: medical NER as QA beats baseline by 0.86 F1","BioBERT with prompt learning: +0.86 F1 on GENIA medical NER","Hard and soft prompts turn BioBERT into medical NER QA","Asking BioBERT to locate spans: medical NER gets +0.86 F1","Prompt-bioMRC: QA-style prompts for medical entity recognition"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000717,"raw_usage":{"total_tokens":3190,"prompt_tokens":883,"completion_tokens":2307,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":499,"completion_tokens_details":{"reasoning_tokens":2202}},"tokens_in":499,"tokens_out":2307,"duration_ms":17663,"temperature":1.0,"reasoning_tokens":2202,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:18:35.565957+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same data split, hyperparameters, and evaluation script for MRCNER and Prompt-bioMRC, then replace the prompt with a generic filler and re-measure. If the F1 gap over MRCNER disappears or the prompt ablation does not lower F1, the improvement is not caused by prompt engineering.","supporting_citations":[{"cited_title":"MRC-Based Medical NER with Multi-task Learning and Multi-strategies","cited_arxiv_id":null,"evidence_quote":"Supplies the MRC-based span-extraction formulation of NER that Prompt-bioMRC adapts to medical texts."},{"cited_title":"W-procer: Weighted Prototypical Contrastive Learning for Medical Few-Shot Named Entity Recognition","cited_arxiv_id":"2305.18624","evidence_quote":"Provides a medical few-shot NER method that prompt-style learning is compared against and positioned relative to."},{"cited_title":"Generative large language models are all-purpose text analytics engines: text-to-text learning is all your need","cited_arxiv_id":null,"evidence_quote":"Establishes the text-to-text and prompt paradigm that motivates using hard and soft prompts rather than task-specific heads."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Gives a medical BERT hybrid baseline that frames the comparison against existing pretrained-model NER systems."},{"cited_title":"Named Entity Recognition of Medical Text Based on the Deep Neural Network","cited_arxiv_id":null,"evidence_quote":"Supplies a deep-neural-network medical NER baseline for the experimental comparisons."}],"review_version":1}