{"id":"1a4d658c-41ec-44ff-8629-3d3db191b0a6","arxiv_id":"2509.00806","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Fine-tuned LLaMA 3 8B reaches ~0.8 concept-level accuracy on MedHopQA development data but only ~0.5 exact match in validation and 0.0 to 0.2 on the test set.","lead":"This paper describes a fine-tuned LLaMA 3 8B system for the MedHopQA biomedical question answering shared task. It reports strong semantic-level accuracy but weak exact-match performance, and introduces a two-stage extraction pipeline to shorten verbose answers.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Validation 0.8 concept accuracy conflicts with official test concept scores (0.11-0.31), and the dev-set size is contradictory; 'strong domain understanding' is unsupported on the actual benchmark.","rationale":"The reader's verdict is CONDITIONAL, with the weakest assumption being that the 10,000-question training mixture is representative of MedHopQA's rare-disease distribution. My stress-test reading agrees that distribution is a key issue, but I see an even more direct problem in the paper's own reported numbers: the concept-level accuracy that anchors the central claim ('up to 0.8') comes from a validation set that is internally described inconsistently (45 vs. 1,000 questions) and is not reproduced on the official test set, where concept scores are 0.3120, 0.1140, and 0.2250. This means the 'strong domain understanding' claim is at best development-only, and the headline gap between understanding and exact match is much smaller on the test distribution. This reinforces the reader's conditional verdict rather than overturning it: the paper should clarify the validation-set size, report confidence intervals, and discuss the dev-to-test drop explicitly. No change to the verdict is needed, but the conditional acceptance should explicitly require these clarifications and ideally artifact release.","tokens_in":6555,"tokens_out":6003,"duration_ms":68453,"concrete_test":"Request from the BioCreative organizers the per-question concept-level scores for the three official submissions, and recompute test concept accuracy. If the test concept accuracy is below 0.4 for all setups (as the reported 0.31/0.11/0.23 suggest), then the validation 0.8 is not representative and the 'strong domain understanding' claim must be withdrawn or restricted to the development distribution. Also report the exact validation-set size used for the 0.8 score and recompute Wilson confidence intervals for both EM and concept accuracy; if n=45, the intervals are wide enough to weaken the claimed gap.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim—'strong domain understanding' with concept-level accuracy up to 0.8 while exact match lags—rests on the validation-set concept scores reported in §4.2. Two problems make this load-bearing. First, §3.1 describes the development set as '45 question instances' with short/long answers, but also says a '1,000-example development set' was used as validation; the denominator for the 0.8 score is never pinned down. If it is 45, 0.8 is 36/45 (Wilson 95% CI ≈ 0.66–0.89) and EM 0.5 is ≈ 23/45 (CI ≈ 0.36–0.64), so the claimed concept-EM gap is based on a handful of items. Second, the same section reports official leaderboard concept scores of 0.3120, 0.1140, and 0.2250 for the three setups—far below 0.8. Thus the validation concept accuracy does not transfer to the actual benchmark, and the 'strong domain understanding' half of the central claim is unsupported on the test distribution. The conclusion that LLMs show a large 'semantic understanding vs exact match' gap is conflated with a train/test distribution shift and small-sample noise.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper describes a supervised fine-tuning system (LLaMA 3 8B with LoRA) for the BioCreative IX MedHopQA multi-hop biomedical QA track. Three models are trained on a self-curated 10,000-question dataset assembled from BioASQ, MedQuAD, TREC, and other public biomedical QA sources, using combined short/long, short-only, and long-only answer formats. A two-stage inference pipeline is introduced to extract concise answers from verbose model outputs. Reported results: validation EM around 0.5 and concept-level accuracy around 0.8; official test EM scores of 0.2, 0.0, and 0.0 for the three setups, with leaderboard concept scores of 0.3120, 0.1140, and 0.2250. The paper concludes that the model has strong semantic understanding but fails to produce exact-match answers.","tokens_in":6863,"tokens_out":3706,"duration_ms":45228,"significance":"If the validation concept-level accuracy of ~0.8 held on the official test, the paper would provide useful evidence on the gap between semantic understanding and exact-match evaluation in biomedical QA. The paper's strengths are its transparent reporting of official leaderboard scores and its clear description of a practical extraction pipeline. However, the central empirical claim is weakened by contradictory development-set descriptions, the absence of per-setup validation tables, a large unexplained drop from validation to test, and no quantitative evaluation of the proposed post-processing pipeline. As a system description, the contribution is modest and primarily of interest to the BioCreative community.","major_comments":[{"comment":"The development set is described twice inconsistently: Section 3.1 first says it includes '45 question instances' and later says the 'official 1,000-example development set' was used as validation. If the ~0.8 concept accuracy and ~0.5 EM are computed on 45 questions, the scores correspond to roughly 36/45 and 23/45, with wide confidence intervals; if computed on the 1,000-example set, the text must say so explicitly. Please state the exact validation denominator and report per-setup validation scores with confidence intervals or at least exact numerators.","section":"§3.1 and §4.2"},{"comment":"The paper's central claim of 'strong domain understanding' relies on the validation concept accuracy of ~0.8, but the official leaderboard concept scores are 0.3120, 0.1140, and 0.2250 for the three setups. The paper attributes the drop to verbosity and formatting issues, yet this conflates formatting errors with a clear train/validation/test distribution shift. As written, the claim that the model 'understands biomedical concepts well' is supported only on the validation distribution and is contradicted on the official benchmark. The authors should either reframe the conclusion to acknowledge the distribution shift or provide an explicit error analysis (e.g., concept scores after answer extraction, human evaluation of a test sample) that separates formatting failures from reasoning failures.","section":"§4.2, abstract, §5"},{"comment":"The two-stage inference pipeline is a central contribution, but the paper provides no quantitative ablation. It reports only examples of verbose outputs and a qualitative statement that the pipeline 'improved output quality'. The 'unofficial test evaluation' that achieved an EM of 0.49 is not described: which test data, which gold labels, how many examples, and what post-processing was applied? Without before/after EM or concept scores for the same model and same evaluation set, the claimed improvement is not verifiable.","section":"§3.3 and §4.2"},{"comment":"The training-data construction is underspecified. The paper lists source datasets with a total of roughly 70,000 QA pairs, states that 10,000 pairs were used, and mentions 'cleaned and filtered' without describing the filtering criteria, per-source counts, duplicate removal, or checks for overlap with the MedHopQA validation/test sets. Given that the paper's premise is that this mixture transfers to MedHopQA, the lack of data-composition details is a reproducibility gap that should be addressed.","section":"§3.1"},{"comment":"The zero-shot baseline results are reported only qualitatively ('near-zero EM', 'approximately 80% accuracy' for medical-specialty questions). No numerical scores, sample sizes, or error counts are provided. This makes the claimed benefit of supervised fine-tuning relative to zero-shot inference not quantitatively established. This is secondary to the main claims but should be fixed with a small table.","section":"§4.2"}],"minor_comments":[{"comment":"The test-set size is inconsistent: Section 1 and Section 3.1 say the test set contains 10,000 questions, while Section 4.2 says the testing phase was evaluated on 1,000 examples only. Please clarify the relationship between the 10,000-question set and the 1,000-example evaluation.","section":"§3.1 / §1"},{"comment":"Figure 1 is referenced but does not appear in the provided text. Please ensure the figure is included and referenced correctly.","section":"§3.3"},{"comment":"There are numerous typos and formatting issues: 'traning', 'envel⌢pe-⌢pen', 'optimiser', 'emphasise', 'generalizing', and an unresolved author token 'L. N. 0001' in reference [13]. A careful proofread is needed.","section":"Throughout"},{"comment":"Reference [4] contains a duplicated phrase 'Overview of overview of the medhopqa track'. Also, several arXiv identifiers or URLs are not consistently formatted.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"This is a workshop-scale system paper. The main problem is that the headline claim ('strong domain understanding', concept accuracy up to 0.8) is not supported on the official test distribution, and the development-set description is internally inconsistent. These issues are fixable by clarifying the validation denominator, reporting per-setup tables, and adding an explicit limitations paragraph. If the authors cannot provide per-setup validation numbers or pipeline ablations, the paper may need to be reframed as a negative result about exact-match evaluation rather than a demonstration of strong understanding."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nHere's the two-line take: this is a routine shared-task system paper that has a genuinely useful negative result hiding in it, but the abstract sells a claim the benchmark numbers don't support.\n\nWhat's actually new: the authors fine-tuned LLaMA 3 8B on a mix of public biomedical QA sets and evaluated on MedHopQA, a brand-new benchmark. The specific numbers—validation EM around 0.5, concept accuracy around 0.8, test EM 0.0–0.2, official leaderboard concept scores 0.312/0.114/0.225—are new. They also tried a two-stage extraction pipeline to rein in verbose outputs and honestly report that it only partially worked. The method is ordinary LoRA SFT; nothing groundbreaking, but that's not a knock.\n\nCredit where due: the paper does not bury the leaderboard scores; they're in Section 4.2. The authors also explicitly acknowledge that their test performance collapsed and that formatting failures caused many mismatches. The qualitative zero-shot baseline comparison is thin but honest.\n\nThe soft spots, in order of severity. First, the dataset description is contradictory: Section 3.1 says the development set has 45 annotated questions, then says a 1,000-example official development set was used as validation. The abstract and conclusions lean on 'up to 0.8 concept-level accuracy,' but if that rests on 45 items, the Wilson 95% CI is roughly 0.66–0.89, and the concept/EM gap could easily be small-sample noise. Second, and more importantly, the official test concept scores are 0.11–0.31, far below 0.8. So the headline 'strong domain understanding' doesn't hold on the actual benchmark, and the 'semantic understanding vs. exact match' conclusion is likely conflated with train/test distribution shift—their external training mixture doesn't transfer to MedHopQA's rare-disease distribution. Third, there are no confidence intervals, no released model or code, and the zero-shot baseline is described only qualitatively.\n\nNone of this is a fatal methodological error. The SFT setup is sound and the failures are reported. But the paper over-claims, and the contradictions need to be resolved before the numbers are citable as reliable evidence.\n\nWho should read this: anyone working on biomedical QA evaluation metrics, or participants in BioCreative IX Track 1. I'd send it to peer review, because the empirical data point is new and the issues are fixable by recalibrating the claims and reporting CIs. It doesn't belong in my own citation list right now.\n\nRecommendation: accept with major revision, or at minimum get the authors to pin down the validation denominator and lead with the official test scores.","headline":"A routine system paper with new MedHopQA numbers, but the central claim about 0.8 concept-level accuracy is undercut by the official test scores it reports.","tokens_in":7383,"tokens_out":5119,"would_cite":false,"duration_ms":54256,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A fine-tuned 8B biomedical QA model reaches 0.8 concept accuracy but 0.0–0.2 exact match, exposing a gap between semantic understanding and strict answer formatting.","keywords":["biomedical question answering","multi-hop QA","supervised fine-tuning","LLaMA 3 8B","low-rank adapters","exact match","concept-level accuracy","answer extraction"],"falsifier":"Re-score the official test submissions after normalizing answer variants (e.g., '2' to 'Chromosome 2', 'Chr.2' to the canonical chromosome label, plus synonym resolution). If normalized EM rises toward concept-level accuracy, the claimed gap is mostly formatting; if normalized EM stays near zero on rare-disease questions, the gap is a content or distribution failure.","tokens_in":6451,"feed_emoji":"🧬","tokens_out":12410,"duration_ms":135789,"temperature":0.7,"pith_summary":"This paper reports a supervised fine-tuning system for a multi-hop biomedical question-answering track centered on diseases, genes, and chemicals. By training a general-purpose 8-billion-parameter LLM on a 10,000-question mixture assembled from public biomedical QA sources, the authors find that their models reach about 0.8 concept-level accuracy on the development set—evidence the model has internalized biomedical knowledge—while exact-match scores stay around 0.5 on validation and drop to 0.0–0.2 on the official test set. The paper's central move is to identify the gap as a formatting problem, not a knowledge problem, and to attack it with a two-stage inference pipeline that extracts concise phrases from verbose first-stage outputs. The pipeline helps in some cases but does not reliably close the gap, and the authors argue that tight output control under strict evaluation metrics remains an open limitation for biomedical LLM question answering.","feed_headline":"Fine-tuned biomedical QA knows concepts (0.8) but misses exact answers (0.0)","feed_subtitle":"It reaches 0.8 on concept accuracy but 0.0 to 0.2 on exact match, showing evaluation rewards formatting, not just knowledge.","key_machinery":"The load-bearing mechanism is a two-stage inference pipeline on top of a supervised fine-tuned model. In the first stage, the model generates a free-form answer to the question; in the second, a follow-up prompt instructs it to extract the exact answer phrase or entity from that response, and if three extraction attempts fail, the system falls back to the first-stage longer output. The model itself is a LLaMA 3 8B adapted with low-rank adapters (LoRA), a parameter-efficient fine-tuning method that tunes small adapter weights instead of all model weights. The pipeline is what attempts to translate the model's semantic competence into the strict short-answer format the evaluation demands.","core_discovery":"The paper's central claim is that a general-purpose 8-billion-parameter LLM, fine-tuned with low-rank adapters on a mixture of external biomedical QA pairs, learns enough biomedical knowledge to score up to 0.8 on concept-level evaluation of multi-hop questions about diseases, genes, and chemicals—yet fails exact-match evaluation because its outputs are verbose, paraphrased, or slightly misformatted. The authors demonstrate this with three fine-tuning setups, reporting validation EM near 0.5 and test EM of 0.2, 0.0, and 0.0 for combined, short-only, and long-only training. Zero-shot general-purpose models score near zero on both metrics, indicating that fine-tuning is what recovers semantic","pith_inferences":["Re-scoring the official predictions with a canonical-name normalizer (gene symbols, chromosome labels, synonym expansion) would quantify how much of the EM gap is pure formatting; the authors' own error examples suggest the share is large.","The development-set concept accuracy (~0.8) and the official leaderboard concept scores (0.11–0.31) diverge sharply; a per-topic breakdown by rare-disease category could reveal whether the model fails on rare entities or on multi-hop composition, and whether the development set is a trustworthy predictor.","The two-stage pipeline is a heuristic form of constrained decoding; comparing it against grammar-guided or constrained decoding would show how much of the remaining format gap is addressable by generation-time control, which the authors leave for future work."],"forward_implications":["Fine-tuned models can reach roughly 0.8 concept-level accuracy on multi-hop biomedical questions, so specialized domain knowledge is achievable from a small curated training mixture.","Exact-match evaluation penalizes surface form as much as content: correct-concept variants like '2', 'Chr.2', and 'Chromosome 2p13' all count as misses, so the metric shapes the measured gap.","Training on combined short and long answers was the only setup with a nonzero official test EM (0.2), suggesting answer-format diversity during training supports generalization.","Prompt refinement and two-stage extraction can lift test EM from near zero to about 0.49 without new training data, so output-control techniques are a promising, low-cost lever.","Zero-shot general-purpose models fail on both concept and exact-match metrics, so instruction-following alone is insufficient for specialized biomedical multi-hop reasoning."],"supporting_citations":[{"why":"Defines the multi-hop biomedical QA task, its question categories, and the concept-level/exact-match metrics the paper's central claim is measured against.","marker":"[4]"},{"why":"Official dataset description for the development and hidden test sets, including the 1,000-question official evaluation and the rare-disease focus.","marker":"[5]"},{"why":"Supplies a large set of external biomedical question–answer pairs used in the 10,000-question fine-tuning mixture.","marker":"[18]"},{"why":"Contributes the clinical-trials QA pairs to the training mixture, one of the external sources the fine-tuning relies on.","marker":"[8]"},{"why":"Supplies long-answer question–answer pairs used in the combined and long-answer fine-tuning conditions.","marker":"[20]"},{"why":"Contributes a manually curated biomedical QA corpus to the training mixture.","marker":"[21]"},{"why":"Supplies a large-scale biomedical QA corpus used as a training source.","marker":"[22]"}],"fun_headline_variants":["Biomedical QA: 0.8 concept, 0.0 exact match","LLM knows concepts but misses exact answers in biomed QA","Fine-tuned LLM: 80% concept accuracy, 0-20% exact match","Biomedical QA: semantic understanding vs strict formatting","Concept accuracy 0.8, exact match 0.0: biomedical QA gap"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The load-bearing premise is that the external training mixture and the small development set represent the same distribution as the hidden test set; if that distribution shift is large, the reported gap between concept accuracy and exact match could be a benchmark artifact rather than a general LLM limitation.","fun_headline_variants_meta":{"raw":{"variants":["Biomedical QA: 0.8 concept, 0.0 exact match","LLM knows concepts but misses exact answers in biomed QA","Fine-tuned LLM: 80% concept accuracy, 0-20% exact match","Biomedical QA: semantic understanding vs strict formatting","Concept accuracy 0.8, exact match 0.0: biomedical QA gap"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000227,"raw_usage":{"total_tokens":1327,"prompt_tokens":784,"completion_tokens":543,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":528,"completion_tokens_details":{"reasoning_tokens":442}},"tokens_in":528,"tokens_out":543,"duration_ms":6181,"temperature":1.0,"reasoning_tokens":442,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T13:11:19.044577+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-score the official test submissions after normalizing answer variants (e.g., '2' to 'Chromosome 2', 'Chr.2' to the canonical chromosome label, plus synonym resolution). If normalized EM rises toward concept-level accuracy, the claimed gap is mostly formatting; if normalized EM stays near zero on rare-disease questions, the gap is a content or distribution failure.","supporting_citations":[{"cited_title":"Islamaj, J","cited_arxiv_id":null,"evidence_quote":"Defines the multi-hop biomedical QA task, its question categories, and the concept-level/exact-match metrics the paper's central claim is measured against."},{"cited_title":"Lu, Medhopqa, 2025","cited_arxiv_id":null,"evidence_quote":"Official dataset description for the development and hidden test sets, including the 1,000-question official evaluation and the rare-disease focus."},{"cited_title":"Ben Abacha, D","cited_arxiv_id":null,"evidence_quote":"Supplies a large set of external biomedical question–answer pairs used in the 10,000-question fine-tuning mixture."},{"cited_title":"Roberts, D","cited_arxiv_id":null,"evidence_quote":"Contributes the clinical-trials QA pairs to the training mixture, one of the external sources the fine-tuning relies on."},{"cited_title":"Krithara, A","cited_arxiv_id":null,"evidence_quote":"Contributes a manually curated biomedical QA corpus to the training mixture."},{"cited_title":"Nentidis, G","cited_arxiv_id":null,"evidence_quote":"Supplies a large-scale biomedical QA corpus used as a training source."}],"review_version":1}