{"id":"58f6390b-9927-454a-b928-d070f80ed255","arxiv_id":"2605.30599","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":8.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"AMNESIA is a benchmark suite of 70,560 medical QA pairs that evaluates unlearning methods and shows that patient-level unlearning erodes disease-shared knowledge.","lead":"This paper introduces AMNESIA, the first large-scale open-source benchmark for machine unlearning in medical LLMs, built from 70,560 QA pairs across real patient notes in 11 disease categories. A smart generalist might read it to see how current unlearning techniques fail to separate individual patient data from shared clinical knowledge.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"No significant objection identified","rationale":"Reader correctly flags the QA/disease-category construction as the key assumption. Because the full manuscript was not supplied in the initial review and the current query provides no contradictory evidence or hidden flaw in the reported results, the UNVERDICTED stance remains appropriate; the benchmark itself is the contribution and its utility hinges on that assumption being reasonable.","tokens_in":1634,"tokens_out":258,"duration_ms":16875,"concrete_test":"Re-run the patient-level unlearning experiments on a 10% random subsample of the AMNESIA test set while holding out all questions whose disease category matches the unlearned patient; measure whether the reported erosion on same-disease questions disappears.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim—that patient-level unlearning erodes performance on other patients sharing the same disease—follows directly from the benchmark design once the QA pairs and disease groupings are accepted. The abstract and described construction (70k QA pairs from 8.8k notes across 11 categories, with both factual and reasoning questions) provide a plausible testbed; no internal inconsistency or missing control is visible from the supplied material that would invalidate the reported pattern.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper introduces AMNESIA, the first large-scale open-source benchmark for medical unlearning, with 70,560 QA pairs derived from 8,820 patient notes across 11 disease categories. It includes both factual recall and clinical reasoning questions, evaluates four standard unlearning methods at the individual-patient and disease levels, proposes a new metric for medical terminology leakage, and reports that patient-level unlearning degrades performance on other patients sharing the same disease.","tokens_in":1699,"tokens_out":490,"duration_ms":26249,"significance":"If the erosion result is robust, the benchmark is significant because it supplies a clinically grounded testbed that existing synthetic or small-scale unlearning suites lack. The scale, the split between factual and reasoning questions, the disease-informed grouping, and the open release constitute concrete strengths that can drive development of methods able to separate patient-specific facts from shared clinical knowledge. The terminology-leakage metric is a useful addition for evaluation in the medical domain.","major_comments":[{"comment":"Benchmark Construction section: the central claim that unlearning one patient erodes performance on others with the same condition rests on the assumption that the QA pairs and 11 disease categories cleanly separate patient-specific facts from shared clinical knowledge. The manuscript supplies no explicit construction protocol, examples of how factual versus reasoning questions were authored, or validation that the groupings achieve this separation; without these details the erosion pattern cannot be interpreted as evidence for the claimed limitation of current methods.","section":"Benchmark Construction"},{"comment":"Evaluation section: the reported erosion finding is presented without statistical significance tests, confidence intervals, or ablation on the choice of disease groupings; because the claim is quantitative and load-bearing for the call for new methods, the absence of these controls weakens the evidential basis.","section":"Evaluation"}],"minor_comments":[{"comment":"The abstract and title both use 'disease-informed analysis' but the manuscript should add a short paragraph clarifying how the 11 categories were chosen and whether they were validated by clinicians.","section":"Abstract"},{"comment":"Figure captions should explicitly state which unlearning method and which metric (including the new leakage metric) are plotted in each panel.","section":null}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive feedback and for recognizing the potential significance of AMNESIA. We address each major comment below and describe the revisions that will be incorporated.","responses":[{"response":"We agree that greater transparency on construction is needed to support interpretation of the erosion results. The current manuscript describes the high-level process (extraction from 8,820 notes into 70,560 QA pairs across 11 ICD-10-aligned categories, with factual items drawn directly from notes and reasoning items derived from clinical guidelines), but does not provide the full protocol, sample pairs, or expert validation steps. In the revision we will add an explicit construction protocol subsection, representative examples of both factual and reasoning questions per disease, and a description of the medical-expert review used to confirm separation of patient-specific versus shared knowledge. These additions will allow readers to assess whether the observed erosion indeed indicates limitations in separating patient facts from disease-level knowledge.","revision_made":"yes","referee_comment":"[Benchmark Construction] Benchmark Construction section: the central claim that unlearning one patient erodes performance on others with the same condition rests on the assumption that the QA pairs and 11 disease categories cleanly separate patient-specific facts from shared clinical knowledge. The manuscript supplies no explicit construction protocol, examples of how factual versus reasoning questions were authored, or validation that the groupings achieve this separation; without these details the erosion pattern cannot be interpreted as evidence for the claimed limitation of current methods."},{"response":"We concur that quantitative claims require statistical controls. The revision will include paired t-tests with p-values and 95% confidence intervals on the performance drops after patient-level unlearning, plus an ablation that varies the disease groupings (e.g., coarser vs. finer partitions) to test robustness of the erosion pattern. These analyses will be reported in the updated Evaluation section and will strengthen the evidential basis for recommending new methods that better isolate patient-specific information.","revision_made":"yes","referee_comment":"[Evaluation] Evaluation section: the reported erosion finding is presented without statistical significance tests, confidence intervals, or ablation on the choice of disease groupings; because the claim is quantitative and load-bearing for the call for new methods, the absence of these controls weakens the evidential basis."}],"tokens_in":1332,"tokens_out":487,"duration_ms":20483,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main takeaway is that this paper ships a 70k-pair benchmark drawn from real patient notes across 11 disease categories, plus a terminology-leakage metric, and reports that unlearning one patient degrades results on others sharing the condition.\n\nThe scale and domain focus are the clear advances. Earlier unlearning work stayed on synthetic or general text; here the authors built both factual recall and clinical-reasoning questions, ran four standard methods at patient and disease granularity, and released the data. That setup lets people measure exactly the cross-patient leakage the abstract flags.\n\nThe soft spot sits in the core assumption. The erosion result only carries weight if the QA pairs truly isolate patient-specific facts from the shared clinical knowledge that any model would learn across the disease category. The abstract states the distinction but gives no construction details, inter-annotator checks, or controls that would show the questions are not just rephrasing general medical facts. Without those, the finding risks being partly an artifact of how the test set was written.\n\nThe new leakage metric is a reasonable addition, yet its calibration and false-positive behavior are not described either. These gaps are fixable with more appendix material rather than fatal.\n\nThe paper is aimed at groups building or auditing unlearning pipelines for clinical LLMs. Anyone who needs a concrete testbed for privacy updates in healthcare models will find the dataset and the reported pattern useful even before the analysis is tightened.\n\nIt deserves a serious referee. The benchmark itself is large enough and open enough to matter, and the empirical observation points to a practical limitation that future methods must address.","headline":"AMNESIA supplies the first sizable open medical unlearning benchmark and documents that patient-level removal hurts same-disease cases, but the patient-versus-shared distinction in the QA pairs needs explicit validation.","tokens_in":2151,"tokens_out":410,"would_cite":true,"duration_ms":23080,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Unlearning one patient's medical data erodes a model's knowledge of other patients sharing the same disease.","keywords":["medical unlearning","machine unlearning","benchmark","patient notes","disease categories","clinical inference","terminology leakage","LLM"],"falsifier":"A result in which unlearning one patient's records leaves model accuracy on other patients with the same disease completely unchanged would falsify the central claim.","tokens_in":2553,"feed_emoji":"🏥","tokens_out":707,"duration_ms":20443,"temperature":0.7,"pith_summary":"The paper presents AMNESIA as a benchmark of 70,560 question-answer pairs drawn from 8,820 patient notes across 11 disease categories to study machine unlearning in medical language models. It tests four standard unlearning methods on both individual patients and entire disease groups, and introduces a metric that tracks whether medical terminology leaks through after unlearning. The central finding is that removing data tied to one patient impairs the model's ability to answer factual and reasoning questions about other patients with the identical condition. This pattern indicates that current unlearning techniques cannot cleanly separate personal patient details from the shared clinical knowledge that defines a disease category. The work therefore calls for unlearning approaches designed specifically for domains where individual records and categorical medical facts overlap.","feed_headline":"Unlearning one patient erodes knowledge of others with same disease","feed_subtitle":"New benchmark of 70k medical QA pairs shows current methods cannot separate personal facts from shared clinical knowledge.","key_machinery":"The AMNESIA benchmark suite, which organizes large-scale QA pairs by disease category to expose interference between patient-specific facts and shared clinical knowledge during unlearning.","core_discovery":"AMNESIA shows that unlearning individual patients erodes knowledge of others with the same condition. The benchmark supplies 70,560 factual and reasoning QA pairs from 8,820 patient notes in 11 disease categories. When four common unlearning methods are applied at the patient level, performance on same-disease cases declines; disease-level unlearning produces different leakage patterns. A new terminology-leakage metric quantifies how medical terms remain accessible after unlearning. These results establish that patient-specific facts and shared clinical knowledge are entangled in trained models and that existing methods do not respect this entanglement.","pith_inferences":["Benchmarks that separate individual and categorical knowledge may be needed in other regulated domains such as legal or financial records.","Training regimes that tag disease-level versus patient-level information at the data stage could reduce the interference observed here.","Regulatory requirements to remove patient data may force periodic re-evaluation of model performance on related conditions."],"forward_implications":["Unlearning methods must be tested at both random-patient and disease-group scales to detect cross-patient interference.","Medical unlearning requires explicit mechanisms to isolate individual records from condition-level clinical patterns.","A terminology-leakage metric provides a practical way to measure whether shared medical vocabulary survives unlearning.","Factual recall and clinical-reasoning questions both reveal the same erosion pattern, indicating the problem is not limited to rote memorization."],"fun_headline_variants":["Patient unlearning erodes same-disease knowledge","Unlearning patients harms same-condition recall","Medical unlearning leaks across same-disease patients","Benchmark finds patient unlearning affects disease knowledge"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The constructed QA pairs and disease categories sufficiently capture the distinction between patient-specific facts and shared clinical knowledge that unlearning methods must respect.","fun_headline_variants_meta":{"raw":{"variants":["Patient unlearning erodes same-disease knowledge","Unlearning patients harms same-condition recall","Medical unlearning leaks across same-disease patients","Benchmark finds patient unlearning affects disease knowledge"]},"model":"grok-4.3","cost_usd":0.006317,"raw_usage":{"total_tokens":2962,"prompt_tokens":655,"num_sources_used":0,"completion_tokens":54,"cost_in_usd_ticks":63174500,"prompt_tokens_details":{"text_tokens":655,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":2253,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":655,"tokens_out":54,"duration_ms":22448,"temperature":1.0,"reasoning_tokens":2253,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-29T08:13:57.614120+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A result in which unlearning one patient's records leaves model accuracy on other patients with the same disease completely unchanged would falsify the central claim.","supporting_citations":[],"review_version":1}