REVIEW 4 major objections 5 minor 45 references
Spurious Correlations and Beyond: Understanding and Mitigating Shortcut Learning in SDOH Extraction with Large Language Models
T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Large language models classify a patient's drug-use status using the mere presence of alcohol or smoking in a clinical note as a shortcut, producing false positives as high as 66% even when the text says nothing about drugs.
desk verdict Useful descriptive study of spurious correlations in SDOH extraction, but the causal trigger-removal claim is confounded by co-occurring drug-related text and the paper lacks statistical rigor. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The argument rides on a two-step extraction pipeline: (1) trigger identification, where the model locates spans for a target substance (e.g., Drug), and (2) argument resolution, a multiple-choice QA prompt that assigns a temporal status (current/past/none). The evaluation metric is the false positive rate (FPR) for classifying a note as current/past drug use when the ground truth is none/unknown. The load-bearing comparison is FPR between substance-positive and substance-negative notes, and the causal probe is a controlled ablation that removes alcohol/smoking trigger spans from the note before re-classification. Chain-of-thought prompting—instructing the model to reason step-by-step before answering—is the mitigation that consistently produces the largest reductions.
What would settle it
A matched counterfactual test: take notes that mention alcohol but explicitly deny drug use, remove the alcohol span, and also remove a matched neutral span (e.g., a date or location) from a control set; if the alcohol removal does not reduce false positives more than the sham removal, the causal role of the trigger is not established. Alternatively, an experiment that controls for note length and lexical overlap using synthetic notes that mention alcohol but never mention drugs would settle whether any alcohol mention—rather than correlated phrasings—drives the effect.
Extended reading notes
Core claim
The central claim is that LLMs treat incidental mentions of alcohol or smoking as evidence of illicit drug use: in the Llama-70B zero-shot setting, alcohol-positive notes yield a false positive rate of 66.21% for drug-status classification versus 28.83% for alcohol-negative notes (smoking: 61.11% vs 29.76%). Removing the alcohol or smoking trigger span from the note lowers the alcohol-positive false positive rate to 55.17%, which the authors take as causal confirmation that the superficial cue itself drives part of the error. The authors further claim that patient gender is used as an additional shortcut, with male patients systematically receiving higher false positive rates (e.g., 71.15% vs 53.66% for alcohol-positive Llama-70B zero-shot), and that shortcuts compound: notes that are both alcohol- and smoking-positive show the highest bias. The paper's prescribed mitigations—chain-of-thought reasoning, warning-based instructions, and additional in-context examples—cut false positives by roughly a third to a half but leave a substantial baseline, which the authors read as evidence that the shortcut is deep-rooted rather than a prompt artifact.
Load-bearing premise
The causal claim rests on the assumption that deleting the alcohol/smoking trigger from a note changes the model's prediction only through the intended cue, and that substance-positive and substance-negative notes are otherwise comparable; since the removal is only partial (66.21% to 55.17% for the flagship case), much of the false-positive rate remains unexplained by this contrast.
Editorial extensions
If this is right
- If LLMs rely on alcohol/smoking mentions as shortcuts, then any social-history extraction system deployed without mitigation will systematically over-report illicit drug use in notes that happen to mention alcohol or tobacco.
- Because the shortcut persists after fine-tuning on the target corpus, in-domain adaptation alone is insufficient; deployers must combine data-level and prompt-level debiasing.
- Chain-of-thought prompting, warnings, and additional examples all reduce false positives in the zero-shot and in-context learning regimes, establishing a low-cost mitigation baseline for clinical NLP.
- The gender disparities in false positive rates imply that shortcut use is not uniform across patient demographics, so fairness evaluations must be part of clinical LLM deployment.
Reading between the lines
- The partial success of trigger removal (alcohol-positive FPR stays at 55.17% after removal) suggests that alcohol/smoking mentions are not the only cue; co-occurring phrasings such as 'substance use', 'recreational use', or even the structured 'Social History' header may also serve as shortcuts—an extension the paper does not test.
- Because trigger removal is an intervention on the input text, the same ablation could be adapted into a data-augmentation or fine-tuning curriculum that explicitly teaches models to ignore such cues, rather than only prompting them.
- The gender disparity may reflect documentation bias in the source corpus itself; if so, auditing clinical text corpora for demographic imbalances in substance-use documentation could be a higher-leverage intervention than any prompt change.
- The paper's FPR-centric evaluation could be extended with a cost-sensitivity analysis weighting false positives against false negatives, since in clinical settings the two errors have very different consequences.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper investigates shortcut learning in LLM-based extraction of drug status from clinical social history notes, using the MIMIC portion of the SHAC corpus. It reports elevated false-positive rates for drug-status classification when alcohol or smoking mentions are present, attempts to establish a causal role for these mentions through trigger-removal experiments, evaluates prompt-based mitigation strategies (chain-of-thought, warning-based instructions, increased examples), and reports gender disparities in false-positive rates. The central empirical claim is that superficial substance mentions causally bias drug-status predictions across several open-source LLMs.
Significance. The paper addresses an important and understudied problem: spurious correlations in social-determinants-of-health extraction, with direct clinical safety implications. Its strengths include the use of an external human-annotated benchmark (SHAC), the breadth of models evaluated (zero-shot, ICL, fine-tuned, and domain-specialized), and the systematic comparison of several prompt-level mitigation strategies. If the causal claim were properly supported, the findings would be valuable for clinical NLP deployment and fairness research. However, the current evidence for causality is incomplete because the removal experiment is confounded by co-occurring drug-related text, and the lack of uncertainty quantification and subgroup sample sizes weakens several headline contrasts.
major comments (4)
- [§4.3, Table 3] The trigger-removal experiment does not control for co-occurring drug-related text. For example, Appendix B shows a note containing both "Patient occasionally uses alcohol" and "Denies any illicit drug use"; removing the alcohol span leaves the drug trigger and its negation intact. The residual FPR of 55.17% for alcohol-positive Llama-70B zero-shot (Table 3) could therefore be driven by misclassification of negated drug mentions rather than by the alcohol cue. To support the causal claim in §4.3 and the statement in §9 that causality was established by removal experiments, the authors should stratify notes by presence or absence of drug-related spans, remove drug-related text as a control condition, or demonstrate that alcohol-positive and alcohol-negative notes have similar distributions of drug-related content.
- [§4.1, Tables 1 and 2] All false-positive rates are reported as point estimates without confidence intervals or significance tests. The headline contrasts (66.21% vs. 28.83% for Llama-70B zero-shot alcohol-positive vs. alcohol-negative; male 71.15% vs. female 53.66% in Table 2) are not accompanied by note counts or uncertainty bounds, so it is unclear whether the gaps exceed sampling noise. The authors should report the number of notes in each cell and compute exact binomial or bootstrap confidence intervals, and use paired tests such as McNemar's test for within-note comparisons.
- [§4.4, Table 2] The gender analysis does not state how MIMIC demographic fields were linked to the SHAC notes, how notes with missing or unknown gender were handled, or the subgroup sizes underlying each reported false-positive rate. Without denominators, the claim of systematic gender disparities that "persist across models" is not verifiable, as some cells could be based on very few notes. Please provide counts and exclusion criteria for each gender-by-substance cell, and ideally interval estimates.
- [§5, Table 1] The mitigation evaluation reports only false-positive rate, so a strategy that simply predicts "none" more often would appear beneficial without any measure of recall or precision. For clinical utility, the trade-off matters: chain-of-thought and warning prompts might reduce false positives at the cost of missing true drug use. Report recall, precision, and F1 (or at least full confusion matrices) for each mitigation condition, alongside the FPR values.
minor comments (5)
- [§3.2] There is a typo in the pipeline description: "models applies" should read "models apply."
- [Throughout] Model naming is inconsistent: "Llama-3.1-70B," "Llama-70B," and "Llama 3.1 70b" are used interchangeably. Please standardize to a single convention.
- [Table 1] The column labeled "Vanilla" for Llama-8B is not defined in the text; please clarify what "vanilla" means for the fine-tuned model.
- [References] The reference "Mateo and Williams" is incomplete; provide a full citation with year, title, venue, and page range.
- [Appendix B] The character offsets in the BRAT example (e.g., "T3 StatusTime 8 19") appear inconsistent with the token spans shown; please verify the example's correctness.
Circularity Check
No significant circularity: the paper's claims are evaluated against an external human-annotated benchmark and no prediction is fitted from or defined in terms of the target outcome.
full rationale
The derivation chain is empirically self-contained. The paper evaluates LLMs on the MIMIC portion of the SHAC corpus, where trigger spans and status labels come from human annotations external to the models and to the paper's own outputs. The false positive rate is computed by comparing model predictions to ground-truth labels, so the headline finding (alcohol-positive notes show higher drug-status FPR than alcohol-negative notes) is an observational comparison on held-out benchmark data, not a quantity reconstructed from fitted parameters. The trigger-removal experiments are ablations of input text rather than fitted corrections, and the mitigation strategies are prompt interventions with no parameters fitted to the test set. The gender analysis similarly uses demographic fields linked from MIMIC, which are external to the model predictions. The Limitations section's claim that causality was established by removal experiments is a causal-inference overstatement given potential confounds such as co-occurring drug-related text, but that is a validity concern, not a circularity: the reported numbers are not forced by construction, by self-definition, or by a self-citation chain. No load-bearing step reduces to its own inputs, so the appropriate finding is no significant circularity.
Assumptions & free parameters
assumptions (3)
- domain assumption SHAC annotations are the gold standard for drug status and trigger spans.
- domain assumption MIMIC-III demographic fields correctly link to all SHAC notes.
- ad hoc to paper Removing trigger mentions is a valid counterfactual that isolates the causal role of those mentions.
Cite this review
Pith. "Pith review of Spurious Correlations and Beyond: Understanding and Mitigating Shortcut Learning in SDOH Extraction with Large Language Models." pith.science (2026). https://pith.science/paper/IIZ5GNYZ
@misc{pith2026250600134,
author = {Pith},
title = {Pith review of: Spurious Correlations and Beyond: Understanding and Mitigating Shortcut Learning in SDOH Extraction with Large Language Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/IIZ5GNYZ}},
note = {Machine review of arXiv:2506.00134}
}
read the original abstract
Social determinants of health (SDOH) extraction from clinical text is critical for downstream healthcare analytics. Although large language models (LLMs) have shown promise, they may rely on superficial cues leading to spurious predictions. Using the MIMIC portion of the SHAC (Social History Annotation Corpus) dataset and focusing on drug status extraction as a case study, we demonstrate that mentions of alcohol or smoking can falsely induce models to predict current/past drug use where none is present, while also uncovering concerning gender disparities in model performance. We further evaluate mitigation strategies - such as prompt engineering and chain-of-thought reasoning - to reduce these false positives, providing insights into enhancing LLM reliability in health domains.
Reference graph
Works this paper leans on
-
[1]
Meta AI. 2024. Llama 3.1 model card. https://github.com/meta-llama/llama-models/blob/main/models/llama3_1/MODEL_CARD.md. Accessed: 2024-12-13
work page 2024
-
[2]
BS Armour, T Woollery, A Malarcher, TF Pechacek, and C Husten. 2005. Annual smoking-attributable mortality, years of potential life lost, and productivity losses—united states, 1997-2001. JAMA: Journal of the American Medical Association, 294(7)
work page 2005
-
[3]
Alexander Brown, Nenad Tomasev, Jan Freyberg, Yuan Liu, Alan Karthikesalingam, and Jessica Schrouff. 2023. Detecting shortcut learning for fair medical ai using shortcut testing. Nature communications, 14(1):4314
work page 2023
-
[4]
Clément Christophe, Praveen K Kanithi, Tathagata Raha, Shadab Khan, and Marco AF Pimentel. 2024. https://arxiv.org/abs/2408.06142 Med42-v2: A suite of clinical llms . Preprint, arXiv:2408.06142
arXiv 2024
-
[5]
Rachel A Dahl, J Priyanka Vakkalanka, Karisa K Harland, and Joshua Radke. 2022. Investigating healthcare provider bias toward patients who use drugs using a survey-based implicit association test: Pilot study. Journal of addiction medicine, 16(5):557--562
work page 2022
-
[6]
Hilary Daniel, Sue S Bornstein, Gregory C Kane, Health, and Public Policy Committee of the American College of Physicians*. 2018. Addressing social determinants to improve patient care and promote health equity: an american college of physicians position paper. Annals of internal medicine, 168(8):577--578
work page 2018
-
[7]
Dina Demner-Fushman, Wendy W Chapman, and Clement J McDonald. 2009. What can natural language processing do for clinical decision support? Journal of biomedical informatics, 42(5):760--772
work page 2009
-
[8]
Robert Geirhos, J \"o rn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. 2020. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11):665--673
work page 2020
Show all 45 references
-
[9]
Sifei Han, Robert F Zhang, Lingyun Shi, Russell Richie, Haixia Liu, Andrew Tseng, Wei Quan, Neal Ryan, David Brent, and Fuchiang R Tsui. 2022. Classifying social determinants of health from unstructured electronic health records using deep learning-based natural language proce...
2022
-
[10]
Elham Hatef, Masoud Rouhizadeh, Iddrisu Tia, Elyse Lasser, Felicia Hill-Briggs, Jill Marsteller, Hadi Kharrazi, et al. 2019. Assessing the availability of data on social and behavioral determinants in structured and unstructured electronic health records: a retrospective analy...
2019
-
[11]
David U Himmelstein and Steffie Woolhandler. 2018. Determined action needed on social determinants. Annals of internal medicine, 168(8):596--597
2018
-
[12]
Yan Hu, Qingyu Chen, Jingcheng Du, Xueqing Peng, Vipina Kuttichi Keloth, Xu Zuo, Yujia Zhou, Zehan Li, Xiaoqian Jiang, Zhiyong Lu, et al. 2024. Improving large language models for clinical named entity recognition via prompt engineering. Journal of the American Medical Informa...
2024
-
[13]
Zalaya K Ivy, Sharon Hwee, Brittany C Kimball, Michael D Evans, Nicholas Marka, Catherine Bendel, and Alexander A Boucher. 2024. Disparities in documentation: evidence of race-based biases in the electronic medical record. Journal of Racial and Ethnic Health Disparities, pages 1--7
2024
-
[14]
Sarah Jabbour, David Fouhey, Ella Kazerooni, Michael W Sjoding, and Jenna Wiens. 2020. Deep learning applied to chest x-rays: exploiting and preventing shortcuts. In Machine Learning for Healthcare Conference, pages 750--782. PMLR
2020
-
[15]
Peter B Jensen, Lars J Jensen, and S ren Brunak. 2012. Mining electronic health records: towards better research applications and clinical care. Nature Reviews Genetics, 13(6):395--405
2012
-
[16]
Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. arXiv preprint arXiv:1707.07328
2017 arXiv
-
[17]
Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. 2016. Mimic-iii, a freely accessible critical care database. Scientific data, 3(1):1--9
2016
-
[18]
Min Kyung Kim, Joy Noel Baumgartner, Jennifer Headley, Julius Kirya, James Kaggwa, and Joseph R Egger. 2021. Medical record bias in documentation of obstetric and neonatal clinical quality of care indicators in uganda. Journal of Clinical Epidemiology, 136:10--19
2021
-
[19]
Zhengliang Liu, Yue Huang, Xiaowei Yu, Lu Zhang, Zihao Wu, Chao Cao, Haixing Dai, Lin Zhao, Yiwei Li, Peng Shu, et al. 2023. Deid-gpt: Zero-shot medical text de-identification by gpt-4. arXiv preprint arXiv:2303.11032
2023
-
[20]
Kevin Lybarger, Nicholas J Dobbins, Ritche Long, Angad Singh, Patrick Wedgeworth, \"O zlem Uzuner, and Meliha Yetisgen. 2023. Leveraging natural language processing to augment structured social determinants of health data in the electronic health record. Journal of the America...
2023
-
[21]
Kevin Lybarger, Mari Ostendorf, and Meliha Yetisgen. 2021. Annotating social determinants of health using active learning, and characterizing determinants using neural event extraction. Journal of Biomedical Informatics, 113:103631
2021
-
[22]
Mingyu Derek Ma, Alexander K Taylor, Wei Wang, and Nanyun Peng. 2022. Dice: data-efficient clinical event extraction with generative models. arXiv preprint arXiv:2208.07989
2022 arXiv
-
[23]
Yubo Ma, Yixin Cao, YongChing Hong, and Aixin Sun. 2023. Large language model is not a good few-shot information extractor, but a good reranker for hard samples! arXiv preprint arXiv:2303.08559
2023 arXiv
-
[24]
David M Markowitz. 2022. Gender and ethnicity bias in medicine: A text analysis of 1.8 million critical care records. PNAS nexus, 1(4):pgac157
2022
-
[25]
Addressing bias and reducing discrimination
CM Mateo and DR Williams. Addressing bias and reducing discrimination. The professional responsibility of health care providers, 2020:95
2020
-
[26]
RT McCoy, E Pavlick, and T Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. arxiv preprint arxiv: 190201007
2019
-
[27]
SA Meyers, VA Earnshaw, Brittany D’Ambrosio, Natasia Courchesne, Dan Werb, and LR Smith. 2021. The intersection of gender and drug use-related stigma: A mixed methods systematic review and synthesis of the literature. Drug and alcohol dependence, 223:108706
2021
-
[28]
Braja G Patra, Mohit M Sharma, Veer Vekaria, Prakash Adekkanattu, Olga V Patterson, Benjamin Glicksberg, Lauren A Lepow, Euijung Ryu, Joanna M Biernacka, Al’ona Furmanchuk, et al. 2021. Extracting social determinants of health from electronic health records using natural langu...
2021
-
[29]
Giridhar Kaushik Ramachandran, Yujuan Fu, Bin Han, Kevin Lybarger, Nicholas J Dobbins, Özlem Uzuner, and Meliha Yetisgen. 2023. https://arxiv.org/abs/2306.07170 Prompt-based extraction of social determinants of health using few-shot learning . Preprint, arXiv:2306.07170
2023 arXiv
-
[30]
Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond accuracy: Behavioral testing of nlp models with checklist. arXiv preprint arXiv:2005.04118
2020 arXiv
-
[31]
Brendan Saloner, Wenshu Li, Michael Flores, Ana M Progovac, and Benjamin L \^e Cook. 2023. A widening divide: Cigarette smoking trends among people with substance use disorder and criminal legal involvement: Study examines cigarette smoking trends among people with substance u...
2023
-
[32]
Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. 2023. Large language models encode clinical knowledge. Nature, 620(7972):172--180
2023
-
[33]
Rachel Stemerman, Jaime Arguello, Jane Brice, Ashok Krishnamurthy, Mary Houston, and Rebecca Kitzmiller. 2021. Identification of social determinants of health using multi-label classification of electronic health record clinical notes. JAMIA open, 4(3):ooaa069
2021
-
[34]
Pontus Stenetorp, Sampo Pyysalo, Goran Topi \'c , Tomoko Ohta, Sophia Ananiadou, and Jun’ichi Tsujii. 2012. Brat: a web-based tool for nlp-assisted text annotation. In Proceedings of the Demonstrations at the 13th Conference of the European Chapter of the Association for Compu...
2012
-
[35]
Ruixiang Tang, Dehan Kong, Longtao Huang, and Hui Xue. 2023. Large language models can be lazy learners: Analyze shortcuts in in-context learning. arXiv preprint arXiv:2305.17256
2023 arXiv
-
[36]
Lifu Tu, Garima Lalwani, Spandana Gella, and He He. 2020. An empirical study on robustness to spurious correlations using pre-trained language models. Transactions of the Association for Computational Linguistics, 8:621--633
2020
-
[37]
\"O zlem Uzuner, Ira Goldstein, Yuan Luo, and Isaac Kohane. 2008. Identifying patient smoking status from medical discharge records. Journal of the American Medical Informatics Association, 15(1):14--24
2008
-
[38]
Leonieke C Van Boekel, Evelien PM Brouwers, Jaap Van Weeghel, and Henk FL Garretsen. 2013. Stigma among health professionals towards patients with substance use disorders and its consequences for healthcare delivery: systematic review. Drug and alcohol dependence, 131(1-2):23--35
2013
-
[39]
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824--24837
2022
-
[40]
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze...
2024 arXiv
-
[41]
Zehao Yu, Xi Yang, Chong Dang, Songzi Wu, Prakash Adekkanattu, Jyotishman Pathak, Thomas J George, William R Hogan, Yi Guo, Jiang Bian, et al. 2022. A study of social and behavioral determinants of health in lung cancer patients using transformers-based natural language proces...
2022
-
[42]
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2017. Men also like shopping: Reducing gender bias amplification using corpus-level constraints. arXiv preprint arXiv:1707.09457
2017 arXiv
-
[43]
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018. Gender bias in coreference resolution: Evaluation and debiasing methods. arXiv preprint arXiv:1804.06876
2018 arXiv
-
[44]
online" 'onlinestring :=
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
-
[45]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.