{"id":"42d3f7a5-4d92-4bd1-8a93-967fd18263dc","arxiv_id":"2412.07924","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"LLM-extracted social determinants from psychosocial notes improve prediction of liver transplant recommendation and listing and explain part of racial disparities in listing.","lead":"Researchers used a language model to pull 23 social and behavioral factors out of liver transplant evaluation notes for thousands of patients. These factors helped predict who was recommended and listed for transplant, and partly explained racial gaps in listing.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline predictive and disparity-explanation claims rest on outcome labels extracted from the same note that supplies the SDOH features; until those labels are validated against an independent source, the AUROC gains and Blinder-Oaxaca shares cannot be attributed to SDOH.","rationale":"The reader's weakest assumption is exactly the load-bearing concern. The paper's main contributions are (1) SDOH extraction, (2) prevalence/temporal patterns, (3) disparity decomposition, and (4) predictive improvements. The first is supported by the 101-note validation, though only for features, not for outcome labels. The second is descriptive and robust. The fourth and the explanatory portions of the third depend on the outcome labels. Because Q25–Q28 extract those labels from the same notes used for features, the predictive improvements are not identifiable as SDOH effects. The paper itself flags related leakage for BOW in Supplement A.3.3 and acknowledges in the Discussion that LLM labels assume a ground truth, but it does not apply that caution to the main AUROC claims. The concrete test—independent outcome validation—would settle the issue. If it passes, the paper is stronger; if not, the predictive and disparity-explanation claims should be downgraded to descriptive associations. This is not an objection to the descriptive prevalence or temporal findings, which do not rely on outcome label independence in the same way. The CONDITIONAL verdict is appropriate: the authors should add external outcome validation (and ideally a redaction-based leakage check) before the predictive claims are accepted as evidence of SDOH influence.","tokens_in":17650,"tokens_out":4604,"duration_ms":47877,"concrete_test":"Validate all outcome labels against sources external to the psychosocial note: UNOS/SRTR or the transplant program's structured registration data for listing status, and a blinded manual review of the transplant committee's final recommendation (or another independent adjudication) for the psychosocial outcome. Re-run the XGBoost prediction models and the Blinder-Oaxaca decomposition with these external labels, keeping the same LLM-derived SDOH features, clinical/demographic covariates, and train/test split. If the AUROC gain from adding SDOH features falls below a pre-specified threshold (e.g., <0.05 for psychosocial recommendation or non-significant for listing), the same-note leakage concern is confirmed and the central predictive and disparity-explanation claims would not survive in their current form.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim—that LLM-extracted SDOH snapshots significantly improve prediction of psychosocial recommendation and listing, and explain racial gaps in listing—requires that the outcome labels be independent of the feature source. That condition is not met. In Supplementary Table 2, questions Q25–Q28 ask the LLM to extract the overall psychosocial risk, the recommendation, whether an addendum records the listing decision, and the listing status from the same note that is queried for the 23 SDOH features (Q2–Q24). Figure 7 defines the listing outcome as 'based on evidence from notes' rather than from UNOS or another independent registry. The LLM prompt supplies the full note for all 30 questions, so the model has access to outcome-bearing text when it labels features such as motivation, coping skills, or caregiver concerns; those labels can simply mirror the note's stated conclusion. The authors' own BOW comparison strengthens this worry: BOW reaches AUROC 0.91 versus 0.87 for recommendation, and Supplement A.3.3 attributes the BOW advantage to terms that 'directly mention recommendation and risk.' The same leakage mechanism applies to the LLM features, because the same note text is the input. There is no validation of Q25–Q28 labels against chart review, UNOS, or structured EHR fields. Consequently, the reported improvements (0.494→0.876 for recommendation; 0.616→0.717 for listing) and the Blinder-Oaxaca 'explained' shares may measure within-note coherence rather than the causal or predictive content of SDOH.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper develops an LLM-based pipeline to extract 23 social determinants of health (SDOH) features from psychosocial evaluation notes of liver transplant candidates at a single academic center (n=3,704 with complete data). Using these features, the authors report substantial improvements in predicting two clinical decisions: the psychosocial recommendation (AUROC 0.494→0.876) and eventual listing (AUROC 0.616→0.717), as well as Blinder-Oaxaca decompositions claiming that SDOH explain 42.6% of the Asian listing gap and 94.6% jointly with clinical features. The paper also presents demographic prevalence analyses, temporal trends, co-occurrence patterns, and SHAP-based interpretability. The central claims rest on LLM-extracted outcome labels obtained from the same psychosocial notes that supply the SDOH features.","tokens_in":17967,"tokens_out":3244,"duration_ms":31340,"significance":"If the central claims hold, the paper offers a scalable method for converting unstructured psychosocial notes into standardized SDOH representations, with clear interpretability advantages over bag-of-words baselines and potential applicability to other clinical decision settings. The authors are appropriately careful in noting limitations such as documentation bias and the assumption that social worker labels are ground truth. However, the validity of the predictive and disparity-explanation claims depends critically on the independence of the outcome labels from the feature source. That independence is not established, and the paper's own supplement acknowledges label leakage in text-based baselines without resolving the same concern for LLM-derived features. The paper's strengths—clinician-informed category design, a 101-note validation set, and transparent SHAP analyses—are real, but they do not outweigh the unresolved circularity at the heart of the main results.","major_comments":[{"comment":"The outcome labels for psychosocial recommendation (Q25–Q26) and listing (Q27–Q28) are extracted by the same LLM from the same psychosocial evaluation note that supplies the 23 SDOH features (Q2–Q24). The paper does not validate these extracted outcomes against an independent source, such as UNOS listing data, structured EHR fields, or chart review. Since the LLM prompt contains the full note, the reported AUROC improvements (0.494→0.876 for recommendation; 0.616→0.717 for listing) may measure within-note correlation between feature labels and outcome labels rather than the predictive power of SDOH. This is load-bearing because the abstract and Section 2.6 base their central claim on these numbers.","section":"§2.6, Supplementary Table 2"},{"comment":"The listing outcome is defined 'based on evidence from notes' (Figure 7) rather than from an independent registry. The Blinder-Oaxaca decomposition presented in Section 2.5 uses this note-derived outcome. The explained shares (e.g., 42.6% for SDOH alone and 94.6% combined for Asian patients) therefore cannot be attributed to SDOH until the listing label is verified against an independent source. If the note's stated listing decision is the same text that informs the SDOH features, the decomposition's 'explained' component is partly a measure of internal consistency of the note rather than a causal or explanatory quantity.","section":"§2.5, Figure 7, Table 2"},{"comment":"The claim that LLM-derived features 'avoid the label leakage' attributed to BOW is not supported. The authors correctly note that BOW features such as 'recommendation' and 'listing' leak outcome information, but the LLM is given the entire note and is asked (Q25–Q28) to produce outcome labels. The same outcome-bearing text is present when the LLM answers Q2–Q24. The fact that BOW outperforms LLM features (AUROC 0.91 vs. 0.87 for recommendation; 0.71 vs. 0.66 for listing in Table 3) does not rule out leakage in LLM features; it merely indicates that BOW captures the outcome terms more directly. The authors should test this by masking or removing outcome-related sections from the notes before extraction and re-running the prediction, or by validating the extracted outcomes against independent chart review.","section":"Supplementary §A.3.3, Table 3"}],"minor_comments":[{"comment":"The sentence 'achieving 0.70-0.98% accuracy' contains an erroneous percentage symbol; it should read '0.70–0.98' as a proportion.","section":"§2.2"},{"comment":"The caption states '28 questions' while the text says 23 SDOH dimensions; Supplementary Table 2 lists 30 questions. The relationship among the number of questions, the 23 SDOH factors, and the additional questions for outcomes should be clarified.","section":"§2.2, Figure 1c"},{"comment":"The validation of LLM extraction against 101 expert annotations would be strengthened by reporting inter-annotator agreement (e.g., Cohen's kappa or Fleiss' kappa) rather than only average accuracy, especially for categories with accuracy as low as 0.70.","section":"§5.2"},{"comment":"Because the outcome base rates are highly imbalanced (93% for recommendation and 81% for listing), the paper should report precision-recall AUC or Brier scores alongside AUROC, as AUROC can be optimistic in this setting.","section":"§2.6, Table 3"}],"recommendation":"major_revision","confidential_remarks":"The circularity issue is the central obstacle. If the authors can validate the extracted psychosocial recommendation and listing outcomes against independent data (e.g., UNOS records, structured transplant committee decisions, or manual chart review), the paper would be much stronger. The revision should also clarify the relationship between the LLM's outcome extraction and feature extraction, and ideally re-run the predictive and decomposition analyses with independently verified outcomes to show that the reported effects persist. The paper is likely to generate interest in the cs.CY community, but the current version's headline numbers cannot be interpreted as evidence about SDOH until this issue is resolved."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe headline result is that LLM-extracted SDOH factors from psychosocial notes substantially improve prediction of transplant recommendation and listing. But you have to read the paper with the co-extraction problem in mind: the outcome labels come from the same notes as the features. The AUROC jumps (0.494 to 0.876 for recommendation, 0.616 to 0.717 for listing) are real in-sample, but they may measure within-note coherence rather than SDOH predictive power. The authors themselves note that BOW gets even higher AUROC and attribute that to label leakage; the same mechanism applies to the LLM features, since the LLM answers both Q2–Q24 (SDOH factors) and Q25–Q28 (psychosocial risk, recommendation, listing) from the same note. Listing is explicitly defined as 'based on evidence from notes' (Figure 7), not from UNOS or structured EHR fields. So the strongest predictive and disparity-decomposition claims rest on a circular design.\n\nThat said, the descriptive parts are genuinely useful and mostly robust. The 23-factor transplant-specific taxonomy, developed with social workers, is a real asset. The extraction accuracy (0.859 mean over 101 notes) is modest but acceptable for prevalence estimation. The temporal trends, co-occurrence patterns, and demographic prevalence results are descriptive and don't depend on outcome-label independence. The Blinder-Oaxaca decomposition for Asian patients showing SDOH explain 42.6% of the listing gap on their own is interesting, though it inherits the circularity for the listing outcome.\n\nThe paper is also honest about some limitations: BOW leakage is acknowledged, the social-worker labels are treated as ground truth, and the streetlight effect is discussed. What's missing is an acknowledgment that the LLM pipeline itself has the same leakage risk, and no validation of the outcome labels against an external source. That should be required before the predictive claims can be taken at face value.\n\nIf I were refereeing, I'd recommend major revision. Ask for (1) validation of the extracted outcomes against UNOS or structured listing data, (2) an analysis that removes outcome-bearing text from the note before feature extraction (e.g., masking the recommendation section), and (3) a sensitivity analysis around the LLM prompt and BOW thresholds. The descriptive sections are worth keeping; the predictive sections should be re-framed as measuring association within notes unless the circularity is resolved.\n\nOverall: worth engaging, should go to peer review, but the headline claims need substantial rework.\n\nBest,\n[Name]","headline":"Useful descriptive SDOH analysis held back by outcome labels co-extracted from the same notes as the features; the predictive claims need external validation before they can be taken at face value.","tokens_in":18574,"tokens_out":2312,"would_cite":true,"duration_ms":22099,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that LLM-extracted social factors from clinical notes substantially improve prediction of liver-transplant recommendations and listing, and explain most of the Asian versus non-Asian listing gap.","keywords":["social determinants of health","liver transplantation","large language models","clinical notes","health disparities","Blinder-Oaxaca decomposition","predictive modeling","electronic health records"],"falsifier":"Take the listing outcome from an independent transplantation registry rather than from the psychosocial note, retrain the same models, and check whether adding the 23 SDOH features still raises AUROC from 0.616 to 0.717; if the gain disappears, the result is an artifact of reading the label and features from the same text.","tokens_in":17430,"feed_emoji":"🩺","tokens_out":7735,"duration_ms":68512,"temperature":0.7,"pith_summary":"The paper tries to show that a large language model can convert a patient's psychosocial evaluation note into a standardized set of 23 social and behavioral factors, and that these factors improve prediction of whether the patient is recommended for and listed for liver transplant. It reports that adding these factors raises the AUROC for psychosocial recommendation from 0.494 to 0.876 and for listing from 0.616 to 0.717. The paper also claims the factors explain 94.6 percent of the Asian versus non-Asian listing gap, with social determinants alone accounting for 42.6 percent and liver-health measures for 36.8 percent. A sympathetic reader would care because transplant eligibility rests heavily on unstructured psychosocial assessments, and quantifying them could show which modifiable circumstances drive disparities.","feed_headline":"AI-read social notes explain 94.6% of an Asian listing gap","feed_subtitle":"Adding 23 extracted social factors lifts psychosocial-recommendation AUROC from 0.494 to 0.876.","key_machinery":"The 'SDOH snapshot' is the central object: a standardized vector of 23 binary social and behavioral factors produced by having a privacy-preserving LLM answer 28 expert-designed questions about each psychosocial evaluation note (23 SDOH factors plus outcome-related questions). It carries the argument by turning unstructured clinical prose into comparable, patient-level features. Those features are then fed to XGBoost (a gradient-boosted tree model) to test predictive value, interpreted with SHAP values, and used in Blinder-Oaxaca decompositions to separate explained from unexplained portions of racial listing gaps.","core_discovery":"On the paper's own terms, the central discovery is that LLM-derived 'SDOH snapshots'—23 standardized factors such as alcohol use, housing stability, caregiver availability, mental health treatment, disease insight, and translator need—carry substantial predictive and explanatory power for liver transplant decisions. Compared with clinical features alone (MELD score, HCC status, age, BMI), adding the snapshots raises the AUROC for psychosocial recommendation from 0.494 to 0.876 and for eventual listing from 0.616 to 0.717; in patients already recommended, SDOH-only models reached AUROC 0.641, beating clinical-only models at 0.589. In a Blinder-Oaxaca decomposition, the combined feature set explains 94.6% of the Asian listing gap, with SDOH alone explaining 42.6% versus 36.8% for liver-health measures, while the gap for patients with unknown or undisclosed race remains 89.2% unexplained. Extraction accuracy against expert annotations averaged 0.859 across categories, ranging from 0.70 for disease insight to 0.98 for housing instability.","pith_inferences":["The paper does not test listing outcomes against an independent registry; a reader should infer that part of the AUROC gain may reflect the LLM reading the decision out of the same note that supplies the features.","A neighboring application would be to run the same note-survey procedure on kidney or heart transplant evaluations, but the extracted labels would first need validation against administrative records rather than the note itself.","The paper's own missing-data analyses imply that excluding patients without complete clinical data may bias disparity estimates; modeling note availability as a predictor would test this directly.","The co-occurrence matrix suggests compound vulnerabilities (e.g., mental-health challenges plus housing instability) could be treated as phenotypes and tested for predicting dropout between recommendation and listing."],"forward_implications":["If the claim is right, transplant programs could rank modifiable social factors such as caregiver support, housing stability, current alcohol use, and disease insight by measured association with outcomes and target resources accordingly.","Since SDOH-only models beat clinical-only models for listing prediction, psychosocial circumstances carry decision-relevant information beyond MELD score, HCC status, age, and BMI.","The decomposition implies most of the Asian listing advantage in this cohort is attributable to measured characteristics, while the low listing rate for patients with unknown or undisclosed race is not explained by the measured features.","Because bag-of-words models achieve higher raw AUROC but appear to leak outcome language, the interpretable LLM features are the safer basis for explaining decisions rather than reproducing them."],"supporting_citations":[{"why":"Supplies the prior demonstration that LLMs can identify social determinants of health from electronic health records, which the extraction pipeline builds on.","marker":"[9]"},{"why":"Defines the liver-transplant-relevant SDOH categories and health disparities that shaped the 23 factors and the expert question set.","marker":"[17, 18]"},{"why":"Provides the XGBoost gradient-boosted tree model used for all prediction tasks and feature comparisons.","marker":"[45]"},{"why":"Provides SHAP values used to interpret which SDOH and clinical features drive predictions.","marker":"[37]"},{"why":"Provides the Blinder-Oaxaca decomposition method used to quantify explained versus unexplained racial listing gaps.","marker":"[48, 49, 50]"},{"why":"Supplies the cTAKES clinical NLP baseline that the LLM features are benchmarked against.","marker":"[38]"}],"fun_headline_variants":["AI-read social factors explain 94.6% of Asian liver transplant gap","Social snapshot from AI notes lifts transplant decision AUROC to 0.876","23 AI-extracted social factors beat clinical data in transplant decisions","AI social snapshot explains most of Asian listing gap in liver transplant"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Both the outcome labels (psychosocial recommendation and listing) and the 23 SDOH features are extracted by the LLM from the same psychosocial evaluation note; if the note does not independently record the decision, the reported AUROC gains could reflect within-note correlation rather than the predictive power of SDOH.","fun_headline_variants_meta":{"raw":{"variants":["AI-read social factors explain 94.6% of Asian liver transplant gap","Social snapshot from AI notes lifts transplant decision AUROC to 0.876","23 AI-extracted social factors beat clinical data in transplant decisions","AI social snapshot explains most of Asian listing gap in liver transplant"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000446,"raw_usage":{"total_tokens":2275,"prompt_tokens":989,"completion_tokens":1286,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":605,"completion_tokens_details":{"reasoning_tokens":1209}},"tokens_in":605,"tokens_out":1286,"duration_ms":10285,"temperature":1.0,"reasoning_tokens":1209,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T18:24:56.025359+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the listing outcome from an independent transplantation registry rather than from the psychosocial note, retrain the same models, and check whether adding the 23 SDOH features still raises AUROC from 0.616 to 0.717; if the gain disappears, the result is an artifact of reading the label and features from the same text.","supporting_citations":[{"cited_title":"Large language models to identify social determinants of health in electronic health records","cited_arxiv_id":null,"evidence_quote":"Supplies the prior demonstration that LLMs can identify social determinants of health from electronic health records, which the extraction pipeline builds on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the XGBoost gradient-boosted tree model used for all prediction tasks and feature comparisons."},{"cited_title":"Lundberg and Su-In Lee","cited_arxiv_id":null,"evidence_quote":"Provides SHAP values used to interpret which SDOH and clinical features drive predictions."},{"cited_title":"Mayo clinical text analysis and knowledge extraction system (ctakes): architecture, component evaluation and applications","cited_arxiv_id":null,"evidence_quote":"Supplies the cTAKES clinical NLP baseline that the LLM features are benchmarked against."}],"review_version":1}