{"id":"1967691a-a37f-412b-81cb-ca73ef9ca687","arxiv_id":"2412.03961","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":2.0,"correctness_risk":"high","formal_verification":"none","parameter_count":10,"one_line_summary":"The authors report that a BiLSTM-CRF feature extractor combined with XGBoost and logistic regression outperforms several baseline models for diabetes risk prediction on a private Beijing health-check dataset.","lead":"This paper applies a BiLSTM-CRF text model plus XGBoost and logistic regression to predict diabetes risk from electronic health records, reporting accuracy around 82 to 89 percent on a private dataset. A reader might look because diabetes prediction from EHR is a common and practical machine learning target, but the paper lacks code, data, and statistical detail.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported 89% accuracy and 0.93 AUC may be an artifact of label/feature leakage: Section 4.2 lists 'diabetes status' among the 48 features while diabetes status is the predicted outcome, and no temporal split or exclusion of diagnostic biomarkers is described.","rationale":"The reader's REJECT is well supported. My pass narrows the decisive issue: the most load-bearing problem is not external generalizability but that the comparison cannot even be interpreted internally because the outcome may be present among the predictors. Section 4.2's feature list explicitly includes 'diabetes status' among the 48 features. If that column is the target, the model has access to the label as an input. The abstract claims 'prognosis,' which requires temporal separation, yet the paper describes only an 80/20 random split and 5-fold cross-validation, not a patient-level temporal split. No code or data are provided, so the reader cannot check this. The manuscript also contains template artifacts ('Font: Open Sans; Font size; 10. Paragraph comes content here.') and an internally contradictory statement in the conclusion ('although this study used a public dataset') versus Section 4.2's private Beijing health-check-center dataset. These inconsistencies reduce confidence but are secondary to the leakage concern. There is no independent support such as machine-checked proofs, re-runnable code, or external replication to offset the missing evaluation details. My proposed experiment—retraining after removing the diagnostic shortcut and using a temporal, patient-level outcome definition—would settle whether the 89%/0.93 result survives. Because this reinforces, rather than redirects, the reader's rejection, the verdict remains REJECT, with UNCHANGED indicating no change to the reader's verdict.","tokens_in":18887,"tokens_out":6093,"duration_ms":57972,"concrete_test":"Ask the authors for the exact cohort definition and data dictionary, then run one controlled experiment on the same data: define the outcome as incident diabetes first recorded at least 12 months after the feature date; remove 'diabetes status' and fasting blood glucose (and any other glucose-related diagnostic biomarkers) from the feature set; use patient-level grouping so no individual appears in both training and test sets; retrain the exact XGBoost-logistic-regression ensemble with the same 5-fold cross-validation; report AUC with 95% confidence intervals. If the AUC remains near 0.93, the leakage concern is resolved; if it drops sharply, the reported superiority is an artifact rather than a prognostic advance.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing weakness is not primarily cohort size or representativeness; it is that the outcome may be both a feature and the label. Section 4.2 defines the cohort as '1,097 individuals with 48 features, including gender, age, ..., fasting blood glucose, ..., and diabetes status.' The paper never defines a separate target variable, a prediction horizon, or a rule for labeling incident versus prevalent diabetes. If 'diabetes status' is one of the 48 features and is also the response, the reported ensemble accuracy (89%) and AUC (0.93) can be produced by reading the label off the input at training time. Even without explicit label inclusion, including fasting blood glucose and other diagnostic lab values among predictors while predicting 'diabetes status' risks reproducing diagnostic criteria rather than forecasting risk. Because the data come from annual checks over 2010–2015 but no temporal split between feature time and outcome time is described (Section 4.1.4 uses a generic 80/20 random split and 5-fold cross-validation), the claim that the model provides 'risk prognosis' is not supported. If the dataset contains per-visit rows rather than one row per patient, random splitting can also put the same patient in both training and test sets. None of this requires assuming author misconduct; the manuscript simply omits the cohort and schema details that would rule out leakage. Table 2 provides no confidence intervals or standard errors, so a favorable random split or leakage cannot be distinguished. The conclusion's statement that 'this study used a public dataset' contradicts Section 4.2's description of a private Beijing health-check cohort, further undermining the verifiability of the evaluation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a two-stage pipeline for diabetes risk prediction from electronic health records: a BiLSTM-CRF model extracts medical entities and temporal features from text, and an ensemble of XGBoost and logistic regression classifies the fused feature set. On a private Beijing health-check dataset reduced from 5,046 to 1,097 individuals, the authors report 82% accuracy for BiLSTM-CRF and 89% accuracy with 0.93 AUC for the ensemble, claiming superiority over LSTM, BiLSTM, CNN-Bi-LSTM, 3D-CNN-SPP, and Med-BERT.","tokens_in":19248,"tokens_out":4740,"duration_ms":43724,"significance":"If the reported results were valid, the pipeline would be a practical engineering contribution to EHR-based diabetes risk stratification. The manuscript does provide explicit hyperparameter settings for the main models (Sections 4.1.2 and 4.1.3), which is helpful for reproducibility. However, as written, the experiments do not establish the central claim of superior predictive performance: the outcome variable is not clearly separated from the features, the evaluation protocol does not respect the temporal structure of the data, and the reported numbers contain internal inconsistencies and no uncertainty quantification. These issues are load-bearing because they directly affect the accuracy and AUC values that the abstract and conclusions emphasize.","major_comments":[{"comment":"The dataset description lists 'diabetes status' as one of the 48 features, and no separate target variable is defined anywhere in Section 4. If 'diabetes status' is also the predicted outcome, the ensemble's 89% accuracy and 0.93 AUC in Table 2 could be obtained simply by reading the label from the input features. The manuscript must explicitly define the outcome, remove it from the feature list, and state the prediction horizon (e.g., whether the task is to predict incident diabetes in a future year from features recorded in prior years). Without this clarification, the headline performance numbers are not interpretable.","section":"Section 4.2"},{"comment":"The evaluation uses a random 80/20 split (for BiLSTM-CRF) and 5-fold cross-validation (for XGBoost and logistic regression) with no temporal split. Since the data are annual health checks from 2010 to 2015, random splitting can place the same patient's visits in both training and test sets, and using diagnostic biomarkers such as fasting blood glucose as predictors of a same-visit 'diabetes status' label reproduces the diagnostic criteria rather than forecasting future risk. A temporal split (e.g., train on 2010-2013, test on 2014-2015) or at least a strict per-patient split with a defined outcome time is required to support the paper's claims of 'risk prognosis' and 'early detection.'","section":"Section 4.1.4"},{"comment":"The text states that 'LSTM achieves the highest recall at 79.82%,' but Table 1 reports LSTM recall of 0.6942 and Med-BERT recall of 0.7982. This internal inconsistency, combined with the absence of confidence intervals or significance tests in Tables 1 and 2, means the stated conclusion that BiLSTM-CRF (or the ensemble) 'outperforms all other models' is not supported by the reported evidence. The table/text should be corrected, and interval estimates or pairwise significance tests should be provided for the accuracy, F1, and AUC comparisons.","section":"Section 4.3 and Table 1"},{"comment":"The manuscript says SMOTE is applied 'to balance class distribution,' but it does not state whether SMOTE is applied before or after the train/test split. If SMOTE runs on the full dataset before splitting, synthetic minority-class examples can appear in both training and test sets, biasing all reported metrics upward. This needs to be clarified, and ideally SMOTE should be applied inside each cross-validation fold or training split only.","section":"Section 4.1.1"}],"minor_comments":[{"comment":"The section opens with placeholder text 'Font: Open Sans ; Font size; 10. Paragraph comes content here.' This unfinished content should be removed before any consideration for publication.","section":"Section 4.2"},{"comment":"Reference [33] is displayed as '!!! INVALID CITATION !!!' and must be replaced with a valid citation.","section":"References"},{"comment":"The LSTM description lists 'three types of gates: forget gate; output gate and input gate' but the equations also include a candidate cell update that is not described as a gate. Please reconcile the wording with the equations.","section":"Section 3.2"},{"comment":"The conclusion says the dataset consists of 'only 1000 clinical electronic health records,' but Section 4.2 reports 1,097 individuals; please make the numbers consistent.","section":"Section 5"}],"recommendation":"reject","confidential_remarks":"This manuscript appears to be an unfinished submission: it contains placeholder text, an invalid citation, and numerous references unrelated to the core topic. More importantly, the experimental protocol as described cannot support the headline accuracy and AUC claims because the outcome may be included among the features and the temporal structure of the EHR data is ignored. Even if these issues were fixed, the absence of code or data and the lack of any uncertainty quantification would make verification difficult. I would not encourage resubmission without a substantially redesigned study."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You asked about 2412.03961. Short version: the paper claims a nice accuracy number, but the evaluation is not credible as written. Section 4.2 lists 48 predictors and includes 'diabetes status' — which is also the outcome they are predicting. Without a separate target definition, a prediction horizon, or a patient-level temporal split, the 0.93 AUC could just be the model reading the label off the input. That is the load-bearing issue; everything else is secondary.\n\nTo be fair, the pipeline itself is standard and clearly explained: BiLSTM-CRF for entity tagging, then XGBoost and logistic regression on the extracted text features plus structured fields. The baselines (LSTM, BiLSTM, CNN-Bi-LSTM, 3D-CNN-SPP, Med-BERT) are reasonable, and the authors do acknowledge the small dataset and the black-box nature of the models in the conclusion.\n\nBut the problems pile up quickly. No confidence intervals, significance tests, or external validation; point estimates only. No code or data, so nothing can be checked. The 80/20 split and 5-fold CV are mentioned, but there is no indication that the split is patient-level or temporal. SMOTE is described without saying whether it was applied before or after splitting, which is another potential leak. The text contains placeholder junk ('Font: Open Sans; ...') and an invalid citation in the references. The conclusion says 'this study used a public dataset,' directly contradicting Section 4.2's description of a private Beijing health-check cohort. The reference list also contains many apparently unrelated papers, which makes the citation pattern look padded.\n\nSo the central claim — 89% accuracy and 0.93 AUC for the ensemble — is unverified and, on the current description, likely an artifact. This is not a matter of a missing error bar; the evaluation design as written cannot distinguish prediction from memorization.\n\nWho is this for? A reader who wants a cautionary example of what happens when EHR prediction papers omit cohort and schema details. As a substantive contribution, it does not hold up. I would not send this to peer review in its current state. The authors need to redesign the experiment, define the outcome and prediction horizon explicitly, apply a proper temporal split, and release code and data. If they do that, the underlying question — whether text-derived features add value over structured labs — is worth an honest test. But this version does not provide one.","headline":"The paper's headline result is unverifiable and likely contaminated by leakage: the feature list includes the outcome variable, and no temporal or patient-level split is described.","tokens_in":19802,"tokens_out":2571,"would_cite":false,"duration_ms":26442,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"EHR text-mining pipeline predicts diabetes risk at 89% accuracy.","keywords":["electronic health records","diabetes risk prediction","BiLSTM-CRF","XGBoost","logistic regression","natural language processing","ensemble learning","clinical decision support"],"falsifier":"Re-running the same pipeline on an independent EHR cohort from a different institution, using the same 48 features and an identical train/test split, would settle the claim: if the ensemble loses its clear margin over logistic regression alone and over Med-BERT on that external data, the reported 89% accuracy does not generalize.","tokens_in":18704,"feed_emoji":"🩺","tokens_out":6955,"duration_ms":53970,"temperature":0.7,"pith_summary":"Electronic health records mix structured lab values with unstructured clinical text, and most predictive models use only one of the two. This paper tries to show that a two-stage pipeline—a BiLSTM-CRF network that extracts medical entities from text, followed by an XGBoost-and-logistic-regression ensemble that classifies—uses both, and that the combination beats the individual models. On a cohort of 1,097 people from a Beijing health check center, the authors report the ensemble reaching 89% accuracy and 0.93 AUC, with BiLSTM-CRF alone at 82% accuracy. If the result holds, it would give clinicians a data-driven way to flag diabetes risk earlier from records they already have.","feed_headline":"EHR text mining model hits 89% accuracy on diabetes risk","feed_subtitle":"BiLSTM-CRF pulls clinical entities from text; an XGBoost-logistic ensemble turns them into risk scores.","key_machinery":"The load-bearing mechanism is a two-stage pipeline. Stage one is a BiLSTM-CRF: a Bidirectional Long Short-Term Memory network (two LSTM passes, forward and backward, to capture surrounding context) topped by a Conditional Random Field layer, which constrains the output to coherent label sequences. It takes clinical text preprocessed with BIO tags (Begin/Inside/Outside entity markers) and word embeddings (dimension 300) and produces medical-entity tags. Stage two fuses those tags with 48 structured fields—demographics, vital signs, and lab results—and feeds them to XGBoost (a gradient-boosted decision tree ensemble) and logistic regression, whose predictions are combined into an ensemble. The CRF layer does the key work: it lets the model choose optimal entity-label sequences rather than independent per-token decisions, which the authors say is why BiLSTM-CRF beats plain BiLSTM.","core_discovery":"The central claim is that a hybrid architecture outperforms both pure deep-learning text models and pure structured-data classifiers on diabetes risk prediction. The paper's evidence is a set of comparisons: BiLSTM-CRF reaches 82% accuracy, 80.90% precision, 74.87% F1, and a Kappa of 0.6219, edging out Med-BERT (81%), 3D-CNN-SPP (80%), CNN-Bi-LSTM (77%), and BiLSTM (76%). When the extracted entities are merged with 48 structured EHR features, the ensemble of XGBoost and logistic regression reaches 89% accuracy, 90% precision, 87% recall, and 0.93 AUC, beating XGBoost alone (86%, AUC 0.91) and logistic regression alone (81%, AUC 0.87). The authors present these numbers as evidence that combining deep contextual text analysis with gradient-boosted and linear classifiers yields an improvement over traditional modeling.","pith_inferences":["The paper does not isolate how much the BiLSTM-CRF tags contribute beyond the 48 structured features; a natural test would run the same XGBoost-logistic ensemble on the structured fields alone and compare the drop.","Because the cohort is a single health check center with 1,097 people after exclusions, the absolute numbers are likely optimistic for broader populations; external validation at another institution would be the decisive check.","The comparison omits established clinical risk scores such as FINDRISC or Framingham-based diabetes scores; adding them would place the 89% accuracy claim in context with tools already in use."],"forward_implications":["A clinical deployment could flag high-risk patients from routine check-up records without additional tests, because the required features are already in the EHR.","The entity tags produced in stage one double as interpretable output, letting clinicians see which symptoms or findings drove a patient's risk score.","The same pipeline could be retrained for other chronic conditions that leave traces in both structured labs and clinical text, such as hypertension or chronic kidney disease.","The reported margins suggest that adding an NLP front end to structured-data classifiers is worth the extra complexity, at least on records similar to this cohort."],"supporting_citations":[{"why":"Supplies the premise of combining structured and unstructured data for predictive modeling, which the two-stage pipeline operationalizes.","marker":"[22]"},{"why":"Provides a BERT-BiLSTM-CRF approach for Chinese EMR named entity recognition that the text-analysis stage builds on.","marker":"[26]"},{"why":"Shows BiLSTM-CRF used for clinical named entity recognition, the direct methodological template for stage one.","marker":"[27]"},{"why":"Source of the XGBoost algorithm used as one of the two risk classifiers.","marker":"[44]"},{"why":"CNN-Bi-LSTM diabetes prediction model used as a comparison baseline in Table 1.","marker":"[45]"},{"why":"3D-CNN-SPP patient risk prediction system used as a comparison baseline in Table 1.","marker":"[46]"},{"why":"Med-BERT, the strongest text baseline at 81% accuracy, which the paper's model must beat.","marker":"[47]"}],"fun_headline_variants":["Hybrid AI model predicts diabetes risk at 89% accuracy","EHR text plus structured data boosts diabetes prediction to 89%","BiLSTM-CRF and XGBoost ensemble edge out traditional diabetes models","From EHR text to risk scores: 89% accuracy with hybrid model","Deep learning plus boosting lifts diabetes risk prognosis to 89%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported superiority rests on the assumption that the 1,097 people who remain after filtering from a single Beijing health check center represent the wider patient population, so the accuracy numbers would transfer to other hospitals and patients.","fun_headline_variants_meta":{"raw":{"variants":["Hybrid AI model predicts diabetes risk at 89% accuracy","EHR text plus structured data boosts diabetes prediction to 89%","BiLSTM-CRF and XGBoost ensemble edge out traditional diabetes models","From EHR text to risk scores: 89% accuracy with hybrid model","Deep learning plus boosting lifts diabetes risk prognosis to 89%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000459,"raw_usage":{"total_tokens":2346,"prompt_tokens":1039,"completion_tokens":1307,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":655,"completion_tokens_details":{"reasoning_tokens":1214}},"tokens_in":655,"tokens_out":1307,"duration_ms":8853,"temperature":1.0,"reasoning_tokens":1214,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T21:53:17.113855+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-running the same pipeline on an independent EHR cohort from a different institution, using the same 48 features and an identical train/test split, would settle the claim: if the ensemble loses its clear margin over logistic regression alone and over Med-BERT on that external data, the reported 89% accuracy does not generalize.","supporting_citations":[{"cited_title":"Predicting patients' satisfaction with doctors in online medical communities: An approach based on XGBoost algorithm,","cited_arxiv_id":null,"evidence_quote":"Source of the XGBoost algorithm used as one of the two risk classifiers."},{"cited_title":"An optimization-based diabetes prediction model using CNN and Bi- directional LSTM in real-time environment,","cited_arxiv_id":null,"evidence_quote":"CNN-Bi-LSTM diabetes prediction model used as a comparison baseline in Table 1."},{"cited_title":"3D-CNN-SPP: A patient risk prediction system from electronic health records via 3D CNN and spatial pyramid pooling,","cited_arxiv_id":null,"evidence_quote":"3D-CNN-SPP patient risk prediction system used as a comparison baseline in Table 1."},{"cited_title":"Med-BERT: pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction,","cited_arxiv_id":null,"evidence_quote":"Med-BERT, the strongest text baseline at 81% accuracy, which the paper's model must beat."}],"review_version":1}