REVIEW 4 major objections 3 minor 6 references
Leveraging Geolocation in Clinical Records to Improve Alzheimer's Disease Diagnosis Using DMV Framework
T0 review · 4 major / 3 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read The paper claims that adding geolocation features to Llama3-70B and GPT-4o embeddings cuts Alzheimer's risk-score prediction error by 28.57% and 33.47%.
desk verdict The paper's central claim—geolocation improves AD risk prediction—is an artifact of predicting CDC aggregate percentages from the question text, not a real clinical result. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The DMV framework is a data-processing, model-training, and validation pipeline: missing values are imputed, categorical fields are one-hot encoded, numeric fields are normalized, the text column is embedded by Llama3-70B or GPT-4o, and the embeddings are concatenated with features including latitude and longitude before a Random Forest Regressor predicts the continuous risk score. The geolocation columns are the feature subset whose removal produces the reported error increases, so they carry the paper's central ablation claim.
What would settle it
Permute the latitude and longitude pairs across rows while keeping the question text intact; if the near-zero mean squared error persists, the model is recovering the stored aggregate value from row-level identifiers rather than from language or location.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is that the spatial context of a health-survey question nearly determines the numeric answer once the question text is embedded by a large language model. The task is framed as regressing the 'data value' column from the 'question' text plus latitude, longitude, and other categorical features. With both Llama3-70B and GPT-4o embeddings, the regressor reaches $R^2 = 0.9999$; removing geolocation raises mean squared error by 70.97% and 50.70% respectively and mean absolute error by 28.57% and 33.47% respectively.
Load-bearing premise
The paper assumes that each row of the public-health survey—a location, a question about older adults, and that location's aggregate percentage answer—can stand in for a patient-level clinical note with an Alzheimer's risk score.
Editorial extensions
If this is right
- If geolocation truly adds this much signal, geographically stratified risk models should outperform text-only models for this kind of survey-derived health outcome.
- If the near-perfect fit is genuine, then in this dataset the question text plus location essentially determines the risk score, leaving little room for further clinical variables.
- The same embedding-plus-geography recipe could be applied to other spatially patterned health outcomes, such as asthma or diabetes risk from community health surveys.
- Clinical deployment would require moving from aggregate survey statistics to individual patient notes, a step the current setup does not test.
Reading between the lines
- Because the target variable is an aggregate survey value that varies by question and location, the near-zero error most plausibly reflects the model memorizing the dataset's structured mapping rather than learning a patient-level diagnostic signal; the paper does not test this distinction.
- A decisive extension would be to obtain patient-level notes with confirmed Alzheimer's diagnoses and addresses, then test whether location still adds predictive power after controlling for age, education, and comorbidities.
- The reported geolocation gains could also be an artifact of the dataset's group structure: each location has many rows, so latitude and longitude may act as row identifiers; a grouped split that holds out all rows for a location together would settle this.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript proposes a 'DMV' framework that concatenates Llama3-70B or GPT-4o text embeddings with geolocation and other tabular features, then trains a Random Forest regressor to predict a continuous Alzheimer's disease risk score. The dataset is described as CDC-derived records with 284K rows, and the authors report near-perfect test performance (R-squared = 0.9999 for both embedding models) as well as geolocation-based error reductions of 28.57% for Llama3-70B and 33.47% for GPT-4o on MAE. The central claim is that geolocation information significantly improves early AD risk prediction from clinical notes.
Significance. If the evidence supported it, combining LLM embeddings with geolocation to improve patient-level Alzheimer's risk prediction would be a practically valuable contribution. The paper is clearly organized, and its experimental protocol (10-fold cross-validation, baseline comparison, and ablation study) is easy to follow. However, no code or data are provided, so reproducibility rests entirely on the authors' description. More importantly, as detailed below, the dataset and target variable do not correspond to patient-level clinical notes or to individual AD risk, so the strong numerical results are not evidence about Alzheimer's diagnosis. The claimed contribution therefore does not currently hold.
major comments (4)
- [§3, Table 1] The dataset described is not a set of clinical notes. Table 1 shows the input 'question' column containing CDC PLACES measure labels such as 'Percentage of older adults who are experiencing frequent mental distress,' and the text states that these 'statistical values' will be used as the risk scores to predict, with higher scores meaning patients 'in this area' are more likely to have Alzheimer's disease. These are area-level aggregate percentages, not individual patient AD risk scores. The abstract and the Data Description claim that the dataset includes patient records and clinical notes, but the examples shown and the features listed in Tables 2 and 3 provide no evidence that any clinical-note text is used. Because the regression target is the numeric answer corresponding to the question text fed as input, the near-perfect R-squared reflects a lookup of the known aggregate statistic rather than a diagnostic model.
- [§5, Table 5] Table 5 reports R-squared = 0.9999 and MSE values of 0.93e-9 and 1.24e-9 for the embedding models. For a patient-level clinical risk score these values would be implausibly high; they are, however, exactly what one expects when the model predicts the published CDC data value from question text and location identifiers. The baseline Random Forest already reaches R-squared = 0.9781 from categorical features such as topic, question, and stratification, which is likewise inconsistent with patient-level diagnosis and consistent with retrieving an area-level percentage from category labels. No confidence intervals or statistical significance tests are reported, so the abstract's claim that geolocation 'significantly decreases' prediction error is unsupported.
- [§5, Tables 6 and 7] The geolocation ablation compares MSE values of order 1e-9 and MAE values of order 1e-7. The reported improvements of 70.97%, 50.70%, 28.57%, and 33.47% are relative differences at a scale where rounding of the underlying metrics could dominate; no confidence intervals are given. These differences are not shown to be statistically or clinically meaningful, and they do not support the conclusion that geolocation data enhance early AD risk assessment.
- [§4, DMV Framework] Section 4 states that text from 'patient records' is tokenized and embedded, but the features listed in Tables 2 and 3 include columns such as locationabbr, topic, question, and data value. The embedding input is therefore the question label, not unstructured clinical narrative. Thus the framework does not demonstrate the claimed analysis of patient records, and the conclusions in Section 6 about 'unstructured medical text' and 'rich features extracted from patient records' are not supported by the experiments actually described.
minor comments (3)
- [§5, Table 5] The table heading 'Matric Comparison' should read 'Metric Comparison,' and the symbol 'R!' is used inconsistently instead of 'R-squared' or an actual superscript-2 notation.
- [Throughout] The model name appears both as 'GPT4-o' and 'GPT-4o'; the paper should adopt one consistent spelling.
- [Limitations] The Limitations section mentions static datasets and computational resources but does not acknowledge that the target variable is an aggregate CDC statistic rather than a patient-level risk score; this omission obscures the main limitation of the study.
Circularity Check
The DMV framework 'predicts' the CDC 'data value' from the question text that defines it: R2=0.9999 is table lookup, and geolocation trivially identifies the area-level statistic, so the AD-risk claim reduces to its input by construction.
-
self definitional
[Abstract; Section 3 Data Description; Table 1]
"Framing the task as a regression problem, we model the relationship between linguistic features in clinical notes (inputs) and a target variable (data value) that answers specific questions related to AD risk... Besides, the dataset contains questions under 39 topics and corresponding values for each question. Some sample questions and values are listed in Table 1. We will use these statistical values as the risk scores to predict."
The input is the question/measure text and the target is the numeric 'data value' that answers exactly that question. The DMV framework embeds the question text with Llama3/GPT-4o and regresses the associated CDC percentage. This is by construction a lookup: the embedding of 'Percentage of older adults who are experiencing frequent mental distress' is trained to emit 9.0, the very value attached to that row. R2=0.9999 and MSE ~1e-9 are therefore inevitable when the label text is the feature; they say nothing about clinical notes or patient-level Alzheimer's risk. The abstract's 'clinical notes (inputs)' is contradicted by Table 1, whose inputs are survey question strings.
-
renaming known result
[Section 3, Data Description (after Table 1)]
"We will use these statistical values as the risk scores to predict. The scores represent the risk or possibility of developing Alzheimer’s disease for the patients in this area. The higher the score, the more likely the patients in this area are to have Alzheimer’s disease."
The CDC rows are public-health aggregate percentages (obesity, frequent mental distress, activity limitations, self-reported health) stratified by geographic area. The paper renames them 'AD risk scores' without calibration, biomarker linkage, or patient-level derivation. Once the target is redefined as these area-level 'statistical values,' every later 'prediction' is just reproducing the CDC table. The claimed geolocation contribution is likewise forced: latitude/longitude identifies the area whose aggregate percentage is the target, so including geolocation trivially improves the lookup. This is renaming a known public-health dataset as a diagnostic risk score, not deriving an AD prediction from first principles.
full rationale
The paper's own equations and tables exhibit the reduction. The target variable is called 'data value' and is described as the value that 'answers specific questions'; the inputs are the same question texts plus categorical area identifiers and lat/long. Training a regressor to map a question string to its associated CDC statistic is equivalent to learning the answer key of the table, which explains the near-perfect R2 and the MSE values near 1e-9. The Random Forest baseline already reaches R2=0.9781 without any text embeddings, consistent with recovering area-level aggregate percentages from the categorical question/stratification fields rather than diagnosing individual patients. The geolocation ablation is computed on absolute MSE differences of order 1e-9 (0.93e-9 vs 1.59e-9; 1.24e-9 vs 1.87e-9), so the headline 70.97%, 50.70%, 28.57%, and 33.47% improvements are relative changes at numerical-precision scale. No self-citation chain is load-bearing here; the circularity is in the definition of the target relative to the input. The central claim therefore reduces by construction to table lookup, not to an independent AD risk prediction.
Assumptions & free parameters
free parameters (1)
- Random Forest regressor hyperparameters =
n_estimators=100, max_depth=None, min_samples_split=2, min_samples_leaf=1, max_features='auto', bootstrap=True…
assumptions (4)
- ad hoc to paper CDC PLACES 'data value' percentages are valid continuous AD risk scores for patients.
- domain assumption The 'question' text column represents the clinical notes from which AD risk should be predicted.
- domain assumption Ten-fold cross-validation on rows with repeated question text gives unbiased estimates of patient-level prediction performance.
- domain assumption Latitude and longitude are predictive of AD risk beyond the information contained in the question text.
Cite this review
Pith. "Pith review of Leveraging Geolocation in Clinical Records to Improve Alzheimer's Disease Diagnosis Using DMV Framework." pith.science (2026). https://pith.science/paper/RXJHVQRS
@misc{pith2026250204288,
author = {Pith},
title = {Pith review of: Leveraging Geolocation in Clinical Records to Improve Alzheimer's Disease Diagnosis Using DMV Framework},
year = {2026},
howpublished = {\url{https://pith.science/paper/RXJHVQRS}},
note = {Machine review of arXiv:2502.04288}
}
read the original abstract
Alzheimer's Disease (AD) early detection is critical for enabling timely intervention and improving patient outcomes. This paper presents a DMV framework using Llama3-70B and GPT-4o as embedding models to analyze clinical notes and predict a continuous risk score associated with early AD onset. Framing the task as a regression problem, we model the relationship between linguistic features in clinical notes (inputs) and a target variable (data value) that answers specific questions related to AD risk within certain topic categories. By leveraging a multi-faceted feature set that includes geolocation data, we capture additional environmental context potentially linked to AD. Our results demonstrate that the integration of the geolocation information significantly decreases the error of predicting early AD risk scores over prior models by 28.57% (Llama3-70B) and 33.47% (GPT4-o). Our findings suggest that this combined approach can enhance the predictive accuracy of AD risk assessment, supporting early diagnosis and intervention in clinical settings. Additionally, the framework's ability to incorporate geolocation data provides a more comprehensive risk assessment model that could help healthcare providers better understand and address environmental factors contributing to AD development.
Figures
Figures from the paper (7 more)
Reference graph
Works this paper leans on
-
[3]
Optimizing and Evaluating Pre-Trained Large Lan- guage Models for Alzheimer’s Disease Detection. In 2024 32nd Euromicro International Conference on Parallel, Dis- tributed and Network-Based Processing (PDP) , 277–284. IEEE. Kaliappan, J.; et al
work page 2024
-
[4]
Cate- gorical data: Need, encoding, selection of encoding method and its emergence in machine learning models—a practical review study on heart disease prediction dataset using pear- son correlation. In Proceedings of International Conference on Data Science and Applications: ICDSA 2022 , volume 1, 369–382. Singapore: Springer Nature Singapore. Liu, D.; et al
work page 2022
-
[5]
Evaluation of General Large Language Models in Contextually Assessing Semantic Concepts Ex- tracted from Adult Critical Care Electronic Health Record Notes. arXiv preprint arXiv:2401.13588. Malakouti, S. M.; Menhaj, M. B.; and Suratgar, A. A
-
[6]
MGH Radiology Llama: A Llama 3 70B Model for Radiology
MGH Radiology Llama: A Llama 3 70B Model for Radiol- ogy. arXiv preprint arXiv:2408.11848. Singh, A.; Kumar, R.; and Tiwari, A. K
-
[2023]
arXiv preprint arXiv:2303.08774
GPT-4 technical report. arXiv preprint arXiv:2303.08774. Al Nazi, Z.; and Peng, W
-
[2024]
arXiv preprint arXiv:2407.21783
The Llama 3 Herd of Models. arXiv preprint arXiv:2407.21783. Esmaeilzadeh, P
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.