{"id":"7da70bd9-f336-4551-a21f-da13f24f2ca8","arxiv_id":"2411.18922","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A compact 15-feature set using LLM-generated content coverage and TF-IDF class similarities reaches 85.4% accuracy for Alzheimer's detection on ADReSS, beating 40 traditional linguistic features.","lead":"The paper proposes a 15-feature set for Alzheimer's screening from Cookie Theft picture descriptions, combining GPT-4o-generated content keywords and TF-IDF similarity scores. The feature set reaches 85.4% accuracy on ADReSS, outperforming a 40-feature traditional baseline, but three features are built from training labels.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed consistent outperformance rests on a single 48-sample test set with no significance testing; a 10.4-point accuracy gap is about 5 subjects and may be within chance. The label-derived TF-IDF reference features add a target-encoding risk that needs out-of-fold validation.","rationale":"The paper's central claim is that a compact 15-dimensional explainable feature set consistently outperforms 40 traditional linguistic features on ADReSS. For that claim to hold, the observed accuracy differences must reflect real signal rather than chance or leakage. The most insecure premise is the statistical status of Table 3: only 48 test samples, no error bars, no paired test, and a 10.4-point gap that corresponds to roughly 5 subjects. The paper's own ablation shows accuracy rising and then declining as features are added, which is the expected pattern of small-sample overfitting rather than a stable advantage. The TF-IDF reference features are constructed from training labels, so they are supervised features; this is not automatically fatal in a fixed train/test split, but it makes the comparison with label-free linguistic features less clean, and the paper does not state how Bayesian optimization used the data. If cross-validation on the training set used reference vectors computed from the full training set, then target-encoding leakage would contaminate model selection. The reader's weakest assumption is therefore plausible but not the only load-bearing issue; I see the lack of significance testing as the most direct threat to the central claim. Because the authors do not release code, data, or per-subject predictions, the reported 85.4% cannot be independently verified or tested for significance. The reader's REJECT verdict remains appropriate; no verdict change is needed.","tokens_in":6853,"tokens_out":8358,"duration_ms":83533,"concrete_test":"Obtain the per-subject test predictions behind Table 3 and run an exact McNemar test between the RF-new-features and RF-linguistic models, and between the corresponding XGBoost pair. If p>0.05, or if the 95% binomial confidence interval for the 85.4% accuracy (n=48) includes the 75.0% baseline, the central 'consistent outperformance' claim is not supported by the evidence as presented.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2 (Table 3) reports RF accuracy 85.4% with 15 new features versus 75.0% with 40 linguistic features on 48 test samples, and XGBoost 83.3% versus 72.9%. The paper treats these differences as 'consistently outperform,' but no confidence interval, bootstrap, or paired significance test is provided. On n=48, a 10.4-point gap is about 5 correct classifications; even if all discordant pairs favored the new set, an exact McNemar test would give a two-sided p around 0.0625, above 0.05. The ablation in Fig. 3 shows accuracy fluctuating non-monotonically as features are added, consistent with small-sample instability. A related risk is that the TF-IDF similarity features (Eqs. 5-7) and the TF-IDF keyword hit rate are constructed from training labels; if hyperparameters were chosen by cross-validation on those same features, validation subjects are included in the reference vectors, creating target-encoding leakage in model selection. The duplicated phrase in the abstract and typos do not affect the argument, but the missing statistical evaluation and the label-derived reference construction are substantive.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a compact, explainable 15-dimensional feature set for Alzheimer's disease (AD) screening from Cookie Theft picture descriptions. The features include LLM-generated topic keyword hit rates, average BLEU and METEOR scores against LLM-generated golden-standard descriptions, TF-IDF similarity to class-specific reference vectors, a TF-IDF keyword hit rate, and four conventional linguistic features (parse-tree depth, filled pauses, filled-pause ratio, and word error rate). On the ADReSS 2020 challenge data, the authors report 85.4% accuracy with random forest and 83.3% with XGBoost using the new features, compared with 75.0% and 72.9% for a 40-feature linguistic baseline, and conclude that the new features consistently outperform traditional linguistic features while being more interpretable and dimensionally efficient.","tokens_in":7169,"tokens_out":4903,"duration_ms":42361,"significance":"If the reported performance is reliable, a 15-dimensional, explainable feature set that approaches fine-tuned BERT accuracy on ADReSS would be a useful contribution to explainable AD screening. The use of a multimodal LLM for content-coverage features is creative, and the proposed features are more transparent than black-box embeddings. The paper also provides a clear feature-generation pipeline with prompts and examples, which facilitates reproducibility. However, the headline claim of consistent outperformance is currently not statistically supported, and the TF-IDF reference features are constructed from training labels in a way that risks target-encoding leakage. No code or data are provided, so the results cannot be independently checked. These issues must be addressed before the central claim can be accepted.","major_comments":[{"comment":"The HC and AD reference vectors vHC and vAD are computed by averaging TF-IDF vectors over training documents grouped by their labels. Consequently, the features 'TF-IDF similarity HC', 'TF-IDF similarity AD', and 'TF-IDF Keywords Hit Rate' are derived from the training labels. When the classifier is trained on these features, it receives a target-encoded representation, and the Bayesian optimization with cross-validation described in Section 3.1 will produce optimistic validation scores whenever validation-fold subjects contribute to the reference vectors. The paper must either compute the reference vectors in a strictly leave-one-out/out-of-fold fashion for every training and validation prediction, or justify why this is unnecessary. As written, the comparison against label-free linguistic features is not a fair test of the new feature set.","section":"Section 2.2.2, Eqs. (5)-(7)"},{"comment":"The reported accuracy differences (85.4% vs 75.0% for RF; 83.3% vs 72.9% for XGBoost) are based on a single 48-sample test set with no confidence intervals, bootstrap resampling, or paired significance testing. On n=48, the 10.4-point gap corresponds to roughly 5 correct classifications, and an exact McNemar test on the discordant pairs would not reach p<0.05. The paper should provide paired significance tests (e.g., McNemar) or confidence intervals for all metrics, and the abstract's 'consistently outperform' claim should be tempered unless such evidence is provided.","section":"Table 3 and Section 3.2"},{"comment":"The ablation study adds features in order of ANOVA F-values computed on the training set. Since the TF-IDF reference vectors already encode training labels, the ANOVA F-values for those features are inflated by the same target-encoding mechanism. The non-monotonic accuracy curve in Figure 3 is therefore not a reliable guide to feature importance, and the conclusion that 'feature selection based on ANOVA F-values may not be suitable' is confounded by the leakage in the feature construction. An ablation that separates the label-derived features from the LLM-based content features would be more informative.","section":"Section 4.2 and Figure 3"}],"minor_comments":[{"comment":"The phrase 'due to their scalability due to their scalability' is duplicated and should be corrected.","section":"Abstract"},{"comment":"The table title 'Fifteen proposed features description' is inconsistent with the text: Section 2.2 states that 11 new features are proposed, and the remaining four are conventional linguistic features added from prior work. Please clarify the composition of the 15-feature set.","section":"Table 1"},{"comment":"The claim 'we pioneered the breakdown of the Cookie Theft picture' overstates novelty; a more cautious phrasing such as 'we propose' or 'we introduce' would be appropriate.","section":"Introduction"},{"comment":"There are several typos and grammatical errors in this section, including 'calculating the the number', 'coresponds', and 'evalaution' in the footnote. Equation (4) is also difficult to parse in the provided text and should be typeset more clearly.","section":"Section 2.2.2"},{"comment":"Reference [3] is malformed ('V . L, R. SH, R. M, P. M, L. J, C. M, and L. G.'), and several other references (e.g., [9], [19]) lack full publication details. Please ensure all references follow a consistent, complete format.","section":"References"},{"comment":"The WER feature uses Whisper-large-v3, but it is not specified whether the word error rate is computed against the manual transcripts of the same audio or against some other reference. Please clarify the reference transcript used.","section":"Section 2.2.2, WER feature"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a relevant problem and the feature-engineering idea is interesting, but the central claim is currently weakened by the target-encoding issue and the lack of statistical testing. Both issues are fixable within the manuscript's scope: the reference vectors can be computed in an out-of-fold manner, and significance tests can be added. However, if the authors cannot demonstrate that the label-derived features are not responsible for the performance gain (e.g., by ablating those features or using nested cross-validation), the 'consistently outperform' claim may need to be substantially downgraded. The paper would also benefit from a limitations section acknowledging the assumptions about LLM-generated golden standards."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"I read the Li et al. paper on compact explainable features for AD screening. The LLM-vision idea is genuinely new: they crop the Cookie Theft picture into three regions, generate topic keywords with GPT-4o, and generate 15 'golden' descriptions to compute BLEU/METEOR content-coverage scores. That's a sensible, interpretable way to quantify description completeness, and it does not appear in prior work.\n\nThe paper also adds TF-IDF cosine similarity to class-average reference vectors and a TF-IDF keyword hit rate. That's where the problems start. Equations 5–7 construct separate reference vectors from AD and HC training transcripts, then use the same training labels to train the classifier. That is target encoding. Features built this way tend to look highly predictive in-sample and can degrade on new data unless computed out-of-fold. The paper neither acknowledges this nor validates it. The TF-IDF keyword hit rate (top-30 from vHC) has the same issue.\n\nThe empirical claim is also shaky. On 48 test samples, the 85.4% vs 75.0% gap is about 5 subjects. The paper reports no confidence intervals, no bootstrap, no paired test. An exact McNemar test would be marginal (p around 0.06). The ablation curve is non-monotonic, consistent with small-sample noise. So 'consistently outperform' is not established.\n\nI want to give credit where it's due: the features are explainable, the use of LLM vision is creative, and the ADReSS setup is standard. But reproducibility is limited—no code, no data, no generated references. For a feature-engineering paper on a small dataset, that matters.\n\nWho is this for? Someone working on explainable AD screening features, or on using LLMs to ground content coverage. But the central comparison needs to be redone with out-of-fold reference vectors, significance testing, and released artifacts. The core idea is worth engaging with, but as presented the results are not reliable.\n\nI'd send it to peer review—the idea is novel and the flaws are fixable. But I would expect a serious revision.","headline":"A genuinely new LLM-vision feature-generation idea for AD screening, but the headline comparison is undermined by label-derived TF-IDF features and a 48-sample test set with no significance testing.","tokens_in":7636,"tokens_out":3711,"would_cite":false,"duration_ms":31050,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fifteen explainable features outperform the 40-feature linguistic baseline in Alzheimer's screening.","keywords":["Alzheimer's disease detection","spoken language features","Cookie Theft picture description","TF-IDF","large language model features","ADReSS","explainable features","dementia screening"],"falsifier":"Recompute the three TF-IDF-based features (similarity to HC, similarity to AD, and keyword hit rate) under nested cross-validation or with reference vectors built from a held-out development set, so that no test transcript's label contributes to the reference vectors; if the accuracy gap over the 40-feature baseline shrinks or disappears, the central claim depends on target encoding.","tokens_in":6670,"feed_emoji":"🧠","tokens_out":4940,"duration_ms":41726,"temperature":0.7,"pith_summary":"The paper tries to establish that a compact, explainable 15-dimensional feature set can screen Alzheimer's disease from Cookie Theft picture descriptions more accurately than a conventional 40-dimensional linguistic feature baseline. The set combines LLM-generated content-coverage scores (topic keyword hit rates, BLEU, METEOR), TF-IDF similarity to class reference vectors, and four standard linguistic measures. The authors report 85.4% accuracy on the ADReSS test set with a random forest, compared with 75.0% for the baseline, and similar gains with XGBoost. If correct, this suggests that task-specific content-coverage features can substantially outperform generic linguistic features at a fraction of the dimensionality, which matters for scalable, interpretable dementia screening.","feed_headline":"15 explainable features beat 40 for Alzheimer's screening","feed_subtitle":"Compact set built from LLM content coverage and TF-IDF reference vectors outperforms the traditional linguistic baseline.","key_machinery":"The carrying object is the 15-dimensional feature vector itself, with two novel components: (1) content-coverage features generated by prompting a multimodal LLM (GPT-4o) with cropped sub-images of the Cookie Theft picture to produce topic keyword sets and full descriptions, scored by keyword hit rate, BLEU, and METEOR; and (2) TF-IDF reference-vector similarity features, where per-subject transcripts are embedded in TF-IDF space and compared by cosine similarity to an HC reference vector and an AD reference vector, each built by averaging training-document vectors within a class. The TF-IDF keyword hit rate counts how many of the top-30 HC-reference terms appear in a subject's transcript. These features make the content of what a speaker says—not just how they say it—the signal for screening.","core_discovery":"The central claim is that the 15 new features consistently outperform 40 traditional linguistic features across two classifiers (Random Forest and XGBoost) on the ADReSS benchmark. The best result, 85.4% accuracy, is achieved by the new features alone in a random forest. The feature set comprises three LLM-generated topic keyword hit rates, averaged BLEU-1 through BLEU-4 and METEOR scores against 15 LLM-generated golden descriptions, two TF-IDF cosine similarities to HC and AD reference vectors, a TF-IDF keyword hit rate, plus averaged parse tree depth, filled pause count and ratio, and word error rate. The authors argue the features are explainable because each step—picture segmentation, keyword generation, reference construction—is traceable.","pith_inferences":["If the TF-IDF reference vectors are a form of target encoding, then the feature-importance ranking (TF-IDF similarity HC in the top five) will not survive a nested cross-validation evaluation; testing that is the direct next step.","The LLM-generated references are a substitute for human gold-standard descriptions; replacing them with human references in the same pipeline could isolate how much of the gain comes from the LLM versus from the content-coverage idea itself.","The topic keyword sets are LLM-version-dependent; scores may drift across GPT-4o versions, so a reproducibility check with another multimodal LLM or a fixed LLM version would clarify how much of the result is tied to the specific model.","Because the ADReSS set is small (156 samples), the reported margins may not be stable; a paired bootstrap or confidence interval on the accuracy difference would show whether the gap over the baseline is robust."],"forward_implications":["A 15-dimensional feature vector can serve as a lightweight, interpretable alternative to fine-tuned BERT for AD screening, at roughly a tenth of the feature count.","Naively concatenating the new features with traditional linguistic features can hurt accuracy, so feature selection is needed when combining feature families.","Because the features are tied to the Cookie Theft task, they transfer to other picture-description cognitive assessments only if equivalent content-coverage pipelines are built for those stimuli.","The TF-IDF reference vectors give a quantitative measure of how a speaker's language resembles the prototypical healthy or AD profile, which may help clinicians see why a screening decision was made."],"supporting_citations":[{"why":"Supplies the ADReSS Challenge 2020 dataset, the evaluation benchmark on which all accuracy numbers are reported.","marker":"[17]"},{"why":"Provides the 40 traditional linguistic features that serve as the baseline feature set the new features are compared against.","marker":"[14]"},{"why":"Provides the TF-IDF formulation from information retrieval that underlies the HC and AD reference vectors and the keyword hit rate feature.","marker":"[19]"},{"why":"Describes the Pitt Corpus within DementiaBank, the source corpus from which the ADReSS dataset is curated.","marker":"[18]"}],"fun_headline_variants":["15 explainable features outperform 40 for Alzheimer's","LLM-based compact features outperform traditional AD screening","Explainable 15-feature set tops 40 baseline in AD test","Alzheimer's: 15 features outscore 40 in screening","Compact feature set achieves 85.4% accuracy for AD"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that constructing the HC and AD reference vectors from the training labels and then training a classifier on those same labels is a legitimate feature-engineering step rather than a form of target leakage; if the TF-IDF similarity features merely encode the training labels, the reported consistent outperformance would not hold on unseen data.","fun_headline_variants_meta":{"raw":{"variants":["15 explainable features outperform 40 for Alzheimer's","LLM-based compact features outperform traditional AD screening","Explainable 15-feature set tops 40 baseline in AD test","Alzheimer's: 15 features outscore 40 in screening","Compact feature set achieves 85.4% accuracy for AD"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000963,"raw_usage":{"total_tokens":4032,"prompt_tokens":810,"completion_tokens":3222,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":426,"completion_tokens_details":{"reasoning_tokens":3138}},"tokens_in":426,"tokens_out":3222,"duration_ms":22481,"temperature":1.0,"reasoning_tokens":3138,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T10:43:59.475547+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the three TF-IDF-based features (similarity to HC, similarity to AD, and keyword hit rate) under nested cross-validation or with reference vectors built from a held-out development set, so that no test transcript's label contributes to the reference vectors; if the accuracy gap over the 40-feature baseline shrinks or disappears, the central claim depends on target encoding.","supporting_citations":[{"cited_title":"Linguis- tic features and automatic classifiers for identifying mild cogni- tive impairment and dementia,","cited_arxiv_id":null,"evidence_quote":"Provides the 40 traditional linguistic features that serve as the baseline feature set the new features are compared against."},{"cited_title":"Leveraging pretrained representations with task-related key- words for alzheimer’s disease detection,","cited_arxiv_id":null,"evidence_quote":"Provides the TF-IDF formulation from information retrieval that underlies the HC and AD reference vectors and the keyword hit rate feature."}],"review_version":1}