{"id":"c00d716b-dc16-48e8-b58c-47fce0b6d4e8","arxiv_id":"2412.06259","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"Prompt-based fine-tuning with pause encoding reaches a maximum 95.8% accuracy for Alzheimer's detection on ADReSS transcripts, while the mean over random seeds is 87.9%.","lead":"This paper applies prompt-based fine-tuning of language models, together with pause information from forced alignment, to detect Alzheimer's disease from speech transcripts, reporting up to 95.8% accuracy on the ADReSS test set. The headline result is a benchmark improvement, but it comes from selecting the best of many random-seeded runs, and the average accuracy is lower than a previously reported state of the art.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The transcript-only SOTA claim rests on the maximum over 15 seeds and many system variants; the mean (87.9%) is below the prior 91.7%, so 95.8% is likely a seed-selection artifact.","rationale":"The reader's weakest assumption identifies the most load-bearing issue: the evaluation treats the maximum over many random seeds and system variants as the representative performance. The paper is transparent about the mean and std, but the SOTA claim is stated using the maximum, and the mean is below the prior SOTA. This is a protocol problem, not an integrity problem. A simple simulation from the reported distribution can settle whether 95.8% is within chance. I also note a secondary concern: the best result uses audio-derived pause annotations via forced alignment, so the 'only transcripts' label is contestable and deserves re-framing or a comparison to multimodal systems. Neither issue requires rejection; the contribution is coherent and the experiments are mostly well described, so a conditional verdict asking for a robust evaluation protocol and a clarified claim remains appropriate.","tokens_in":10662,"tokens_out":11164,"duration_ms":118139,"concrete_test":"Use the reported mean and std to simulate 15-run experiments under a binomial null with accuracy 87.9% on 48 subjects, and compute the probability that the maximum reaches 95.8% or higher; if this one-sided p-value exceeds 0.05, the headline is within selection noise. Independently, rerun the best configuration (Sys 12, large PLMs, Before+After, subject-only transcript with pause encoding) with a single pre-registered seed and report the median and mean accuracy; if the median is not at or above 91.7%, the SOTA claim is not reproducible under a fixed protocol.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is the 95.8% SOTA accuracy on ADReSS using only transcripts. That number is the maximum test accuracy over 15 random seeds, two prompt positions, BERT/RoBERTa fusion, base/large sizes, and voting choices, selected after seeing test-set results. Under the reported distribution (mean 87.9%, std 3.3%, n=48), the expected maximum of 15 independent runs is roughly 93.5%, so 95.8% is not strong evidence of a real advantage. The prior transcript-only SOTA is 91.7% (44/48 subjects), and 95.8% is 46/48: a two-subject difference. The mean accuracy is below the prior SOTA, so the usual average-over-seeds protocol does not support the SOTA claim. No correction for multiple comparisons and no paired significance test (e.g., McNemar) is reported. The claim also deserves scrutiny because the best input uses pause annotations derived from forced alignment on audio, making 'only transcripts' contestable; without pause encoding, the same row's maximum drops to 91.7%. These issues make the headline claim conditional at best.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes prompt-based fine-tuning (PBFT) of BERT and RoBERTa for Alzheimer's disease detection on the ADReSS benchmark, together with pause encoding derived from forced alignment, a comparison of manual versus Whisper ASR transcripts, and majority voting/ensemble over models, prompt positions, and random seeds. The central claim is that the method reaches a maximum test accuracy of 95.8% on manual transcripts with pause encoding, reported as state-of-the-art for transcript-only AD detection on the ADReSS test set, with mean accuracy 87.9% and standard deviation 3.3%. \n\nThe paper also reports WER comparisons across several ASR models, an analysis of prompt template position, the effect of removing interviewer speech, and the contribution of pause encoding.","tokens_in":10925,"tokens_out":2070,"duration_ms":23134,"significance":"If the 95.8% result were robustly supported, the paper would make a useful contribution: it addresses the objective mismatch between standard sequence classification fine-tuning and masked language modeling, demonstrates a systematic comparison of prompt positions, and provides a careful WER benchmark for ASR options on pathological speech. The WER analysis and the inclusion of pause encoding are informative and likely useful to the community. However, the headline SOTA claim is not yet supported by the reported evaluation: the number is the maximum over many seeds and system variants, the mean accuracy is below the previously reported transcript-only result of 91.7%, and no statistical test is provided on a 48-subject test set. The contribution is therefore promising but the evaluation protocol needs revision before the central claim can be accepted.","major_comments":[{"comment":"The headline result of 95.8% is the maximum test accuracy selected over 15 random seeds, two prompt positions, BERT/RoBERTa fusion, base- and large-sized models, and epoch voting choices, while the reported mean is 87.9% with std 3.3% on a 48-subject test set. Under this distribution, the expected maximum over independent runs is much higher than the mean, so the gap between 95.8% and the prior 91.7% (a difference of two subjects) is not strong evidence of a real improvement. The paper should report the full selection protocol, correct for multiple comparisons, and provide a paired significance test such as McNemar's test against [15]; otherwise the SOTA claim should be withdrawn or explicitly framed as a best-of-many configuration rather than as the expected performance of the proposed method.","section":"§4.2.2, Table 2, Sys. 12 and Abstract"},{"comment":"The claim of 'state-of-the-art performance for AD detection using only transcripts' is contestable because the best input includes pause encoding derived from forced alignment of the audio recordings, which is audio-derived information, not transcript-only information. The manuscript should either reclassify this setting as using transcripts plus pause timing features, or demonstrate that the 95.8% result is achieved without pause encoding. Notably, the corresponding row without pause encoding reports a maximum of 91.7%, which matches the prior SOTA and highlights that the headline result is specifically tied to the audio-derived pause information.","section":"§3.3 and §4.2.2, Table 2"},{"comment":"The evaluation protocol reports mean, std, and maximum accuracy over 15 seeds, but the paper does not state whether any configuration selection or hyperparameter tuning was performed using the test set. Since Table 2 reports eleven system rows with multiple inputs and model sizes, and the headline number is the best cell among them, the risk of overfitting to the test set is substantial. The authors should describe exactly how the configurations were selected, report confidence intervals for the mean accuracy, and justify why the maximum rather than the mean is the appropriate performance estimate for the SOTA comparison.","section":"§4.1, Experimental Setup"}],"minor_comments":[{"comment":"There are several typographical errors: 'mjority' in Section 1.3, 'probablities' in Section 3.2, 'hign WER' in Section 5, and 'classifcation' in the caption of Figure 1. These should be corrected.","section":"Global"},{"comment":"Table 2 is very dense and difficult to parse, especially with four input conditions separated by slashes and large-model results in parentheses. A separate table or a clearer layout with explicit row labels per input type would improve readability.","section":"Table 2"},{"comment":"The description of manual transcript processing would benefit from an example showing a CHAT-format raw segment and the corresponding processed transcript, particularly to illustrate how '[x n]' repetition and pause markers are converted.","section":"§2.1"},{"comment":"The WER comparison is informative, but the paper does not report confidence intervals or significance tests for the WER differences between ASR models; given the small number of test utterances, these would strengthen the claim that Whisper is the best ASR choice.","section":"§4.2.1"},{"comment":"Reference [3] is malformed ('S. Luz and F. Haide, Fuente s. dl, fromm d, macwhinney b'). It should be corrected to a proper citation of the ADReSS challenge paper.","section":"Reference [3]"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a worthwhile problem and contains useful empirical comparisons, but the central SOTA claim is not yet robustly established under the reported evaluation protocol. The main issue is that the headline accuracy is the maximum over many seeds and configurations while the mean is below the prior SOTA; this is fixable by reanalyzing the results and reframing the claim, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Let me give you the short version. The paper is a clean empirical study of prompt-based fine-tuning (PBFT) versus traditional fine-tuning for AD detection on ADReSS, with pause encoding as an extra input. The useful core is the systematic comparison: PBFT generally beats TFT, pause encoding helps, interviewer transcripts hurt, and Whisper's ASR transcripts underperform manual ones. That last set of findings is worth having.\n\nWhat is genuinely new is modest. PBFT for AD detection is already in Wang et al. [17], which the paper cites, and pause encoding comes from Yuan et al. [30]. The combination is a reasonable extension, not a new capability. The paper is honest about its own components and reports mean/std alongside the maximum, which I appreciate.\n\nThe soft spot is the headline. The 95.8% is the maximum test accuracy over fifteen random seeds, two prompt positions, two model families, two sizes, and voting choices, selected after seeing test results. The reported mean is 87.9% ± 3.3 on a 48-subject test set. That puts the prior SOTA of 91.7% above the mean, and the gap to 95.8% is two subjects. Under the reported distribution, the expected maximum of fifteen independent runs is roughly 93.5%, so 95.8% is within the range of seed-selection artifact. There is no McNemar or paired test, and no multiple-comparison correction. The claim as worded—'state-of-the-art performance'—is not supported by the evidence; the claim that the method is competitive and prompt learning helps is supported.\n\nOne nuance: calling the 95.8% 'using only transcripts' is a bit slippery because the pause encoding comes from forced alignment on the audio. The pauses are written into the text, so it is transcript plus pause tags, not raw text. Not fatal, but it should be stated.\n\nThe citation pattern is fine; self-citations are to their own earlier systems and are not load-bearing. The experimental setup is transparent enough for someone to reproduce.\n\nWho should read it: anybody working on AD detection from language, especially on the ADReSS benchmark. It deserves a serious referee, but the referee should push the authors to report the mean as primary, add a significance test, and stop treating the maximum as the system's performance. I would send it to review with a recommendation to revise.","headline":"Useful incremental comparison of prompt learning and pause encoding for AD detection, but the 95.8% SOTA claim is the maximum over fifteen seeds and sits on a two-subject difference; the mean accuracy is below the prior SOTA.","tokens_in":11430,"tokens_out":1922,"would_cite":true,"duration_ms":19504,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Prompt-based fine-tuning with pause encoding reaches 95.8% accuracy for Alzheimer's detection from transcripts alone.","keywords":["Alzheimer's disease detection","prompt-based fine-tuning","pause encoding","pre-trained language models","ADReSS benchmark","forced alignment","ensemble voting","transcript-only classification"],"falsifier":"Running the same pipeline with a single fixed seed and reporting the test accuracy, or computing the median over the 15 seeds, would settle it: if the fixed-seed accuracy lands near the 87.9% mean rather than near 95.8%, the state-of-the-art claim rests on selection over seeds rather than on a systematic gain. A second check is whether the pause-encoding benefit survives when the forced-alignment timestamps are replaced by random punctuation inserted at the same rate.","tokens_in":10474,"feed_emoji":"🧠","tokens_out":4180,"duration_ms":39845,"temperature":0.7,"pith_summary":"The paper argues that Alzheimer's disease can be detected more reliably from speech transcripts when a pre-trained language model is fine-tuned with a prompt template rather than a standard classification head, and when word-boundary pauses recovered from forced alignment are written into the text as punctuation. On the ADReSS benchmark, this combination reaches a maximum accuracy of 95.8% using manual transcripts, which the authors report as the best transcript-only result on that test set. The practical stakes are that a cheap, non-invasive screening signal—spontaneous speech—can be squeezed harder without any acoustic features.","feed_headline":"Prompt learning lifts transcript-only AD detection to 95.8%","feed_subtitle":"Pause marks plus BERT/RoBERTa voting beat prior text-only systems on the ADReSS test set.","key_machinery":"The load-bearing device is the prompt template \"The diagnosis result is [MASK]\", which turns the classification decision into a word-choice prediction and lets the PLM use its masked-language-modeling head instead of a new classification layer. A second component is pause encoding: forced alignment supplies word-boundary 'SIL' intervals, which are grouped into three duration bins and transcribed as punctuation marks so that disfluency patterns enter the text itself. The third component is the ensemble: majority voting over the last three epochs, late fusion of two template positions, and voting across BERT and RoBERTa with 15 random seeds, which stabilizes the result and is the source of the reported maximum.","core_discovery":"The central claim is that recasting binary AD classification as a masked-language-modeling task closes the gap between fine-tuning and pre-training objectives, and that this change, together with pause encoding and cross-model voting, is what pushes transcript-only accuracy on the ADReSS test set to 95.8% (mean 87.9%, std 3.3%). The authors insert the template \"The diagnosis result is [MASK]\" before or after each transcript, fine-tune BERT or RoBERTa to predict the label word \"alzheimer\" or \"healthy\", and then fuse predictions across the two models, two template positions, and multiple random seeds. Pauses from forced alignment are binned into short, medium, and long and encoded as ',', '.', and '...', giving the model explicit disfluency cues. The paper also reports that adding interviewer speech hurts accuracy, and that Whisper ASR transcripts, despite being the most accurate of the tested recognizers, still underperform manual transcripts.","pith_inferences":["Inference: the headline 95.8% is the maximum over 15 seeds, so the expected accuracy of a single run is closer to the reported mean of 87.9%, and the two-subject gap over the prior baseline could shrink if seed selection is removed.","Inference: the pause-encoding gain suggests that explicit disfluency markers, not just word content, drive the improvement; a direct test would be to add random punctuation at the same rate and check that accuracy does not rise.","Inference: transfer to other AD datasets with more speakers and ASR conditions is the natural next experiment, since the 48-subject test set makes the best-of-seeds figure brittle."],"forward_implications":["Transcript-only AD detection need not rely on a classification head; a prompt-formulated MLM objective can outperform the standard fine-tuning setup.","Pause duration markers add complementary signal to the words themselves, improving accuracy in both fine-tuning paradigms.","Including interviewer speech in the model input lowers accuracy, so subject-only transcripts are the safer input.","Even the best ASR transcripts degrade performance relative to manual transcripts, implying recognition errors currently cost more than the convenience of automation."],"supporting_citations":[{"why":"Supplies the ADReSS benchmark dataset and the train/test split used for all evaluations.","marker":"[3]"},{"why":"Provides the prior transcript-only baseline of 91.7% that this paper extends and compares against.","marker":"[15]"},{"why":"Earlier prompt-learning application to AD detection that the authors build on and differentiate from.","marker":"[17]"},{"why":"Defines BERT and the masked-language-modeling objective that motivates the prompt-based fine-tuning.","marker":"[18]"},{"why":"Defines RoBERTa, the second pre-trained model used in the ensemble.","marker":"[19]"},{"why":"Supplies the Whisper ASR model selected to generate the ASR transcripts compared with manual ones.","marker":"[25]"},{"why":"Provides the prompt-learning software used to implement the prompt-based fine-tuning.","marker":"[28]"},{"why":"Introduces the three-bin pause encoding method applied to the forced-alignment timestamps.","marker":"[30]"}],"fun_headline_variants":["Prompt learning lifts transcript-only AD detection to 95.8%","Prompt learning plus pauses: 95.8% AD from transcripts alone","Prompt learning achieves 95.8% AD detection on transcript text","Pause encoding and prompt tuning: AD transcripts hit 95.8%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the best result over 15 random seeds, two prompt positions, two models, and epoch voting is a fair representative of system performance; on 48 test subjects, the 95.8% maximum differs from the prior 91.7% by only about two people, while the mean over seeds is 87.9%.","fun_headline_variants_meta":{"raw":{"variants":["Prompt learning lifts transcript-only AD detection to 95.8%","Prompt learning plus pauses: 95.8% AD from transcripts alone","Prompt learning achieves 95.8% AD detection on transcript text","Pause encoding and prompt tuning: AD transcripts hit 95.8%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001007,"raw_usage":{"total_tokens":4272,"prompt_tokens":973,"completion_tokens":3299,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":589,"completion_tokens_details":{"reasoning_tokens":3221}},"tokens_in":589,"tokens_out":3299,"duration_ms":22422,"temperature":1.0,"reasoning_tokens":3221,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T19:51:00.144034+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Running the same pipeline with a single fixed seed and reporting the test accuracy, or computing the median over the 15 seeds, would settle it: if the fixed-seed accuracy lands near the 87.9% mean rather than near 95.8%, the state-of-the-art claim rests on selection over seeds rather than on a systematic gain. A second check is whether the pause-encoding benefit survives when the forced-alignment timestamps are replaced by random punctuation inserted at the same rate.","supporting_citations":[{"cited_title":"BertForSequenceClassifica- tion","cited_arxiv_id":null,"evidence_quote":"Supplies the ADReSS benchmark dataset and the train/test split used for all evaluations."},{"cited_title":"Prompt- based learning framework for zero-shot cross-lingual text classi- fication,","cited_arxiv_id":null,"evidence_quote":"Provides the prompt-learning software used to implement the prompt-based fine-tuning."}],"review_version":1}