{"id":"18520ed1-7348-458c-97e5-bd2219597bbb","arxiv_id":"2412.15230","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"The PROCESS challenge provides a new corpus of spontaneous speech from three prompts, with baseline F1 55.0% for MCI/dementia classification and RMSE 2.98 for MMSE regression.","lead":"A new challenge releases a corpus of spontaneous speech from people with early dementia, mild cognitive impairment, and healthy controls, using three neurologist-designed prompts. Baseline systems reach an F1 of 55.0% on classification and an RMSE of 2.98 on MMSE prediction, establishing a benchmark for the field.","discovery_kind":"new_application","skeptic_critique":null,"referee_report":{"model":"deepseek-v4-flash","summary":"The paper announces the PROCESS Signal Processing Grand Challenge for early dementia detection using spontaneous speech. It describes a new corpus collected through an online platform, with three neurologist-designed prompts (semantic fluency, phonemic fluency, and the Cookie Theft picture description). Two tasks are defined: a three-way classification task (healthy, MCI, dementia) evaluated by macro F1, and a regression task predicting MMSE scores evaluated by RMSE. Baseline models are presented using eGeMAPS acoustic features with SVM and Random Forest, and Whisper-transcribed text processed by RoBERTa. The paper reports a best classification F1 of 55.0% and a best regression RMSE of 2.98.","tokens_in":2940,"tokens_out":4618,"duration_ms":45581,"significance":"If the PROCESS corpus is real, professionally collected, and accessible, this challenge would be a useful community resource, particularly for early-stage (MCI) detection where existing datasets such as Dementia Bank are noted to be limited. The paper provides simple, reproducible baselines, which is appropriate for a challenge overview. However, the paper lacks any quantitative description of the corpus—no participant counts, class distribution, demographic information, recording conditions, or label-validation procedures. This makes the reported baseline scores uninterpretable outside the specific sample and prevents the paper from serving as a self-contained benchmark. The contribution is therefore conditional on information that is currently missing.","major_comments":[{"comment":"The corpus is described only qualitatively. There is no statement of the number of speakers, the number of recordings per prompt, the distribution across healthy/MCI/dementia classes, age and sex distribution, recording conditions (microphone, environment, duration), or the total size of the train/dev/test splits. For a challenge paper whose central claim is to provide a new benchmark corpus, these numbers are essential: they determine whether the task is class-balanced, how much data participants have to train on, and how much confidence one can place in the reported 55.0% F1 and 2.98 RMSE. This information should be added in a dedicated data section.","section":"Section 2, Tasks and Corpus"},{"comment":"All baseline results are reported as single point estimates with no error bars, confidence intervals, or significance tests. Moreover, the paper does not describe how the train/development/test splits were constructed (e.g., random split, number of folds, stratification, or the number of speakers in the test set). Given that the differences between several configurations are small (e.g., F1 of 55.0 vs 53.3 in Table 1, and RMSE of 2.99 vs 2.98 in Table 2), the reader cannot determine whether these differences are meaningful. The paper should report variance across cross-validation splits or, at minimum, the test-set size and split protocol.","section":"Section 3, Baseline Models and Tables 1-2"},{"comment":"The ground-truth labels (diagnoses and MMSE scores) are taken as given, but the paper does not describe how they were obtained: what clinical assessment was used, whether MMSE was administered at the time of recording or later, how the healthy/MCI/dementia status was confirmed, and whether any quality control such as inter-rater agreement was performed. For a medical benchmark, noisy or non-contemporaneous labels would directly undermine both tasks. The authors should report the labeling procedure and any available reliability measures.","section":"Section 2, Corpus, and Section 3, Text Baseline"},{"comment":"There is an internal inconsistency in the reported best classification result. The text states that 'the best F1-score of 55.0% for VF (semantic + phonemic fluency)', but Table 1 shows that the 55.0% F1 belongs to the SVC (eGeMAPS) model on the Cookie Theft (CT) prompt; the VF results are 38.3% (SVC), 33.9% (RFC), and 35.6% (RoBERTa). This discrepancy affects the interpretation of the abstract's headline claim and must be corrected.","section":"Section 3, first paragraph"}],"minor_comments":[{"comment":"The task descriptions in the Introduction ('distinguishing early cognitive decline and dementia from healthy volunteers') suggest a binary setup, while Section 2 lists three classes (healthy, MCI, dementia). Please clarify whether the classification task is three-way or binary (e.g., MCI+dementia vs healthy), and define how macro metrics are computed over the classes.","section":"Section 2, Tasks"},{"comment":"The paper refers to the Cognospeak website but does not describe the data-access mechanism for challenge participants (e.g., registration, data use agreement, ethics approval). This information is necessary for the challenge to be reproducible.","section":"Section 2, Corpus access"},{"comment":"The acoustic baseline averages eGeMAPS features over the entire prompt. For fluency tasks, temporal dynamics (e.g., number of items produced in successive intervals) are known to be informative; the averaging step may discard this information. This is acceptable for a simple baseline, but it should be acknowledged as a limitation.","section":"Section 3, Acoustic baseline"},{"comment":"The text baseline relies on Whisper transcriptions, but the paper does not report transcript quality (e.g., word error rate on a held-out subset). For spontaneous speech from individuals with cognitive impairment, ASR errors could significantly affect downstream text models; a brief quality check would strengthen the baseline description.","section":"Section 3, Text baseline"}],"recommendation":"major_revision","confidential_remarks":"This manuscript is a challenge description rather than a full research paper. The main weakness is the complete absence of dataset statistics and validation details, which would be essential for a benchmark paper in a journal. The internal inconsistency about which prompt yields the 55% F1 is also concerning and should be fixed. If the dataset is to be described in a separate data paper, the authors should refer to it and provide at least basic summary statistics here. The paper may be better suited for a workshop or challenge-proceedings format unless the dataset description is substantially expanded."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: this is a Signal Processing Grand Challenge description, not a methods paper. The genuinely new thing is the PROCESS corpus: spontaneous speech from healthy, MCI, and dementia speakers, elicited with three neurologist-designed prompts (semantic fluency, phonemic fluency, Cookie Theft), with manual transcripts for train/dev and MMSE scores. The baseline models are off-the-shelf (eGeMAPS + SVC/RF, Whisper + RoBERTa), and the reported numbers are what they are: best F1 55.0 on the VF prompt with SVC, best RMSE 2.98 with RoBERTa regression. That the textual baselines beat the acoustic ones on regression is a useful data point.\n\nWhat the paper does well: it is honest about the baselines being weak and frames the challenge as needing better models. The prompt design based on neurology is sensible, and the corpus addresses a real gap—late-stage AD data in Dementia Bank makes detection too easy, and audio quality is poor. The two-task setup (classification plus MMSE regression) is a good structure for a challenge.\n\nThe soft spots are real but not fatal. The paper reports no error bars, significance tests, or cross-validation details; for a benchmark paper that is a genuine omission. More importantly, Section 2 gives no participant counts per class, no age/gender breakdown, no description of how diagnoses were made or MMSE collected, and no inter-rater agreement. Without these, a reader cannot judge whether the corpus is balanced or whether the ground truth is reliable. The phrase 'state-of-the-art corpus' in the introduction is an overclaim—'new' is defensible, 'state-of-the-art' is not established. Also, data and code are not yet available at the time of writing, which limits reproducibility.\n\nMy assessment: the central claim—that a corpus was collected and baselines were run—is plausible and the results are direct measurements, so there is no circularity. The paper would benefit from a supplementary data sheet (counts, demographics, label protocol, audio quality) and a note on how the corpus will be archived. The missing details are fixable, not structural.\n\nWho is this for? Researchers working on speech-based cognitive screening who need a newer, multi-class benchmark. As a challenge description, it deserves serious peer review—conditional on the authors providing dataset statistics and access details. I'd bring it to our reading group, but I would not cite the baseline numbers as established facts until the data description is public.","headline":"New corpus for early dementia detection from speech, with honest baselines but under-reported dataset statistics; worth engaging if the data description is tightened.","tokens_in":3462,"tokens_out":2052,"would_cite":true,"duration_ms":20718,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The PROCESS speech corpus is proposed as a shared benchmark for early dementia detection, with baselines reaching 55.0% F1 on classification and 2.98 RMSE on MMSE prediction.","keywords":["dementia detection","mild cognitive impairment","spontaneous speech","speech corpus","MMSE regression","grand challenge","semantic fluency","phonemic fluency"],"falsifier":"Re-annotating a random sample of the recordings with an independent clinical diagnosis, or publishing per-class participant counts and demographic distributions, would test whether the healthy/MCI/dementia labels are trustworthy and whether the 55.0% F1 and 2.98 RMSE reflect real signal rather than class imbalance or labeling noise.","tokens_in":2859,"feed_emoji":"🧠","tokens_out":6546,"duration_ms":60585,"temperature":0.7,"pith_summary":"The paper introduces a spontaneous-speech corpus, called PROCESS, built for a grand challenge on early dementia detection. The corpus collects English speech from three neurologist-designed prompts—naming animals, saying words beginning with 'P', and describing the Cookie Theft picture—from speakers who are healthy, have mild cognitive impairment (MCI), or have dementia. The authors report baseline systems: an acoustic-feature pipeline reaches 55.0% macro F1 on the three-way classification task, and a transcript-based regression model predicts MMSE scores with an RMSE of 2.98. The goal is to provide a modern common benchmark that reflects early-stage cases and realistic audio quality, so that competing methods can be compared fairly. If the corpus holds up, it gives researchers a shared testbed for spotting cognitive decline before dementia becomes severe.","feed_headline":"New speech corpus targets early dementia detection with baseline F1 of 55%","feed_subtitle":"Three neurologist-designed prompts let researchers classify healthy, MCI, and dementia and predict MMSE scores from voice.","key_machinery":"The load-bearing object is the PROCESS corpus itself: a collection of English speech recordings obtained through an online platform, with each speaker answering three prompts—semantic fluency (name as many animals as possible in a minute), phonemic fluency (say words beginning with 'P' for a minute), and the Cookie Theft picture description. The corpus is split into training, development, and test sets, with manual transcripts provided for the first two and withheld from the test set, so the challenge can evaluate real speech-understanding conditions. The baseline machinery uses two routes: acoustic features averaged over each prompt and processed by support-vector or random-forest models, and manual or automatically transcribed text processed by a pretrained transformer for classification and regression.","core_discovery":"The central claim is that a new corpus of spontaneous speech, elicited by three short prompts designed with neurological assessment in mind, can serve as a valid shared benchmark for early dementia detection. The corpus supplies training and development sets with audio and manual transcripts, an independent test set, diagnostic labels for classification, and MMSE scores for regression. Baseline experiments show that averaged acoustic features fed to linear support-vector machines and random forests reach 55.0% macro F1 at best, while a text pipeline that first transcribes speech and then applies a transformer language model reaches 36.8% F1 on classification and an RMSE around 2.98 on regression. The authors present these results as reference points for challenge participants to improve on, not as a solved problem.","pith_inferences":["A natural next step, not pursued in the paper, is to fuse acoustic and textual features at the model level; the reported numbers hint that they capture complementary information.","The paper's reliance on averaged feature vectors discards temporal structure, so sequence-aware models such as recurrent or attention networks may change the baseline picture.","If the corpus grows over time, it could support longitudinal studies of the same speakers, which would be more directly relevant to early detection than a single cross-sectional recording.","The absence of reported demographic or per-class counts means external validation is needed before treating the baseline F1 as a stable property of the corpus; independent re-labeling would test label reliability."],"forward_implications":["Challenge participants can train on the same corpus and compare their models against the published baselines, making future results directly comparable.","The three-prompt design lets researchers ask which prompt—semantic fluency, phonemic fluency, or picture description—carries the most signal for early-stage detection.","The gap between the acoustic baseline (55.0% F1) and the text baseline (36.8% F1) suggests that combining audio and transcript features may be a promising route.","The modest baseline scores indicate that early-stage dementia detection from spontaneous speech is not yet solved, so the corpus is a target for improvement rather than a finished product.","MMSE regression with an RMSE near 2.98 provides a quantitative reference for severity estimation, not just binary or three-way classification."],"supporting_citations":[{"why":"Establishes the context of prior speech-based dementia recognition challenges that the new corpus updates.","marker":"[1]"},{"why":"Provides the acoustic feature extraction tool used to turn recordings into feature vectors for the baseline models.","marker":"[2]"},{"why":"Defines the specific compact acoustic parameter set from which the baseline feature vectors are computed.","marker":"[3]"},{"why":"Supplies the automatic speech recognition system used to create transcripts for the text-based baselines.","marker":"[4]"},{"why":"Provides the pretrained transformer language model used to process transcripts for classification and regression.","marker":"[5]"}],"fun_headline_variants":["PROCESS challenge: new speech corpus for early dementia detection","Baseline F1 55% in new dementia speech benchmark","Three prompts, one corpus: early dementia detection challenge","Early dementia detection via spontaneous speech: PROCESS challenge","Dementia detection challenge: new speech corpus, baseline F1 55%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The diagnostic labels and MMSE scores attached to each recording are accurate, and the number of speakers in each group is large and balanced enough for the reported baseline numbers to be meaningful.","fun_headline_variants_meta":{"raw":{"variants":["PROCESS challenge: new speech corpus for early dementia detection","Baseline F1 55% in new dementia speech benchmark","Three prompts, one corpus: early dementia detection challenge","Early dementia detection via spontaneous speech: PROCESS challenge","Dementia detection challenge: new speech corpus, baseline F1 55%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000902,"raw_usage":{"total_tokens":3803,"prompt_tokens":790,"completion_tokens":3013,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":406,"completion_tokens_details":{"reasoning_tokens":2929}},"tokens_in":406,"tokens_out":3013,"duration_ms":23691,"temperature":1.0,"reasoning_tokens":2929,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T21:32:40.042252+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotating a random sample of the recordings with an independent clinical diagnosis, or publishing per-class participant counts and demographic distributions, would test whether the healthy/MCI/dementia labels are trustworthy and whether the 55.0% F1 and 2.98 RMSE reflect real signal rather than class imbalance or labeling noise.","supporting_citations":[{"cited_title":"Dementia is typically recognized only when symp- toms become pronounced; however, its effects on the brain begin much earlier","cited_arxiv_id":null,"evidence_quote":"Establishes the context of prior speech-based dementia recognition challenges that the new corpus updates."},{"cited_title":"Please name as many animals as you can in a minute","cited_arxiv_id":null,"evidence_quote":"Provides the acoustic feature extraction tool used to turn recordings into feature vectors for the baseline models."},{"cited_title":"Acoustic features were extracted with OpenSmile [2] and eGeMAPS feature set [3], converting recordings into a se- quence of vectors X = {⃗ x1, ⃗ x2,","cited_arxiv_id":null,"evidence_quote":"Defines the specific compact acoustic parameter set from which the baseline feature vectors are computed."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the automatic speech recognition system used to create transcripts for the text-based baselines."},{"cited_title":"An overview of the ADReSS-M signal processing grand challenge on multilingual Alzheimer’s dementia recognition through spontaneous speech,","cited_arxiv_id":null,"evidence_quote":"Provides the pretrained transformer language model used to process transcripts for classification and regression."}],"review_version":1}