REVIEW 4 major objections 8 minor 39 references
Optimizing Speech-Input Length for Speaker-Independent Depression Classification
T0 review · 4 major / 8 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read This paper establishes that the accuracy of speaker-independent NLP depression classifiers depends on speech-input length in a structured way: responses need at least 30–50 words, stop improving beyond about 120 words for a weaker system…
desk verdict A useful, concrete empirical study of how much speech a depression classifier needs, but the headline thresholds come from gating full-length-trained models and carry no error bars. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central measurement device is the cumulative gated length metric: for any word-count threshold x, all samples longer than x are truncated to x, and AUC is recomputed on the truncated data, so the curve shows how much information is available 'so far' in a response or session. This gating makes the floor, saturation, and ordering effects visible. Two supporting mechanisms carry the comparison: a single aggregate speaking rate (2.39 words/second) that converts word counts to elapsed time, and two deliberately different NLP systems (an SVM over averaged word-vector embeddings and a deep fine-tuned language model) whose performance gap tests whether the threshold values depend on model quality.
What would settle it
Train the same two systems on a corpus where depression labels come from structured clinical interviews rather than PHQ-8, and re-run the gated-length analysis; if the 30–50 word minimum, the 120/200 response saturation points, or the ~1,000 word session ceiling do not recur, the reported thresholds are specific to PHQ-8 agreement rather than to depression classification.
Extended reading notes
Core claim
Performance depends on natural length (words), elapsed length (time), and ordering of the response within a session. Both systems share a minimum threshold of 30–50 words per response; below this, AUC falls sharply. The better deep-learning system (System 2) keeps gaining from additional words until about 200 words per response, while the weaker SVM-based system (System 1) saturates at about 120 words; sessions of concatenated responses saturate near 1,000 words. At saturation, moving to a new question is better than continuing the current response, with a maximum relative gain of about 4% early in the session. Additionally, within a session, speakers tend to increase response lengths over time, and within a response, the later part of longer utterances carries more classification value than the early part, with threshold lengths of about 80 words (System 1) and 150 words (System 2) below which cutting off a response is harmful.
Load-bearing premise
The analyses take the PHQ-8 self-report score as the gold-standard depression label and assign that session-level label to every response inside the session; if this questionnaire is a noisy or biased measure of depression, all the length effects measure agreement with the questionnaire rather than with clinical depression.
Editorial extensions
If this is right
- Deployed depression screening applications should elicit responses of at least 30–50 words (about 20 seconds) before deciding anything from the text.
- For the strong system, a response can be cut off after roughly 200 words; for a weaker system, after 120 words, because additional words within that response add less than asking a new question.
- Session-level assessments should budget about 1,000 words (~8 minutes) of total speech; beyond that, additional responses or words contribute little.
- Concatenating many shorter responses is better than a few longer ones, provided every response clears the minimum length; the largest gain from a new response (about 4% AUC) comes early in the session.
- Model improvements raise the saturation threshold, so updating a classifier also changes the optimal elicitation design: a better model can justify asking for longer responses.
Reading between the lines
- By the same gating logic, similar length thresholds are likely to appear in other paralinguistic screening tasks (anxiety, stress, cognitive load), but the absolute values should scale with task difficulty and model capacity; this is a testable extension.
- The warm-up effect (responses lengthening across a session) suggests an elicitation design that orders topics from easier to harder could harvest longer, more informative responses later in the session.
- Inferred cost-benefit: since the marginal value of a new response decays with session length, an adaptive interviewer that asks the next question when the current response's gated AUC flattens could approach the 1,000-word session ceiling with fewer, shorter responses.
- If transferred to other languages or demographics, the absolute word thresholds would need re-estimation, but the qualitative shape (floor, saturation, new-question advantage) probably persists.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper investigates how the length of speech input affects speaker-independent depression classification. Using a proprietary corpus of over 1,400 hours of speech from a human-machine screening app, the authors evaluate two NLP systems (a weaker SVM/Word2Vec system trained on 650 hours and a stronger ULMFiT system trained on 1,400 hours) on a held-out test set with no overlapping speakers. They define cumulative gated length as a test-time truncation measure and report AUC curves for responses and sessions, finding a minimum length threshold of 30-50 words, response saturation around 120-250 words depending on system, session saturation near 1,000 words, and various within-session effects including increasing response lengths over a session and a 6% advantage of the second half over the first half in longer responses. The paper concludes with recommendations for eliciting and processing speech at optimal lengths.
Significance. If the reported thresholds are robust, they provide actionable design constraints for speech-based depression screening systems, which is a practical and understudied problem. The study is unusually large (over 1,400 hours) and the within-session control analyses (e.g., comparing shortest vs. longest responses within the same session) are a valuable attempt to separate length effects from speaker identity. The paper explicitly holds the test set constant across systems, which strengthens the system comparison. However, the central quantitative claims are weakened by the test-time truncation protocol, the absence of confidence intervals, and some unsupported or inconsistent threshold reports. The work is therefore promising but not yet conclusive.
major comments (4)
- [Section 3.2, Figures 3-4, Observations 1-3, 8] The cumulative gated length analysis truncates every test response or session at x words, but Systems 1 and 2 are trained on full-length responses. This creates a train/test length mismatch, so the observed AUC declines and saturation points may reflect the models' sensitivity to partial inputs rather than intrinsic information content in shorter speech. Since the paper's stated purpose is to guide system design ('elicit and process optimal input lengths'), the thresholds must be shown to be stable when models are trained on inputs whose length matches the evaluation length; please retrain at fixed lengths (or otherwise balance training length) and report whether the 30-50 word minimum, the 120/200/250 word response saturation values, and the ~1,000 word session saturation persist.
- [Section 3.3, Observations 7 and 11-14] The 6% first-half/second-half AUC difference, the 4% maximum gain from a new response, and the threshold values are presented without confidence intervals, error bars, or significance tests. Given the large number of responses, bootstrap or other resampling CIs should be straightforward to compute. Without them, it is impossible to tell whether the differences between lengths, between early/late portions, and between systems are reliable or within sampling noise.
- [Observations 6 and 8, and Section 4 summary] Observation 6 ('having more responses is better') and Observation 8 (saturation lengths) are stated to apply to both systems, but Figure 4 shows System 2 only, and the System 1 saturation value is explicitly 'not shown.' Additionally, Observation 2 reports response saturation at 'about 250 words' while Observation 8 and the summary say 200 words for System 2 (120 for System 1). Please resolve the inconsistency and show the supporting data for both systems, or restrict the claims to System 2.
- [Section 2.1] The PHQ-8 score for each session is assigned to every response within that session, and all length analyses inherit that label. Because PHQ-8 is a self-report instrument, not a clinical diagnosis, the AUC curves measure agreement with the questionnaire; the paper should at least state this limitation and, ideally, check robustness using continuous PHQ-8 scores or a sensitivity analysis around the cut-off.
minor comments (8)
- [Section 2.1] Please clarify how the test set was balanced and whether any class re-weighting was applied; also define what 'speaker-independent' means exactly (no overlapping speakers between train and test is stated, but validation details are missing).
- [Section 3.1] The speaking rate differences (about 5 wpm between classes, 3-4 wpm by length) are reported without statistical tests or CIs; please add them.
- [Figures 3/4/5] The curves are difficult to read at the exact threshold locations; consider adding vertical/horizontal markers and numeric annotations.
- [Section 3.3] The statement 'We searched empirically for natural length values of the transition' describes a form of multiple comparisons; please describe any correction or report the full transition curve.
- [References] Reference [30] is incomplete ('R. Soche'); also Reference [20] (personal communication) is not a public data source and should be marked as such.
- [Section 2.2] The sentence 'We are exploring updated approaches such as [32]' is not relevant to the current results and should be moved to future work.
- [Acknowledgements] The name 'Oliveiro' appears to be a typo; please verify the spelling.
- [Section 4] The sentence 'Within-speaker analysis also showed that while longer responses perform better overall, once the utterances are completed, shorter responses perform better initially' is confusingly worded; please rephrase to clarify the comparison.
Circularity Check
No significant circularity; the length thresholds are empirical observations from gated AUC curves, not fitted parameters or self-referential predictions.
full rationale
The paper's central claims (minimum thresholds of 30–50 words, response saturation near 120/200 words, session saturation near 1000 words, within-session ordering effects) are descriptive results obtained by evaluating two fixed, independently trained NLP systems under a cumulative gated length protocol. The gating metric is explicitly defined as removing any additional length beyond x before computing AUC, and the reported thresholds are read off the resulting curves; they are not parameters fitted to maximize those curves, nor are they defined in terms of the target result. The global speaking rate of 2.39 words/second is an aggregate constant used only to convert word counts to approximate seconds, and it is not optimized against AUC or otherwise used to construct the predicted labels. No load-bearing premise rests on a self-citation: references to prior work concern standard methods (e.g., Word2Vec, ULMFiT, PHQ-8 cutoff) or external benchmarks, and the paper's own corpus statistics are reported as descriptive. The PHQ-8 session label assigned to each response is a measurement assumption about the gold standard, not a circular derivation. Similarly, the skeptic's concern about testing on truncated inputs from models trained on full-length inputs is a potential train/test distribution-shift confound, but it does not make the reported thresholds true by construction: the thresholds could in principle differ under retraining, so the claims remain empirically testable rather than definitionally forced. No equation in the paper reduces to its inputs, and no fitted value is renamed as a prediction.
Assumptions & free parameters
free parameters (2)
- Global speaking rate estimate =
2.39 words/second
- Length thresholds =
30-50, 120, 150, 200, and 1000 words
assumptions (3)
- domain assumption PHQ-8 self-report score is a valid gold standard for depression status.
- domain assumption A single aggregate speaking rate of 2.39 words/second approximates speaking rate for all speakers and lengths.
- domain assumption Speaker-independent partitioning with no overlapping speakers prevents speaker leakage.
Cite this review
Pith. "Pith review of Optimizing Speech-Input Length for Speaker-Independent Depression Classification." pith.science (2026). https://pith.science/paper/2PUSIUFG
@misc{pith2026250100608,
author = {Pith},
title = {Pith review of: Optimizing Speech-Input Length for Speaker-Independent Depression Classification},
year = {2026},
howpublished = {\url{https://pith.science/paper/2PUSIUFG}},
note = {Machine review of arXiv:2501.00608}
}
read the original abstract
Machine learning models for speech-based depression classification offer promise for health care applications. Despite growing work on depression classification, little is understood about how the length of speech-input impacts model performance. We analyze results for speaker-independent depression classification using a corpus of over 1400 hours of speech from a human-machine health screening application. We examine performance as a function of response input length for two NLP systems that differ in overall performance. Results for both systems show that performance depends on natural length, elapsed length, and ordering of the response within a session. Systems share a minimum length threshold, but differ in a response saturation threshold, with the latter higher for the better system. At saturation it is better to pose a new question to the speaker, than to continue the current response. These and additional reported results suggest how applications can be better designed to both elicit and process optimal input lengths for depression classification.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
Introduction Depression is a prevalent disabling condition and a major global public health concern [1], [2]. Mobile AI technology could play an important role in expanding screening for depression, especially as an aid to providers who could follow up with appropriate care. Speech technology offers promise because speaking is natural, can be used at a di...
-
[2]
Method 2.1. Data It was necessary to use a new corpus to obtain enough data to study length effects. To facilitate comparative research we have initiated discussion with the Linguistic Data Consortium on future release of data from this corpus to the community [20]. Corpus statistics are given in Table 1. For training we used a larger (1400 speech hours, ...
-
[3]
Both systems show sharp decline below 30 to 50 words. 2. Responses saturate in AUC at about 250 words. 3. Sessions appear to saturate at closer to 1000 words. To understand the contribution of responses as they accumulate within a session, see Figure 4. As noted earlier, our data contain a mean of roughly 4.5 responses per session. Figure 4: Session-level...
-
[4]
F. Scibelli, G.Roffo, M. Tayarani, L. Bartoli, G. De Mattia, and A. Vinciarelli "Depression Speaks: Automatic Discrimination Between Depressed and Non-Depressed Speakers Based on Nonverbal Speech Features" In Conf. International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2018
work page 2018
-
[5]
Depression and other common mental disorders: global health estimates,
World Health Organization, “Depression and other common mental disorders: global health estimates,” World Health Organization, pp. 1–24, 2017
work page 2017
-
[6]
In: Diagnostic and Statistical Manual of Mental Disorders DSM-5
Major depressive disorder. In: Diagnostic and Statistical Manual of Mental Disorders DSM-5. 5th ed. Arlington, Va.: American Psychiatric Association; 2013
work page 2013
-
[7]
Detecting Depression with Audio/Text Sequence Modeling of Interviews
T. Alhanai, M. Ghassemi, and J. Glass, "Detecting Depression with Audio/Text Sequence Modeling of Interviews." Proc. Interspeech. 2018
work page 2018
-
[8]
A. Pampouchidou, O. Simantiraki, A. Fazlollahi, M. Pediaditis, D. Manousos, A. Roniotis, G. Giannakakis, F. Meriaudeau, P. Simos, and K. Marias, et al. Depression Assessment by Fusing High and Low Level Features from Audio, Video, and Text. In Proceedings of the 6th International Workshop on Audio/Visual Emotion Challenge. ACM, 27–34. 2016
work page 2016
Show all 39 references
-
[9]
Detecting Depression using Vocal, Facial and Semantic Communication Cues,
J. R. Williamson, E. Godoy, M. Cha, A. Schwarzentruber, P. Khorrami, Y. Gwon, H.-T. Kung, C. Dagli, and T. F. Quatieri, “Detecting Depression using Vocal, Facial and Semantic Communication Cues,” in Proceedings of the 6th International Workshop on Audio/Visual Emotion Challeng...
2016
-
[10]
L. Yang, D. Jiang, L. He, E. Pei, M.C. Oveneke, and H. Sahli
-
[11]
This was consistent for both Systems
Long utterances perform better than short ones—eventually. This was consistent for both Systems. 12. Short utterances perform better than long, initially. 13. There is a threshold length below which one should not cut off a current response. This length is about 80 words for S...
-
[12]
Resnik, A
P. Resnik, A. Garron, and R. Resnik. 2013. Using topic modeling to improve prediction of neuroticism and depression. In Proceedings of the 2013 Conference on Empirical Methods in Natural, pages 1348–1353
2013
-
[13]
Coppersmith
G. Coppersmith. CLPsych 2015 shared task: Depression and PTSD on Twitter. Proceedings of the 2nd Workshop on Computational Linguistics and Clinical Psychology: From Linguistic Signal to Clinical Reality. 2015
2015
-
[14]
Multimodal assessment of depression from behavioral signals
J.F. Cohn, N. Cummins, J. Epps, R. Goecke, J. Joshi, and S. Scherer. "Multimodal assessment of depression from behavioral signals." In The Handbook of Multimodal-Multisensor Interfaces, pp. 375-417. Association for Computing Machinery and Morgan & Claypool, 2018
2018
-
[15]
Stasak, J
B. Stasak, J. Epps, and N. Cummins. Depression prediction via acoustic analysis of formulaic word fillers. Polar, 77(74):230. 2016
2016
-
[16]
Valstar, B
M. Valstar, B. Schuller, K. Smith, F. Eyben, B. Jiang, S. Bilakhia, S. Schnieder, R. Cowie, and M. Pantic. AVEC 2013: the continuous audio/visual emotion and depression recognition challenge. In Proceedings of the 3rd ACM international workshop on Audio/visual emotion challeng...
2013
-
[17]
Valstar, B
M. Valstar, B. Schuller, K. Smith, T. Almaev, F. Eyben, J. Krajewski, R. Cowie, and M. Pantic. AVEC 2014: 3D Dimensional Affect and Depression Recognition Challenge. In Proceedings of the 4th International Workshop on Audio/Visual Emotion Challenge (AVEC '14). ACM, New York, N...
2014
-
[18]
Detecting depression: a comparison between spontaneous and read speech,
S. Alghowinem, R. Goecke, M. Wagner, J. Epps, M. Breakspear, and G. Parker, "Detecting depression: a comparison between spontaneous and read speech," ICASSP 2013, Vancouver, B.C. Canada, 2013, pp. 7547-7551
2013
-
[19]
Valstar, J
M. Valstar, J. Gratch, B. Schuller, F. Ringeval, D. Lalanne, M. Torres, S. Scherer, G. Stratou, R. Cowie, and M. Pantic. AVEC 2016: Depression, Mood, and Emotion Recognition Workshop and Challenge. In Proceedings of the 6th International Workshop on Audio/Visual Emotion Challe...
2016
-
[20]
Ringeval, B
F. Ringeval, B. Schuller, M. Valstar, J. Gratch, R. Cowie, S. Scherer, S. Mozgai, N. Cummins, M. Schmitt, and M. Pantic. AVEC 2017: Real-life Depression, and Affect Recognition Workshop and Challenge. In Proceedings of the 7th Annual Workshop on Audio/Visual Emotion Challenge ...
2017
-
[21]
Neumann and N
M. Neumann and N. T. Vu, Attentive convolutional neural network based speech emotion recognition: A study on the impact of input features, signal length, and acted speech, arXiv:1706.00612, 2017
2017 arXiv
-
[22]
Short utterance based speech language identification in intelligent vehicles with time-scale modifications and deep bottleneck features
Z. Ma, H. Yu, W. Chen, J. Guo, "Short utterance based speech language identification in intelligent vehicles with time-scale modifications and deep bottleneck features", IEEE Trans. Veh. Technol., vol. 68, no. 1, pp. 1-8, 2019
2019
-
[23]
Manea, S
L. Manea, S. Gilbody, and D. Mcmillan, Optimal cut-off score for diagnosing depression with the Patient Health Questionnaire (PHQ-9): A meta-analysis. CMAJ: Canadian Medical Association journal. 184. E191-6. 10.1503/cmaj.110829. 2011
-
[24]
Wang and I
Z. Wang and I. Tashev, Learning utterance-level representations for speech emotion and age/gender recognition using deep neural networks. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), New Orleans, LA, 2017, pp. 5150-5154. 2017
2017
-
[25]
Personal communication with Mark Liberman, Linguistic Data Consortium. 2019
2019
-
[26]
Stasak, J
B. Stasak, J. Epps, and R. Goecke, Elicitation design for acoustic depression classification: An investigation of articulation effort, linguistic complexity, and word affect. in INTERSPEECH, pp. 834–838. 2017
2017
-
[27]
J. Yuan, M. Liberman, and C. Cieri. Towards an integrated understanding of speaking rate in conversation. INTERSPEECH 2006
2006
-
[28]
Pennington, R
J. Pennington, R. Socher, and C.D. Manning. Glove: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, 2014
2014
-
[29]
Kroenke, T.W
K. Kroenke, T.W. Strine, R. Spitzer, J.B.W. Williams, J.T. Berry, A.H. Mokdad, The PHQ-8 as a measure of current depression in the general population, Journal of Affective Disorders, Volume 114, Issues 1–3, 2009
2009
-
[30]
Carey, K
M. Carey, K. Jones, G. Meadows, R. Sanson-Fisher, C. D’Este, et al. Accuracy of general practitioner unassisted detection of depression. Aust N Z J Psychiatry 48(6):571–8. 2014
2014
-
[31]
Google Cloud STT at https://cloud.google.com/speech-to-text/
-
[32]
Mikolov, I
T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111–3119, 2013
2013
-
[33]
Rücklé, S
A. Rücklé, S. Eger, M. Peyrard, and I. Gurevych. Concatenated p-mean word embeddings as universal cross-lingual sentence representations. CoRR, abs/1803.01400, URL http://arxiv.org/abs/1803.01400. 2018
2018 arXiv
-
[34]
Peters, M
M.E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer. Deep contextualized word representations. In Proc. of NAACL, 2018
2018
-
[35]
Merity, N
S. Merity, N. Keskar, R. Soche Regularizing and Optimizing LSTM Language Models. URL https://arxiv.org/abs/1708.02182
-
[36]
Howard and S
J. Howard and S. Ruder. Fine-tuned language models for text classification. CoRR, abs/1801.06146, 2018
2018 arXiv
-
[37]
Devlin, M
J. Devlin, M. Chang, K. Lee, K. Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805. 2018
2018 arXiv
-
[39]
Cummins, S
N. Cummins, S. Scherer, J. Krajewski, S. Schnieder, J. Epps, and T.F. Quatieri, A review of depression and suicide risk assessment using speech analysis, Speech Communication, vol. 71, pp. 10–49, 2015
2015
-
[2016]
In Proceedings of the 6th International Workshop on Audio/Visual Emotion Challenge
Decision Tree Based Depression Classification from Audio Video and Language Information. In Proceedings of the 6th International Workshop on Audio/Visual Emotion Challenge. ACM, 89–96
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.