{"id":"9db128c0-d9b0-4c50-a339-ff07a36baad5","arxiv_id":"2411.10761","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Generic open-source LLMs can classify speakers, engaged activities, language skill levels, and age ranges in ASD child-adult clinical transcripts, and sometimes outperform non-expert human raters.","lead":"This paper tests whether off-the-shelf open-source LLMs can understand transcripts of clinical conversations between adults and children with autism. It finds the models can often match or beat non-expert humans at recognizing activities and language skills from these transcripts.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claim that generic LLMs 'often surpass' non-expert humans rests on a tiny, statistically unquantified comparison: 10 transcripts, no confidence intervals, and one F1=1.000 that may reflect class imbalance.","rationale":"The paper's central claim is comparative: generic LLMs 'often surpass' non-expert human evaluators. The evidence for this is Table 6, computed on a very small convenience sample (10 Remote-NLS transcripts for activity/language; 50 utterance examples for speaker classification), with no uncertainty quantification. The language-skill result (LLM F1=1.000 vs human average 0.592) is the strongest evidence in the paper, but with n=10 sessions it can be driven by one annotator's difficulty or by class imbalance; no confusion matrix or class frequencies are provided. A paired bootstrap or permutation test would settle whether the difference is real. Without it, the headline is not justified. I considered the reader's alternative concern, that expert transcripts are incomplete because non-verbal cues are absent (Section 5.6): that is a real limitation for clinical deployment and should be stated more prominently, but it does not invalidate the more modest claim that LLMs can analyze transcripts. The statistical gap does invalidate the 'surpassing' phrasing. I also noticed the internal inconsistency in Section 5.2, which cites an ADOSMod3 activity F1 of 0.961 that does not match Table 3's best value of 0.984; this should be fixed but is not the central issue. The right verdict remains CONDITIONAL: with per-sample statistics and a more carefully scoped abstract, the claim could be supported, but as written the strongest claim is not yet established.","tokens_in":9356,"tokens_out":6290,"duration_ms":63538,"concrete_test":"Reconstruct per-transcript predictions and gold labels for the 10 Remote-NLS sessions used in Table 6, for both the best LLM and each non-expert annotator. Run a paired bootstrap (10,000 resamples) on the F1 difference (LLM minus human average) and report the 95% confidence interval, plus a McNemar or exact binomial test on the language-skill predictions. If the confidence interval includes zero, or if the LLM's F1=1.000 is explained by predicting the majority class, the 'surpassing' claim must be downgraded in the abstract and conclusion.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's central claim is comparative: generic LLMs are 'highly capable' and 'often surpassing the performance of non-expert human evaluators.' The only direct support is Table 6, based on a small convenience sample: 10 Remote-NLS transcripts for activity/language predictions and 50 utterance examples for speaker classification, with non-experts being IRB-trained co-authors rather than a sampled population. No confidence intervals, significance tests, per-transcript/per-annotator variance, or class frequencies are reported. The language-skill row is the most fragile: the best LLM achieves F1=1.000 on 10 transcripts while the human average is 0.592. With n=10, this can arise from class imbalance or from averaging over a few atypical annotators, and the difference may be statistically indistinguishable from zero. Section 5.5 does not report the label distribution or individual human scores. Thus the strong phrasing in the abstract is not supported by the evidence as presented. A defensible claim would be limited to 'on this 10-session sample, the best LLM scored higher on language-skill F1 than the non-expert average.' The gold-standard incompleteness noted in Section 5.6 is a separate deployment concern; the decisive weakness here is the absent statistical grounding of the human comparison.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper evaluates six locally deployable open-source LLMs on four tasks derived from child-adult dyadic interaction transcripts in two ASD-relevant datasets (Remote-NLS and ADOSMod3): utterance-level child/adult speaker classification, session-level engaged-activity recognition, language-skill classification, and binarized age-trait prediction. The authors also report a small human comparison using IRB-trained co-authors on 50 utterance-classification examples and 10 Remote-NLS transcripts, and they find that the best LLM outperforms non-expert humans on activity recognition and language-skill prediction while humans do slightly better on utterance classification. The paper additionally explores chain-of-thought prompting, majority voting, and documents hallucination and prompt-confusion failure modes.","tokens_in":9604,"tokens_out":4613,"duration_ms":65484,"significance":"This is an early exploration of generic LLMs on clinically relevant child-adult interaction transcripts, and the study is transparent about models, prompts, and compute. Strengths include the use of locally deployable models for privacy-sensitive data, a fine-tuned RoBERTa baseline for utterance classification, a two-dataset design, and explicit discussion of hallucinations and limitations. However, the headline comparative claim rests on a very small, statistically unquantified human sample and on post hoc selection of the best LLM, and several reported numbers do not match the tables. If these issues are addressed, the paper can serve as a useful pilot for LLM-assisted clinical observation analysis, but in its current form the central claim is not sufficiently supported.","major_comments":[{"comment":"The central comparative claim that generic LLMs 'often surpass' non-expert human evaluators is not statistically grounded. The comparison uses only 10 transcripts for activity and language skills and 50 utterances for speaker classification, and the paper reports no confidence intervals, significance tests, per-transcript or per-annotator variance, or class frequencies. The language-skill F1=1.000 for the best LLM on 10 transcripts is especially fragile, since it could arise from class imbalance or from a few atypical annotators; with n=10 the difference from the human average of 0.592 may be statistically indistinguishable from zero. Moreover, the 'best LLM' is selected after inspecting all six models on the same test samples, which inflates the reported performance. The abstract should be tempered to 'on this small sample, the best LLM scored higher than the non-expert average on language skills' unless the authors add appropriate statistical characterization, such as bootstrap confidence intervals, per-annotator scores, and label distributions.","section":"§5.5, Table 6"},{"comment":"Several reported maxima do not match the results in Table 3. The text states that the highest activity-recognition F1 is 0.961 in ADOSMod3, but the table shows a maximum of 0.984 (Qwen1.5-14B); the text states that the highest age-prediction F1 is 0.748 in ADOSMod3, but the table's maximum is 0.688. These discrepancies prevent readers from verifying the main session-level results and must be corrected or explicitly explained.","section":"§5.2, Table 3"},{"comment":"The non-expert human baseline is not characterized in a way that supports the comparative claim. The raters are IRB-trained co-authors selected by convenience rather than a sampled population, and the number of raters per item and the inter-rater agreement are not reported. Without per-rater variability, the 'non-expert human' performance is a single averaged point that may depend strongly on a few individuals, making the comparison with LLMs difficult to interpret.","section":"§4.1, §5.5"},{"comment":"The Remote-NLS activity-recognition task is multilabel (select 1 to 5 primary activities from 13 options), but the paper does not specify how F1 is computed for this multilabel setting, such as macro versus micro averaging or how variable-size predictions are matched to the gold standard. This omission matters because different aggregation choices can materially change the reported F1 values; please state the exact computation.","section":"§3, §5.2"}],"minor_comments":[{"comment":"The abstract says LLMs are 'often surpassing' non-expert humans, but Table 6 shows that humans outperform the best LLM on utterance classification; 'often' is supported only for two of the four tasks and should be rephrased to accurately reflect the scope.","section":"Abstract, §5.5"},{"comment":"Table 3 appears to contain embedded option-list text from the prompts ('1) Conversation only; 2) Figure play; ...'), which is likely a layout artifact and makes the table difficult to read; the table should be regenerated so that only numerical results appear.","section":"Table 3"},{"comment":"The caption states that blue, red, and purple denote repetitions, language skills, and engaged activity, but the transcript excerpt in the figure does not visibly use these colors; please add colored markup or revise the caption.","section":"Figure 1"},{"comment":"It is unclear whether the human raters saw the same prompt templates and option lists as the LLMs; if they did not, the comparison is not apples-to-apples and this should be stated.","section":"§4.1, §5.5"}],"recommendation":"major_revision","confidential_remarks":"This manuscript reads as a workshop-style pilot rather than a completed journal study. The main barrier is the statistical fragility of the human-LLM comparison, which is the basis of the abstract's headline claim; the text/table inconsistencies in §5.2 add to the need for a thorough revision. I would encourage the editor to send it back with clear requests for confidence intervals, label distributions, per-annotator results, and a pre-specified or at least honestly described model-selection procedure, rather than treating the current numbers as final."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one thing to know: this is a genuinely first look at local open-source LLMs on four transcript-analysis tasks in ASD clinical sessions, and it is transparent about prompts and models. But the central claim that LLMs \"often surpass\" non-expert humans is built on 10 transcripts and 50 utterances, with no confidence intervals or significance tests. Take the headline with a grain of salt.\n\nWhat's new: they define four well-motivated tasks—speaker classification, activity recognition, language-skill level, and age range—on two real clinical datasets (Remote-NLS and ADOSMod3), use only locally deployable models under 15B for IRB reasons, and report per-model F1 scores. They also compare against fine-tuned RoBERTa for the utterance task, which is a sensible baseline. They document hallucinations and prompt confusions, and they explicitly note the lack of multimodal information and bias analysis. That's more honest than most LLM papers.\n\nThe soft spots are real, though. The human comparison in Table 6 is the load-bearing weakness. Ten transcripts is far too few to claim superiority; with n=10 and a binary label, a single model can get a perfect F1 by chance if the class distribution is skewed. They also chose the best LLM post hoc from six models, which is a form of test-set selection. No per-transcript variation, no CIs, no inter-annotator agreement. Additionally, the text says ADOSMod3 activity top F1 is 0.961 but Table 3 shows 0.984; that discrepancy needs fixing. The human \"non-experts\" are co-authors, which limits generalizability. Finally, as they themselves note in Section 5.6, activity recognition often depends on non-verbal cues absent from transcripts, so the transcript-only evaluation may overestimate real-world usability.\n\nWho this is for: researchers working on LLM-assisted clinical assessment, especially in autism or child-adult interaction. They'll get a reasonable task setup and initial numbers, but should not treat the performance as established. With a bigger human sample, pre-registered model selection, and confidence intervals, this could be a solid benchmark paper. As is, it deserves a serious referee, but the referee should push for those fixes.\n\nRecommendation: send it to peer review with a request for major revisions, mainly around statistics and internal consistency. It's a legitimate contribution, just not yet a proof of LLM superiority.","headline":"First benchmark of local LLMs on ASD clinical transcripts, but the 'better than non-experts' claim rests on 10 transcripts and no statistics.","tokens_in":10115,"tokens_out":2542,"would_cite":false,"duration_ms":23786,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Generic open-source LLMs can analyze child-adult clinical transcripts involving children with autism, often beating non-expert human raters on activity and language-skill recognition.","keywords":["large language models","autism spectrum disorder","child-adult interaction","clinical observation","activity recognition","language skill prediction","transcript analysis","speaker classification"],"falsifier":"The simplest check is to take a random subset of Remote-NLS sessions, have clinicians relabel activities from the videos, and compare the best LLM's transcript-only predictions to those video-grounded labels; if agreement drops well below the reported F1, the transcript-only ceiling overstates what is usable in real observation.","tokens_in":9193,"feed_emoji":"🤖","tokens_out":6405,"duration_ms":56164,"temperature":0.7,"pith_summary":"The paper tries to establish that generic, locally run large language models can make sense of long, messy child-adult conversations recorded in autism diagnostic and natural-language-sampling sessions. Across four tasks, from telling which utterances come from the child to inferring the activity and the child's language level from a full transcript, the best open models match or beat non-expert human annotators. The authors argue this points toward a scalable assistive tool that could help clinicians segment and evaluate clinical interactions, while noting that hallucinations and prompt confusion are real obstacles. The paper's own evaluators also report that activity recognition often depends on non-verbal cues absent from the text, which is a key caveat for interpreting the results.","feed_headline":"LLMs beat non-experts on autism clinical transcript tasks","feed_subtitle":"Local open-source models identify activities and language levels in ASD clinical transcripts, often beating non-experts.","key_machinery":"The load-bearing object is a session-level prompt template that feeds an entire expert transcript of a child-adult conversation into a local open-weight LLM (Mistral-7B, LLaMa 2/3, Qwen) with a small set of multiple-choice options. Each prompt converts a clinical annotation job into a constrained text-generation task, letting the model draw on conversation context to decide speaker, engaged activity, language level, and age range. The mechanism that carries the argument is this prompt-plus-model setup: no fine-tuning, no cloud API, just local inference on transcripts.","core_discovery":"The central claim is that generic LLMs, without any clinical fine-tuning, are highly capable of analyzing child-adult dyadic interactions in ASD-relevant clinical observations, often surpassing non-expert human evaluators. On the Remote-NLS naturalistic dataset, the best LLM scored 0.448 vs 0.332 for non-experts on activity recognition and 1.000 vs 0.592 on language-skill recognition in F1; on the ADOSMod3 semi-structured data, activity/observation recognition reached 0.984 with the best model. The paper also finds that chain-of-thought prompting hurts performance and that majority voting across models helps. It identifies hallucinations and prompt confusion as the main error sources.","pith_inferences":["Because the gold labels come from expert transcripts and the paper's own human evaluators say non-verbal behavior is missing, the reported F1 likely overstates what a transcript-only LLM would deliver if tested against video-grounded clinician judgments.","A direct test would run the same prompts on ASR-plus-diarization transcripts; the gap between those scores and the expert-transcript scores would quantify how much the claimed capability depends on clean input.","The age-prediction weakness on Remote-NLS children under 8 (best F1 0.487) suggests the models rely on verbal complexity, which is sparse in minimally verbal children; adding acoustic or visual features might be required for reliable trait inference."],"forward_implications":["Clinicians could use local open-source LLMs to pre-label activities and language levels from transcripts, cutting the manual labor that non-expert evaluators found draining.","In naturalistic home sessions (Remote-NLS), the best LLM matched or beat non-experts on activity (0.448 vs 0.332 F1) and language skills (1.000 vs 0.592 F1), so the models could serve as a scalable second reader rather than a replacement for experts.","Simple 0-shot multiple-choice prompts outperform chain-of-thought prompts in this setting, so the practical protocol is to avoid COT when deploying.","Combining the three best models by majority vote raises F1 further on session-level tasks, giving an ensemble strategy that needs no extra training.","Hallucinated and confused outputs still appear, meaning any real deployment needs output validation before use in assessments."],"supporting_citations":[{"why":"supplies the Remote-NLS dataset of 89 home-based child-adult interactions used for the naturalistic evaluations","marker":"[5]"},{"why":"defines the ADOS protocol and the ADOSMod3 dataset of child-clinician interactions","marker":"[13]"},{"why":"provides the spoken-language benchmark stages (pre-verbal, first words, word combinations) used as Remote-NLS labels","marker":"[14]"},{"why":"supplies the non-expert human evaluation methodology the paper adapts for comparison","marker":"[7]"},{"why":"the RoBERTa fine-tuned baseline for child-adult utterance classification","marker":"[11]"},{"why":"one of the evaluated open-weight LLMs (Mistral-7B), the best performer on several tasks","marker":"[8]"},{"why":"provides the LLaMa 2 chat models evaluated as baselines","marker":"[15]"},{"why":"provides the Qwen chat models evaluated as baselines","marker":"[1]"}],"fun_headline_variants":["Generic LLMs outscore non-experts on autism transcript analysis","LLMs beat humans in autism clinical interaction analysis","Unfine-tuned LLMs excel at ASD clinical observation tasks","LLMs surpass non-experts in child-adult autism interactions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that the expert-written transcripts and their activity and language labels are complete and correct, even though the paper's own non-expert feedback says these judgments often depend on non-verbal cues not present in the text.","fun_headline_variants_meta":{"raw":{"variants":["Generic LLMs outscore non-experts on autism transcript analysis","LLMs beat humans in autism clinical interaction analysis","Unfine-tuned LLMs excel at ASD clinical observation tasks","LLMs surpass non-experts in child-adult autism interactions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000164,"raw_usage":{"total_tokens":1186,"prompt_tokens":824,"completion_tokens":362,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":440,"completion_tokens_details":{"reasoning_tokens":294}},"tokens_in":440,"tokens_out":362,"duration_ms":4580,"temperature":1.0,"reasoning_tokens":294,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T19:19:48.941980+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"The simplest check is to take a random subset of Remote-NLS sessions, have clinicians relabel activities from the videos, and compare the best LLM's transcript-only predictions to those video-grounded labels; if agreement drops well below the reported F1, the transcript-only ceiling overstates what is usable in real observation.","supporting_citations":[{"cited_title":"Remote natural language sampling of parents and children with autism spectrum disorder: Role of activity and language level","cited_arxiv_id":null,"evidence_quote":"supplies the Remote-NLS dataset of 89 home-based child-adult interactions used for the naturalistic evaluations"},{"cited_title":"The autism diagnostic observation sched- ule—generic: A standard measure of social and communication deficits associated with the spectrum of autism","cited_arxiv_id":null,"evidence_quote":"defines the ADOS protocol and the ADOSMod3 dataset of child-clinician interactions"},{"cited_title":"Defining spoken language benchmarks and selecting measures of expressive language development for young children with autism spectrum disorders","cited_arxiv_id":null,"evidence_quote":"provides the spoken-language benchmark stages (pre-verbal, first words, word combinations) used as Remote-NLS labels"},{"cited_title":"understanding","cited_arxiv_id":null,"evidence_quote":"supplies the non-expert human evaluation methodology the paper adapts for comparison"}],"review_version":1}