{"id":"d450100d-2b15-41c7-bb3e-199c4065303c","arxiv_id":"2505.14356","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"An LLM prompted with transcripts, laughter, turn-taking, backchannel, and emotion statistics predicts Big Five personality traits from two-person phone calls, agreeing with human ratings more closely than three text-only baselines.","lead":"This paper builds a personality prediction system for two-person phone conversations by combining transcripts, laughter, turn-taking, backchannels, and emotions into prompts for a large language model. The model's personality ratings matched human listener judgments more closely than three text-only baselines in the reported experiments.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline comparison rests on a human-label benchmark whose reliability is never documented; with no inter-rater statistics or per-dialogue rater counts, the small correlations in Table 3 cannot establish 'stronger alignment with human judgments.'","rationale":"The paper's central claim is explicitly about agreement with human judgments, so the trustworthiness of those judgments is the most load-bearing element. The reader's weakest_assumption identifies exactly this. I agree with that assessment and sharpen it: the problem is not merely absence of inter-rater agreement but the lack of any information about how many raters contribute to each ground-truth label. With 190 speakers and 20 raters, per-speaker ratings could range from one rater per dialogue to several, and the effective reliability of the criterion is unknown. The ablation table's internal inconsistency (e.g., in Table 4 the full configuration has Trend 0.186 versus 0.277 for Emotion/Sentiment-only, and Cosine 0.503 versus 0.551 for Samples-only) is a real reporting problem, but it does not directly undercut the comparison to baselines in Table 3; the human-label benchmark is the load-bearing measurement for the abstract's 'stronger alignment' claim. A concrete reliability check and a bootstrap significance test would settle whether the observed differences are meaningful. Since this concern is real but does not by itself disprove the claim, the verdict remains CONDITIONAL rather than moving to ACCEPT or REJECT.","tokens_in":8832,"tokens_out":6038,"duration_ms":66655,"concrete_test":"Obtain the per-rater, per-speaker raw ratings behind Table 3 (the project page says the annotated dataset and supplementary materials are available) and compute inter-rater reliability, e.g., ICC(2,k) or Krippendorff's alpha, per Big Five dimension and per dialogue. Then recompute Table 3 correlations using only items with at least two raters and consensus labels, and bootstrap 95% confidence intervals for the Ours-vs-best-baseline difference. If ICC is below roughly 0.4, or if the confidence interval includes zero, or if the ranking flips, the claimed stronger alignment with human judgments is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is comparative: the proposed LLM pipeline with PersonaTAB features aligns better with human judgments than three text-only baselines (Table 3: avg Corr 0.183 vs 0.096, -0.016, -0.053; Cosine 0.503). The criterion for 'human judgments' is the MUSHRA listening-test labels described only in Sec 4.2. The paper never reports how many of the 20 raters judged each 90-second dialogue, whether raters were screened for consistency, or any inter-rater agreement statistic (ICC, Krippendorff's alpha, etc.). Personality ratings from a single listener on 90 seconds of dialogue are notoriously noisy, and self-other agreement in personality research is typically modest. If labels are averaged over few raters with low agreement, the observed correlations are attenuated and unstable, and the margin over BERT (0.087 on avg Corr) may be within noise. The absence of confidence intervals or significance tests makes 'stronger' a bare point estimate. This is not an internal inconsistency, but it is the load-bearing measurement for the headline claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes PersonaTAB, a pipeline that takes two-channel audio recordings of fully-duplex telephone conversations (a 95-conversation subset of Fisher) and automatically produces a dialog dataset with timestamps, laughter labels, turn/overlap/backchannel/interjection labels, and emotion/sentiment annotations. It then uses GPT-4o to predict Big Five personality traits from speaker attributes drawn from these annotations, including textual samples, emotion/sentiment percentages, and conversational statistics such as turn frequency, laughter, backchannels, and interjections. The system is evaluated against three text-only baselines (LM, BERT, MiniLM) using two metrics: a 'Trend' score based on human-derived expected attribute–trait associations, and correlations/cosine similarity with human ratings from a MUSHRA-style listening test. The paper reports that the proposed system achieves stronger alignment with human judgments than the baselines, and includes an ablation study of the attribute categories.","tokens_in":9010,"tokens_out":6550,"duration_ms":59310,"significance":"If the central claim is supported, the paper offers a useful and relatively complete pipeline for personality prediction from speech-only dialog data, integrating acoustic, behavioral, and textual cues in a way that prior text-only systems do not. The dataset preprocessing and the use of an LLM to consume structured conversational statistics are of interest to the spoken dialog and personality computing communities. However, the strength of the empirical evidence is currently limited by missing reliability and uncertainty analyses for the human ground-truth labels, an under-specified evaluation metric, and an ablation conclusion that is contradicted by the reported numbers. The pipeline itself is a reasonable contribution, but the validation needs substantial strengthening before the headline claim can be accepted.","major_comments":[{"comment":"The central comparative claim rests on correlations between model predictions and human labels from the listening test, yet the paper does not report the number of raters per 90-second dialogue, any rater-screening procedure, or any inter-rater agreement statistic (e.g., ICC, Krippendorff's alpha). With average correlations around 0.18 and a margin over BERT of only 0.087 on avg Corr, the reported difference may be within noise. Please report per-item rater counts, inter-rater reliability, and bootstrap confidence intervals or significance tests for the differences in Table 3.","section":"Sec. 4.2, Sec. 5.1, Table 3"},{"comment":"The text states that \"the full prompt consistently outperforms other conditions across all metrics,\" but Table 4 does not support this. The condition with only Emotion and Sentiment (\"- - O O\") has Trend=0.277, higher than the full prompt's 0.186; the condition with Basics, Emotion, and Sentiment (\"- O O O\") has Trend=0.210; and the condition with Samples and Basics (\"O O - -\") has Cosine=0.603, higher than the full prompt's 0.503. The ablation conclusion is therefore internally inconsistent with the reported results and must be corrected or reinterpreted.","section":"Sec. 5.2, Table 4"},{"comment":"The Trend metric is not defined precisely enough to be reproducible or interpretable. The description \"we computed a weighted sum of correlation coefficients between personality traits and raw speaker attributes\" does not specify which correlation coefficient is used (Pearson vs. Spearman), the observations over which it is computed, how the weights are normalized from Table 1, or exactly how negative trends are handled. Additionally, because the weights are derived from Table 1, which is itself the output of the human survey (Sec. 4.2), the Trend metric is not an independent measure of human alignment; it measures agreement with the same human priors used to construct the metric. Please provide a formal definition and discuss this dependency.","section":"Sec. 5.1, Eq./text preceding Table 2"},{"comment":"The cosine similarity metric is undefined. It is not clear what vectors are compared (e.g., the predicted five-dimensional alignment scores vs. the human-rated scores for each speaker, averaged across speakers) or why BERT yields a negative cosine similarity of -0.543 while LM's entry is simply \"-\" with no explanation. Please specify the exact computation and either provide LM's value or explain its omission.","section":"Sec. 5.1, Table 3"}],"minor_comments":[{"comment":"There are two obvious typos: \"These agents are generally into two models\" should read \"divided into two models,\" and \"in the blankets\" should read \"in the brackets.\"","section":"Sec. 1, Sec. 2.1"},{"comment":"The footnotes defining \"overlappee\" and \"overlapper\" are awkward; these terms would be better defined directly in the main text.","section":"Sec. 2.2"},{"comment":"The paper describes the evaluation as \"two MUSHRA tests\" but does not explain how the standard MUSHRA protocol (e.g., reference condition, anchors, or grading scale) was adapted to personality alignment. Please clarify.","section":"Sec. 4.2"},{"comment":"The statement \"We averaged five responses to yield reliable alignment scores\" does not specify the GPT-4o temperature or the variance across the five repeats. Reporting the standard deviation or agreement across repeats would help assess reliability.","section":"Sec. 5.1"},{"comment":"LM's cosine similarity is reported as \"-\"; either provide the value or explain why it is not reported, since all other entries are numeric.","section":"Table 3"},{"comment":"Reference [24] is cited with the author field \"Anonymous\" and venue \"Submitted to ICLR, under review\"; this is not an acceptable citable reference and should be replaced with a published version or removed.","section":"References"},{"comment":"The 700 ms sentence-merge threshold is justified by reference [30], which reports language generation latencies exceeding 600 ms; the connection between turn-taking latency and sentence boundary segmentation should be clarified, as these are distinct phenomena.","section":"Sec. 3.2, Step 3"},{"comment":"The sentence reporting deviations for \"surprise\", \"angry\", and \"sadness\" (values -0.166, 0.0176, 0.079) is not tied to any table or equation; please explain the origin of these numbers or remove them.","section":"Sec. 5.1"}],"recommendation":"major_revision","confidential_remarks":"The paper's novelty claim of being \"the first to integrate interjection behaviors, acoustics, and other conversational traits with textual data for personality prediction\" is not substantiated by a comprehensive literature review; I would encourage the authors to survey existing speech-based personality recognition work and temper the claim if necessary. The evaluation size (95 dialogues, 20 raters) is modest, and the missing reliability statistics are a concern for a venue with standard empirical rigor. The central idea is promising, but the paper needs a corrected ablation analysis and more rigorous evaluation before it can be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper is a useful engineering contribution with a genuinely new integration of fully-duplex conversational cues into LLM-based personality prediction, and the preprocessing pipeline alone is worth something. But the headline comparison to human judgments rests on a benchmark whose reliability is never documented, and the paper's own ablation undercuts its 'consistently outperforms' claim. It deserves a serious referee, with the expectation of major revision.\n\nWhat's actually new: prior LLM personality work (e.g., Ji et al.) is text-only; here they convert two-channel audio into structured dialogue annotations—overlaps, backchannel types, laughter tokens, interjections, emotion/sentiment per response—and feed those as prompts to GPT-4o. The dataset construction is a real contribution; the Table 1 human survey of expected trait-behavior trends is a useful resource. They also ship code and data on a project page, which is more than many papers do.\n\nWhere it's soft: the central claim 'stronger alignment with human judgments' is supported by correlations around 0.18 with no confidence intervals, no significance tests, and no inter-rater reliability. Twenty MUSHRA listeners rated 90-second dialogues, but the paper never says how many raters judged each dialogue or whether raters were screened. Personality ratings from a single listener on 90 seconds are notoriously noisy; the margin over BERT (0.087 average correlation) could be within noise. Also, the Trend metric in Table 2 is partially circular because its weights come from the same human survey used to define the expected trends. Table 3 is an independent benchmark, so this is not fatal, but the Trend metric shouldn't be advertised as independent validation. The ablation table is internally inconsistent with the text: the full prompt does not consistently outperform—Emotion/Sentiment alone gives Trend 0.277 vs 0.186 for the full prompt, and Samples+Basics gives Cosine 0.603 vs 0.503. That needs to be fixed or the text toned down. Minor: thresholds like the 700ms merge gap are plausible but arbitrary, and there's no sensitivity analysis.\n\nOverall: the idea is sound, the implementation is careful, and the resource is useful. The evaluation is the weak link, and it's fixable. For a spoken-dialogue or affective-computing audience, this is worth citing and worth refereeing, but the authors need to report inter-rater agreement, add confidence intervals, and correct the ablation claim.","headline":"Useful pipeline and resource, but the headline comparison to human judgments is not statistically supported; worth refereeing with major revisions.","tokens_in":9596,"tokens_out":2356,"would_cite":true,"duration_ms":21276,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a cue-rich pipeline combining ASR transcripts with laughter, backchannel, interjection, and emotion/sentiment cues lets an LLM predict Big Five personality from fully-duplex speech dialogues with stronger alignment…","keywords":["personality prediction","Big Five","speech dialogue","fully-duplex conversation","large language models","backchannel classification","laughter detection","MUSHRA evaluation"],"falsifier":"Re-run the evaluation with a larger, screened panel of listeners and report inter-rater agreement, while also checking whether the full prompt's cosine similarity beats every ablated subset; Table 4 already shows the Samples-only condition at 0.551 versus 0.503 for the full prompt, so the claim that the full prompt consistently outperforms all conditions is directly testable.","tokens_in":8559,"feed_emoji":"🎙️","tokens_out":8793,"duration_ms":80980,"temperature":0.7,"pith_summary":"The paper tries to establish that conversational personality can be read from speech-only recordings by extracting structured behavioral cues and feeding them to an LLM. It introduces a pipeline that converts two-channel recordings, where both speakers can talk at once, into timed dialogue transcripts with laughter tokens, turn and overlap labels, backchannel types, and emotion and sentiment percentages. The central claim is that this cue-rich prompt yields Big Five predictions more aligned with human listener judgments than three text-only baselines, with an average correlation of 0.183 and average cosine similarity of 0.503. If correct, this gives conversational agents a way to infer personality from natural interaction without needing any personality-labeled speech dataset at training time. The paper also positions the work as the first to integrate interjection behaviors, acoustics, and other conversational traits with textual data for personality prediction.","feed_headline":"Speech cues let LLMs beat text-only personality prediction","feed_subtitle":"A pipeline adds laughter, turns, backchannels, and emotions to LLM prompts, aligning Big Five scores with human judgments.","key_machinery":"The central object is the PersonaTAB prompt, a text representation of a speaker assembled from four attribute categories: Emotion and Sentiment percentages, Basics statistics that include turns, speaking duration, laughter, backchannels, and interjections normalized into five relative groups, and sampled responses. The prompt is fed to an LLM that returns five-level alignment scores for each Big Five trait. The pipeline that builds the prompt, using ASR word timestamps, 700 ms response boundaries, overlap detection, and LLM-based backchannel classification, is the enabling mechanism because it turns raw audio into features an LLM can reason over.","core_discovery":"The discovery is that combining textual, acoustic, and behavioral cues within a single LLM prompt produces personality scores that track human evaluations more closely than text-only models. The pipeline first reconstructs dialogue structure from raw two-channel audio: word-level timestamps from ASR, laughter tokens, response boundaries from silence gaps, overlap and interjection labels, and LLM-classified backchannels. It then summarizes each speaker with emotion and sentiment percentages, relative-group statistics for turns, speaking duration, laughter, backchannels, and interjections, plus sampled responses, and lets an LLM rate each Big Five dimension on a five-level alignment scale. The paper reports that this system outperforms the text-only baselines on both trend alignment and ground-truth correlation, interpreting these results as evidence that conversational behavior carries personality signal beyond what text alone provides.","pith_inferences":["If the reported alignment transfers, the same prompt template could label other two-channel or diarized speech corpora, effectively turning personality prediction into a data-annotation service for training personality-conditioned agents.","The relative-group normalization of numeric statistics, which converts counts into labels such as Normal, Many, Few, and Very Many, is a generally reusable way to make continuous conversational features legible to an LLM and could apply to other social attributes like rapport or dominance.","The per-trait results suggest that different cues matter differently for different traits, so a system that weights cues per trait rather than using a single prompt might improve on the reported correlations."],"forward_implications":["Personality-aware conversation agents could be built from the pipeline's labels without needing pre-annotated personality datasets, since the labels are derived automatically from raw speech.","The identified cue list gives concrete behavioral indicators for each trait, such as extraversion aligning with frequent turns and laughter, and neuroticism aligning with anger and fear emotions.","The reported gains over text-only baselines suggest that speech-specific signals, including laughter, backchanneling, and interjections, carry personality information that text-only models miss.","The resulting dataset, with timestamps, response types, and emotion and sentiment labels, is reusable for other conversational-analysis tasks beyond personality prediction."],"supporting_citations":[{"why":"It supplies the language-model baseline trained on Reddit comments, one of the text-only systems the proposed method must beat.","marker":"[16]"},{"why":"It is the prior demonstration that LLMs can infer personality from text, which this work extends by adding conversational behavior and acoustic cues.","marker":"[18]"},{"why":"It provides ASR with word-level timestamps, the foundation for both transcripts and response boundary construction.","marker":"[28]"},{"why":"It provides the laughter detector whose timestamps yield the laughter tokens integrated into the transcripts.","marker":"[29]"},{"why":"It supplies the finding on language generation latency that justifies the 700 ms response-boundary threshold.","marker":"[30]"},{"why":"It supplies the two-channel telephone conversation corpus from which the 95-dialogue subset and human evaluations are drawn.","marker":"[37]"},{"why":"It provides the BERT-based multi-label text baseline used in comparison.","marker":"[38]"},{"why":"It provides the MiniLM-based text baseline used in comparison.","marker":"[39]"}],"fun_headline_variants":["LLMs read laughter, turns, emotion to nail personalities","Speech + text cues in LLMs outpredict text alone for personality","Personality from voice: LLMs beat text with laughter, turns, emotion","Beyond words: LLMs trained on speech cues nail personality traits"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the 20 human listeners' personality ratings are reliable enough to serve as ground truth, and if those ratings are noisy or idiosyncratic, the reported correlations lose their benchmark value.","fun_headline_variants_meta":{"raw":{"variants":["LLMs read laughter, turns, emotion to nail personalities","Speech + text cues in LLMs outpredict text alone for personality","Personality from voice: LLMs beat text with laughter, turns, emotion","Beyond words: LLMs trained on speech cues nail personality traits"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000433,"raw_usage":{"total_tokens":2155,"prompt_tokens":841,"completion_tokens":1314,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":457,"completion_tokens_details":{"reasoning_tokens":1240}},"tokens_in":457,"tokens_out":1314,"duration_ms":51109,"temperature":1.0,"reasoning_tokens":1240,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:35:33.937722+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the evaluation with a larger, screened panel of listeners and report inter-rater agreement, while also checking whether the full prompt's cosine similarity beats every ablated subset; Table 4 already shows the Samples-only condition at 0.551 versus 0.503 for the full prompt, so the claim that the full prompt consistently outperforms all conditions is directly testable.","supporting_citations":[{"cited_title":"Listeners as co-narrators,","cited_arxiv_id":null,"evidence_quote":"It supplies the language-model baseline trained on Reddit comments, one of the text-only systems the proposed method must beat."},{"cited_title":"Personality in interaction: how the big five relate to the reception of interactive narratives,","cited_arxiv_id":null,"evidence_quote":"It is the prior demonstration that LLMs can infer personality from text, which this work extends by adding conversational behavior and acoustic cues."},{"cited_title":"A novel personality detection method based on high-dimensional psycholinguistic features and improved distributed gray wolf optimizer for feature selection,","cited_arxiv_id":null,"evidence_quote":"It provides ASR with word-level timestamps, the foundation for both transcripts and response boundary construction."},{"cited_title":"Modeling dyadic conversations for personality infer- ence,","cited_arxiv_id":null,"evidence_quote":"It provides the laughter detector whose timestamps yield the laughter tokens integrated into the transcripts."},{"cited_title":"Personality structure: Emergence of the five- factor model,","cited_arxiv_id":null,"evidence_quote":"It supplies the finding on language generation latency that justifies the 700 ms response-boundary threshold."},{"cited_title":"Timing in turn-taking and its im- plications for processing models of language,","cited_arxiv_id":null,"evidence_quote":"It supplies the two-channel telephone conversation corpus from which the 95-dialogue subset and human evaluations are drawn."},{"cited_title":"Universals and cultural variation in turn-taking in conversation,","cited_arxiv_id":null,"evidence_quote":"It provides the BERT-based multi-label text baseline used in comparison."},{"cited_title":"Sequence organization in interaction: A primer in conversation analysis,","cited_arxiv_id":null,"evidence_quote":"It provides the MiniLM-based text baseline used in comparison."}],"review_version":1}