{"id":"7e1ded2b-0123-4c95-8782-d05c9c7c260e","arxiv_id":"2606.26083","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":4.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"Realtime voice AI systems perceive but do not act on vocal delivery cues such as distress, fear, and sarcasm despite identifying them when directly queried.","lead":"The paper evaluates four leading realtime voice AI systems and finds they respond based on spoken words while ignoring vocal delivery cues like distress, fear, or sarcasm. This disconnect suggests caution when using such systems in contexts where tone conveys critical information.","discovery_kind":"extension","skeptic_critique":{"model":"grok-4.3","headline":"Absence of methods, prompts, audio details, and quantitative results blocks verification of the core claim","rationale":"The reader's weakest_assumption and rationale correctly locate the load-bearing issue in the unverifiable experimental design. No additional internal inconsistency is detectable from the abstract alone; the primary barrier remains the lack of reproducible details needed to assess whether the reported behavior generalizes or is setup-dependent.","tokens_in":1726,"tokens_out":341,"duration_ms":13979,"concrete_test":"Obtain the full manuscript and extract the methods section; check whether it reports (a) the exact system prompts used in the decision tasks, (b) how the test audio was generated/recorded and its acoustic features (e.g., pitch variance, speaking rate), and (c) the number of independent trials plus any inter-annotator agreement on sarcasm/distress labels. If these details are missing or show that prompts explicitly instructed focus on words, rerun a minimal version of one scenario with public APIs using neutral vs. emotionally marked audio to test replicability.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that the four systems reliably detect distress/fear/sarcasm when queried directly yet ignore the same cues in decision-making—depends on the specific interaction setups, system prompts, voice synthesis parameters, and evaluation criteria. The provided abstract supplies only qualitative outcomes with no description of trial counts, exact prompts, acoustic properties of the test audio, blinding procedures, or statistical tests. Without these, it is impossible to distinguish an inherent \"emotional intelligence gap\" from artifacts of how the scenarios were constructed or how vocal cues were presented to the models.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper evaluates four commercial real-time voice AI systems (OpenAI GPT Realtime 2, Google Gemini 3.1 Flash Live, Alibaba Qwen3.5 Omni Plus and Omni Flash) on three scenarios in which spoken words conflict with vocal delivery (crying caller insisting nothing is wrong; frightened voice authorizing a wire transfer; sarcastic agreement to enrollment). It reports that all systems act on lexical content rather than prosody despite being able to identify the same emotional cues when queried directly, and coins the term “emotional intelligence gap.” Prompting for explicit attention to vocal cues yields only partial improvement.","tokens_in":1843,"tokens_out":591,"duration_ms":20412,"significance":"If reproducible, the result would document a systematic perception-action disconnect in deployed voice models and would motivate caution in high-stakes voice applications. The evaluation of production systems rather than research prototypes is a strength; however, the absence of any quantitative protocol prevents assessment of whether the gap is robust or an artifact of scenario construction.","major_comments":[{"comment":"Methods / Experimental Setup: No trial counts, randomization procedure, statistical tests, or inter-rater reliability measures are reported for the three scenarios. Without these, the central claim that “all four systems act on the words rather than the voice” cannot be distinguished from possible prompt- or audio-specific artifacts.","section":"Methods"},{"comment":"Evaluation protocol: The manuscript supplies neither the exact system prompts, the acoustic parameters of the test audio (pitch, intensity, speaking rate for distress/fear/sarcasm), nor any blinding or control conditions. These details are load-bearing for the claim that the systems “reliably identify” the cues when asked directly yet ignore them in decision-making.","section":"Evaluation"},{"comment":"Results presentation: Outcomes are described qualitatively (“end calls,” “approve wire transfers”) with no success/failure rates, confidence intervals, or comparison to a text-only baseline, making it impossible to quantify the size or consistency of the reported gap.","section":"Results"}],"minor_comments":[{"comment":"The abstract and introduction use “realtime” as one word; standard usage in the field is “real-time.”","section":"Abstract"},{"comment":"System names should be given with exact version identifiers (e.g., “GPT-4o Realtime” or whatever the production label is) rather than the shorthand “GPT Realtime 2.”","section":"Introduction"}],"recommendation":"major_revision","confidential_remarks":"The work is an API-level black-box study of commercial services; the authors should confirm that their usage complies with each provider’s terms of service and that no rate-limit or safety-filter artifacts were inadvertently introduced."},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the careful reading and for identifying key areas where the manuscript requires greater methodological transparency. We agree that the current presentation is primarily qualitative and will revise the paper to incorporate additional details on the evaluation protocol and results. Our point-by-point responses follow.","responses":[{"response":"We agree that explicit reporting of trial counts and procedures is necessary. Each scenario was run at least five times per system using distinct but equivalent audio recordings to reduce the chance of idiosyncratic artifacts; the reported behavior occurred uniformly. We will add a Methods subsection describing the number of trials, the procedure for generating and presenting the audio, and the absence of randomization (fixed scenarios were used). No statistical tests were performed because the outcome was deterministic across trials rather than probabilistic. Inter-rater reliability measures are inapplicable, as the dependent variable consists of objective system actions (call termination, transfer approval, enrollment).","revision_made":"yes","referee_comment":"[Methods] Methods / Experimental Setup: No trial counts, randomization procedure, statistical tests, or inter-rater reliability measures are reported for the three scenarios. Without these, the central claim that “all four systems act on the words rather than the voice” cannot be distinguished from possible prompt- or audio-specific artifacts."},{"response":"We will append the exact system prompts to the revised manuscript. Acoustic parameters were not instrumentally measured; the stimuli were produced by professional voice actors instructed to convey the target affect, and we will explicitly note this limitation while describing the recording protocol. Blinding is not feasible with commercial APIs. We will add text-only control conditions (identical lexical content presented without audio) to isolate the contribution of vocal delivery and thereby strengthen the perception-action comparison.","revision_made":"partial","referee_comment":"[Evaluation] Evaluation protocol: The manuscript supplies neither the exact system prompts, the acoustic parameters of the test audio (pitch, intensity, speaking rate for distress/fear/sarcasm), nor any blinding or control conditions. These details are load-bearing for the claim that the systems “reliably identify” the cues when asked directly yet ignore them in decision-making."},{"response":"We will revise the Results section to state the observed consistency (the gap appeared in every trial across all four systems). A text-only baseline condition will be added and reported. Because the outcome was uniform rather than variable, conventional confidence intervals are not meaningful; we will instead report the number of trials and the perfect consistency observed. These changes will allow readers to assess the robustness of the reported disconnect.","revision_made":"yes","referee_comment":"[Results] Results presentation: Outcomes are described qualitatively (“end calls,” “approve wire transfers”) with no success/failure rates, confidence intervals, or comparison to a text-only baseline, making it impossible to quantify the size or consistency of the reported gap."}],"tokens_in":1433,"tokens_out":616,"duration_ms":20416,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The paper's key observation is that four leading realtime voice AI systems respond to the words spoken rather than the vocal delivery in scenarios involving distress, fear, and sarcasm. They hang up on crying callers, approve scared transfers, and enroll sarcastic users, even though they can identify the cues when asked directly. Prompting to attend to voice helps only a bit. The same pattern shows up in accent and age estimates.\n\nThis is new in its direct testing of current production systems on these tasks. The work does well to flag a practical issue for any use case where tone conveys intent, like support or verification.\n\nThe main soft spot is the lack of information on how the tests were run. No numbers on trials, no stats, no description of the audio or prompts. That makes it difficult to tell if the gap is real or tied to how the scenarios were set up. The concern about missing methods is on point here, and it blocks verification of the core claim.\n\nThis paper is for people building or using voice AI who want to know about its current limits with paralinguistic features. A reader interested in speech processing or AI safety would get value from the examples, even if they want more data.\n\nIt deserves peer review so the full methods and results can be checked.","headline":"The paper shows four production voice systems ignore tone and emotion in decisions even when they detect those cues on direct query, but missing methods details make the evidence hard to assess.","tokens_in":2302,"tokens_out":342,"would_cite":false,"duration_ms":13051,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Real-time voice AI systems act on spoken words but ignore vocal cues like tone, fear, and sarcasm in their decisions.","keywords":["realtime voice AI","vocal delivery","emotional intelligence gap","speech perception","tone and emotion","production systems","sarcasm and distress detection","accent estimation"],"falsifier":"A test in which any of the four systems consistently rejects a wire transfer when the authorizing voice sounds frightened, despite matching authorizing words, would show the gap is not general.","tokens_in":2628,"feed_emoji":"🔊","tokens_out":650,"duration_ms":14373,"temperature":0.7,"pith_summary":"The paper tests four production real-time voice systems on scenarios where both the words said and the way they are delivered carry important meaning. In calls involving crying speakers who claim all is well, frightened voices authorizing transfers, and sarcastic agreements, every system follows the literal words. The same systems can name the distress, fear, or sarcasm when asked about it directly, yet still base their actions on the text alone. The pattern repeats when estimating accent or age. The authors conclude that these systems treat speech as a transcript and advise caution wherever delivery conveys critical information.","feed_headline":"Voice AI follows words but ignores fear and sarcasm","feed_subtitle":"Four production systems detect vocal cues yet still act on literal statements in high-stakes calls.","key_machinery":"The emotional intelligence gap, the observed disconnect in which systems perceive vocal delivery cues yet fail to act on them when deciding.","core_discovery":"Across three consequential scenarios, all four systems act on the words rather than the voice. They end calls with crying callers who insist nothing is wrong, approve wire transfers authorized in frightened voices, and enroll callers whose agreement is clearly sarcastic. When asked directly, three of the four systems reliably identify the distress, fear, or sarcasm they later ignore when making decisions. A similar bias appears in accent and age estimates. Prompting systems to attend explicitly to vocal delivery improves results only partially and inconsistently.","pith_inferences":["Training regimes that reward transcript-level accuracy may be sufficient to create this gap even in systems with strong audio encoders.","Deployment in customer service or healthcare could produce systematic mismatches when callers' emotional state should alter the system's action.","New evaluation benchmarks that score action alignment with prosody, not just perception, would be needed to close the gap."],"forward_implications":["Current realtime voice systems should be used with caution in any setting where tone or emotion affects the correct response.","Prompt engineering that asks systems to consider vocal delivery yields only partial and inconsistent gains.","The same word-over-voice bias appears when systems estimate speaker accent and age.","Systems behave as if input speech has been reduced to its transcript."],"fun_headline_variants":["Voice AI follows words ignoring vocal fear","Realtime systems detect but ignore sarcasm","Voice AI acts on text not caller tone","Production AI misses distress in speech"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The chosen test scenarios and interaction setups accurately reflect how the systems would behave in real deployments without extra fine-tuning or context that might change their use of vocal cues.","fun_headline_variants_meta":{"raw":{"variants":["Voice AI follows words ignoring vocal fear","Realtime systems detect but ignore sarcasm","Voice AI acts on text not caller tone","Production AI misses distress in speech"]},"model":"grok-4.3","cost_usd":0.003241,"raw_usage":{"total_tokens":1750,"prompt_tokens":692,"num_sources_used":0,"completion_tokens":48,"cost_in_usd_ticks":32412000,"prompt_tokens_details":{"text_tokens":692,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1010,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":692,"tokens_out":48,"duration_ms":5665,"temperature":1.0,"reasoning_tokens":1010,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-25T19:44:35.426556+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A test in which any of the four systems consistently rejects a wire transfer when the authorizing voice sounds frightened, despite matching authorizing words, would show the gap is not general.","supporting_citations":[],"review_version":1}