{"id":"0fd71ce7-693a-42f6-a3ff-322551a66a8e","arxiv_id":"2502.08848","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A multi-microphone phone case with real-time localization enables direction-aware, diarized captions that deaf and hard-of-hearing users rate as valuable in group conversations.","lead":"SpeechCompass adds speaker direction to live captions on mobile phones using a four-microphone phone case that estimates where sound comes from. The authors show that frequent captioning users value directional cues and diarized transcripts in group conversations.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The system assigns one angle per frame and diarization is tested only on turn-taking speech with no overlap (Sec. 5.6); real group conversations contain overlap, so the central group-conversation benefit remains unverified.","rationale":"The Reader's weakest assumption (no-overlap testing) matches the most load-bearing condition for the central claim. I agree and sharpen it with the system mechanism: one TDOA/KDE peak per frame means overlapping speakers cannot be separately labeled, and the paper's own Sec. 5.3 states GCC-PHAT locks onto the loudest sound. The no-overlap DER of 0.30-0.37 is already high; overlap will likely worsen it. This is not a disagreement with current consensus; it is an untested internal condition of the proposed pipeline. The systems contribution is genuinely solid: localization accuracy is carefully measured, the hardware is low-power and reproducible, and the reported code release supports replication. Those strengths do not resolve the overlap gap. Keeping the Reader's CONDITIONAL verdict is appropriate; the condition should be an overlap-inclusive technical evaluation or a scoped claim. I would not move to REJECT because a targeted evaluation could plausibly show the approach still helps, e.g., by showing that dominant-speaker diarization remains accurate and even imperfect direction cues improve comprehension. An objective comprehension measure in the lab study would also strengthen the usability claim, but the overlap test is the more fundamental check.","tokens_in":22064,"tokens_out":5137,"duration_ms":53253,"concrete_test":"Build an overlap-controlled diarization test using LibriSpeech at the four azimuths of Sec. 5.6: create 100 conversations at each overlap ratio (0%, 10%, 25%, 50%) and each SNR (clean, 18, 12, 6 dB), with two speakers active during the overlap intervals. Compute DER with pyannote as in the paper and separately compute the secondary speaker's missed-content fraction in the ASR output. If DER rises steeply with overlap or secondary-speaker content is largely absent at 25% overlap, the claim must be scoped to turn-taking conversations (or the system needs a multi-source/beamforming extension).","verdict_should_be":"UNCHANGED","load_bearing_attack":"SpeechCompass's only speaker-separation cue is azimuth. The localization pipeline produces one TDOA peak per microphone pair (Eqs. 1-4), and the KDE layer is described as taking the highest peak (Sec. 4.3); the UI then colors/labels one active direction per ASR segment. The diarization evaluation (Sec. 5.6) explicitly uses synthesized conversations 'with each speaker speaking in turn (i.e., no overlap in the speech content)' and still reports DER of 0.30-0.37 for the four-microphone configuration. In overlapping speech, a single peak cannot represent two active directions; Sec. 5.3 notes that GCC-PHAT picks up the loudest sound, so an overlapped segment would be attributed to the dominant speaker and the other speaker's content would be lost. Because the motivating scenario is group conversation (dinner table, Sec. 1), overlap is a core condition, not an edge case. The lab study also does not measure objective comprehension and does not control or report overlap. The central claim that multi-microphone localization 'allows diarization' for group conversations is therefore only supported for clean, turn-taking speech, which is exactly the condition where the accessibility problem is least severe.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"SpeechCompass is a mobile captioning system that augments automatic speech recognition with real-time azimuth localization from a custom four-microphone phone case and a low-power microcontroller. The estimated direction of speech is used to diarize transcripts by color-coding or visually separating speakers, to display directional indicators (arrows, edge dot, minimap), and to suppress speech from selected directions. The paper contributes the embedded hardware and GCC-PHAT-based localization pipeline with kernel density estimation, an Android application, technical evaluations of localization accuracy, latency, power consumption, and diarization error, as well as a foundational survey (n=263), an online interface survey (n=494), and an in-person lab study with eight deaf or hard-of-hearing frequent users of captioning technology.","tokens_in":22256,"tokens_out":4144,"duration_ms":41444,"significance":"If the technical claims hold, this is a valuable accessibility contribution: it shows that low-power, privacy-preserving, calibration-free microphone-array processing can add spatial speaker cues to mobile ASR without requiring conversation partners to enroll or wear devices. The paper's strengths include the released code and design files, the concrete low-latency implementation (2.9 ms per microphone pair, ~263 ms onset-to-estimate for speech), localization errors of 11-22 degrees at conversational loudness that are comparable to human azimuth accuracy, and the user evaluation with the target deaf and hard-of-hearing population. The main weakness is that the diarization evaluation, which is central to the group-conversation claim, is limited to turn-taking speech without overlap and still reports diarization error rates of 30-37% for the four-microphone configuration; the benefit for realistic overlapping group conversations is therefore not yet demonstrated.","major_comments":[{"comment":"The diarization evaluation uses only synthesized turn-taking conversations with no overlap in the speech content, as explicitly stated. For the four-microphone configuration, the reported DER is 0.30-0.37 even in this simplest scenario. Because the localization pipeline selects a single KDE peak per frame (Sec. 4.3) and GCC-PHAT picks up the loudest sound (Sec. 5.3), overlapping speech from different directions cannot be represented or diarized. Since the motivating scenario in Sec. 1 is a dinner-table group conversation, where overlap is common, the central claim that localization 'allows diarization' for group conversations is only supported for the least challenging condition. I recommend either evaluating with overlapping speech and reporting per-speaker coverage, or explicitly limiting the claim to turn-taking conversations.","section":"Sec. 5.6"},{"comment":"The lab study has only eight participants, and while a Kruskal-Wallis test shows a significant overall effect on visualization preferences, the post-hoc pairwise comparisons are not significant after Bonferroni correction. The user-facing claim that 'the value of diarization and visualizing localization was consistent across participants' rests entirely on subjective Likert ratings from n=8, with no objective measure of comprehension, speaker-attribution accuracy, or reading speed. The manuscript should either temper this claim to 'initial preference evidence' or add a task-based outcome measure to support the accessibility benefit.","section":"Sec. 6.3"},{"comment":"The technical evaluations use a single stationary source at 1.5 m with the device rotated for localization, and four fixed sources at cardinal angles for diarization. Real group conversations involve moving speakers, multiple simultaneous sources, and reverberation, which are exactly the conditions in the motivating scenario. The paper does not characterize performance under these conditions, and the discussion in Sec. 5.3 notes that the algorithm tracks the loudest sound. The authors should either add experiments with moving or concurrent sources, or explicitly state these as scope limitations in Sec. 7.","section":"Secs. 5.1, 5.3, 5.6"},{"comment":"The KDE bandwidth (25), the KDE buffer size (600 samples), and the diarization running histogram window (522 ms) are fixed values with no reported sensitivity analysis or independent justification. If these parameters were tuned on the evaluation data, the reported DER and localization errors may be optimistic. A sensitivity sweep over these parameters, or a statement that they were chosen a priori, is needed to support the robustness of the reported numbers.","section":"Secs. 4.3 and 5.6"}],"minor_comments":[{"comment":"The text says 'as can be seen in Table 12', but the DER results are presented in a figure, not a table; this should be 'Figure 12'.","section":"Sec. 5.6"},{"comment":"The text reports 'power consumption of the whole system was 28 mAh'; mAh is a charge unit, so this should likely be '28 mA' or 'mW'.","section":"Sec. 5.5"},{"comment":"References [31] and [32] are duplicate entries for the same paper by Jain et al. (2015) on head-mounted display visualizations; one should be removed and the in-text citations updated.","section":"References"},{"comment":"The paragraph describing the lab study mentions the 'SoundCompass UI', but the system is consistently named SpeechCompass elsewhere; this should be corrected.","section":"Sec. 6.2"},{"comment":"The conclusion states 'All the participants found the diarization, localization, and visualization features to be useful'; given n=8 and the lack of significant pairwise results, the wording 'all eight participants in our study' would be more precise than the unqualified 'all participants'.","section":"Sec. 8"}],"recommendation":"major_revision","confidential_remarks":"This is a solid CHI submission with a working prototype, open-source code, and user evaluations with the target population. The main gap is that the diarization evaluation avoids overlapping speech and still shows high error rates, which directly undermines the strongest claim about group conversations. The authors should be asked to either add overlap-aware evaluation or substantially reframe the claims. The Google Surveys methodology and the small n=8 lab study are acceptable for an HCI audience if the claims are calibrated, but the current wording overstates the evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a solid systems and accessibility paper with a working low-power prototype, but the core claim about group conversations rests on an evaluation that deliberately avoids overlapping speech. The stress-test note is right: the diarization test is turn-taking only, and the DER is still 30–37% with four mics. That gap should be fixed or explicitly scoped.\n\nWhat's new: the authors built a four-microphone phone case with real-time GCC-PHAT localization on a microcontroller, and a captioning UI that maps azimuth to text color and arrows. The localization evaluation is careful — error, elevation, latency, power. They also ported the algorithm to a two-mic phone to show a limited 180° version. Code and design files are on GitHub, which is good practice.\n\nWhat's well done: the technical evaluation is reproducible in principle, and the paper is honest about current limits. The n=263 and n=494 surveys are large but measure self-reported challenges, not comprehension. The lab study with eight participants is a preference probe — useful, but no objective measure of whether the diarization helps.\n\nSoft spots: one angle per frame means overlapping speakers cannot be separated, and the evaluation avoids that condition entirely. The DER is high even in the clean turn-taking case, so it's unclear whether readers can actually follow better than with a single stream. The abstract slightly overstates the survey's import. None of these are fatal for a prototype paper, but they are real.\n\nWho it's for: people working on mobile accessibility, captioning, or embedded audio. It deserves a serious referee; the prototype and measurements are valuable enough to justify revision rather than rejection. My recommendation: send it to review, with a request to either add a minimal overlap test or state plainly that the system targets turn-taking conversations only.","headline":"Solid prototype with a real gap: diarization only works for turn-taking speech, and the paper never tests overlap.","tokens_in":22850,"tokens_out":2839,"would_cite":false,"duration_ms":28611,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Sound direction lets mobile captions separate speakers in group conversations.","keywords":["assistive technology","hearing accessibility","sound localization","speaker diarization","microphone array","mobile captioning","real-time speech recognition"],"falsifier":"An experiment that feeds the system a two-speaker conversation with a known fraction of overlapping speech (e.g., 20 to 30 percent of speaking time) and computes diarization error rate would show whether direction alone can keep speakers separate when turns overlap; if the error rate rises to near-chance levels under overlap, the central benefit would fail in exactly the group conversations where it is needed.","tokens_in":21829,"feed_emoji":"🧭","tokens_out":8354,"duration_ms":71459,"temperature":0.7,"pith_summary":"This paper claims that adding multi-microphone localization to mobile speech-to-text can fix a problem single-microphone captioning cannot: in group conversations, the transcript loses who said what and from where. The authors build SpeechCompass, a phone case with four microphones and a low-power microcontroller that estimates the 360-degree direction of speech in real time, and a captioning app that uses that direction to color text, place arrows, and let users suppress unwanted directions. They report that a survey of 263 frequent captioning users identifies speaker separation as a top challenge, and that eight frequent users in a lab study all agreed that directional guidance is valuable. If correct, mobile captioning becomes useful in group conversations without requiring conversation partners to wear or enroll devices, at lower computational cost and with less privacy risk than speaker-embedding approaches.","feed_headline":"Sound direction lets mobile captions separate speakers","feed_subtitle":"Four microphones compute a 360° angle per utterance, letting captions show who is talking and from where.","key_machinery":"The load-bearing mechanism is the time-difference-of-arrival (TDOA) angle estimate: for each of six microphone pairs, GCC-PHAT (generalized cross-correlation with phase transform) finds the delay that maximizes the normalized cross-correlation; with known microphone geometry and the far-field approximation, the delay is converted into an azimuth angle, and a kernel density estimate over the most recent 600 samples selects the most likely source angle. That angle is bound to automatic speech recognition output so each transcript segment inherits a speaker direction, which the interface renders as text color, a directional glyph, or selective suppression of entire directions.","core_discovery":"The paper's central claim is that the direction of arrival of speech, measured with a small microphone array, is sufficient to diarize a group conversation into separate speakers and to guide the user's attention with directional visual cues. The authors demonstrate this by implementing a complete system: four synchronized microphones on a phone case feed a GCC-PHAT time-delay estimator running on a low-power microcontroller, which fuses six inter-microphone delays through kernel density estimation into a 360-degree azimuth angle with mean error 11 to 22 degrees at conversational loudness, comparable to human localization error. The angle drives a captioning interface that colors transcript lines, shows arrows or moving indicators, and lets the user hide speech from chosen directions. In a lab study with eight deaf or hard-of-hearing frequent users of captioning, all participants rated directional feedback as valuable and said they would recommend it, with colored text and arrows the most preferred visualization styles.","pith_inferences":["The same direction signal could be combined with speaker-embedding diarization to handle overlapping speech, where pure angle assignment fails; this is a natural next step the paper leaves open.","If phones gain more microphones with wider spacing, the approach could become a pure software layer on existing devices, making directional captioning widely available without the phone-case hardware.","Because the localization is language-agnostic and works for non-speech sounds, the compass interface could be extended to alert users to environmental sounds such as alarms with spatial cues.","The system's privacy profile—no voice enrollment and no audio leaving the device—could make it attractive beyond accessibility, for example in meeting transcription or language learning at a table."],"forward_implications":["If direction-based diarization works, mobile captioning apps can separate speakers in turn-taking group conversations without requiring speakers to install apps, wear microphones, or enroll voice samples.","Users can know where to look when the speaker changes, and can filter out speech from a given direction, directly addressing the two top challenges reported by frequent users: background noise and merged text.","The localization runs on a low-power microcontroller at under 20 milliseconds per frame, and the whole system draws little enough current to run for about 18 hours on a 500 milliamp-hour battery.","Diarization error rate improved by roughly 32 percent relative with four microphones versus three, suggesting that modest hardware additions materially improve speaker separation.","A two-microphone phone can already provide limited 180-degree directional guidance in the app without the extra phone-case hardware."],"supporting_citations":[{"why":"Supplies the GCC-PHAT cross-correlation method used to estimate inter-microphone time delays for all localization.","marker":"[37]"},{"why":"Provides the human azimuth error baseline (up to about 20 degrees) used to set the accuracy target and to compare SpeechCompass's localization error.","marker":"[45]"},{"why":"Provides the diarization evaluation toolkit used to compute the diarization error rate in the technical evaluation.","marker":"[12]"},{"why":"Supplies the audiobook speech material used to synthesize the turn-taking test conversations for the diarization experiment.","marker":"[51]"},{"why":"Represents the existing consumer mobile captioning app whose lack of speaker separation motivates the problem SpeechCompass addresses.","marker":"[2]"},{"why":"Prior work on acoustic beamforming for speaker diarization that SpeechCompass adapts to mobile, on-the-go use.","marker":"[4]"}],"fun_headline_variants":["Direction of speech separates speakers in mobile captions","Microphone array points captions to each speaker","Speech direction drives speaker-aware captions","Four mics give captions a sense of direction"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The diarization evaluation only tests turn-taking speech with no overlap, so the central benefit—separating speakers by direction—is unproven for overlapping conversations, where two people speak at once.","fun_headline_variants_meta":{"raw":{"variants":["Direction of speech separates speakers in mobile captions","Microphone array points captions to each speaker","Speech direction drives speaker-aware captions","Four mics give captions a sense of direction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000331,"raw_usage":{"total_tokens":1821,"prompt_tokens":900,"completion_tokens":921,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":516,"completion_tokens_details":{"reasoning_tokens":863}},"tokens_in":516,"tokens_out":921,"duration_ms":8011,"temperature":1.0,"reasoning_tokens":863,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T23:27:32.496296+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"An experiment that feeds the system a two-speaker conversation with a known fraction of overlapping speech (e.g., 20 to 30 percent of speaking time) and computes diarization error rate would show whether direction alone can keep speakers separate when turns overlap; if the error rate rises to near-chance levels under overlap, the central benefit would fail in exactly the group conversations where it is needed.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the GCC-PHAT cross-correlation method used to estimate inter-microphone time delays for all localization."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the human azimuth error baseline (up to about 20 degrees) used to set the accuracy target and to compare SpeechCompass's localization error."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the diarization evaluation toolkit used to compute the diarization error rate in the technical evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Represents the existing consumer mobile captioning app whose lack of speaker separation motivates the problem SpeechCompass addresses."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior work on acoustic beamforming for speaker diarization that SpeechCompass adapts to mobile, on-the-go use."}],"review_version":1}