Pith. sign in

REVIEW 4 major objections 4 minor 25 references

Moravec's Paradox: Towards an Auditory Turing Test

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read An auditory Turing test of 917 human-easy challenges defeats every current AI audio model, with the best score at 6.9% versus a human average of 52%.

desk verdict Useful diagnostic idea, but the headline numbers don't add up: 917 vs 131 and 'near-perfect' humans vs 52%. read the letter →

arxiv 2507.23091 v1 pith:MZ7OGBZM submitted 2025-07-30 cs.AI cs.SDeess.AS

David Noever , Forrest McKee This is my paper
classification cs.AIcs.SDeess.AS
keywords auditoryTuringtestcocktailpartyeffectaudioCAPTCHAselectiveattentionspeechrecognitionMoravec'sparadoxmultimodallanguagemodelsceneanalysis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that current AI audio systems, including the best multimodal models available, cannot perform basic listening tasks that human listeners handle with relative ease. It introduces an auditory Turing test made of 917 short audio challenges across seven categories -- overlapping speech, speech in noise, temporal distortion, spatial audio, coffee-shop noise, phone distortion, and perceptual illusions -- and reports that the best model solved 6.9% of the challenges while nine human volunteers averaged 52%. The authors locate the failure in the hearing stage rather than the reasoning stage: giving the model a source-separation preprocessing step helped on overlapping speech, while telling it the category or giving it multiple guesses did not. If the result holds, it gives the field a concrete measure of the human-machine listening gap and a diagnostic benchmark for what would count as human-level machine hearing.

What carries the argument

The central object is the auditory Turing test itself: 917 audio challenges arranged into seven categories and scored as binary transcription successes. It is built on the cocktail party effect -- the human ability to follow one voice inside a noisy mixture -- and on the history of audio CAPTCHAs, which already exploited the human-machine hearing gap. The benchmark does its work by isolating the perceptual front-end: category hints and multiple attempts do not move machine scores, but a source-separation preprocessing step does, which lets the authors attribute the failures to hearing rather than to language reasoning.

What would settle it

A decisive check would be to run the same 917 clips through a larger, independent human panel and through models with and without a strong source-separation front-end, using a blinded scoring rubric; if human accuracy falls toward the machine level, or if the front-end-augmented model exceeds 50% accuracy, the central claim of a fundamental front-end auditory gap would be weakened.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that a human-easy auditory Turing test defeats every model tested: across 917 challenges the top performer, GPT-4 audio, reached 6.9% accuracy, and the Whisper variants scored between 0.8% and 4.6%, while nine human listeners averaged 52%. The failures are not random transcription noise but systematic focusing failures: models cannot hold onto one voice in overlap, cannot normalize distorted or spatialized speech, and do not use context to resolve ambiguous sounds. The authors read this as evidence that current architectures lack the selective-attention and auditory-scene-analysis machinery humans use automatically, and they support that reading by showing that source separation before transcription improves the overlap task.

Load-bearing premise

The benchmark's claim rests on the assumptions that the 917 audio clips are genuinely human-solvable, that binary scoring is fair to humans and machines alike, and that nine listeners give a trustworthy human baseline; if a meaningful fraction of clips are mislabeled or unfairly scored, the reported 93% machine failure rate would be an artifact of the benchmark rather than a fundamental gap.

Editorial extensions

If this is right

  • If the benchmark is a fair measure, current multimodal AI is far from human-level listening, and near-chance scores on cluttered audio should not be read as a model failure that scaling alone will fix.
  • The front-end locus of the failures implies that progress will come from source separation, selective attention, and scene analysis, not from better language models alone.
  • The benchmark gives the field a shared instrument: a 917-item, seven-category auditory Turing test that can be re-run to track progress on each failure mode.
  • The results suggest audio CAPTCHAs built on human-favored distortions still separate humans from current machines, reversing the earlier pattern where automated recognizers broke audio CAPTCHAs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct test of the paper's front-end diagnosis would be to feed the same 917 clips through a strong source-separation module before transcription; the paper's own overlap experiment predicts a large accuracy jump without any change to language reasoning.
  • If the audio files and scoring rubric are ever made available, the same items could serve as a live audio CAPTCHA, since machine success below 10% with human success around 50% is the classic CAPTCHA trade-off.
  • The human baseline comes from only nine listeners; before treating 52% as the definitive human reference point, a larger and more diverse sample is needed, since the reported confidence interval (40-64%) leaves room for the gap to shrink or grow.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes an "auditory Turing test" benchmark of audio challenges across categories such as overlapping speech, speech in noise, temporal distortion, spatial audio, and perceptual illusions, and evaluates several commercial and open speech models (GPT-4 audio, Whisper base/small/tiny, and a large commercial ASR). The authors report that even the best model achieves only 6.9% accuracy, that aggregate machine accuracy is 2.6%, and that human performance is 52% on average, which they interpret as a 7.5x human advantage and as evidence that current AI lacks human-like auditory scene analysis. The paper frames this as a diagnostic framework for machine listening and draws on the audio CAPTCHA literature to motivate the task design.

Significance. If the reported results were reliable, the benchmark would be a useful diagnostic instrument for quantifying progress in machine listening and for connecting Moravec's paradox to a concrete, falsifiable evaluation. The paper also provides a useful synthesis of the audio CAPTCHA literature. However, the empirical core is not currently credible: the paper's own numbers are internally inconsistent (917 total tests versus 131 machine-scored items), the human data contradict the "near-perfect" and "effortless" characterizations, the human and machine scores appear to come from different test sets, and no stimuli or scoring artifacts are released. These issues preclude an assessment of whether the claimed human-machine gap is real, so the significance of the contribution cannot be established from the manuscript as written.

major comments (4)
  1. [Abstract; Discussion "Overall Statistics"; Results (Figure 2)] The abstract and the "Overall Statistics" block in the Discussion state that the benchmark comprises 917 total tests, but the Results section reports model accuracy as fractions of 131 (9/131 = 6.9%, 6/131 = 4.6%, 2/131 = 1.5%, 1/131 = 0.8%) and Figure 2 is titled "Model Accuracy for 131 Decoding Challenges." The Methods section says the suite contains "over one hundred audio challenges (approximately 20–25 per category)" for five categories. The number 917 is never reconciled with 131, and no derivation of 917 is given. Since the headline 93% failure rate and the 7.5x human-machine ratio depend on these conflicting totals, the central quantitative claim is internally inconsistent.
  2. [Methods (human evaluation paragraph)] The same paragraph reports that "Human listeners correctly recognized the speech in almost all cases" and "human performance was near-perfect," while also reporting a 52% average with a minimum of 30%, a maximum of 83%, and a 95% confidence interval of 40–64% for N=9 human participants. These statements are flatly contradictory: a 52% average with a 30% minimum is not near-perfect and does not support the abstract's characterization of the tasks as ones that "humans perform effortlessly." The human data need to be reported per category and reconciled with these qualitative claims.
  3. [Methods; Results (human versus machine subsets)] The paper reports human data for "decoding 23 audio challenges" and machine data for 131 challenges, but never states whether the 23 human-evaluated items are a subset of the 131 machine-scored items, nor how the scoring rubric differed between humans and machines (e.g., exact-word matching versus semantic equivalence, permitted number of attempts, or partial credit). The headline ratio 52/6.9 = 7.5 therefore compares accuracies that may have been computed on different test sets with different scoring rules. This is the paper's central empirical finding, so the missing overlap information is load-bearing.
  4. [Methods; general availability] No stimuli, reference transcripts, scoring rubric, model prompts, or code are released or described in sufficient detail to reproduce the evaluation. For a paper whose stated contribution is a benchmark, this omission is critical: without the artifact, a reader cannot check whether the 131 items are genuinely human-solvable, whether the machine failures are due to prompt formatting or scoring stringency rather than perceptual deficits, or how the human data were collected and scored. The internal inconsistencies described above make the absence of these materials particularly damaging to the paper's claims.
minor comments (4)
  1. [Abstract; Methods; Discussion "BEST PERFORMING CATEGORIES"] The abstract and the "Overall Statistics" list seven categories, while the Methods section says "we focused on five representative categories" and enumerates five; the category names in the "BEST PERFORMING CATEGORIES" block (e.g., speech_blending, hidden_speech) do not match the names in the Methods list. The taxonomy should be unified and every item mapped to a category.
  2. [Introduction; Discussion; References] The "Turing Eye Test" work by Gao et al. is cited as [25] in the Introduction but as [20] in the Discussion, while reference [20] in the bibliography is a different paper (Çakmak and Balcilar). The citation numbering and the reference list need to be corrected and cross-checked.
  3. [Methods] There is a typo in the Methods paragraph: "In addition to previously constructed audio CAPTCHA tests [20], e e focused on five representative categories" should read "we focused." The caption of Figure 4, "Human reports on easy-hard captcha challenges," is also vague and should be clarified.
  4. [Methods (benchmark assembly)] The Methods says "we assembled a machine-only benchmark of over one hundred audio challenges," but the paper then describes human evaluation on the same suite. This phrasing is confusing; the authors presumably mean the machine evaluation was run on the full set while humans saw a subset, but this needs to be stated explicitly.

Circularity Check

1 steps flagged · score 2.0 of 10

No derivation-level circularity: the benchmark contains no equations, fitted parameters, or load-bearing self-citations, but its headline machine-failure rate is mildly self-confirming because the tasks were deliberately selected from the audio-CAPTCHA lineage that is defined by machine failure.

  1. other [Methods (benchmark construction) and Background (task-selection rationale)]
    "we designed a suite of auditory Turing test challenges encompassing a range of perceptual feats that humans excel at... By focusing on those human strengths, we can create challenges that remain trivial for people but hard for machines... Using these carefully constructed tasks, we assembled a machine-only benchmark of over one hundred audio challenges."

    The reported failure rate is not derived from a model or a fitted parameter, so this is not a formal derivation-level circularity. However, the benchmark is assembled from audio-CAPTCHA-style stimuli whose defining purpose, per the cited CAPTCHA literature, is to be hard for machines while easy for humans. The Methods explicitly state that tasks were chosen because humans excel at them and that the goal is to create challenges that are trivial for people but hard for machines. Reporting a 93% machine failure rate on items selected by that criterion is therefore a restatement of the inclusion criterion rather than an independent empirical discovery.

full rationale

The paper contains no equations, no fitted parameters, and no load-bearing self-citations; references [20] and [25] are not by the authors and are not used to force the conclusions. The central human-machine comparison is an evaluation rather than a derivation, so it cannot reduce algebraically to its inputs. The closest analogue to circularity is the construction of the benchmark: the Methods state that the suite builds on 'previously constructed audio CAPTCHA tests' and on challenges that deliberately 'remain trivial for people but hard for machines,' so a high machine-failure rate is partly guaranteed by item selection. That is a benchmark-design tautology, not a logical circularity. The headline claim is further weakened by non-circularity problems: the human data report 52% average accuracy (with a 30% minimum), contradicting the abstract's 'effortlessly' and the Methods' 'near-perfect' human performance, and the denominators are unreconciled (917 total tests vs 131 machine-scored items vs 23 human-timed items), so the 7.5x human-machine gap may rest on mismatched test sets. These are correctness and validity concerns, not circularity. Because the failure rate is partly baked into the task-selection criterion, a low score of 2 is appropriate; none of the enumerated circularity patterns 1-5 apply.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No fitted free parameters or invented physical entities appear. The benchmark rests on domain assumptions about human data, ground-truth labels, and model representativeness, all of which are weakened by unreleased artifacts and inconsistent counts.

assumptions (3)
  • domain assumption The nine volunteer listeners provide a representative and stable human baseline.
    The Methods section reports only nine listeners with a 52% average and a 95% confidence interval of 40 to 64%, which is too wide to support statements like human performance was near-perfect.
  • domain assumption The 917 audio challenges are correctly labeled, categorized, and scored.
    No stimuli or scoring rubric are released, and the paper gives inconsistent totals (about 100 to 125, then 131, then 917) and category counts (five versus seven), so the correctness of the ground truth cannot be verified.
  • domain assumption The tested model versions represent current state-of-the-art audio AI.
    The paper says seven models were tested but names five; it does not specify GPT-4o audio API versions, Whisper model checkpoints beyond size names, decoding configurations, or the large-scale commercial system that scored zero.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Moravec's Paradox: Towards an Auditory Turing Test." pith.science (2026). https://pith.science/paper/MZ7OGBZM

@misc{pith2026250723091,
  author       = {Pith},
  title        = {Pith review of: Moravec's Paradox: Towards an Auditory Turing Test},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MZ7OGBZM}},
  note         = {Machine review of arXiv:2507.23091}
}
read the original abstract

This research work demonstrates that current AI systems fail catastrophically on auditory tasks that humans perform effortlessly. Drawing inspiration from Moravec's paradox (i.e., tasks simple for humans often prove difficult for machines, and vice versa), we introduce an auditory Turing test comprising 917 challenges across seven categories: overlapping speech, speech in noise, temporal distortion, spatial audio, coffee-shop noise, phone distortion, and perceptual illusions. Our evaluation of state-of-the-art audio models including GPT-4's audio capabilities and OpenAI's Whisper reveals a striking failure rate exceeding 93%, with even the best-performing model achieving only 6.9% accuracy on tasks that humans solved at 7.5 times higher success (52%). These results expose focusing failures in how AI systems process complex auditory scenes, particularly in selective attention, noise robustness, and contextual adaptation. Our benchmark not only quantifies the human-machine auditory gap but also provides insights into why these failures occur, suggesting that current architectures lack fundamental mechanisms for human-like auditory scene analysis. The traditional design of audio CAPTCHAs highlights common filters that humans evolved but machines fail to select in multimodal language models. This work establishes a diagnostic framework for measuring progress toward human-level machine listening and highlights the need for novel approaches integrating selective attention, physics-based audio understanding, and context-aware perception into multimodal AI systems.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

25 extracted references · 25 canonical work pages

  1. [1]

    (2010, July)

    Gao, H., Liu, H., Yao, D., Liu, X., & Aickelin, U. (2010, July). An audio CAPTCHA to distinguish humans from computers. In 2010 third international symposium on electronic commerce and security (pp. 265 - 269). IEEE

    work page 2010

  2. [2]

    Tam, J., Simsa, J., Hyde, S., & Ahn, L. (2008). Breaking audio captchas. Advances in Neural Information Processing Systems, 21

    work page 2008

  3. [3]

    Fanelle, V., Karimi, S., Shah, A., Subramanian, B., & Das, S. (2020). Blind and human: Exploring more usable audio {CAPTCHA} designs. In Sixteenth Symposium on Usable Privacy and Security (SOUPS

    work page 2020

  4. [4]

    Alnfiai, M. (2020). A novel design of audio CAPTCHA for visually impaired users. International Journal of Communication Networks and Information Security, 12(2), 168 -179

    work page 2020

  5. [5]

    (2018, July)

    Kulkarni, S., & Fadewar, H. (2018, July). Audio captcha techniques: A review. In Proceedings of the Second International Conference on Computational Intelligence and Informatics: ICCII 2017 (pp. 359 - 368). Singapore: Springer Singapore

    work page 2018

  6. [6]

    P., & Cavender, A

    Bigham, J. P., & Cavender, A. C. (2009, April). Evaluating existing audio CAPTCHAs and an interface optimized for non-visual use. In Proceedings of the SIGCHI conference on human factors in computing systems (pp. 1829-1838)

    work page 2009

  7. [7]

    Soupionis, Y., & Gritzalis, D. (2010). Audio CAPTCHA: Existing solutions assessment and a new implementation for VoIP telephony. computers & security, 29(5), 603-618

    work page 2010

  8. [8]

    (2011, May)

    Bursztein, E., Beauxis, R., Paskov, H., Perito, D., Fabry, C., & Mitchell, J. (2011, May). The failure of noise-based non-continuous audio captchas. In 2011 IEEE symposium on security and privacy (pp. 19-31). IEEE Computer Society

    work page 2011

Show all 25 references
  1. [9]

    S., & Kim, H

    Choi, J., Oh, T., Aiken, W., Woo, S. S., & Kim, H. (2018, May). POSTER: I can't hear this because i am human: A novel design of audio CAPTCHA system. In Proceedings of the 2018 on Asia Conference on Computer and Communications Security (pp. 833-835)

    2018

  2. [10]

    & Ekedebe, N

    Lazar, J., Feng, J., Brooks, T., Melamed, G., Wentz, B., Holman, J., ... & Ekedebe, N. (2012, May). The SoundsRight CAPTCHA: an improved approach to audio human interaction proofs for blind users. In Proceedings of the SIGCHI conference on human factors in computing systems (p...

    2012

  3. [11]

    (2008, July)

    Tam, J., Simsa, J., Huggins-Daines, D., Von Ahn, L., & Blum, M. (2008, July). Improving audio captchas. In Symposium On Usable Privacy and Security (SOUPS)

    2008

  4. [12]

    H., Holz, T., & Kolossa, D

    Meutzner, H., Gupta, S., Nguyen, V. H., Holz, T., & Kolossa, D. (2016). Toward improved audio captchas based on auditory perception and language understanding. ACM Transactions on Privacy and Security (TOPS), 19(4), 1-31

    2016

  5. [13]

    (2022, June)

    Hossen, I., & Hei, X. (2022, June). aaecaptcha: The design and implementation of audio adversarial captcha. In 2022 IEEE 7th European Symposium on Security and Privacy (EuroS&P) (pp. 430-447). IEEE

    2022

  6. [14]

    Shekhar, H., Moh, M., & Moh, T. S. (2019, December). Exploring adversaries to defend audio captcha. In 2019 18th IEEE International Conference On Machine Learning And Applications (ICMLA) (pp. 1155- 1161). IEEE

    2019

  7. [15]

    (2017, November)

    Solanki, S., Krishnan, G., Sampath, V., & Polakis, J. (2017, November). In (cyber) space bots can hear you speak: Breaking audio captchas using ots speech recognition. In Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security (pp. 69-80)

    2017

  8. [16]

    Wang, P., Gao, H., Guo, X., Yuan, Z., & Nian, J. (2023). Improving the security of audio captchas with adversarial examples. IEEE Transactions on Dependable and Secure Computing, 21(2), 650-667

    2023

  9. [17]

    (2019, October)

    Jain, M., Tripathi, R., Bhansali, I., & Kumar, P. (2019, October). Automatic generation and evaluation of usable and secure audio ReCAPTCHA. In Proceedings of the 21st International ACM SIGACCESS Conference on Computers and Accessibility (pp. 355-366)

    2019

  10. [18]

    (2015, April)

    Meutzner, H., Gupta, S., & Kolossa, D. (2015, April). Constructing secure audio captchas by exploiting differences between humans and machines. In Proceedings of the 33rd annual ACM conference on human factors in computing systems (pp. 2335-2338)

    2015

  11. [19]

    H., Lazar, J., & Brooks, T

    Olalere, A., Feng, J. H., Lazar, J., & Brooks, T. (2014). Investigating the effects of sound masking on the use of audio captchas. Behaviour & Information Technology, 33(9), 919 -928

    2014

  12. [20]

    F., & Balcilar, M

    Çakmak, A. F., & Balcilar, M. (2019). Audio CAPTCHA recognition using RASTAPLP features by SVM. arXiv preprint arXiv:1901.02153

    2019 arXiv

  13. [21]

    Arons, B. (1992). A review of the cocktail party effect. Journal of the American Voice I/O society, 12(7), 35-50

    1992

  14. [22]

    cocktail party

    Shapiro, K. L., Caldwell, J., & Sorensen, R. E. (1997). Personal names and the attentional blink: a visual" cocktail party" effect. Journal of Experimental Psychology: Human Perception and Performance, 23(2), 504

    1997

  15. [23]

    Agrawal, K. (2010). To study the phenomenon of the Moravec's paradox. arXiv preprint arXiv:1012.3148

    2010 arXiv

  16. [24]

    Graham, C., & Roll, N. (2024). Evaluating OpenAI's Whisper ASR: Performance analysis across diverse accents and speaker traits. JASA Express Letters, 4(2)

    2024

  17. [25]

    & Zhang, W

    Gao, H., Huang, Z., Xu, L., Tang, J., Li, X., Liu, Y., ... & Zhang, W. (2025). Pixels, Patterns, but No Poetry: To See The World like Humans. arXiv preprint arXiv:2507.16863. Figure 5. CAPTCHA Accuracy by Category Across All Models Figure 6. Summary of Human Test Results by Ch...

    2025 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.