I’ve been noodling on ideas for reliably distinguishing humans from machines, and went down a rabbit hole leading to Moravec’s paradox: the idea that certain tasks which are trivial for humans are difficult for computers, because human abilities have been honed over millions of years via evolution. That led me to pose the question: computers can transcribe speech, but can they pick out and understand one voice in a loud, crowded environment? I decided to test it out.

Selective auditory attention is the trick human brains use to listen to a speaker at a loud cocktail party. If humans can do this easily and computers can’t, that’s a neat reverse Turing test. For the experiment, I generated a corpus of synthetic audio clips such as the following to run through a suite of speech recognition models:

From 40 synthetic cocktail-party-like audio clips tested against 8 speech recognition model configurations compared with 146 human answers, humans performed roughly as expected, and machines performed far better than expected.

A brief summary of the findings:

• Humans outperformed the best model¹: 0.71 versus 0.54 word accuracy for Gemini 3.8 Flash; 77% human pass rate (word accuracy ≥ 0.5) versus 55% for Gemini
• Performance gaps exist across every model, but shrink as models get stronger: compared against the same clips, humans led Whisper small by +0.31 word accuracy, narrowing to +0.13 for Gemini (95% CI +0.03 to +0.23)
• Instructing an LLM (Gemini) to focus on the main speaker improves word accuracy, rising from 0.34 to 0.54
• Allowing humans to listen to clips repeatedly didn’t lead to measurable improvements
• The human-machine gap in picking out speech from chatter is not viable to serve as a reliable reverse Turing test

Code & data for this article can be found at github.com/cloudkj/crosstalk

Hypothesis

My goal was to test two aspects of selective auditory attention. First, whether human success rates for picking out speech in chatter will be high and machine speech recognition success rates will be low. As rough targets, I picked over 80% for humans and under 20% for machines; these are simple gut-feel numbers for what could constitute a useful test, but are not derived from anywhere.

Second, if the gap between human and machine performance, however narrow or wide, holds up across a variety of speech recognition models spanning different levels of capacity and complexity.

Landscape

A 1953 paper from E.C. Cherry was the original work that introduced the cocktail party problem, and highlighted the human ability to selectively focus on a single auditory message while tuning out background noise. A number of previous experiments and analyses in the modern era were also discovered in the literature, a number of which do not support the core hypothesis. Previous papers as early as the late 2000s up to early 2020s showed the ability of automatic speech recognition (ASR) models to break audio CAPTCHAs.

A recent study from researchers at PeopleTec (Noever & McKee, 2025) covered a number of efforts related to selective auditory attention, and their takeaways with regard to overlapping speech recognition were directionally promising; average model performance achieved 17.9% compared with 61% baseline human accuracy, specifically for the task of identifying overlapping speech. On the other hand, the best performing model in that study (GPT-4o) performed roughly on par with humans. However, the audio generation methods were unspecified and no underlying datasets or code were available, and so this study mostly served as a reference point.

Approach

The task at hand was to devise a method to produce audio that reliably exhibits a clear gap in performance on the selective auditory attention task between humans and speech recognition models. In short, I had to generate overlapping speech clips then measure model task performance. I went through multiple rounds of audio-mix tuning in order to generate synthetic audio clips that were intelligible to my ears and simultaneously plausibly hard for machines. Once the audio synthesis seemed sufficient, I gathered actual transcription responses from human participants that listened to these clips which were compared against transcription results from models.

espeak-ng

An initial proof-of-concept was built using espeak-ng to flesh out the end-to-end pipeline and provide a demo for a small number of human testers. The first version used simple two-word phrase pairs, along with various adjustments on pan, distractor, and stagger parameters. The initial model comparison baseline was the lightweight Whisper base model.

These first clips turned out to be largely unintelligible, and difficult to transcribe for both models and humans alike. In the next version, I switched to full sentences with multiple overlapping voices and decibel adjustments. Qualitatively, these sounded more like plausible speech, but the underlying voices still sounded too artificial; the background chatter was also too garbled to resemble an actual cocktail party environment.

Ultimately, a full parameter sweep produced a number of variants which were successful at producing speech that was clear enough for my ears, using interjection fragments with a handful of voices, along with a low-pass muffle. However, the resulting clips turned out to also be easier for the models, in which Whisper base was able to achieve 90% pass rate based on simple word overlap.

Piper neural TTS

In the next round, I switched to more distinct, human-sounding, natural voice sources using Piper TTS as a way to address robotic voices with unexpected specific accent variations. I landed on a stable candidate to produce discernible audio with an initial 10% model success rate (Whisper base on word overlap), using a fixed target voice and a pool of five babble voices. The babble voices spoke randomly selected sentence fragments from a pool of filler text, and used 200 ms gaps with jittered start delays of up to 800 ms per background speaker. Subsequent tuning was applied to make the audio clips more qualitatively tractable to my own ears.

Ultimately, I ended up with a corpus of 40 target sentences (e.g. “the old man walked slowly to the store”) combined with a filler pool of 22 sentence fragments (e.g. “no way, really?”) as the evaluation dataset.

Model evaluation

Model evaluation centered on small open models plus one frontier API-backed model in order to replicate an environment that could be readily employed in adversarial or automated settings. The task for each model was to transcribe the given audio clip in a single pass. Specifically, the models included the Whisper base/small variants, Wav2Vec2, Vosk, Qwen3-ASR 0.6B, and Parakeet CTC 0.6B.

Gemini 3.8 Flash was later introduced as a representative candidate for hosted frontier variants, and also as a way to test the feasibility of an instruction-augmented LLM in the loop. The naive, simple prompt was Transcribe this audio. Output only the transcript text, nothing else. whereas the full instructed prompt was:

This recording contains one main speaker reading a single sentence, with several
other people chattering in the background. Transcribe only the main speaker's
sentence, word for word, and ignore the background voices. Output only the
transcript text, with no commentary or labels.

Whisper was also given a short hint prompt to ignore background chatter, which it treats as preceding context rather than an instruction. To keep everything close to common off-the-shelf setups, I also opted to skip source separation as a preprocessing step.

Human evaluation

Human responses were gathered via a web application from participants, either via direct outreach (friends and family) or through survey subreddits. I did not have any predetermined target demographics or selection criteria. In each session, a sample of 10 clips from the 40-clip corpus was presented to test participants, with inverse frequency sampling to balance responses.

Participants could optionally indicate their age bracket and native English speaker status, and play each audio clip as many times as they wanted before submitting their transcribed result. The feedback after each submission was masked to show limited information so as to avoid biased or skewed results from repeated participation and leakage through domain familiarity.

Results

Scoring

Standard preprocessing was done on the input results, such as lowercasing, removal of non-alphanumerics, and whitespace collapse, to normalize the human vs. model results. The primary metric is word accuracy, defined as 1 − WER (word error rate), equivalent to the share of words transcribed correctly in order. When considering binary pass/fail outcomes for each transcription task, a default threshold of word accuracy ≥ 0.5 was used to indicate a passing answer.

Various preprocessing techniques were also considered to help increase tolerance of human errors, such as the use of stemming, hyphenation and compound word fixes, along with typo guarding. Common errors included early sentence truncation, inflection differences (e.g. “stacks” vs. “stacked”), mishearings, and spelling variations (e.g. “practiced” vs. “practised”).

Ultimately, the human outperformance results against models held up across every scoring variant.

Data

Overall results

The human dataset comprises 146 transcriptions from 18 participant sessions, covering all 40 clips. Each model transcribed each clip exactly once.

Using word accuracy as the basis and the default 0.5 passing threshold, there’s a pass rate of 77% for humans (target 80%) versus 55% for Gemini (target 20%), and no threshold comes close to the hypothesized split (see below). On the machine side, this is well above initial target performance of 20% and likely any other reasonable choice.

Here are the human and Gemini pass rates for varying word accuracy threshold values:

Prompting

I also included a frontier hosted LLM model in Gemini 3.8 Flash to assess any prompting benefits by using directed hints to the model to focus on a specific target in the audio clip. With a single prompt, there was an improvement of +0.20 in mean word accuracy for Gemini, and a pass rate improvement from 35% to 55%.

Human vs. model gap

To quantify the human-machine performance gap, I compare each human answer against the model’s transcription of the same clip, and average the difference in word accuracy. Since clips received different numbers of human answers (2 to 6), this matched gap isn’t simply the difference of the overall averages. Weighted the same way as the human answers, Gemini scores 0.58 rather than 0.54 overall, so its gap is +0.13. For the same reason, Qwen3-ASR has a larger gap than Gemini despite its marginally higher overall average.

I also use AUC, which ignores which clip an answer belongs to: it’s the probability that a randomly chosen human answer scores higher than a randomly chosen model answer, with ties counted as half. An AUC of 0.5 means the two are indistinguishable, and 1.0 means every human answer beats every model answer. Gemini’s AUC of 0.62 means a random human answer beats a random Gemini answer only 62% of the time.

Here’s a summary for the best representative from each of the top three model families:

In comparing human and best model performance, on a word accuracy bucket basis, another interesting insight appears: Gemini performance is fairly bimodal: about a third of its answers score near zero, and another third are perfect.

Repeated plays

One idea I had prior to running these tests was that human performance could improve with repeated plays of the target clip, since I found that listening to the same clip a few times made each task easier, and replays are effectively “free”. However, the data can’t show whether replays help. Instead, it appears humans replay the clips they find the most difficult, and the most-replayed clips are also the most difficult for the best performing model.

Positional effects

Another interesting finding is that mistakes generally occur towards the end of the audio clip sentences, and these mistakes show up for humans and every model alike. As visualized in Figure 1, this is partly due to the way clips are generated and the design choices I made around voice babble start, delay, and offsets. The intuition I have is that clip openings are likely to have fewer overlapping voices and thus are less difficult than clip endings.

Limitations

The dataset here is based on a relatively small sample (18 sessions) of adult, mostly native English speakers. Some were people I knew; others were random strangers on the internet willing to take surveys. Thus, there’s a potential for bias towards participants that were attentive and motivated to try hard and succeed at the task, and so the results are better interpreted as a rough upper bound on human performance. Under everyday conditions, humans may be distracted or in noisy environments that will likely only hinder task performance.

Additionally, I tested a single synthetic setup (albeit honed over a few iterations), with one fixed target voice and background chatter stitched from sentence fragments, and limited to a 40-clip corpus. I also did not try the tests on actual chatter clips from real-world audio.

Future Work

The landscape is constantly evolving, and this test was largely focused on local-first, hardware-constrained models. When tested on larger open models or commercial speech-to-text services, it is entirely plausible the human-machine gap completely disappears or even shifts in favor of models outperforming humans.

On the other hand, further exploration in the audio clip synthesis and task construction could reveal patterns in selective auditory attention that are more tuned to human abilities; for example, alternating speakers, voice changes, or simply larger corpora of target sentences and filler fragments. There’s also room for testing on more specific speech recognition tasks, such as identifying certain keywords or picking out inflections.

Conclusion

From the test results, humans appear to perform better than every model based on the word accuracy metric. However, the gap is narrower than my original hypothesis, and is certainly not reliable enough to serve as a viable CAPTCHA; the best performing model provides answers that are hard to tell apart from humans on any given clip. More than half of the clips that were straightforward for humans were also easy for the best performing model. Additionally, my initial attempts at generating these synthetic audio clips showed that changes which improved human performance also tended to help models, which suggests difficulty in striking a right balance.

In the end, I’m pretty convinced that picking out speech from chatter is unlikely to be a task that falls neatly under Moravec’s paradox. Humans performed just about what I’d expect, models far exceeded my expectations, and these models are only likely to get better over time.

Appendix

Raw overall result metrics. A simple overlap metric is also included, which is the fraction of unique expected words that appear in each response.

(55%), had the most perfect transcriptions (30%), and had the smallest gap to humans on the same clips (+0.13 versus +0.16).