Scale AI’s Voice Showdown: New Global Benchmarks for Voice AI Models

Scale AI launched Voice Showdown today, a new platform designed to benchmark voice artificial intelligence models through real human interaction, offering users free access to leading frontier models in exchange for participation in blind comparison tests.

The move comes as voice AI development rapidly outpaces the tools used to evaluate it, according to Scale AI product manager Janie Gu. “Voice AI is really the fastest moving frontier in AI right now,” Gu said. “But the way that we evaluate voice models hasn’t kept up.”

Voice Showdown is built on Scale’s ChatLab platform, already used by a community of over 500,000 annotators, and is now opening to the public via a waitlist. The platform allows users to interact with high-tier models—typically requiring multiple paid subscriptions—at no cost. Users are periodically presented with blind, head-to-head “battles” where they choose which of two anonymized voice models provides a better experience. This data will form a human-preference leaderboard for voice AI.

The platform addresses shortcomings in existing benchmarks, which often rely on synthetic speech, English-only prompts, and scripted test sets. Voice Showdown utilizes real human speech, with accents, background noise, and conversational filler, across more than 60 languages and six continents. A significant portion – over a third – of the comparison battles occur in non-English languages, including Spanish, Arabic, Japanese, Portuguese, Hindi, and French.

The evaluation mechanism is designed to incentivize honest voting. After a user votes for their preferred model, the app automatically switches them to that model for the remainder of their conversation. This “alignment of consequence with preference” aims to discourage casual or dishonest voting, a criticism leveled at similar text-based benchmarks like Chatbot Arena (LM Arena).

Initial results, based on data collected through March 18, 2026, reveal performance gaps that traditional benchmarks have missed. In the Dictate mode (speech-to-text), Google’s Gemini 3 Pro and Gemini 3 Flash are statistically tied for the top rank, followed by OpenAI’s GPT-4o Audio. In Speech-to-Speech (S2S) mode, Gemini 2.5 Flash Audio and GPT-4o Audio are also statistically tied, with GPT-4o Audio pulling ahead after adjusting for response length, and formatting.

Qwen 3 Omni, an open-weight model from Alibaba’s Qwen team, consistently performs well across both modes, exceeding expectations based on its relative profile. “When people come in, they go for the big names,” Gu noted. “But for preference, lesser-known models like Qwen actually pull ahead.”

A key finding highlighted by Voice Showdown is the significant disparity in multilingual capabilities. OpenAI’s GPT Realtime 1.5 model responds in English to non-English prompts roughly 20% of the time, even for widely supported languages like Hindi, Spanish, and Turkish. Gemini 2.5 Flash Audio and GPT-4o Audio exhibit lower rates of language mismatch, around 7%.

The platform also evaluates models at the individual voice level, revealing substantial performance variance within a single model’s voice catalog. For one unnamed model, the best-performing voice won 30 percentage points more often than the worst-performing voice, despite both sharing the same underlying reasoning and generation capabilities.

The study also indicates that model performance degrades over the course of extended conversations, with content quality becoming a primary failure point in later turns. GPT Realtime variants, however, showed marginal improvement on later turns, aligning with their known strengths in longer contexts.

Scale AI plans to introduce a Full Duplex evaluation mode, designed to capture real-time, interruptible conversations, in the near future. This mode will further differentiate Voice Showdown from existing benchmarks, which primarily focus on turn-based interactions.

The Voice Showdown leaderboard is available at scale.com/showdown. A public waitlist for access to ChatLab is now open.

Earlier this year, Meta invested $14.3 billion into Scale AI, acquiring a 49% stake in the company, and recruited Scale AI founder Alexandr Wang as its new chief AI officer to lead its Superintelligence Lab. OpenAI has been winding down its work with Scale AI over the last year, according to an OpenAI spokesperson, seeking alternative data providers. Google is also reportedly reducing its ties with Scale AI following the Meta partnership, though Google declined to comment.