引言
全球超过一半的人口会说不止一种语言。而对于许多双语使用者来说，语码转换——在不同语言之间无缝切换，甚至在一句话中间切换——是日常交流中自然而然的一部分。无论是在闲聊、联络中心还是 IT 服务台，说话者都会流畅地切换到当下感觉最自然的语言。
尽管双语使用者遍布全球，但针对语音智能体在企业场景中如何处理语码转换语音的研究却很少。因此，当一位客户询问我们，我们的语音智能体在他们以双语为主、经常进行语码转换的客户群体中表现如何时，我们决定构建自己的基准测试和数据集来评估模型。
我们聚焦于自动语音识别（ASR）——任何语音智能体流水线的第一步——因为转写错误会向前传播到每一个下游组件。在企业场景中，一张被错误分流的工单或一个被误解的政策问题都会带来真实的运营后果，因此把转写做对是语音智能体流水线中尤为关键的一步。
我们的基准覆盖了四组与我们的客户群体最相关的语言对：西班牙语-英语、法语-英语、加拿大法语-英语，以及德语-英语。它以非英语语言作为矩阵框架语言，并在其中以不同长度嵌入英语。数据涵盖广泛的人力资源（HR）和 IT 服务管理（ITSM）场景，包括员工关于福利或薪资的咨询，以及诸如重置密码、VPN 访问或设备故障排查之类的支持请求。
为了衡量各模型的表现，我们报告三项指标：词错误率（WER）、语义词错误率（SWER）和答案错误率（AER）。我们选择这些指标，是为了同时捕捉（1）模型在转写上的精确准确率，以及（2）模型为下游任务保留话语含义的能力。
我们通过用于评估语音模型的测试框架 AU-Harness 发布我们的基准和数据。我们还提供了七套 ASR 系统的结果，其中包括一些大型音频语言模型（LALM）、前沿 ASR 和开源 ASR。我们的主要发现是，语码转换的代价因所测试的语言对和模型而异。ElevenLabs Scribe V2、Google Gemini 3 Flash 和 Assembly AI Universal 3-Pro 在该任务的各项指标上表现最为突出。
基准
数据流水线
我们从内部的 IT 支持和 HR 交互语料库入手。为了生成每一句语码转换话语，我们先从英语与四种非英语语言之一的平行用户话语出发，然后筛选出合适的语码转换候选。我们保留长度在 12 到 40 个词之间的话语——短到足以像自然的口语轮次，又长到足以包含真正的语码转换机会。
我们还会排除那些由实体主导的话语——电子邮件、电话号码、ID 或 URL，这些会让文本因必要而半英语化，而非出于双语选择。最后，我们要求至少有 3 个可转换的实词——非实体或产品名称的名词、动词或形容词——以便为生成模型提供足够的素材，生成有意义的语码转换版本。
在此基础上，我们测试了多种以真实方式组合语言的策略，最终选定了一个简单的 persona 提示词，发送给 LLM（OpenAI/GPT-5）来生成语码转换文本。随后，我们使用一次 LLM 口语化处理，将文本转换为其口语形式，并使用 ElevenLabs Multilingual V2 合成音频。每一句话语随后都会由一位 AI/NLP 语言学家审阅，该语言学家是主体语言的母语者；被标记的话语会被排除或重新生成并再次审阅。最终数据集包含 259 条西班牙语-英语记录、298 条法语-英语记录、188 条加拿大法语-英语记录，以及 173 条德语-英语记录。
评估方法
我们针对每个模型、每种语言对报告三项指标，这些指标旨在捕捉转录准确率、语义保留度和下游任务表现：
- 词错误率（WER）。除了每种语言对的整体 WER 之外，我们还按单种语言报告 WER。
- 语义 WER（SWER）。该分数表示被判定为具有语义意义的错误率。我们的实现主要基于 Pipecat 的 STT 基准测试，并使用 Gemma-4-31B 作为评判模型。
- 答案错误率（AER）。该指标直接衡量转录错误是否会传导为下游失败。它是一个问答指标，遵循 Bhushan 等人（IISc/ARTPARK，arXiv 2507.16456）的方法论。对于每段话语，我们生成三个下游理解问题，并衡量阅读 ASR 转录文本的 LLM 能否正确回答这些问题。流程如下图所示。
发现
我们评估了以下模型：
- AssemblyAI / Universal 3-Pro
- Deepgram / Nova 3 Multilang
- ElevenLabs / Scribe V2
- Google / Gemini 3 Flash
- Mistral AI / Voxtral Small 24B-2507
- Nvidia / Parakeet TDT 0.6b V3
- OpenAI / Whisper Large V3 Turbo
A. 模型在我们的语码转换基准测试中表现如何？
我们从两个维度分析了错误：
- 词级准确率，通过 WER 衡量。WER 是标准方法：它将真实转录文本与模型输出对齐，并量化两者之间的距离。尽管它简单且被广泛使用，但无法区分微小的拼写差异和完全错误的词。
- 语义准确率，通过 SWER 和 AER 衡量。SWER 让我们从整体视角了解话语级别的表现，不过它反映的是评判模型的评估，而非直接的下游测试。相比之下，AER 是一种功能性测试：对于每段话语，三个理解问题衡量转录中是否保留了最关键的细节——案件编号、姓名、日期、请求原因。
当模型在不同指标上出现分歧时，各指标之间的差异才最有意义。
WER 结果（越低越好）
- ElevenLabs/Scribe V2 和 AssemblyAI/Universal-3 Pro 是转录准确率最高的两个模型。它们在西班牙语-英语上持平，在所有其他语言对上相差 0.02-0.13 个百分点，其中 ElevenLabs/Scribe V2 在每一对上都以微弱优势领先。
- Google/Gemini 3 Flash 在每一组语言对上都紧随其后，在加拿大法语-英语上落后最多，比 Scribe 低 0.14 分，比 AssemblyAI 低 0.12 分。Deepgram/Nova-3、Mistral/Voxtral Small 和 Nvidia/Parakeet 占据中间名次，各自至少在一组语言对上领先。Nvidia/Parakeet 总体上是三者中最弱的，但在德语-英语上缩小了差距，在该语言对上同时优于 Deepgram/Nova-3 和 Mistral AI/Voxtral。
- OpenAI/Whisper Large V3 Turbo 位居末位，WER 范围从 0.16 到 0.61。虽然差距明显，但这反映了 Whisper 已知的局限性。在处理语码转换音频时，若调用时未显式指定语言参数，Whisper 会默认将内容翻译成英语而非转写，无法保留音频中所说的语言。
SWER 和 AER 结果（越低越好）
语义指标与 WER 呈现的总体情况大致相似，但有几处反转。
- ElevenLabs/Scribe V2 仍位居第一，SWER 和 AER 分数都非常低。
- 尽管 Assembly AI/Universal-3 Pro 在 WER 上于各语言对中排名第一或第二，但 Google/Gemini 3 Flash 在 AER 上始终优于它，将 AssemblyAI/Universal-3 Pro 挤到第三位。SWER 上也出现同样的模式，不过 AssemblyAI/Universal-3 Pro 在西班牙语-英语上优于 Gemini。作为 LALM，Google/Gemini 3 Flash 针对语言理解和推理进行了优化，这很可能让它在语义敏感指标上占据优势，即便其原始转写准确率有所不足。
- 在 Whisper 上也观察到了类似的性能变化。尽管它仍然始终排在最后，但在语义指标下，其表现不佳的差距明显缩小，这直接源于它倾向于将语码转换的音频翻译成英语，而不是将其转写下来。
语义结果还揭示了 SWER 与 AER 之间值得注意的一致性。这两个指标在不同的粒度上运作——SWER 聚合每个词上的错误，而 AER 衡量的是每句话的三个理解问题能否被正确回答——因此规模上的差异是意料之中的。值得注意的是，模型相对排名在两者之间有多么稳定。唯一明显的例外是 Deepgram/Nova-3，它在 SWER 上处于中游，但在所有语言对上的 AER 排名都是最后或倒数第二。这一差距在西班牙语-英语上最为显著：Nova-3 的整体语义错误率低于它在最关键细节上的错误率。
B. 与纯单语语音相比，语码转换会带来哪些额外成本？
尽管这些结果清晰地展现了各模型在语码转换语音上的相对性能，但它们并未揭示这些错误究竟源于转写本身的固有难度，还是源于语言切换所带来的额外挑战。
为了分离语码转换带来的成本，我们将每一句话都通过评估流水线以三种音频运行：语码转换音频、相同内容的单语矩阵语言音频，以及单语英语音频。对于每一句话，我们测量了语码转换条件与单语条件之间的 WER 差异，并在整个基准上汇总了这些差值。结果如下。
- ElevenLabs/Scribe V2、Google/Gemini 3 Flash 和 AssemblyAI/Universal-3 Pro 整体差值最小，其中 ElevenLabs/Scribe V2 的表现尤为突出，甚至优于其自身的 L2 基线，表明其对双语输入具有真正的鲁棒性。
- 语码转换的影响也遵循一个直观的规律：表现最好的系统相对于单语基线仅承受很小的损失，而排名较低的模型退化更为显著，这表明语码转换主要暴露的是鲁棒性方面的差异，而非对所有模型统一增加了难度。
- 所有语言对中都呈现出一致的结构性规律：绿色柱（相对于英语的成本）几乎总是大于红色柱（相对于 L2 的成本），这在意料之中——对大多数模型而言，L2 基线本身就比英语更难，因此以 L2 为基准衡量时，净转换损失更小。最明显的异常值是 Whisper，它相对于英语的退化最大，在德语-英语上达到 +0.85 的峰值。它也是唯一一个在语码转换语音上表现优于单语 L2 的模型——这是默认执行翻译的直接结果，因为翻译完全绕过了矩阵语言。
C. 语码转换是如何破坏 ASR 系统的？
既然我们已经知道语码转换可能导致模型出错，接下来我们转而研究这些错误所关联的具体条件。为回答这一问题，我们拟合了一个两部分模型：
- 首先，我们使用逻辑回归来探究哪些变量与至少发生一次转写错误相关联。
- 其次，在至少发生一次错误的条件下，我们使用普通最小二乘法（OLS）回归来考察哪些变量与错误幅度相关联。
这种两部分方法让我们能够区分使错误更可能发生的因素与影响错误一旦发生后其幅度的因素。两个步骤都包含相同的预测变量：（1）话语中的语言转换次数，以及（2）话语的语码混合指数（CMI）——即相对于基质语言而言取自次要语言的词汇比例，遵循Gambäck 和 Das的方法。我们还纳入了话语长度作为控制变量，因为更长的话语提供了更多出错的机会。
与转写错误相关的变量
从我们模型的第一部分可以发现，语言切换次数是始终与转写错误是否发生关联最紧密的预测因子。每一次语言切换似乎都为转写过程引入了一次额外的失败机会。这一关系在法语-英语语言对中尤为显著，七个模型中有六个都表现出这一关系。其他预测因子——CMI 和话语长度——与错误发生之间几乎没有显著关系。
当问题转向错误严重程度时，呈现出不同的模式。此时更强的预测因子不再是切换次数，而是CMI。具体在德语-英语语言对中，七个模型里有四个显示出 CMI 与 WER 之间存在显著的正相关关系。这表明，一旦错误发生，其严重程度并非由说话者切换语言的频率决定，而是由混合的整体密度决定：一段话语将两种语言交织得越彻底，由此产生的转写错误往往越大。
语码转换话语中导致转写错误的部分
这个两部分模型解释了哪些因素与错误的发生和恶化相关。我们的最后一个实验考察了语码转换话语中哪些部分对这些错误贡献不成比例。为了检验错误在话语的英语部分和非英语部分之间是否分布不同，我们使用 GPT-5 按语言为每个词打标签，然后将每个转写错误归因到其发生所在词的语言，计算每种语言的 WER。下面的热力图展示了结果。
这一模式在所有模型和语言对中都是一致的：错误集中在话语的英语部分，而非主体语言部分。这与直觉相悖——在单语环境下，英语往往是这些模型处理得最好的语言。一种解释是，语码转换语音中的英语片段可能不成比例地包含更难转写的技术词汇或命名实体。另一种解释是，嵌入式语言片段无论嵌入的是哪种语言，都会制造出具有挑战性的上下文：当模型过渡到一段非主体语言的语音时，它必须在话语中途适应不同的音系和词汇语域，从而恰好在这一跨度上增加了出错的可能性。
这一结果表明，语码转换 ASR 中的转写难度并非仅集中在切换点上，而是更广泛地延伸到嵌入式语言跨度之中。厘清这一模式究竟反映的是英语片段的词汇特征、其作为嵌入式语言的结构性角色，还是当前模型在话语中途适应能力的局限，是未来工作中一个有前景的方向。
局限性
有若干局限性值得说明：
- 该基准测试是合成数据。所有音频均通过文本转语音（TTS）模型生成，而非由自然双语说话人录制。因此，该基准测试可能无法完全捕捉真实语码转换语音的韵律和音系特征。
- 所有模型仅使用“自动语言检测”进行评估。部分系统提供了可配置选项——强制语言 token、多语言提示等——这些可能提升语码转换音频的转录质量。我们选择自动检测，是因为它符合生产环境中的实际设置，即系统事先不知道来电者将使用哪种语言对。
- 分语言 WER 不计入插入错误。我们的分语言 WER 通过将每个参考词标注为英语或非英语，并将错误归入对应的类别来计算。插入错误无法在不额外调用模型识别插入词语言的情况下归入某一语言，因此我们将其排除在分语言计算之外。它们仍会计入总体 WER。
结论
语码转换长期以来一直是语音模型的压力测试。我们的结果表明，对于最顶尖的前沿 ASR 系统而言，它正日益成为一种常态。
当企业审慎选择其 ASR 系统时，双语客户可以自然地说话——在对话需要时于句子中途切换语言——而不会牺牲转录质量或下游任务表现。在我们基准测试中排名靠前的模型，处理语码转换语音时的性能损失相对于其单语基线小得惊人，而语义指标所呈现的情况甚至更令人鼓舞。
但情况并非一片向好。在做出生产决策之前，你必须针对客户实际所说的语言进行基准测试——不同模型和语言对之间的表现差异很大，对西班牙语—英语使用者而言的最佳选择，未必是德语—英语使用者的最佳选择。
Introduction
Over half of the world's population speaks more than one language. And for many bilingual speakers, code-switching — seamlessly switching between languages, even mid-sentence — is a natural part of everyday communication. Whether in casual conversations, contact centers, or IT helpdesks, speakers fluidly adapt to whichever language feels most natural in the moment.
Despite the prevalence of bilingual speakers across the world, there has been little work focused on how voice agents handle code-switched speech in enterprise settings. So, when a customer asked us how our voice agents would perform for their largely bilingual customer base who routinely code-switched, we decided to build our own benchmark and dataset to evaluate models. We focused on automatic speech recognition (ASR) — the first step in any voice agent pipeline — because transcription errors propagate forward into every downstream component. In enterprise settings, where a misrouted ticket or misunderstood policy question has real operational consequences, getting the transcript right is an especially important step of the voice agent pipeline.
Our benchmark covers four language pairs that were most relevant for our customer base: Spanish-English, French-English, Canadian French-English, and German-English. It uses the non-English language as the matrix framing, with English embedded at varying lengths. The data covers a wide range of Human Resources (HR) and IT Service management (ITSM) scenarios, including employee inquiries about benefits or payroll, and support requests such as password resets, VPN access, or device troubleshooting. To measure how various models perform, we report three metrics: Word Error Rate (WER), Semantic Word Error Rate (SWER), and Answer Error Rate (AER). We choose these metrics to capture both (1) the models' exact accuracy in transcription, as well as (2) their ability to preserve the meaning of the utterance for downstream tasks.
We release our benchmark and data through our harness for evaluating voice models, AU-Harness. We also provide results from seven ASR systems, including some Large Audio Language Models (LALMs), frontier ASRs, and open-source ASRs. Our main finding is that the cost of codeswitching varies depending on the language-pair and model tested. ElevenLabs Scribe V2, Google Gemini 3 Flash, and Assembly AI Universal 3-Pro surface as the top models across metrics for the task.
The Benchmark
Data Pipeline
We start with an internal corpus of IT support and HR interactions. To create each code-switched utterance, we begin with parallel user utterances in English and one of our four non-English languages, then filter for good code-switching candidates. We keep utterances between 12 and 40 words — short enough to be natural spoken turns, long enough to contain real switching opportunities. We also exclude utterances where entities dominate — emails, phone numbers, IDs, or URLs that make text half-English by necessity rather than bilingual choice. Finally, we require at least three switchable content words — nouns, verbs, or adjectives that are not entities or product names — to give the generation model enough material to produce a meaningful code-switched version.
From here, we tested various strategies for combining languages in a realistic way and ultimately selected a simple persona prompt sent to an LLM (OpenAI/GPT-5) to produce the code-switched text. We then used an LLM verbalization pass to convert the text into its spoken form and used ElevenLabs Multilingual V2 to synthesize the audio. Every utterance is then reviewed by an AI/NLP linguist who is a native speaker of the matrix language; flagged utterances are excluded or regenerated and re-reviewed. The final dataset has 259 Spanish-English records, 298 French-English records, 188 Canadian French-English records, and 173 German-English records.
Evaluation Methodology
We report three metrics per model per language pair, chosen to capture transcription accuracy, meaning preservation, and downstream task performance:
- Word Error Rate (WER). Along with overall WER per language pair, we report WER by individual language.
- Semantic WER (SWER). This score represents the rate of errors that are judged as semantically meaningful. Our implementation is largely based on Pipecat's STT benchmark, and we use Gemma-4-31B as our judge.
- Answer Error Rate (AER). This metric directly captures whether transcription errors propagate into downstream failures. It is a question-answer metric that follows the methodology in Bhushan et al. (IISc/ARTPARK, arXiv 2507.16456). For each utterance, we generate three downstream comprehension questions and measure whether an LLM reading the ASR transcript can answer them correctly. The flow is shown in the diagram below.
Findings
We evaluated the following models:
- AssemblyAI / Universal 3-Pro
- Deepgram / Nova 3 Multilang
- ElevenLabs / Scribe V2
- Google / Gemini 3 Flash
- Mistral AI / Voxtral Small 24B-2507
- Nvidia / Parakeet TDT 0.6b V3
- OpenAI / Whisper Large V3 Turbo
A. How well do models perform on our benchmark for codeswitching?
We analyzed errors along two dimensions:
- Word-level accuracy, measured through WER. WER is the standard approach: it aligns the ground truth transcript with the model's output and quantifies the distance between them. Although it is simple and widely used, it can't distinguish a minor spelling difference from a completely wrong word.
- Semantic accuracy, captured through SWER and AER. SWER gives us a holistic view of utterance-level performance, though it reflects a judge model's assessment rather than a direct downstream test. AER, by contrast, is a functional test: for each utterance, three comprehension questions measure whether the most consequential details — case numbers, names, dates, the reason for a request — were preserved in the transcription.
The differences between metrics become most meaningful when models diverge across them.
WER results (lower is better)
- ElevenLabs/Scribe V2 and AssemblyAI/Universal-3 Pro are the top two models on transcription accuracy. They are tied on Spanish-English and separated by 0.02-0.13 percentage points across all other language pairs, with ElevenLabs/Scribe V2 taking a narrow lead on each.
- Google/Gemini 3 Flash follows closely in every language pair, trailing most on Canadian French-English, where it falls 0.14 points behind Scribe and 0.12 points behind AssemblyAI. Deepgram/Nova-3, Mistral/Voxtral Small, and Nvidia/Parakeet occupy the middle ranks, each pulling ahead on at least one language pair. Nvidia/Parakeet is the weakest of the three overall but closes the gap on German-English, where it out performs both Deepgram/Nova-3 and Mistral AI/Voxtral.
- OpenAI/Whisper Large V3 Turbo sits at the bottom, with WER ranging from 0.16 to 0.61. While it's a significant drop, it reflects known limitation of Whisper. When called without an explicit language parameter on code-switched audio, Whisper defaults to translating into English rather than transcribing, failing to preserve the language spoken in the audio.
SWER and AER results (lower is better)
The semantic metrics tell a broadly similar story to the WER, with a few inversions.
- ElevenLabs/Scribe V2 remains at the first place, with very low SWER and AER scores.
- While Assembly AI/Universal-3 Pro ranked first or second across language pairs in WER, Google/Gemini 3 Flash consistently outperforms it in AER and pushes AssemblyAI/Universal-3 Pro down to third place. The same pattern appears in SWER, although AssemblyAI/Universal-3 Pro outperforms Gemini on Spanish-English. As an LALM, Google/Gemini 3 Flash is optimized for language understanding and reasoning, which likely gives it an advantage on meaning-sensitive metrics even where its raw transcription accuracy falls short.
- A similar shift in performance is noticed in Whisper. While it still consistently ranks last, the margin of its underperformance narrows considerably under semantic metrics, a direct consequence of its tendency to translate code-switched audio into English rather than transcribe it.
The semantic results also reveal notable consistency between SWER and AER. The two metrics operate at different granularities — SWER aggregates error across every word, while AER measures whether three comprehension questions per utterance can be answered correctly — so differences in scale are expected. What's notable is how stable the relative model rankings are across both. The one clear outlier is Deepgram/Nova-3, which sits mid-tier on SWER but ranks last or second-to-last on AER across all language pairs. The gap is most pronounced on Spanish-English: Nova-3's overall rate of semantic errors is lower than its error rate specifically on the details that matter most.
B. What additional cost does code-switching add compared to plain monolingual speech?
While these results provide a clear picture of relative model performance on code-switched speech, they do not reveal whether the errors stem from the inherent difficulty of transcription itself, or from the additional challenge introduced by language switching.
To isolate the cost of codeswitching, we ran every utterance through our evaluation pipeline in three audios: the code-switched audio, a monolingual matrix-language audio of the same content, and a monolingual English audio. For each utterance, we measured the difference in WER between the code-switched and monolingual conditions and aggregated the deltas across the benchmark. Below are the results.
- ElevenLabs/Scribe V2, Google/Gemini 3 Flash, and AssemblyAI/Universal-3 Pro show the smallest deltas overall, with ElevenLabs/Scribe V2 notably outperforming its own L2 baseline, pointing to genuine robustness to bilingual input.
- The effect of code-switching also follows an intuitive pattern: top-performing systems incur only a small penalty relative to monolingual baselines, while lower-ranked models degrade more substantially, suggesting that code-switching primarily exposes differences in robustness rather than uniformly raising difficulty across all models.
- A consistent structural pattern emerges across all language pairs: the green bars (cost relative to English) are almost always larger than the red bars (cost relative to L2), which is expected — the L2 baseline is itself harder than English for most models, so the net switching penalty is smaller when measured against it. The clearest outlier is Whisper, which shows the largest degradation relative to English, peaking at +0.85 on German-English. It is also the only model that performs better on code-switched speech than on monolingual L2 — a direct consequence of defaulting to translation, which sidesteps the matrix language entirely.
C. How does code-switching break ASR systems?
Now that we know code-switching can cause models to make mistakes, we turn to investigating the specific conditions associated with those mistakes. To address this question, we fit a two-part model:
- First, we use a logistic regression to ask what variables are associated with at least one transcription error occurring.
- Second, conditional on at least one error occurring, we use an ordinary least squares (OLS) regression to examine which variables are associated with error magnitude.
This two-part approach lets us distinguish between factors that make an error more likely to occur and factors that influence how large the error becomes once it has. Both steps include the same predictors: (1) the number of language switches in the utterance, and (2) the utterance's Code-Mixing Index (CMI) — the proportion of words drawn from a secondary language relative to the matrix language, following Gambäck and Das. We also include utterance length as a control, since longer utterances provide more opportunities for error.
Variables associated with transcription errors
From the first part of our model, we find that the number of language switches within an utterance is the predictor most consistently associated with whether the occurrence of a transcription error. Each language change appears to introduce an additional opportunity for the transcription process to fail. This relationship was significant in the French-English language pair in particular, where six out of seven models exhibited it. Other predictors — CMI and utterance length — showed few significant relationships with error occurrence.
When the question shifts to error magnitude, a different pattern emerges. Rather than switch count, CMI surfaces as the stronger predictor. In the German-English language pair specifically, four out of seven models showed a significant positive relationship between CMI and WER. This suggests that once errors occur, their severity is shaped not by how often the speaker switches languages but by the overall density of mixing: the more thoroughly an utterance interweaves the two languages, the larger the resulting transcription errors tend to be.
Portions of a code-switched utterance contributing to transcription errors
The two-part model explains what factors are associated with errors occurring and worsening. Our final experiment examines which portions of a code-switched utterance contribute disproportionately to those errors. To test whether errors distribute differently across the English and non-English parts of an utterance, we used GPT-5 to tag each word by language, then attributed each transcription error to the language of the word on which it occurred, computing a per-language WER. The heatmap below shows the results. The pattern is consistent across all models and language pairs: errors concentrate on the English portions of utterances rather than the matrix-language portions. This is counterintuitive — English is the language these models tend to handle best in monolingual settings. One explanation is that English segments in code-switched speech may disproportionately contain technical vocabulary or named entities that are harder to transcribe. Another is that embedded-language segments create a challenging context regardless of which language is embedded: when a model transitions into a stretch of non-matrix speech, it must adapt to a different phonological and lexical register mid-utterance, increasing the likelihood of error at exactly that span.
This result suggests that transcription difficulty in code-switched ASR is not concentrated at switch points alone, but extends across embedded-language spans more broadly. Disentangling whether this pattern reflects the lexical characteristics of English segments, their structural role as embedded language, or current models' limited ability to adapt mid-utterance is a promising direction for future work.
Limitations
Several limitations are worth acknowledging:
- The benchmark is synthetic. All audio is generated via Text-to-Speech (TTS) model rather than recorded by natural bilingual speakers. So, the benchmark may not fully capture the prosodic and phonological characteristics of real code-switched speech.
- All models were evaluated with "auto language detection" only. Some systems expose configurations — forced language tokens, multi-language hints, and similar — that might improve transcription quality on code-switched audio. We chose auto-detection because it matches the production setting where the system has no prior knowledge of which language pair a caller will use.
- Per-language WER excludes insertions. Our per-language WER is computed by tagging each reference word as English or non-English and attributing errors to the corresponding bucket. Insertions cannot be attributed to a language without an additional model call to identify the inserted word's language, so we exclude them from per-language calculations. They are still counted in the aggregate WER.
Conclusion
Code-switching has long been a stress test for voice models. Our results suggest that for the best frontier ASR systems, it is increasingly becoming a normal condition.
When enterprises choose their ASR systems carefully, bilingual customers can speak naturally — switching languages mid-sentence as the conversation demands — without sacrificing transcription quality or downstream task performance. The top models in our benchmark handle code-switched speech with surprisingly small penalties relative to their monolingual baselines, and the semantic metrics tell an even more encouraging story.
But the picture is not uniformly positive. Before making production decisions, you must benchmark the languages your customers actually speak — performance varies substantially across models and language pairs, and the best choice for Spanish–English speakers is not necessarily the best choice for German–English speakers.