Apple 研究：用语音基础模型的语种选择性维度直接检测语码转换
Code-Switching Detection via Language-Selective Dimensions in Speech Foundation Models
Apple 研究团队提出一种轻量方法，直接从语音中检测语码转换（CS），无需依赖多语种转录。该方法利用预训练语音编码器的帧级语种信息，通过单语录音找出语种选择性隐藏状态维度，再聚合其帧级激活值生成 CS 预测。在 CS-FLEURS 的 14 个语种对上，Whisper-small 编码器取得 0.798 的平均 F1，高于 CNN 的 0.776 和 HuBERT 线性基线的 0.751。
AuthorsJie Chi, Zakaria Aldeneh, Masha Fedzechkina, Maureen de Seyssel, Natalie Schluter
Code-switching (CS), the alternation between languages within an utterance, is common in multilingual speech but underrepresented in pretraining corpora. Increasing CS coverage requires finding relevant utterances in large speech collections so that they can be included in training data. One way to find such utterances is to transcribe the speech and detect language alternation in the text, but this depends on obtaining accurate multilingual transcriptions. We propose a lightweight method that detects CS directly from speech. It uses frame-level language information in pretrained speech encoders to determine whether both languages are present within an utterance. Specifically, we identify language-selective hidden-state dimensions using monolingual recordings and aggregate their frame-level activations into a CS prediction. With a small CS set used to tune a decision rule, we show that the Whisper-small encoder achieves a mean F1 of 0.798 across 14 language pairs in CS-FLEURS, compared with 0.776 for a CNN and 0.751 for a HuBERT linear baseline trained for monolingual language identification. Our findings suggest that pretrained speech encoders already encode the information needed to identify CS utterances, and they provide a lightweight tool for building more representative multilingual corpora.
Related readings and updates.
Code-switching (CS), alternating languages within the same utterance, poses significant challenges for automatic speech recognition (ASR) due to limited CS training data. This paper applies an iterative pseudo-labeling training approach to CS-ASR for the first time, demonstrating its effectiveness in leveraging unlabeled data to improve CS-ASR performance. The approach comprises three phases: pseudo-label generation, two-stage bilingual model…
Code switching (CS) refers to the phenomenon of interchangeably using words and phrases from different languages. CS can pose significant accuracy challenges to NLP, due to the often monolingual nature of the underlying systems. In this work, we focus on CS in the context of English/Spanish conversations for the task of speech translation (ST), generating and evaluating both transcript and translation. To evaluate model performance on this task,…
来源：Apple Machine Learning Research · machinelearning.apple.com