# Functionalizer：面向子词分词的无损功能分解预分词框架

- 来源：HuggingFace Daily Papers（社区热门论文）
- 发布时间：2026-09-18 08:00
- AIHOT 分数：34
- AIHOT 链接：https://aihot.news/items/cmucvjz7f0pz9roedye27xt8m
- 原文链接：https://arxiv.org/abs/2609.15991

## AI 摘要

Functionalizer 是一种无损预分词框架，在分词前将大小写、变音符号和字符重复等拼写变化拆解为 Unicode 私有使用区中的 opcode/operand 前缀流，且完全可逆。在自然语言与代码语料上，它最多可减少 19.7% 的实际词表槽位需求；在 98M 参数 GPT-2 上，Python 代码语法有效率从 7.70% 提升至 9.12%，并减少了自然语言中的重复 n-gram。

## 正文

Standard subword tokenizers either treat every orthographic variation of a word (such as hello, Hello, HELLO, and Héllo) as unrelated vocabulary entries, which fragments the embedding space, or discard this variation through lossy normalization. We present the Functionalizer, a lossless pre-tokenizer framework that factors orthographic and structural variations into a compositional opcode/operand prefix stream before tokenization: a canonical base token (operand) prefixed by parametric transformation operators (opcodes) encoded in the Unicode Private Use Area.

We introduce operators covering casing (CAPITALIZE), diacritics (13 dedicated opcodes), and character repetition (REPEAT, MULTIREPEAT), which are fully reversible. Across natural language and code corpora, the Functionalizer enables complete corpus coverage with significantly smaller vocabularies under unconstrained exhaustion conditions, reducing actual vocabulary slot requirements by up to 19.7%. Downstream evaluations on 98M-parameter GPT-2 models show that the Functionalizer improves Python code syntax validity (9.12% vs. 7.70%) while reducing duplicate n-gram repetition in natural language prose.

These findings demonstrate that functional decomposition can be an effective mechanism for vocabulary-efficient, structurally aware language modeling, and motivate further validation at production scale.