# NCP-ArchPreview 技术报告：用 Next Concept Prediction 迈向潜在空间语言模型

- 来源：HuggingFace Daily Papers（社区热门论文）
- 发布时间：2026-09-09 08:00
- AIHOT 分数：49
- AIHOT 链接：https://aihot.news/items/cmtwdbmaz0cctrolkwr0tzswg
- 原文链接：https://arxiv.org/abs/2609.10715

## AI 摘要

NCP-ArchPreview 是一个潜在空间语言模型，在标准 next-token prediction（NTP）之外引入 Next Concept Prediction（NCP），预测跨多个 token 的离散概念，并保留 token 级自回归生成。

## 正文

Abstract:We introduce NCP-ArchPreview, a latent-space language model that pushes autoregressive pretraining beyond standard next-token prediction (NTP). Alongside NTP, the model learns through Next Concept Prediction (NCP) to predict discrete concepts that span multiple tokens, introducing an explicit and more challenging concept-level objective while preserving standard token-level autoregressive generation. NCP-ArchPreview builds a latent space by constructing a product-quantized concept vocabulary directly from its hidden states, and subsequently learns to predict future concepts via a dedicated Concept Module. These predicted concepts are then fed back to the token level to guide subsequent generation, with NTP and NCP trained jointly end-to-end. We scale this architecture to 8.9B parameters and train it on 5.73T tokens from the Dolma-3 dataset, marking the largest demonstration of a latent-space language model to date. Remarkably, by consuming only 51.3% of the total training tokens, NCP-ArchPreview achieves the final pretraining loss of OLMo-3-7B. Following full pretraining, it outperforms OLMo-3-7B by 2.45 points on the downstream macro-average, including a notable 5.99-point gain on GSM8K. Controlled experiments isolate a clear progression of performance gains stemming from both the latent architecture and the NCP objective. Furthermore, utilizing only 85% of the standard computation, NCP-ArchPreview approaches the training loss of a strictly parameter-aligned 8.9B baseline. The learned latent space remains highly valuable after the pretraining stage: updating just the 17M-parameter VQ module yields a novel, lightweight interface for domain adaptation, while a simple injection of concept representations into a DFlash2 drafter improves the mean accepted length by 4.17% with negligible overhead.

Subjects: Computation and Language (cs.CL)

Cite as: arXiv:2609.10715 [cs.CL]

(or arXiv:2609.10715v1 [cs.CL] for this version)

https://doi.org/10.48550/arXiv.2609.10715

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Yuliang Liu [

Wed, 9 Sep 2026 18:12:43 UTC (2,701 KB)

Access Paper:

View PDF

TeX Source

Current browse context:

References & Citations

NASA ADS

Google Scholar

Semantic Scholar

Bookmark

Bibliographic and Citation Tools

Code, Data and Media Associated with this Article