我们如何用 Python 改造应用存储平台 Habitat，以应对前所未有的增长。
作者：Jon Lee、Chaomin Yu 和 Ben Ries，技术团队成员
每一款 OpenAI 产品都依赖于快速、可靠的数据访问，无论是用户登录、查看其 Codex 设置，还是在 ChatGPT 中开启一段新对话。这些操作中的每一个，在产品能够响应之前，都可能需要许多次独立的数据查询。如果这些请求很慢，产品就会显得很慢。如果这些请求失败，产品就会彻底停止工作。
Habitat 是我们构建的在线存储平台，让 OpenAI 产品能够快速、可靠地访问所需信息。Habitat 现在每秒处理超过 7000 万次请求，支撑着每周超过 10 亿人使用的产品，覆盖近 40 个地理区域。两年前，Habitat 还只是一个连接到单一数据库的简单 Python 客户端库。如今，它已是一个复杂的分布式系统，服务于超过 500 PB 的数据。
图 01 · 什么是 Habitat？
在线存储平台
Habitat 是我们构建的在线存储平台，让 OpenAI 产品能够快速、可靠地访问所需信息。
- 请求
- 响应
- 变更（CDC）
客户
在线存储平台
存储资源
- ChatGPT
- API
- Codex
- 内部服务
- 以及更多
Habitat
缓存
ACL 策略 授权
放置与数据驻留 数据驻留
加密 数据安全
隔离 多租户
限流 请求整形
路由 Schema 查找 · 数据驻留
Azure Cosmos DB 在线存储
Nanobase 在线存储
Valkey 缓存
Blob 存储 存储资源
CDC 服务 变更数据捕获
Databricks
Rockset
Kafka
以及更多
在这一规模上构建和运营基础设施绝非易事，但也并非特别具有挑战性。真正让我们的情况独一无二的是，我们不得不以空前的速度扩展，以支撑惊人的用户增长和产品需求，同时还要构建一个成熟的平台。通常情况下，系统工程师会按照 10 倍规模来构建，并希望它能支撑几年，同时为下一个 10 倍做准备。而在我们的情况中，过去三年我们每年都实现了超过 10 倍的增长。因此，构建和运营 Habitat 的过程是一系列战术决策和排序：从最低层级理解每个组件，以尽可能从现有技术栈中榨取更多性能，同时抵御存储和计算容量的紧缺，为基础性投入争取时间。
- 每秒 7000 万+ 请求
- 每周 10 亿+ 用户
- 500 PB+ 数据
随着 OpenAI 的发展，Habitat 也必须随之成长：首先要可靠到足以承载关键业务的产品流量，然后要快到足以服务全球用户，最后还要能够娴熟地支撑超大规模运行。本文是关于我们如何扩展在线存储的两部分系列文章中的第一篇。在这篇文章中，我们将分享 Habitat 是如何演进的，我们为什么把它从一个库转变为一个服务，以及我们如何将一个用不常见的服务栈语言——Python——编写的服务，扩展为一个可靠的存储平台层。
在后续文章中，我们将详细讲述我们如何实现大规模下的多租户可靠性、我们优化读取性能的分层策略，以及我们如何扩展与 Azure Cosmos DB 的合作关系，以可靠地应对前所未有的需求。
什么是 Habitat？
Habitat 源于一个简单的想法：产品工程师不应该需要操心数据库管理。Habitat 于 2024 年年中起步，最初只是一个与 ChatGPT 主服务器交互的小型 Python 库。它支持一小组操作，这些操作在底层映射到数据库应用 Azure Cosmos DB。
这个库的职责是为产品团队提供一种简单的方式来存储和检索数据，而无需掌握底层的细节。Habitat 负责处理必要的工作：判断涉及的是哪种数据、数据应该来自哪里（或去往哪里）、该请求是否被允许，等等。
产品工程师无需关心 schema 查找、路由、授权、加密、序列化、请求塑形和连接池。他们甚至无需考虑数据来自哪里：Azure Cosmos DB、缓存，还是其他类型的存储。
图 02 · Habitat 服务
简化的 Habitat 请求流程
通过将存储逻辑解耦为独立服务，我们为部署、可观测性和平台增强建立了单一控制点。
- 请求
- 响应
客户端
OpenAI
Azure Cosmos DB
Habitat 客户端 SDK
envoy
habitat-service 进程 1
habitat-service 进程 2
habitat-service 进程 3
habitat-envoy
habitat-cosmos-db-us0
habitat-cosmos-db-us1
habitat-cosmos-db-eu0
这个 Python 库运行良好，Habitat 在 OpenAI 的产品工程师中得到了快速采用，尽管并没有任何集中的推动来促使大家放弃使用自助式 Postgres 和 Azure Cosmos DB。
随着产品需求的演变，产品开发者甚至可以轻松地为这个共享库添加诸如客户端缓存、压缩或加密等功能支持。
构建一个服务，以更好地支持多个复杂产品
到 2025 年中期，Habitat 作为客户端侧实现已经达到了极限。随着 Habitat 层变得越来越复杂，OpenAI 的服务数量不断增加，向后兼容的协议变更已经变得不可行。
在一个案例中，我们希望将最关键的数据集迁移到一组按区域分布的 Azure Cosmos DB 账户上，以缩小任何单一区域故障的影响范围。进行这一变更需要在客户端中引入额外的路由逻辑，将其隐藏在功能开关之后，确保它推广到所有客户端，然后再启用该功能开关。
跨数十个服务协调部署，并与每个团队逐一推进上线，花了数天时间。在启用之前，我们意识到需要引入一些影子流量机制，以确保分片逻辑正确无误。这又花了几天才完成部署。修复一个我们后来发现并不正确的 bug？又是几天。最终，我们准备好启用该开关，结果其中一个团队因无关原因将他们的服务回滚到了一个此前有 bug 的客户端版本，导致了我们一直努力避免的故障。
客户端库的变更需要在数十个服务之间进行复杂的协调，这一过程被证明越来越脆弱、低效，且容易发生运维故障。为了减少未来部署中的这种运维扩散，我们决定将 Habitat 拆分为一个独立服务。
通过将存储逻辑解耦为一个独立服务，我们建立了一个统一的控制点，用于部署、可观测性和平台增强。我们不再需要管理碎片化的更新，而是可以集中实施改进，让每个 OpenAI 产品立即受益。
集中式服务还为我们提供了一个单一卡点，以提供最强的数据安全和隐私原语。Habitat 服务是我们集中执行访问控制策略、进行审计日志记录，以及限制对 Azure Cosmos DB 等底层存储资源访问的地方。Habitat 在保护用户数据和防止来自外部、内部及智能体行为者的未授权访问方面发挥着关键作用。
大规模启动一个 Python 服务
我们知道需要一个服务，但还不想完全从 Python 迁移出去，即便 Python 作为服务会带来额外的开销。与本地库执行相比，用 Python 跑高吞吐服务会增加网络延迟，并带来可观的 CPU 和内存扩容成本。此外，我们意识到 Python 的低效在 100 倍规模下将无法接受，最终几乎必然要重写。
不过，我们把这视为一次战略性的技术债引入。当时我们的首要目标不是成本或资源优化，而是为产品开发者解除阻塞、实现平台稳定性。通过在短期内接受 Python 服务的性能取舍，我们得以优先处理更紧迫的挑战、建立核心 API，并搭建起稳健的基础设施。
我们还做了一次经过盘算的押注：我们自己的编码模型快速进步，会在未来简化技术路径。我们赌的是，等到必须彻底迁离 Python 时，Codex 和 GPT 能让这次迁移变得可行。这个赌注最终被证明是对的。
从性能角度看，把 Habitat 作为 Python 服务来运行并非最优，但却是必要的选择。Python 让我们能快速推进，但这并不意味着我们可以不顾一切、接受明显更差的延迟。当用户的平均请求会触发数百次数据库调用时，用户感受到的是最慢的那次数据库调用。我们发现，在这一规模下运行 Python 服务的主要挑战，就在于管理这些尾部延迟。
追踪 asyncio 延迟
Asyncio 帮助 Python 并发执行 I/O 密集型工作负载，但无法绕过 Python GIL 提供 CPU 并行能力。除了 I/O 密集型的请求代理之外，Habitat 还承担大量 CPU 密集型职责和后台任务：路由、压缩、加密、校验和计算、下游健康检查、请求影子复制以及对冲请求。
由于我们的服务中存在如此多的 CPU 密集型工作负载和后台任务，asyncio 调度延迟很容易成为尾部请求延迟的主要因素。在为初始服务上线进行调优之前，我们在 p99 及以上延迟的请求链路追踪中看到，虽然下游存储响应很快，但请求经常停滞在等待负责解析响应的协程被重新调度上。
图 03 · 追踪 asyncio 延迟
并发不等于 CPU 并行
Python asyncio 允许并发处理请求，但在 CPU 线程上同一时间只有一个请求在执行。当有大量 CPU 工作需要完成时，这对请求延迟有很大影响。
重放 暂停
CPU 请求/响应处理 Python 网络读写 等待 Cosmos
低 CPU 工作量
简短的 Python 步骤；I/O 等待相互重叠
Python：请求 A · CPU 请求处理
时间 →
0 10 20 30 40
Python 线程
A A B B C C
请求 ACPU 请求
Cosmos
请求 B等待首个 Python 轮次
Cosmos
请求 C等待首个 Python 轮次
Cosmos
Cosmos + 网络等待：两种场景下每个请求均为 6 个单位。
高 CPU 负载
较长的 Python 步骤使已就绪的响应持续等待
Python：请求 A · CPU 请求处理
时间 →
0 10 20 30 40
Python 线程
A A B B C C
请求 ACPU 请求
CPU 请求 Cosmos 就绪 · CPU 响应受阻
请求 B等待首个 Python 轮次
CPU 请求 Cosmos 就绪 · CPU 响应受阻
请求 C等待首个 Python 轮次
CPU 请求 Cosmos 就绪 · CPU 响应受阻
Cosmos + 网络等待：两种场景下每个请求均为 6 个单位。
示意时间 0.0 / 40 个示意单位
对于 OpenAI 的 Python 服务，我们发现，除了测量内存、CPU、网络和磁盘使用率的标准利用率与饱和度指标之外，同样关键的是还要监控 asyncio 事件循环及其繁忙程度，并据此进行调优。
通过定期调度后台任务并记录预期执行时间与实际执行时间之间的差值，我们能够以实证方式实时测量事件循环的调度延迟。在高利用率下，当存在大量高开销任务时，即使每个进程的并发请求数量适中，也足以产生显著的调度抖动，最高可达数百毫秒，在某些边缘情况下甚至可达数秒。
因此，我们只能让每个进程仅处理少量并发请求，转而大规模扩展 Python 工作进程的数量。
降低功能开关配置中的尾部延迟
在服务最初上线时，我们通过线上服务的 CPU 性能剖析发现了一个导致 asyncio 高延迟（以及由此产生的高尾部延迟）的根本原因：通过 Statsig（一款用于管理功能开关、并可用于运行 A/B 测试等功能的工具）对功能开关配置进行周期性的 JSON 解析。
默认情况下，Statsig 被配置为每分钟轮询一次刷新后的配置，且没有抖动，而该配置包含了所有服务中的每一条生产规则。另外，在架构上还做出了一项决策：每个 pod 运行最多 8 个 Python 进程，以提高 CPU 使用率并提供更低的延迟。两者叠加意味着，每分钟每个 pod 都会出现某个时刻，其所有工作进程都停止处理正在进行的请求，转而将 CPU 周期花在解析一个巨大的配置文件上。
一旦 CPU 性能剖析帮助我们定位了根本原因，修复就很直接了：部署一个更小的、有针对性的配置，延长刷新间隔，并为这类后台任务添加一些抖动。
负载均衡与连接池管理
为了保持较低的 asyncio 延迟，在服务器进程之间维持良好的请求负载均衡同样至关重要；如果不加以调优，连接池最终可能会与此背道而驰。
在客户端连接池场景下，一个发起大量并发请求的客户端进程可能只建立少量服务器连接，结果将其全部负载只发送给少数几个进程。在调整我们的负载均衡方式之前，我们的服务利用率差异很大，一些尾部进程承载的并发请求数量是平均值的 5-10 倍。
我们是在一次偶然事件中发现这一点的：尽管已经停止了那个让部分服务过载的客户端，仍有一小部分进程在突发流量过去很久之后依然处于降级状态。事实上，我们注意到这些进程出现了失控式降级，收到的请求越来越多，直到我们重启它们才恢复。一旦某个 pod 过载，某些行为就会把更多流量钉到那个过载的 pod 上。这是一类我们一些同事在以往工作中非常熟悉的故障：亚稳态故障。
我们怀疑连接池是罪魁祸首，并通过限制最大连接复用时长来验证这一猜测，结果确实减轻了性能退化，从而确认了我们的排查方向。进一步调查发现，Python 的 aiohttp TCPConnector 默认采用 LIFO 连接复用：最近归还的连接会被选用于下一个请求。这通常是一个合理的默认行为：复用较新的连接可以让为应对突发流量而额外创建的连接进入空闲超时，从而降低维护额外连接的开销。但在我们的场景中，它造成了一种亚稳态故障。在一波请求突发期间，发往较慢的过载服务器的请求会更晚将连接归还到池中，因此这些连接被后续请求选中的频率更高，逐渐将更多流量集中到那些本已不堪重负的 pod 上。将连接池修补为使用 FIFO 复用后，打破了这一反馈回路，甚至降低了我们稳态下的请求方差。
图 04A · 客户端连接池
LIFO 将新工作送回慢进程
在一波请求突发之后，较慢的服务器最后将连接归还到池中。LIFO 促使更多工作集中到那些同样较慢的服务器上。
初始突发流量到达 A、B 以及较慢的进程 C。
初始突发流量到达 A、B 以及较慢的进程 C。
A 和 B 先归还连接。C 的连接归还得更晚。
LIFO 会选择最近归还的连接，反复把新工作发回给 C。
C 会从每个连接到它的进程那里收到后续请求中不成比例的份额。
01 初始突发 02 复用连接 03 结果
客户端进程
B3 = 到服务器进程 B 的连接 3。每个进程有三个连接；仅显示可用的连接。
最先归还 最后归还
A1 A2 A3 B1 B2 B3 C1 C2 C3
↑ 连接归还**—**
服务器进程A
0并发请求
请求 请求 请求 请求 请求 请求 请求 请求 请求 请求 请求 请求
服务器进程B
0并发请求
请求 请求 请求
服务器进程C
较慢的服务器
0个并发请求
请求 请求 请求 请求 请求 请求 请求 请求 请求 请求 请求 请求
后续请求
本已吃力的 pod 上流量进一步增加。
图 04B · 客户端连接池
FIFO 打破了连接复用的反馈循环
FIFO 在突发流量之后维持更多活跃连接，但在所有服务器之间公平地均衡负载。
初始突发流量到达 A、B 以及较慢的进程 C。
初始突发流量到达 A、B 以及较慢的进程 C。
A 和 B 先归还连接。C 的连接归还较晚。
FIFO 选择最早归还的连接，在 C 追赶期间分散新请求。
在相同的到达量和服务器工作量下，FIFO 降低了请求在 C 上的集中程度。
01 初始突发 02 复用连接 03 结果
客户端进程
B3 = 到服务器进程 B 的连接 3。每个进程有三条连接；仅显示可用的连接。
最先返回 最后返回
A1 A2 A3 B1 B2 B3 C1 C2 C3
↑ 连接返回**—**
服务器进程A
0并发请求
请求 请求 请求 请求 请求 请求 请求 请求 请求 请求
服务器进程B
0并发请求
请求 请求 请求 请求 请求 请求 请求 请求 请求 请求
服务器进程C
较慢的服务器
0并发请求
请求 请求 请求 请求 请求 请求 请求
后续请求
FIFO 复用打破了这一反馈回路。
如今，我们主要依赖 Istio 和 Envoy 在 OpenAI 整个基础设施中提供连接池和更好的服务器负载感知均衡策略，从而彻底避免这一问题。
避免下游资源被淹没
针对低 asyncio 延迟进行调优，再加上拥有如此多的 Python 进程，一个副作用就是：海量的连接非常容易压垮下游依赖（即所谓的“惊群效应”）。
一次常规的每日部署——如果没有调优得足够慢——就可能因连接反复建立与断开而引发显著的 CPU 抖动。或者，一处连接泄漏就可能因占满 NAT 网关而拖垮整个网络。这些问题对其他服务来说也并不罕见，但由于进程数量多了一个数量级，触发阈值被大幅拉低，往往会占满那些客户端在稳态下仅凭纯吞吐量并不预期需要处理的网络相关资源。
我们还依赖 Envoy 来最大化连接扇入。我们用它把 Python 的 HTTP/1 连接升级为 HTTP/2，以利用多路复用，然后将这些连接池化并延长连接生命周期。Envoy 还为我们提供了一个集中位置来实现限流和熔断，而这些如果在每个独立的 Python 进程中实现，效果会大打折扣。
图 05 · 连接扇入
相同的请求，更少的连接
连接池化和 HTTP/2 连接多路复用有助于减轻下游的连接负载。
请求 响应 空闲 keep-alive
直接从 Python 发起
独立的连接池会保留每个 pod 的峰值。当全部 6 个 pod 都达到 3 个请求的峰值后：18 个连接保持打开，但只有 6 个处于忙碌状态。
请求 →
Python pod
Pod 13 个活跃 · 峰值 3
Pod 20 个活跃 · 峰值 3
Pod 30 个活跃 · 峰值 3
Pod 40 个活跃 · 峰值 3
Pod 53 个活跃 · 峰值 3
Pod 60 个活跃 · 峰值 3
18 个打开的连接
存储
Azure Cosmos DB
6 个忙碌 · 12 个空闲 · 6 个并发上游请求
Envoy 连接池 HTTP/1 连接
一个共享连接池增长至整个集群的峰值：6 个连接。稳定流量下，每个连接一旦收到响应返回就会被立即复用。
请求 →
Python pod
Pod 12 个活跃 · 峰值 3
Pod 20 个活跃 · 峰值 3
Pod 31 个活跃 · 峰值 3
Pod 40 个活跃 · 峰值 3
Pod 53 个活跃 · 峰值 3
Pod 60 活跃 · 峰值 3
本地流量
共享池
EnvoyHTTP/1 → HTTP/1
6 个打开的连接
存储
Azure Cosmos DB
6 个忙碌 · 0 个空闲 · 6 个并发上游请求
Envoy 升级至 HTTP/2
在稳定负载下，同样的 6 个请求共享 1 条保留的 HTTP/2 连接，每个请求各自使用独立的并发流。
请求 →
Python pod
Pod 12 活跃 · 峰值 3
Pod 20 活跃 · 峰值 3
Pod 31 活跃 · 峰值 3
Pod 40 活跃 · 峰值 3
Pod 53 个活跃 · 峰值 3
Pod 60 个活跃 · 峰值 3
本地流量
共享池
EnvoyHTTP/1 → HTTP/2
1 个打开的连接
存储
Azure Cosmos DB
1 个忙碌 · 0 个空闲 · 6 个并发上游请求
为什么 Habitat 做得更少
我们能够将 Python 扩展到如此规模的一个原因，是 Habitat 受限的 API，它让请求成本保持可预测。Habitat 没有允许客户端构造任意 SQL 查询——这类查询可能导致大规模全表扫描或跨多张表的连接——而是暴露了一个简单的 NoSQL API。缺少强大的 API 是 Habitat 设计中有意做出的权衡。
我们力求针对简单、可预测、工作量恒定的请求进行优化。根据我们的经验，这类系统在扩展上要容易得多，也不容易出错或被误用。扇出不可预测的请求在运维上很危险：它们会让隔离和负载均衡变得复杂，并引入延迟悬崖，无论对服务本身还是其客户端来说都难以扩展。
在我们迁移到 Habitat 和 Azure Cosmos DB 之前，OpenAI 的大部分在线数据都存储在 Postgres 上。当时，审查所有查询和 schema 变更很容易，可以在发布到生产环境之前确保它们行为良好、且针对已建索引的数据运行。随着团队和产品的增长，这很快变得难以管理，并成为频繁导致故障的原因——热路径上一个开销高昂的新查询就能拖垮整个数据库。
这里的问题在于成本失衡：编写开销高昂、难以运行的 SQL 查询既便宜又容易。在 Habitat 中，我们避免了这一点，让开销高昂的查询在客户端一侧变得极其明显。不存在可能让 Habitat 过载的无界查询，而复杂的 join 和图遍历需要产品团队承担一部分繁重工作，这有助于从整体上优化出更高效的设计。
Habitat 对外暴露一个 NoSQL API，其建模围绕客户端定义的对象类型和边类型，灵感来自 TAO 。客户端预先定义对象和边以及它们之间的相互关系，但不定义每种类型的内容。由此产生的关系类似图，但 Habitat 本身并不支持典型的图遍历查询，只能查询某个特定对象的直接边。
我们对这个图进行分区，使每个对象及其对应的边在存储层分区中共置，但我们并未在数据库层面做出一致努力，将对象与其边所指向的远程对象共置。结果是，该模型可以轻松分区以实现水平扩展，但图遍历效率低下，因为对象之间的任意一次特定跳转都可能需要从存储在不同区域的两个完全不同的 Azure Cosmos DB 账户中获取数据。
对于查询需求更复杂的客户，我们确实通过 Rockset 提供了一个离线的 Habitat 二级视图。我们使用变更数据捕获（CDC）将近乎实时地将在线存储中的变更流式传输到隔离的 Rockset 实例。每个客户团队负责为自己的复杂查询需求扩展其各自的 Rockset 实例。
这种 Rockset 配置给我们的客户带来了额外的摩擦，但我们认为在当前这个特定时刻这是正确的权衡：让简单查询成为默认选项，同时为需要复杂查询的用户提供一个逃生通道。这一设计将我们的在线存储与读取密集型的分析和搜索工作负载隔离开来。
从 Python 迁移到 Rust
将 Python 重写推迟一年，使我们能够在超高速增长期间专注于更紧迫、更具影响力的挑战。随着平台日趋成熟、增长持续加速，并且它已成为 OpenAI 按核心数计算的第二大服务（按 Envoy 部署规模计算为第四大），终于到了告别 Python 的时候。在巅峰时期，Python 曾帮助我们每秒处理超过 2000 万次请求。
2026 年第二季度，仅凭 2 名工程师、Codex 和 GPT‑5.5，我们就用 Rust 重写了整个服务。这个新的 Rust 服务目前承载了我们 95% 的生产请求；我们将在未来几周内彻底弃用 Python。我们的数据显示，Rust 服务的 CPU 效率是 Python 版本的 6 倍，内存效率是 15 倍，平均延迟和尾部延迟都显著更低。我们计划在未来的博客中分享更多经验。
优化我们的数据库层：Azure Cosmos DB
这个 Python——如今是 Rust——服务只是 Habitat 的一个侧面。在本系列的第二部分中，我们将讲述我们如何快速扩展在线存储以服务超过 10 亿 ChatGPT 用户，届时会谈到存储层，以及 Habitat 如何每秒处理超过 500 PB 的数据和超过 7000 万次请求。
如果你想在前沿规模的 OLTP 系统上工作，并对这类工程感兴趣，欢迎查看我们团队的这个开放职位。
How we adapted our application storage platform, Habitat, in Python to manage unprecedented growth.
By Jon Lee, Chaomin Yu, and Ben Ries, Members of Technical Staff
Every OpenAI product depends on fast, reliable access to data, whether someone is logging in, checking their Codex settings, or starting a new conversation in ChatGPT. Each of those actions may require many separate data lookups before the product can respond. If those requests are slow, the product feels slow. If those requests fail, the product stops working entirely.
Habitat is the online storage platform we built so OpenAI products can quickly and reliably access needed information. Habitat now handles more than 70 million requests every second, supporting products used by over 1 billion people each week, across almost 40 geographic regions. Two years ago, Habitat started as a simple Python client-side library connected to a single database. Today, it’s a complex distributed system that serves more than 500 petabytes of data.
Figure 01 · What is Habitat?
Online storage platform
Habitat is the online storage platform we built so OpenAI products can quickly and reliably access needed information.
- Request
- Response
- Changes (CDC)
Clients
Online storage platform
Storage resources
- ChatGPT
- API
- Codex
- Internal services
- And more
Habitat
Caching Caches
ACL policies Authorization
Placement & data residency Data residency
Encryption Data security
Isolation Multi-tenancy
Rate limiting Request shaping
Routing Schema lookup · Data residency
Azure Cosmos DB Online storage
Nanobase Online storage
Valkey Caches
Blob storage Storage resources
CDC Services Change Data Capture
Databricks
Rockset
Kafka
And more
Building and operating infrastructure at this scale is no easy feat, but also not particularly challenging. What made our situation unique is the unprecedented rate at which we’ve had to scale to support staggering user growth and product demand while simultaneously building out a mature platform. Often, system engineers build for 10x scale, and hope for it to hold for a few years while preparing for the next 10x. In our case, we've grown more than 10x year-over-year for the last three years. As a result, building and operating Habitat has been a series of tactical decisions and sequencing: understanding each component at the lowest level to squeeze as much juice out of our existing stack, while fending off storage and compute capacity crunches to buy time for foundational investments.
- 70M+ requests per second
- 1B+ people each week
- 500 PB+ data
As OpenAI grew, Habitat had to grow with it: first by becoming reliable enough for mission-critical product traffic, then fast enough for global users, and finally, to deftly operate at massive scale. This post is the first in a two-part series on how we scaled online storage. In this post, we’ll share how Habitat evolved, why we turned it from a library into a service, and how we stretched a service written in an uncommon serving stack language—Python—into a reliable storage platform layer.
In a future post, we’ll go into detail about how we made multi-tenancy reliability at scale, our layered strategy for optimizing read performance, and how we scaled our partnership with Azure Cosmos DB to reliably handle unprecedented demand.
What is Habitat?
Habitat started from a simple idea: product engineers shouldn’t need to think about database management. Habitat began in mid-2024 as a small Python library that interacted with ChatGPT’s main server. It supported a small set of operations that mapped under the hood to the database application, Azure Cosmos DB.
The library’s job was to give product teams a simple way to store and retrieve data without needing to master the underlying details. Habitat took care of the necessary work: figuring out what kind of data was involved, where it should come from (or go), whether the request was allowed, and so on.
Product engineers need not concern themselves with schema lookup, routing, authorization, encryption, serialization, request shaping, and connection pooling. They didn’t even need to consider where the data comes from: Azure Cosmos DB, caches, or other types of storage.
Figure 02 · Habitat service
Simplified Habitat request flow
By decoupling the storage logic into a standalone service, we established a single point of control for deployments, observability, and platform enhancements.
- Request
- Response
Client
OpenAI
Azure Cosmos DB
Habitat client sdk
envoy
habitat-service process 1
habitat-service process 2
habitat-service process 3
habitat-envoy
habitat-cosmos-db-us0
habitat-cosmos-db-us1
habitat-cosmos-db-eu0
This Python library worked well and Habitat saw rapid adoption among product engineers at OpenAI, despite no concerted central push away from using self-serve Postgres and Azure Cosmos DB.
As product needs evolved, it was even easy for product developers to add to the shared library support for features like client-side caching, compression, or encryption.
Build a service to better support multiple, complex products
By the middle of 2025, Habitat had reached its limits as a client-side implementation. As the Habitat layer had grown more complex and OpenAI’s services count increased, backward-compatible protocol changes had become infeasible.
In one instance, we wanted to reduce the blast radius of any single region outage for our most critical data sets by migrating them to a set of regionally distributed Azure Cosmos DB accounts. Making this change required introducing extra routing logic into the client, disabled behind a feature flag, ensuring it rolled out to all clients, and then enabling the feature flag.
Coordinating deployments across dozens of services and working with each team to roll it out took days. Before enabling this, we realized we wanted to introduce some shadowing to ensure the sharding logic would be correct. That took another couple of days to roll out. A bug fix for something we realized was incorrect? Another couple of days. Eventually, we were ready to enable the flag, only for one of the teams to roll back their service for unrelated reasons to a previously buggy client, causing the outage we had worked so hard to avoid.
Changes to the client library necessitated complex coordination across dozens of services, a process that proved increasingly brittle, inefficient, and susceptible to operational failures. To reduce this operational fan out for our future deployments, we decided to pull Habitat into its own service.
By decoupling the storage logic into a standalone service, we established a single point of control for deployments, observability, and platform enhancements. Instead of managing fragmented updates, we could implement improvements centrally, providing immediate benefits to every OpenAI product.
A centralized service also gives us a single chokepoint to provide the strongest data security and privacy primitives. Habitat service is where we can centrally enforce access control policies, perform audit logging, and limit access to underlying storage resources like Azure Cosmos DB. Habitat plays a critical role in protecting user data and preventing unauthorized access from external, internal, and agent actors.
Launching a Python service at scale
We knew we needed a service, but we didn’t want to migrate off Python quite yet, even with Python’s additional overhead as a service. Using Python for a high-throughput service increased network latency and added substantial CPU and memory scaling costs compared to local library execution. Moreover, we recognized that the inefficiencies of Python would not be acceptable at 100x scale, making an eventual rewrite almost certain.
However, we viewed this as a strategic incursion of technical debt. Our primary objective then was not cost or resource optimization, but rather unblocking product developers and achieving platform stability. By accepting the performance trade-offs of a Python service in the short term, we were able to prioritize more immediate challenges, establish our core APIs, and build out a robust infrastructure.
We also made a calculated wager that the rapid advancement of our own coding models would simplify the technical path in the future. We bet that by the time a full migration off Python was required, Codex and GPT would make that migration achievable. That bet eventually proved correct.
Running Habitat as a Python service would be suboptimal, performance-wise, but a necessary choice. Python lets us move quickly, but it doesn’t mean we could throw caution to the wind and accept meaningfully worse latencies. When the average user request results in hundreds of database calls, the slowest database call is the one the user feels. We’ve found the main challenge in running a Python service at this scale is in managing these tail latencies.
Tracking the asyncio delay
Asyncio helps Python execute I/O-bound workloads concurrently, but does not help work around the Python GIL and provide CPU parallelism. In addition to I/O-heavy request proxying, Habitat handles many CPU-heavy responsibilities and background tasks: routing, compression, encryption, checksumming, downstream health checking, request shadowing, and hedging.
With so many CPU-heavy workloads and background tasks in our service, asyncio scheduling delay can easily dominate tail request latency. Before tuning for our initial service launch, we saw in traces for requests with p99 and higher latency that while downstream storage responded quickly, requests frequently stalled while waiting for the responsible coroutine to be rescheduled to parse the response.
Figure 03 · Tracking the asyncio delay
Concurrency is not CPU parallelism
Python asyncio allows concurrent request processing, but only a single request executes on the CPU thread at a time. This has high impact on request latencies when there's a lot of CPU work to be done.
Replay Pause
CPU request/response processing Python network read/write Wait for Cosmos
Low CPU work
Brief Python steps; I/O waits overlap
Python: request A · CPU request processing
Time →
0 10 20 30 40
Python thread
A A B B C C
Request ACPU request
Cosmos
Request BWaiting for first Python turn
Cosmos
Request CWaiting for first Python turn
Cosmos
Cosmos + network wait: 6 units per request in both scenarios.
High CPU work
Long Python steps keep ready responses waiting
Python: request A · CPU request processing
Time →
0 10 20 30 40
Python thread
A A B B C C
Request ACPU request
CPU request Cosmos Ready · blocked CPU response
Request BWaiting for first Python turn
CPU request Cosmos Ready · blocked CPU response
Request CWaiting for first Python turn
CPU request Cosmos Ready · blocked CPU response
Cosmos + network wait: 6 units per request in both scenarios.
Illustrative time 0.0 / 40 illustrative units
For Python services at OpenAI, we find that in addition to measuring standard utilization and saturation metrics on memory, CPU, network, and disk usage, it is critical to also monitor the asyncio loop and how busy it is, then tune accordingly.
By periodically scheduling background tasks and recording the delta between expected and actual execution time, we are able to empirically measure event loop scheduling delay in real time. At high utilization, with many expensive tasks, even modest numbers of concurrent requests per process are enough to produce significant scheduling jitter, up to hundreds of milliseconds and in some edge cases several seconds.
As a result, we resort to keeping each process serving only a small number of concurrent requests and instead massively scale out the number of Python worker processes.
Reducing a tail latency in our feature flag configurations
In our initial service launch, we discovered through live service CPU profiling one root cause of high asyncio delay (and resulting high tail latencies): periodic JSON parsing of our feature flag configurations via Statsig (a tool that manages feature flags, and can be used to run A/B tests and more).
By default, Statsig was configured to poll for refreshed configs every minute with no jitter, and the config included every production rule across every service. Elsewhere, an architectural decision was made to run up to 8 Python processes per pod to push higher CPU usage and provide lower latencies. Combined, this meant that every minute each pod would have some moment where all of its workers stalled processing in-flight requests and instead would spend their CPU cycles parsing a giant configuration file.
The fix was straightforward once CPU profiling helped us root cause the issue: deploy a smaller targeted config, lengthen the refresh interval, and add some jitter to background tasks like these.
Balancing loads and managing connection pools
In order to maintain low asyncio delay, it is also critical to maintain good load balancing of requests across server processes; connection pooling can end up being antithetical to this without tuning as well.
With client-side connection pooling, a single client process that does many concurrent requests might establish only a handful of server connections and as a result send all of its load to only a handful of processes. Prior to adjusting how we do load balancing, our service had a wide variance of utilization with some tail processes serving 5-10x the number of concurrent requests as the average.
We discovered this in a chance incident where, despite stopping the client that was overloading part of our service, a subset of processes remained degraded well past the bursty traffic. In fact, we noticed those processes experienced runaway degradation, receiving increasingly more requests until we restarted them. Once a pod became overloaded, some behavior was pinning more traffic onto the overloaded pod. This was a class of failures some of our teammates were well-acquainted with from prior work: metastable failure .
We suspected the connection pool was to blame and tested this suspicion by capping max connection reuse duration, which indeed limited the degradation and confirmed our investigation direction. Further investigation found that Python’s aiohttp TCPConnector defaults to LIFO connection reuse: the most recently returned connection is selected for the next request. This is normally a reasonable default: reusing recent connections allows the extra connections created to handle bursty traffic to idle timeout, reducing overhead to maintaining extra connections. In this case, it created a metastable failure for us. During a burst of requests, requests to slower overloaded servers returned connections to the pool later and were therefore selected more frequently by subsequent requests, gradually concentrating more traffic on the pods already struggling. Patching the connection pool to use FIFO reuse broke this feedback loop and even reduced our steady state request variance as well.
Figure 04A · Client-side connection pooling
LIFO sends new work back to the slow process
After a burst of requests, slower servers return connections to pool last. LIFO encourages more work to concentrate on those same slower servers.
An initial burst reaches A, B, and the slower process C.
An initial burst reaches A, B, and the slower process C.
A and B return connections first. C’s connections return later.
LIFO picks the most recently returned connection, repeatedly sending new work back to C.
C receives a disproportionate share of subsequent requests from every process that connects to it.
01 Initial burst 02 Reusing connections 03 Outcome
Client process
B3 = connection 3 to server process B. Each process has three connections; only available connections are shown.
Returned first Returned last
A1 A2 A3 B1 B2 B3 C1 C2 C3
↑ Connections return**—**
Server processA
0Concurrent requests
Request Request Request Request Request Request Request Request Request Request Request Request
Server processB
0Concurrent requests
Request Request Request
Server processC
Slower server
0Concurrent requests
Request Request Request Request Request Request Request Request Request Request Request Request
Subsequent requests
More traffic on the pods already struggling.
Figure 04B · Client-side connection pooling
FIFO breaks the connection-reuse feedback loop
FIFO maintains more active connections after a burst, but balances workloads fairly across all servers.
An initial burst reaches A, B, and the slower process C.
An initial burst reaches A, B, and the slower process C.
A and B return connections first. C’s connections return later.
FIFO picks the oldest returned connection, spreading new requests while C catches up.
With the same arrivals and server work, FIFO reduces the concentration of requests on C.
01 Initial burst 02 Reusing connections 03 Outcome
Client process
B3 = connection 3 to server process B. Each process has three connections; only available connections are shown.
Returned first Returned last
A1 A2 A3 B1 B2 B3 C1 C2 C3
↑ Connections return**—**
Server processA
0Concurrent requests
Request Request Request Request Request Request Request Request Request Request
Server processB
0Concurrent requests
Request Request Request Request Request Request Request Request Request Request
Server processC
Slower server
0Concurrent requests
Request Request Request Request Request Request Request
Subsequent requests
FIFO reuse broke this feedback loop.
Today, we mostly depend on Istio and Envoy to provide connection pooling and better server-load-aware balancing strategies throughout OpenAI infrastructure and avoid this problem altogether.
Avoiding flooding downstream resources
One side effect of tuning for low asyncio delay and having so many Python processes is that it becomes very easy to overwhelm downstream dependencies with the vast number of connections (known as a “thundering herd”).
A regular daily deployment—if not tuned to be slow—can cause significant CPU churn from connection cycling. Or a connection leak can take out the network by saturating the NAT gateway. These are not uncommon problems for other services too, but the threshold for triggering is lowered significantly by having an order of magnitude more processes, often saturating network related resources that clients are not expecting to need to handle in a steady state based on pure throughput alone.
We also rely on Envoy to maximize our connection fan-in. We use it to upgrade Python’s HTTP/1 connections to HTTP/2 to take advantage of multiplexing and then to pool those connections and extend connection lifetimes. Envoy also gives us a central place to implement rate limits and circuit breakers that would be less effective in each standalone Python process.
Figure 05 · Connection fan-in
The same requests, fewer connections
Connection pooling and HTTP/2 connection multiplexing help reduce connection load on downstreams.
Request Response Idle keep-alive
Direct from Python
Separate pools keep each pod’s peak. After all 6 pods peak at 3 requests: 18 connections stay open, but only 6 are busy.
Requests →
Python pods
Pod 13 active · peak 3
Pod 20 active · peak 3
Pod 30 active · peak 3
Pod 40 active · peak 3
Pod 53 active · peak 3
Pod 60 active · peak 3
18 open connections
Storage
Azure Cosmos DB
6 busy · 12 idle · 6 concurrent upstream requests
Envoy pools HTTP/1 connections
One shared pool grows to the fleet’s peak: 6 connections. Steady traffic reuses each connection as soon as its response returns.
Requests →
Python pods
Pod 12 active · peak 3
Pod 20 active · peak 3
Pod 31 active · peak 3
Pod 40 active · peak 3
Pod 53 active · peak 3
Pod 60 active · peak 3
Local traffic
Shared pool
EnvoyHTTP/1 → HTTP/1
6 open connections
Storage
Azure Cosmos DB
6 busy · 0 idle · 6 concurrent upstream requests
Envoy upgrades to HTTP/2
At steady load, the same 6 requests share 1 retained HTTP/2 connection, each on its own concurrent stream.
Requests →
Python pods
Pod 12 active · peak 3
Pod 20 active · peak 3
Pod 31 active · peak 3
Pod 40 active · peak 3
Pod 53 active · peak 3
Pod 60 active · peak 3
Local traffic
Shared pool
EnvoyHTTP/1 → HTTP/2
1 open connection
Storage
Azure Cosmos DB
1 busy · 0 idle · 6 concurrent upstream requests
Why Habitat does less
One reason we could scale Python this far was Habitat’s constrained API, which keeps request cost predictable. Rather than allowing clients to construct arbitrary SQL queries that could result in large table scans or joins across many tables, Habitat exposes a simple NoSQL API. The lack of a powerful API is an explicit tradeoff in Habitat’s design.
We aim to optimize for simple, predictable, constant-work requests. In our experience, these systems are substantially easier to scale and difficult to get wrong or misuse. Requests with unpredictable fanout are operationally dangerous: they complicate isolation, load balancing, and introduce latency cliffs that are hard to scale for both the service and its clients.
Before we moved to Habitat and Azure Cosmos DB, most of OpenAI’s online data was stored on Postgres. At that time it was easy to review all query and schema changes to make sure they were well-behaved and operated against indexed data before shipping to production. As the team and products grew, this quickly became unmanageable and was a frequent cause of outages where a single expensive new query on a hot path took out the database.
The problem here is in cost imbalance: it is cheap and easy to write SQL queries that are expensive and hard to run. In Habitat, we avoid this and make expensive queries exceedingly obvious client-side. There are no unbounded queries that can overload Habitat and complex joins and graph traversals require product teams to do some of the heavy-lifting which helps overall optimize for more efficient designs.
Habitat exposes a NoSQL API modeled around client-defined object and edge types, inspired by TAO . Clients predefine objects and edges and how they relate to each other, but not the content of each type. The resulting relationships resemble a graph, but Habitat itself does not support typical graph traversal queries outside of querying direct edges of a particular object.
We partition this graph so that each object and its corresponding edges are colocated in a storage-level partition, but we make no concerted database-level effort to colocate objects and the remote objects to which their edges point. The result is that the model easily partitions for horizontal scalability, but graph traversals are inefficient since any particular hop between objects may require fetching from two entirely different Azure Cosmos DB accounts stored in different regions.
For clients with more complex querying needs, we do provide an offline secondary view of Habitat exposed via Rockset. We use change data capture (CDC) to stream changes from the online storage out to isolated Rockset instances in near-real-time. Each client team is responsible for scaling their own Rockset instance for their complex querying needs.
This Rockset provisioning introduces extra friction to our clients, but we think is the right tradeoff to make at this particular moment: making simple queries the default while providing an escape hatch for those who need complex queries. This design isolates our online storage from read-heavy analytical and search workloads.
Migrate from Python to Rust
Deferring a Python rewrite for a year allowed us to focus on more urgent and impactful challenges during our hypergrowth. With the platform maturing and our growth continuing to accelerate, and being the second largest service by core count at OpenAI (and fourth for our Envoy footprint), it was finally time to move past Python. At its peak, Python helped us serve more than 20 million requests every second.
In Q2 2026, with just 2 engineers, Codex, and GPT‑5.5, we were able to rewrite the entire service in Rust. This new Rust service is now handling 95% of our production requests; we’ll be deprecating Python entirely in the coming weeks. Our data shows the Rust service is 6x more CPU efficient and 15x more memory efficient than the Python version, with significantly lower average and tail latencies. We plan to share more learnings in a future blog.
Optimizing our database layer, Azure Cosmos DB
The Python—and now Rust—service is only one facet of Habitat. In part II of this series explaining how we rapidly scaled our online storage to serve over 1 billion ChatGPT users, we’ll talk about the storage layer and how Habitat serves more than 500 petabytes and over 70 million requests every second.
If you want to work on OLTP systems at frontier scale and are interested in this kind of engineering, check out this open role on our team.