OpenAI revealed the architecture that serves more than 70 million requests per second. See scale, latency, governance, and cost lessons for AI products.
Direct answer
OpenAI reported on September 11, 2026 that Habitat, its online storage platform, serves more than 70 million requests per second, more than 1 billion people per week, and more than 500 petabytes of data. The key business lesson is not to copy this scale, but to separate the data layer from the product, limit unpredictable queries, and treat authorization, residency, observability, and tail latency as architectural requirements.
Scale starts with a predictable interface
Habitat exposes strict NoSQL operations instead of arbitrary queries. This design makes cost and fan-out more predictable, reducing the risk of an expensive query degrading a critical route. In enterprise products, simple contracts also facilitate auditing and per-customer limits.
Centralizing controls reduces coordination
OpenAI moved storage logic from a client library to a central service. Thus, routing, authorization, encryption, auditing, data residency and traffic limiting can evolve without coordinating updates across dozens of applications.
Tail latency is what the user feels
A single answer may depend on many readings. Therefore, average latency hides the problem: p95, p99, event loop delay, saturation and concentration of connections need to be observed by route, region and dependency.
Efficiency must follow stability
The team temporarily accepted the cost of Python to stabilize interfaces and then migrated the service to Rust. According to OpenAI, the new implementation was six times more efficient in CPU and fifteen times in memory. The strategic point is to sequence risk, not adopt a language for fashion.
Nexus Reading
Reliable AI products depend on both the data layer and the model. An enterprise pilot should test isolation, authorization, recovery, and cost under load from the beginning, even when the initial scale is small.
FAQ
What is OpenAI Habitat?
It is the internal online storage platform used by products like ChatGPT, API and Codex to access data with low latency and centralized controls.
Does every company need a platform of this size?
No. The useful pattern is to separate responsibilities and make cost, access, and failures predictable at the actual scale of the product.
Which technical metric deserves attention?
Tail latency, error rate, saturation, and dependency impact are more informative than just average latency.
Essential guides to delve deeper into the decision
This editorial analysis was produced by Nexus from the official sources below, consulted on September 12, 2026. The text is original and interprets practical implications for companies.
- OpenAI — Rapidly scaling online storage to serve over 1 billion ChatGPT users: architecture, volume, service evolution and official Habitat metrics.
- Microsoft Learn — Azure Cosmos DB overview: official documentation of the database service cited by the architecture.
Date reported by the main source: September 11, 2026.