Engineering
AI App Scalability
Short answer
Most AI-built apps do not need Kubernetes. They need indexes, pagination, a queue for LLM jobs, and a cap on tokens per user.
What is AI app scalability?
The ability to add users without the bill or the latency becoming a surprise. For AI products, the model invoice is usually the first ceiling — not CPU.
Prototype vs scalable AI app
| Layer | Prototype | Scalable |
|---|---|---|
| LLM calls | On every keystroke, no cache | Debounced, cached, queued |
| DB | Select * , no indexes | Indexes on foreign keys + filters |
| Files | Public bucket | Signed URLs, size limits |
| Jobs | In the request | Background worker |
Fix the product loop before the cluster
If one user can trigger 200 GPT-4o calls, you do not have a scale problem — you have a product bug. Add usage limits first. See the API cost calculator.
FAQ
When do I need a queue?
When a user action can take more than a few seconds (embeddings, reports, video) or when you cannot afford to lose the job if the tab closes.
Related
Latency or OpenAI bill exploding?
Those are usually the same ticket. We’ll find the unbounded loop before we talk about infra.