Production AI System Design: From Model Call to Reliable Product

const example = true;

request -> prompt -> model -> response

That is enough to prove that a model can do something useful. It is not enough to prove that a product can do it reliably.

The gap matters because LLM adoption is no longer a niche engineering activity. GitHub's 2025 Octoverse reported more than 1.1 million public repositories importing LLM SDKs, up 178% year over year. At the same time, the 2025 Stack Overflow Developer Survey reported that distrust of AI-tool accuracy exceeded trust. The production problem is therefore not simply how to call a capable model. It is how to make model behavior observable, bounded, recoverable, testable, secure, and economically predictable.

This article presents a vendor-neutral production AI system design built around one principle:

Treat the LLM as a nondeterministic external dependency inside an otherwise engineered software system.

The architecture below was implemented as a TypeScript reference example and exercised with deterministic fake providers and fault injection. The measured results in this article are local harness measurements, not benchmarks of OpenAI, Anthropic, Google, AWS, or any other external model provider.

The Model Call Is Not the Product#

A prototype puts the model in the center of the architecture. A production system puts a contract around it.

The model is still important, but it is only one dependency. Around it sit conventional software-engineering concerns: authentication, authorization, deadlines, validation, retries, concurrency control, observability, cost accounting, deployment gates, and rollback behavior.

AWS's Generative AI Lens reliability guidance explicitly treats inference availability, response consistency, recovery, scaling, prompt/model management, and continuous performance evaluation as reliability concerns. Google's guidance for deploying and operating generative AI applications similarly emphasizes versioning, CI/CD, monitoring, automated evaluation, and continuous feedback.

A production design should therefore begin with the behavior your application promises, not with the model SDK you happened to prototype with.

Define the Product Contract First#

Before choosing an orchestration framework or adding a model gateway, define the application contract.

At minimum, distinguish:

• successful result
• explicit refusal
• incomplete or degraded result
• retryable infrastructure failure
• permanent infrastructure failure
• syntactically invalid output
• schema-valid but business-invalid output

The reference implementation keeps provider response types behind the application boundary in:

src/contracts.ts

That separation is important. If every route, job, or service directly depends on a provider SDK's response object, provider details leak into business logic and make testing, migration, and failure handling harder.

Structured generation helps, but it is not sufficient. OpenAI's current API supports schema-constrained structured outputs using JSON Schema, which can reduce malformed output. The important production pattern is to validate again at the application boundary and then apply domain rules.

The distinction was visible in the reference experiment:

The test used only five deterministic samples, so these percentages are not a model-quality benchmark. Their value is architectural: parsing, structure, and business correctness are different checks.

One response can be perfectly valid JSON and still be unusable because a confidence threshold is too low, a citation is missing, an identifier is invalid, or a domain constraint was violated.

The tested reference files are:

• src/validation/schema.ts
• src/validation/business.ts

The takeaway is simple: structured output makes a model easier to integrate, but the product contract still belongs to your application.

Reference Production AI Architecture#

A useful baseline request path looks like this:

This architecture separates four concerns.

Application concerns decide what the product is trying to do.

Model concerns handle provider interaction, model identity, parameters, token usage, and model-level failures.

Deterministic controls validate inputs, outputs, permissions, and business rules.

Cross-cutting controls provide observability, evaluation, security, cost controls, rate limiting, and caching.

The exact deployment topology can differ. A small application might run most of these concerns in one service. A larger platform might split them across gateways, queues, policy services, evaluation services, and telemetry pipelines. The important property is not the number of services. It is whether each failure boundary has an owner and an observable result.

Use a Small Model Gateway#

Calling a provider SDK from one prototype function is fine. Scattering provider calls throughout a production codebase is much harder to operate.

A small model gateway gives you one place to enforce:

• model and provider identity
• timeout policy
• error classification
• retry policy
• token and usage accounting
• trace metadata
• structured-output parsing
• routing and fallback policy

The tested reference uses:

• src/model-gateway.ts
• src/providers/adapter.ts

The goal is not to create an enormous internal AI framework. Over-abstraction can hide capabilities you actually need. Provider-specific features should remain accessible through thin adapters when they are valuable.

A good gateway should make the common production path consistent without pretending every model is interchangeable.

Build Deterministic Boundaries Around Model Output#

The most useful mental model is:

model output
   ↓
parse
   ↓
schema validation
   ↓
domain validation
   ↓
policy validation
   ↓
application result

Each step answers a different question.

Parsing: Can the application read the output?

Schema validation: Does it have the required shape and data types?

Domain validation: Does it satisfy business constraints?

Policy validation: Is the result allowed to produce the requested outcome?

This layered approach prevents a common mistake: treating "the model returned valid JSON" as equivalent to "the model returned a valid product result."

It also gives observability a better vocabulary. Instead of recording a generic "LLM failed" event, the system can record whether a failure occurred during transport, parsing, schema validation, domain validation, authorization, or downstream execution.

Timeouts, Retries, Backoff, and Error Classification#

Retries are useful only when the failure is likely to be transient.

A retry policy should distinguish at least:

• network interruption
• timeout
• rate limiting such as HTTP 429
• transient provider 5xx
• authentication or authorization failure
• malformed output
• deterministic schema failure
• business-rule failure

A permanent authentication error should not be retried. A business rule violation should not be retried merely because the output came from a model. A temporary 429 or 5xx may be retryable, but only inside a bounded latency and attempt budget.

The reference reliability envelope looks like this:

What the timeout experiment showed#

Four controlled delay cases were exercised: 500 ms, 2 seconds, 5 seconds, and one request exceeding the configured deadline.

Measured locally:

• 4 total cases
• 75% completion rate
• 25% timeout rate
• 5007.8 ms measured p95 latency
• the over-deadline request was cancelled with AbortSignal

The p95 value should not be read as a real production target. With four synthetic cases, it only confirms that the harness captures end-to-end delay and that the deadline cancels work instead of allowing it to run indefinitely.

What the retry experiment showed#

The retry harness compared no retry, fixed retry, and bounded exponential backoff with jitter against simulated 429/500 failures.

Measured provider calls for one logical task:

The recorded retry amplification factor was 3, and the simulated p95 latency was 60 ms.

Again, this is not evidence that exponential backoff is faster than a real provider. Backoff time was virtual and deterministic. The test verifies something more fundamental: transient failures recovered inside the configured retry budget, while a permanent failure was not amplified by useless retries.

That distinction is what a production retry policy needs to prove.

Protect Side Effects With Idempotency and Approval Boundaries#

As soon as a model can trigger a mutation—send a message, update a ticket, create a deployment, issue a credit, modify a record—the architecture changes.

The model should propose an action. Deterministic application code should decide whether that action is allowed and whether it has already happened.

This design protects against two independent sources of repetition:

1. infrastructure retries
2. model or workflow loops that propose the same action more than once

The reference implementation uses a safe mock action in:

src/actions/idempotent.ts

For higher-impact operations, add explicit approval, stronger authorization, narrower tool permissions, or human review. Anthropic's engineering guidance on effective agents emphasizes grounding agent behavior in environmental feedback and using clear stopping conditions; the same principle applies here: autonomous execution should have deterministic boundaries.

Model Routing and Fallback Need Measurement#

Fallback is often described as an availability feature:

primary fails -> call secondary -> success

But models are not identical replicas. A fallback can change quality, latency, output format, safety behavior, context limits, or cost.

The reference test compared primary-only behavior with a primary-plus-fallback policy across a healthy primary, timeout, rate limit, and 5xx scenario.

Measured locally:

The latency is tiny because the test uses deterministic fake providers; it must not be compared with an external model API. The cost units are synthetic metadata, not currency.

The useful finding is behavioral: fallback recovered configured retryable failures and did not mask an authentication failure.

That is the right design goal. Fallback should be triggered by explicit policy, traced every time it happens, and evaluated as a separate production path.

The tested implementation lives in:

src/routing/fallback.ts

Observe the Whole AI Request#

Traditional service telemetry answers questions such as:

• did the endpoint return?
• how long did it take?
• did a dependency fail?

AI systems need those answers plus more context.

A useful trace should be able to associate a request with:

• application version
• prompt or configuration version
• provider and model identity
• model parameters when relevant
• input/output token usage
• model-call latency
• retry count
• fallback use and reason
• retrieval or tool calls
• output-validation result
• business-validation result
• error classification
• evaluation result or later user feedback

OpenTelemetry's GenAI observability guidance and semantic conventions provide a standards-oriented basis for instrumenting these flows.

The reference implementation uses:

src/observability/tracing.ts

A critical privacy rule is that telemetry does not need to store full prompts and completions by default. Model content can contain user data, secrets, proprietary documents, or other sensitive material. Capture content only when the product has a clear reason, appropriate controls, and a defined retention policy.

The feedback loop should connect runtime behavior back into engineering:

Reliability SLOs Must Include Quality#

A service can return HTTP 200 quickly and still fail its user.

That means production AI reliability needs at least two classes of signals.

Operational signals#

• availability
• request latency
• model latency
• timeout rate
• dependency error rate
• rate-limit rate
• retry count
• fallback rate

Product-quality signals#

• schema-valid response rate
• business-valid response rate
• task success
• groundedness or correctness where measurable
• safe refusal behavior
• tool-selection accuracy
• cost per successful task

The exact SLOs are product-specific. A document summarizer, code migration agent, medical workflow assistant, and autonomous deployment tool should not share the same quality or risk threshold.

The point is to define the threshold before a release, measure it consistently, and make regressions visible.

Load testing the failure boundary#

The reference load experiment sent 80 tasks to a capacity-limited fake provider.

Measured locally:

• throughput: 187.58 tasks/second
• p50 latency: 15.24 ms
• p95 latency: 41.03 ms
• p99 latency: 41.84 ms
• error rate: 90%
• rate-limit rate: 90%

Those numbers intentionally describe an overloaded synthetic dependency. They are not performance goals. The high error and rate-limit rates demonstrate that a capacity boundary is visible to the harness and that tail latency, throughput, and failure behavior can be measured together.

A production load test should repeat the same pattern against your own service architecture and approved provider quotas, using realistic request sizes and concurrency.

Security, Cost, and Evaluation Are Architecture Concerns#

Security#

AI security should not be reduced to prompt injection.

OWASP's current GenAI guidance covers prompt injection as well as improper output handling, sensitive-data exposure, excessive agency, and unbounded resource consumption. In architectural terms, treat these boundaries separately:

user input
retrieved content
tool output
model output
downstream interpreter
business side effect

Do not assume retrieved documents are trusted because they came from your RAG pipeline. Do not assume model output is safe because the prompt asked for safe behavior. Do not pass generated text directly into shells, SQL interpreters, HTML renderers, or privileged tools without the deterministic validation appropriate to that context.

Cost#

Token price is only part of the unit economics.

A useful request-cost model can include:

input tokens
+ output tokens
+ retries
+ retrieval
+ reranking
+ tool calls
+ fallback calls
+ evaluation overhead

The metric that matters most is often cost per successful task, not cost per model call.

That makes retry, routing, context size, and evaluation architecture economically observable rather than being treated as separate billing concerns.

Evaluation#

Evaluation belongs in the deployment path, not in a notebook that someone runs occasionally.

Google's production guidance recommends automated evaluation in delivery workflows and continuous evaluation after deployment. The reference implementation includes a CI-oriented release gate in:

src/eval/release-gate.ts

A useful release decision should be able to fail independently on:

• quality
• safety
• schema or contract compliance
• latency
• reliability
• cost

A single aggregate score can hide a critical regression.

This is developed further in the companion cornerstone How to Evaluate LLM Applications Before Production.

What Changes With RAG, Tools, and Agents?#

The architecture does not become invalid when you add more capable AI patterns. The number of failure surfaces grows.

RAG#

Retrieval-augmented generation adds:

• ingestion quality
• parsing
• chunking
• index freshness
• retrieval quality
• authorization filters
• evidence assembly
• citation validity
• deletion propagation

A model can be healthy while the retrieval layer returns the wrong evidence. That is why RAG needs component-level evaluation in addition to end-to-end answer evaluation.

The companion cornerstone Production RAG Architecture: Ingestion, Retrieval, Citations, and Evaluation treats RAG as an evidence system rather than a vector-database feature.

Tools#

Tools add deterministic side effects and additional trust boundaries.

For each tool, define:

• who may call it
• what arguments are valid
• what the tool is allowed to mutate
• whether the call is idempotent
• how failure is reported
• what telemetry is captured
• whether human approval is required

Agents#

Agents add multi-step state, repeated tool use, stopping conditions, partial progress, and more opportunities for errors to compound.

The right default is not "make everything an agent." Add autonomy only when the task benefits from it and when the evaluation, observability, permissions, and recovery model can support the larger failure surface.

Production Readiness Checklist#

Before exposing an AI feature to real users, verify that you can answer these questions.

Contract#

• Is the input contract explicit?
• Is the output contract explicit?
• Can the system distinguish success, refusal, degraded output, retryable failure, and permanent failure?
• Are schema validation and business validation separate?

Reliability#

• Is there a total request deadline?
• Are retries limited to classified transient failures?
• Is exponential backoff or equivalent recovery bounded?
• Can retry amplification be measured?
• Is fallback policy explicit and separately evaluated?

Side effects#

• Are actions authorized outside the model?
• Are business mutations idempotent?
• Are high-impact actions gated appropriately?

Observability#

• Can a request be tied to app, prompt/config, model, and schema versions?
• Are retries, fallbacks, latency, token usage, and validation outcomes visible?
• Is sensitive model content excluded from telemetry by default?

Quality and evaluation#

• Is there a representative evaluation set?
• Do release gates cover quality as well as latency and availability?
• Can production failures become regression cases?
• Can the team compare candidate behavior with a known baseline?

Security#

• Are user input, retrieved context, tool output, and model output treated according to their trust level?
• Is generated output validated before it reaches an interpreter or privileged action?
• Do tools use least privilege?

Cost#

• Can you measure retries and fallback calls?
• Can you estimate cost per successful task?
• Are expensive paths visible in traces or metrics?

Operations#

• Have load and dependency-failure paths been tested?
• Is graceful degradation defined?
• Is rollback possible?
• Can you identify which model/config version produced a problematic result?

If several of these answers are unknown, the system may still be a useful prototype—but it is not yet a well-defined production system.

Final Reference Architecture#

The complete reference architecture combines the request path, reliability envelope, trust boundaries, observability, and evaluation loop:

The key design choice is not any individual box. It is the separation between nondeterministic model behavior and deterministic product guarantees.

A model can be upgraded. A provider can change. RAG can be added. Tool calling can be introduced. An agent loop can replace a single generation step.

The surrounding engineering principles should remain recognizable:

contract -> bounded execution -> validation -> observability -> evaluation -> controlled release.

That is the foundation for building AI systems that behave like products instead of demos.

Reference Implementation#

This article is backed by a tested TypeScript reference implementation published in ProdAI Stack Labs. It demonstrates typed application contracts, a model gateway, structured-output and business validation, deadline-aware retries, idempotency, fallback routing, observability, evaluation release gates, and deterministic experiments. The relevant project modules are:

• src/contracts.ts
• src/model-gateway.ts
• src/providers/adapter.ts
• src/validation/schema.ts
• src/validation/business.ts
• src/reliability/retry.ts
• src/actions/idempotent.ts
• src/routing/fallback.ts
• src/observability/tracing.ts
• src/eval/release-gate.ts

The verified commands were:

npm run typecheck
npm run lint
npm run test
npm run example:production-ai
npm run example:diagrams

The import package does not embed the exact TypeScript source bodies in its codeExamples[].code fields, so this article does not reproduce unverified snippets. The repository paths above are the canonical implementation locations.

Sources and Further Reading#

• AWS — Reliability, Generative AI Lens
• AWS — Graceful recovery for generative AI workloads
• AWS — Generative AI Lens design principles
• AWS — Maintaining model performance
• Google Cloud — Deploy and operate generative AI applications
• OpenTelemetry — GenAI observability
• OpenTelemetry — Semantic conventions
• OWASP — Top 10 for Large Language Model Applications
• OpenAI — Structured Outputs / Responses API
• Anthropic — Building Effective AI Agents
• GitHub — Octoverse 2025
• Stack Overflow — 2025 Developer Survey: AI

git clone https://github.com/sarvab-dev/prodai-stack-labs
cd prodai-stack-labs/production-ai-system-design
npm install
npm test