Typed AI Application Contract
Explicit success/refusal/degraded/error result types.
Hands-on lab
Production AI System Design Lab
Build and test the reliability layer around a production LLM application, including contracts, validation, retries, idempotency, fallback routing, observability, and release gates.
What you will build
Build and test the reliability layer around a production LLM application, including contracts, validation, retries, idempotency, fallback routing, observability, and release gates.
Minimal Model Gateway
Provider-neutral generation boundary for timeout, telemetry, configuration, usage, and parsing.
Provider Adapter
One concrete SDK adapter isolated from business logic.
Prerequisites
- HTTP and API fundamentals
- Node.js or comparable backend experience
- Basic TypeScript
- Basic LLM API usage
- Async error handling
- Basic logging and metrics
Capabilities you will leave with
- Typed AI Application Contract
- Minimal Model Gateway
- Provider Adapter
- Structured Output Validation
- Business Rule Validation
- Deadline, Retry Classification, and Backoff
Project modules
Typed AI Application Contract
Explicit success/refusal/degraded/error result types.
src/contracts.tsMinimal Model Gateway
Provider-neutral generation boundary for timeout, telemetry, configuration, usage, and parsing.
src/model-gateway.tsProvider Adapter
One concrete SDK adapter isolated from business logic.
src/providers/adapter.tsStructured Output Validation
Request schema-constrained output and validate again locally.
src/validation/schema.tsBusiness Rule Validation
Apply deterministic domain checks after schema validation.
src/validation/business.tsDeadline, Retry Classification, and Backoff
Bounded retry policy with error classification, jitter, and total deadline.
src/reliability/retry.tsIdempotent Side-Effect Boundary
Safe mock action guarded by policy, authorization, and idempotency.
src/actions/idempotent.tsPrimary and Fallback Routing
Explicit failure-triggered fallback with telemetry.
src/routing/fallback.tsOpenTelemetry AI Request Trace
Trace application/model boundaries and usage without capturing sensitive content by default.
src/observability/tracing.tsEvaluation Release Gate CLI
Fail CI when quality, reliability, latency, safety, or cost thresholds regress.
src/eval/release-gate.tsExperiment results
Experiment 01
PASSEDTimeout Behavior
Inject 500 ms, 2 s, 5 s, and over-deadline latency.
- Cases
- 4
- Completion Rate
- 0.75
- Timeout Rate
- 0.25
- P95 Latency Ms
- 5,007.8
Experiment 02
PASSEDRetry Amplification
Compare no retry, fixed retry, and exponential backoff under 429/5xx faults.
- Provider Calls
- None: 1 · Fixed: 3 · Exponential: 3 · Non Retryable: 1
- Successful Tasks
- None: 0 · Fixed: 1 · Exponential: 1
- P95 Simulated Latency Ms
- 60
- Retry Amplification Factor
- 3
Experiment 03
PASSEDSchema Validity vs Business Validity
Classify outputs as parseable, schema-valid, and business-valid.
- Samples
- 5
- Parse Success Rate
- 0.8
- Schema Validity Rate
- 0.6
- Business Validity Rate
- 0.2
- Refusal Or Failure Rate
- 0.8
Experiment 04
PASSEDPrimary-Only vs Primary-With-Fallback
Run identical tasks under healthy and failed-primary conditions.
- Primary Only Completion Rate
- 0.25
- Fallback Completion Rate
- 0.75
- Average Deterministic Quality
- 0.87
- P95 Latency Ms
- 0.64
- Fallback Rate
- 0.5
- Cost Units Per Successful Task
- 1.67
Experiment 05
PASSEDConcurrency and Tail-Latency Load Test
Use a mock provider by default; live calls only when explicitly authorized.
- Tasks
- 80
- Throughput Per Second
- 187.58
- P50 Latency Ms
- 15.24
- P95 Latency Ms
- 41.03
- P99 Latency Ms
- 41.84
- Error Rate
- 0.9
- Rate Limit Rate
- 0.9
Run locally
git clone https://github.com/sarvab-dev/prodai-stack-labs
cd prodai-stack-labs/production-ai-system-design
npm install
npm testca6908c