The Reference Design: An Executable Blueprint for the AI Platform
The design to hand a Staff MLOps engineer on day one — the stack and the seams that keep it replaceable, six pipelines with the gates wired in, gateway and policy configuration, and a 90-day plan with definitions of done.
On this page
Parts 1 and 2 quantified the value and risk mechanisms of an AI platform; this part specifies the implementation. What follows is the design I would hand a Staff MLOps engineer on day one: a reference stack with the seams that make every component replaceable, the repository and environment layout, six pipelines with the gates wired in, the gateway and policy configuration, and a 90-day execution plan with definitions of done — plus the headcount and run-cost sizing an executive sponsor will ask for before any of it starts. The reference implementation is open-source-forward so it reads vendor-neutrally; every row lists managed swap-ins, because the seam outlives the product choice. Configuration snippets are illustrative and minimal; adapt syntax to your chosen engines.
Reading paths: executives — the traceability table just below, then the investment scenarios near the end. Engineers — everything in between.
Component-to-claim traceability
Each component below is justified by a specific, quantified claim from Parts 1 and 2. Components without such a claim were excluded.
| Design element | The claim it delivers |
|---|---|
| AI gateway in every request path, metering per team/use case | Cost per decision; the −88% routing/caching/batching reduction; budgets that block (Part 1) |
| Credentials issued only via registry entry | Inventory complete by construction; shadow AI removed as a provisioning path (Part 2) |
| Evaluation harness gating CI, portable against any endpoint | Gate B; quality held constant across routing tiers; the evidence that makes provider swaps safe (Parts 1–2) |
| Drift + data-contract monitors with paging thresholds | Detection lag reduced from weeks to days: ~$300k avoided per silent episode (Part 2) |
| Champion–challenger aliases + tested rollback | Pre-approved remediation; controlled launches (Parts 1–2) |
| Open formats at rest, OCI artifacts, configs in Git | Exit tests pass; two or three engine swaps per decade without a rewrite (Part 2) |
| Baseline capture + unit-economics join as template steps | Gate A/C attribution; the $880k→$330k verification gap (Part 1) |
Architecture overview
Reference stack and replacement seams
| Capability | Reference choice | Managed swap-ins | The seam that keeps it replaceable |
|---|---|---|---|
| Source, CI/CD | Git + GitHub Actions | GitLab CI, Azure DevOps | Pipelines-as-code; OIDC to cloud (no stored secrets) |
| Infrastructure | Terraform + Kubernetes | EKS/AKS/GKE; managed node pools | Declarative state; K8s API; GPU pools with spot tolerations |
| Lakehouse | Object storage + Parquet/Iceberg (or Delta) | Databricks, Snowflake, BigLake | Open table format readable in place by a second engine |
| Tracking + registry | MLflow (self-hosted) | Managed MLflow, SageMaker/Vertex registries | OSS API; models exported as OCI artifacts + safetensors |
| Training compute | Containerized jobs on K8s (+ Ray for scale-out) | AzureML, SageMaker, Vertex jobs | Job = container + config + data URI; checkpoints to object storage; no proprietary SDK inside the training loop |
| Serving | KServe (predictive) + vLLM (self-hosted LLMs) | Managed endpoints per cloud | REST/gRPC + OpenAPI contract; image deploys on vanilla K8s |
| AI gateway ★ | LiteLLM or Envoy AI Gateway | Kong AI, APIM GenAI policies, Bedrock gateway | OpenAI-compatible facade; provider swap = config + eval run |
| Vector/knowledge index | pgvector (start), OpenSearch at scale | Managed search/vector services | Canonical docs + chunk/embed configs in Git; index rebuildable in under a week |
| Evals ★ | promptfoo / lm-eval + pytest harness; golden sets in Git (DVC/LFS) | Cloud eval suites (as runners, never as system of record) | Runs headless against any endpoint URL; results to the spine |
| Data quality/contracts | Great Expectations or dbt tests | Cloud-native equivalents | Expectations versioned beside the pipeline they guard |
| Drift/quality monitoring | Evidently + custom detectors as config | Vendor ML-monitoring suites | Detector definitions in Git; alerts via standard incident API |
| Observability | OpenTelemetry → Prometheus/Grafana/Loki (or Tempo) | Datadog, cloud APM | OTel collector owns exporters; dashboards as code; pin GenAI semconv version |
| Lineage/catalog ★ | OpenLineage events → DataHub or OpenMetadata | Unity Catalog, Purview | Event format + stable IDs; consumers read via facade API |
| Policy | OPA/Conftest bundles in CI + admission | Cloud policy engines alongside | Same Rego evaluates anywhere |
| Supply chain | Dependency proxy (e.g., Nexus/Artifactory), syft SBOM, cosign signing | Cloud-native registries + signing | OCI registry API; SPDX/CycloneDX; verifiable provenance |
★ marks the three seams to own outright (Part 2). All other components are expected to be re-evaluated, and potentially replaced, at annual review.
Repositories and environments
One platform repository, plus one repository per use case generated from a template. The platform repository owns everything shared; use-case repositories stay thin, which is what makes the sixth use case a three-week job.
platform/
├── infra/ # Terraform: cluster, GPU pools, registry, gateway, observability
├── gateway/ # routing tiers, budgets, guardrails — config-as-code
├── policies/ # OPA bundles: registration, tagging, deployment gates
├── pipelines/ # reusable CI/CD workflows (called by use-case repos)
├── templates/ # scaffolds: classical-ml/ rag-app/ batch-scoring/
├── evals/ # shared harness + golden datasets (versioned)
├── monitors/ # drift + data-contract detector definitions
└── docs/adr/ # decision records: one page each, with revisit triggers
use-case repo (generated from template):
├── VALUE.md # the signed hypothesis — Gate A artifact, PR-blocking if absent
├── conf/ # params, routing tier, budget, tags — the single config surface
├── src/ # features, training, inference handler
├── data_contracts/ # input schema + expectations
├── evals/ # use-case golden set + thresholds (extends shared harness)
├── pipelines/ # thin wrappers calling platform reusable workflows
└── cards/ # model card — generated at release, never hand-editedThree environments — dev, staging, prod — each with its own credentials, data scope, and gateway tenant. Promotion moves code, and the pipeline retrains in each environment from reviewed code, which keeps lineage clean and satisfies the strictest audit posture. Where retraining is expensive (large fine-tunes), promote the artifact and re-run the full evaluation suite in the target environment: the requirement is evidence in the target environment; the procedural form may vary. In either pattern, a release is a registry alias transition (challenger → champion) with the evaluation report attached, and the author of a change can never be its production approver.
Pipeline definitions and gates
| Pipeline | Trigger | Function | Gate enforced |
|---|---|---|---|
| 1 · CI | Pull request | Lint, unit tests, data-contract validation, policy check (Conftest), VALUE.md present | Gate A artifacts exist |
| 2 · Train | Merge / schedule / retrain signal | Containerized training from config; checkpoints to object storage; run + lineage logged; candidate registered | — |
| 3 · Evaluate | New candidate | Full eval suite vs thresholds: quality, safety, bias, cost-per-decision estimate; report attached to registry entry | Gate B (blocks promotion) |
| 4 · Release | Alias transition approved | Canary 5%→50%→100% with automatic rollback on SLO or quality-guard breach; model card generated | Gate B (independent approver) |
| 5 · Monitor | Continuous | Drift, nulls, score distribution, token cost anomalies; breaches open incidents with runbook links | Feeds Gate C/D evidence |
| 6 · Measure | Nightly | Unit-economics join: gateway token metrics + infra share ÷ decisions; benefits ledger update | Gate C/D (quarterly review pack generated automatically) |
The evaluation gate in CI (GitHub Actions syntax; any CI system with the same structure works):
# .github/workflows/evaluate.yml (called by use-case repos)
jobs:
evaluate:
runs-on: [self-hosted, gpu] # spot-backed runner pool
steps:
- uses: actions/checkout@v4
- name: Run evaluation suite against candidate endpoint
run: |
python -m platform_evals.run \
--endpoint "$CANDIDATE_URL" \
--suite evals/ --shared-suite "$PLATFORM_EVALS" \
--thresholds evals/thresholds.yaml \
--report out/eval_report.json
- name: Attach evidence to registry
run: python -m platform_evals.attach --model "$MODEL_URI" --report out/eval_report.json
- name: Enforce thresholds # non-zero exit = no promotion, no exceptions
run: python -m platform_evals.enforce --report out/eval_report.jsonA drift monitor as configuration — reviewable, versioned, and inexpensive enough to remove schedule pressure as a justification for omission:
# monitors/complaint-triage.yaml
model: complaint-triage # registry ID — monitor won't load without one
detectors:
- name: feature_null_rate
type: data_contract
columns: [product_code, channel]
threshold: 0.02 # 2% nulls in 1h window
- name: score_distribution_shift
type: psi # population stability index
baseline: training_reference # snapshot stored at release
threshold: 0.2
window: 6h
- name: token_cost_per_case
type: cost_anomaly
threshold_pct_over_baseline: 40
on_breach:
open_incident: severity_3 # model-quality incident type (Part 2)
runbook: runbooks/triage-drift.md
auto_action: route_to_challenger # pre-approved remediation onlyGateway configuration
# gateway/config.yaml (LiteLLM-style; Kong/APIM/Envoy equivalents exist)
model_list:
- model_name: tier-small # 80% of document traffic (Part 1 cost table)
litellm_params: { model: <small-tier-model>, rpm: 5000 }
- model_name: tier-frontier # escalations only
litellm_params: { model: <frontier-model> }
router_settings:
fallbacks: [{ tier-frontier: [tier-frontier-alt] }] # provider swap = this line + eval run
enable_pre_call_checks: true
litellm_settings:
cache: true # prefix/semantic cache
success_callback: [otel] # tokens → telemetry spine, per key
# per-team keys carry: budget (blocking), tags, allowed model tiersEvery property Parts 1 and 2 priced — metering, blocking budgets, caching, routing, provider swappability, and the kill switch (disable a key or tier in one place) — is implemented by these few dozen lines, provided applications cannot reach providers except through the gateway. Enforce this with network policy; do not rely on convention. To size the routing and caching tiers against current provider list prices, use the LLM Cost Calculator.
Policy and controls implementation
The policy bundle runs in CI (Conftest) and at deploy time (admission control). The core rule — nothing undocumented reaches production — is about ten lines of Rego:
package deploy
deny[msg] {
not input.model.registry_id
msg := "model has no registry entry — credentials require registration"
}
deny[msg] {
not input.model.eval_report.passed
msg := "no passing evaluation report attached to this version"
}
deny[msg] {
input.model.tier >= 2
not input.approvals.independent_reviewer
msg := "material model requires approval independent of the author"
}
deny[msg] {
not input.resource.tags.cost_center
msg := "untagged resources cannot be created"
}Combined with the provisioning rule from Part 2 — model-calling credentials are minted only by the registration workflow — the inventory, the tagging, the evidence trail and the segregation-of-duties gate are all structural. Model cards are rendered from registry metadata at release (owner, data lineage, evaluation results, thresholds, monitoring configuration), so documentation is a build artifact with zero marginal cost, and it is never out of date because it is never separately maintained.
90-day execution plan
| Weeks | Build | Definition of done (measurable) |
|---|---|---|
| 1–2 | Terraform the foundations: cluster, OCI registry, object storage, OTel stack, OIDC federation, dependency proxy | A hello-world container builds, signs, deploys via CI with zero stored secrets; traces visible end-to-end |
| 3–4 | Gateway + registry + registration-gated credentials; tag-or-deny policy live | Direct provider egress blocked; a test key shows per-team token metrics; untagged resource creation fails; unregistered model cannot obtain a key |
| 5–8 | Paved-road template v1 (the pattern the first use case needs) + lighthouse use case #1 in shadow mode, baseline capture running | Use case scaffolds to first pipeline run in under a day; shadow predictions logged against the incumbent process; VALUE.md signed |
| 9–10 | Eval harness gating CI; monitors + model-quality incident type wired to on-call | A deliberately degraded candidate is blocked by Gate B; a synthetic drift injection pages within one detection window and opens an incident with the correct runbook |
| 11–12 | Lighthouse #2 on the template (reuse test); nightly unit-economics join; showback report v1 | Use case #2 reaches shadow in ≤3 weeks; cost per decision for #1 appears on a dashboard finance has reviewed |
| 13 | First quarterly review pack — generated from registry, monitors and ledger; exit rehearsal #1 (index rebuild on the reference alternative) | Review pack contains hypotheses, baselines, verdicts, and at least one kill or descope; rehearsal has a measured duration and eval delta, filed as an exit card |
Deliberately deferred, with the reasons a review will ask for: a feature store (add when feature reuse across ≥3 use cases is demonstrated; premature feature stores become shelfware), multi-region serving (until an availability requirement names a number), fine-tuning infrastructure (rent until evaluation evidence shows grounded prompting exhausted), and autonomous remediation (Part 2’s graduated-autonomy ladder starts at recommend). Each deferral carries a named re-entry trigger; documented deferrals are part of the plan and will be asked about at review.
Investment scenarios: headcount and technology cost
Platform proposals fail executives in two common ways: understated requests that are funded and later double, or comprehensive requests that are never funded. The alternative is a scenario table with fully loaded numbers and the return calculation attached. Figures below are illustrative mid-2026 USD at a blended fully-loaded engineering cost of ~$220k/year (adjust ±40% by geography and seniority mix); each run rate already includes the 10–15% renewal allocation from Part 2 — a run rate quoted without a renewal allocation understates the cost of keeping the platform viable past year five. One boundary matters for clean accounting: these are platform and shared-run costs only. Use-case delivery squads are funded by the business lines whose Gate-A hypotheses they serve, and their cost sits inside each use case’s own business case (Part 1).
| Scenario 1 · Prove | Scenario 2 · Federate | Scenario 3 · Institutionalize | |
|---|---|---|---|
| Ambition | One business area; 5–15 use cases on-pattern; managed services throughout | Several business lines; 20–60 use cases; hybrid managed + selective self-hosting; formal on-call | Enterprise/regulated scale; 100+ use cases; self-hosted LLM serving + GPU fleet; 24/7; automated audit evidence |
| Platform headcount | 5 — 1 staff lead, 2 platform eng, 1 MLOps/SRE, 1 data eng; security & FinOps as fractional partners | ~12 — adds platform PM, eval engineer, 2nd SRE/AIOps, lineage/data eng, FinOps analyst; small enablement rotation | ~25–30 — 3–4 product teams (road, gateway/serving, spine/governance), 2–3 enablement, sustained on-call; ML-literate validators partnered in 2nd line |
| People cost / yr | ≈ $1.1M | ≈ $2.6M | ≈ $5.5–6.6M |
| Technology run cost / yr | $0.3–0.6M — modest GPU, low token volume, OSS-forward tooling | $1–2.5M — GPU pools, gateway at volume, observability at retention, first provider commitments | $4–10M — GPU fleet incl. self-hosted serving, multi-environment + DR, premium observability, committed capacity |
| Total run rate / yr | ≈ $1.4–1.8M | ≈ $3.5–5M | ≈ $10–16M |
| Clears a 2× return with… | ~a dozen mid-size wins plus 2–3 large ones (fraud/ops-scale, $1M+ net) | a portfolio where several use cases each clear $1M+ net — routine at this scale in risk, fraud and operations |
First finance-verified benefit: one to two quarters in every scenario — the 90-day plan above is scenario-independent.
Four checks apply to any version of this table. People dominate, at roughly 60–70% of run rate: a proposal whose cost is mostly licenses has usually omitted the engineers, and one padded with generic headcount has missed that the team stays small by design — platform headcount grows sub-linearly with use cases (Scenario 3 serves 20× the portfolio of Scenario 1 on ~5× the team, consistent with the declining marginal cost described in Part 1). The role mix is diagnostic: an eval engineer and SREs indicate a team that intends to operate what it ships, while a plan staffed entirely with data scientists describes a research group rather than an operating platform. Consumption grows with success: tokens and GPU scale with adoption, so a rising technology line with falling unit costs indicates adoption rather than overrun — budget the winners’ growth explicitly, or the quarterly review will starve the best use cases at the moment they scale. Fund it as a run rate rather than a project: staffing a platform like a project and disbanding at go-live converts the entire first-year spend into an unowned proof of concept.
Acceptance criteria
- An unregistered model cannot obtain credentials — verified by attempting it, quarterly.
- A provider swap behind the gateway completes in a dev environment in under a day, with the evaluation suite as the acceptance test.
- A synthetic drift injection pages on-call within one detection window, and the runbook’s first step works as written.
- A degraded candidate is blocked by CI with a human-readable evaluation report rather than a stack trace.
- Cost per decision for every live use case appears on one dashboard, and finance has agreed with at least one of the numbers.
- The quarterly review pack generates from systems, and the meeting argues about decisions rather than data.
Each check maps to a claim in Parts 1 and 2; together they are the difference between having a platform and having installed some tools. Run them internally on a schedule; the alternative is having an auditor, a CFO, or an incident run them for you.
Part 1 attached a P&L owner, four gates and a unit cost to every use case; Part 2 priced silent failures and made replaceability the longevity strategy; Part 3 specified the implementation that enforces both in code. None of it requires exotic technology; it requires making the compliant path and the fast path identical.
References
- MLflow — model registry & aliases; KServe — serving on Kubernetes; vLLM — self-hosted LLM serving.
- LiteLLM — gateway/proxy configuration (budgets, routing, caching, callbacks); comparable: Kong AI Gateway, Azure APIM GenAI policies, Envoy AI Gateway.
- Open Policy Agent — Rego & Conftest; Sigstore — cosign signing; SPDX/CycloneDX SBOM via syft.
- OpenTelemetry — GenAI semantic conventions (pin versions; stabilizing through 2026).
- OpenLineage — lineage event standard; DataHub / OpenMetadata as catalog backends.
- Evidently — drift detection; Great Expectations — data contracts; promptfoo / lm-eval — LLM evaluation harnesses.
- Google Cloud, MLOps: continuous delivery and automation pipelines in machine learning — the maturity framing this design’s pipelines align to.
Snippets are illustrative and minimal, not production manifests; adapt to your engines and security baseline. Tool choices are July 2026 defaults, selected together with the seams that make them replaceable.