Silent Failures and the Ten-Year Platform: AI Risk, Resilience, and Renewal
The most expensive AI failures return HTTP 200 — pricing model drift as a financial exposure, running statistical failures with incident discipline, and designing a platform that survives to 2036.
On this page
Scope: the two dimensions that rarely appear in launch business cases — loss avoidance and longevity. Part 1 covered benefits reaching the P&L and the cost of each decision; Part 3 specifies the implementation that enforces both.
The most expensive AI failure in your estate is returning HTTP 200 right now. Conventional software fails loudly, with exceptions, error rates, and pages. A machine-learning system can fail while every infrastructure dashboard stays green — low latency, clean logs, HTTP 200s throughout — because what broke is not the service but the statistics. The world shifted, an upstream feed changed encoding, a fraud pattern mutated, and the model’s answers went quietly from good to plausible to wrong. Confirmation often arrives only weeks later, when ground-truth labels catch up. By then the failure has been running in production with a green status indicator.
Financial exposure from model drift
Drift becomes a manageable risk once it is denominated. A worked composite from card fraud, with round numbers:
| Quantity | Value |
|---|---|
| Fraud prevented by the model | $1.2M / month |
| Recall degradation after an upstream schema change silently nulls two features | −15% |
| Incremental fraud leakage while degraded | ≈ $180,000 / month |
| Added false positives (customer friction, review workload) | ≈ $40,000 / month equivalent |
| Exposure per month of non-detection | ≈ $220,000 |
The controlling variable is detection lag. Quarterly model review, the traditional cadence, implies an expected lag of six weeks: ~$330,000 per silent episode. Continuous monitoring with alerting thresholds reduces the lag to days: ~$20–40k. The monitor that closes this gap is a few hundred lines of code and negligible compute. Few risk controls have a better loss-avoided-per-dollar ratio, yet drift monitoring is routinely the first scope cut when a delivery date slips, because the cost of its absence is invisible until a loss event.
Incident management for model-quality failures
The remedy is organizational as much as technical: model degradation becomes a first-class incident type, with detectors, thresholds, an on-call rotation, runbooks, and postmortems. In the strongest implementations I have worked in, the loop has five stages:
- Observe: one telemetry spine (OpenTelemetry, whose GenAI conventions are stabilizing through 2026) carries infrastructure signals, model quality metrics, data-contract checks, and token/cost counters in the same trace context.
- Detect and correlate: drift and quality monitors emit events; correlation links “recall falling” to “upstream feed changed at 01:47”, converting an unexplained quality drop into a diagnosable event.
- Triage: an incident opens automatically with an LLM-drafted summary and probable cause. AI-assisted operations tooling is now generally available from the major clouds; treat vendor MTTR-reduction claims of 40–70% as hypotheses to verify against your own incident data.
- Remediate with graduated autonomy: pre-approved runbook steps first — route traffic to the challenger model, quarantine the feature, roll back the index build. Automation earns autonomy in steps: recommend, then act with approval, then act with audit. An agent that restarts services in a regulated estate is itself a governed model with an inventory entry and a kill switch.
- Learn in code: the postmortem’s deliverable is a set of artifacts — a new data-contract test, a new detector, a new evaluation case — so every incident permanently hardens the estate. For LLM systems this is the compounding advantage: each production failure becomes a regression test, which prevents recurrence of previously observed failure modes.
Governance obligations and controls
Banking developed the relevant discipline decades ago: model risk management. The US supervisory framework was overhauled in April 2026 — SR 26-2 replaced the fifteen-year-old SR 11-7, moving from calendar-based revalidation to oversight proportional to materiality — and the EU’s AI Act timeline is being re-sequenced through the Digital Omnibus while its core requirements stand. DORA adds an operational-resilience regime for financial entities. The pattern across all three is relevant well beyond finance: regulatory documents are revised frequently, but the underlying obligations are stable — inventory what you run, validate in proportion to potential loss, trace your data, retain the ability for a human to intervene, and prove all of it on demand.
The engineering translation is an obligations registry mapped to controls-as-code: each regulation’s requirements normalized into a table, each requirement mapped to a control that a pipeline enforces or evidences automatically. Inventory entry is required before credentials issue; lineage is emitted by every pipeline run; evaluation reports attach to every release; model cards are generated from registry metadata rather than written as documents. When rules change (2026 changed them twice within months), the update is a mapping revision, not a re-platforming. Organizations outside regulated industries have a commercial reason to adopt the same kit: AI-governance certification (ISO/IEC 42001) now appears in vendor due-diligence questionnaires.
The inventory deserves specific attention because reviews start there. The gap is rarely concealment: nothing prevents a team from wiring a model API into a workflow without registering it. The fix is structural: credentials to call models are issued only through the registration workflow, so the inventory is complete by construction rather than by survey. Shadow AI persists where ungoverned provisioning paths exist; closing the path is more effective than restating the policy.
Technology half-lives and replaceability design
The arithmetic of longevity is unfavorable to any fixed technology selection. The transformer architecture is nine years old. “MLOps” as a term is about eight. The dominant agent frameworks are under three. A platform built today is expected to serve for ten to fifteen years. These numbers do not reconcile, and institutions that ignore the mismatch typically face a forced rewrite around year six, at crisis prices, after the original architects have left.
SQL, Git, HTTP, OCI, Parquet, OIDC
The resolution is to make long-term commitments only where longevity is empirically supported, and to engineer everything else for replacement:
- Commit at the capability level. Train, evaluate, register, serve, monitor, govern: these functions will exist in 2036 regardless of which products perform them. The capability map is the durable architecture; product names are annotations reviewed annually.
- Own three seams outright: the AI gateway (every model call passes one facade, so a provider swap is configuration plus an evaluation run, and Part 1’s metering is a side effect of the same chokepoint), the evaluation harness (the portable evidence that a swap is safe; without it every migration is unverifiable), and the metadata/lineage spine (the record that must survive every engine swap). If these three are ceded to proprietary black boxes, a temporary vendor choice hardens into a dependency that cannot be exited on acceptable terms.
- Open formats at rest: tables in open formats another engine can read in place; models as standard artifacts in OCI registries; prompts, evaluations and configs as versioned text in Git. Data gravity should accrue to the institution rather than to a vendor.
- Write exit tests, then rehearse them. Every major component carries a card: the seam it sits behind, the concrete exit test (“rebuild the vector index on the reference alternative in under five days with retrieval evals within 2%”), the measured result of the last rehearsal, and the triggers that would initiate a real exit. Two rehearsals a year keeps the capability current, and the results serve as third-party-risk evidence under resilience regimes such as DORA.
- Replace on a calendar, strangler-fig style. An annual radar re-scores components; a biennial re-tender of one major seam keeps pricing honest. Expect two or three major engine replacements per decade, absorbed at the seams without a rewrite.
Renewal versus rewrite cost comparison
The renewal budget is the line most likely to be cut in year three, so the alternative should be priced explicitly. For a platform with an $8M build cost (illustrative; scale to your estate):
| Continuous renewal | No renewal → forced rewrite | |
|---|---|---|
| Annual investment | ~12% of build ≈ $1M/yr, planned | $0 for 5–6 years |
| Year 5–6 event | Components replaced incrementally; no event | Rewrite at 1.5–2× original in a worse market: $12–16M |
| Delivery impact | None structural | 12–18 months of feature freeze across the AI portfolio |
| Risk profile | Small, reversible migrations | Big-bang migration of models, data and controls; re-validation of everything at once |
| 10-year cash cost | ≈ $8M + $10M = $18M, smooth | ≈ $8M + $14M = $22M, lumpy — before pricing the freeze, which usually dwarfs it |
The cash difference is real but modest; the larger difference is option value. A renewing platform can adopt each technology wave (as agents were to 2024–26) in the year it matures; a frozen platform meets each wave as a crisis. The budgeting implication: a permanent renewal allocation of 10–15% is the cost of preserving the next decade’s adoption options, and should be protected on those grounds rather than classified as deferrable maintenance.
Assessment questions for executives and engineers
For executives:
- Can we produce, today, a complete list of models and agents in production with named owners?
- What is our mean time to detect a quality failure, as distinct from an outage?
- Which of our AI components could we exit in ninety days, and when did we last demonstrate it?
- Is renewal a protected budget line?
For engineers, the checklist: drift and data-contract monitors as code with paging thresholds; evaluation suites gating every release in CI; kill switches and champion–challenger rollback tested rather than assumed; one telemetry standard end to end; exit cards with measured rehearsal results.
These items are unglamorous; collectively they determine whether a platform survives its second audit, its first silent failure, and its fifth technology wave.
Next — Part 3: The Reference Design: the implementation that delivers the mechanisms in Parts 1 and 2 — the stack, repositories, pipelines, gateway and policy configuration, and a 90-day execution plan.
References
- Board of Governors of the Federal Reserve System, SR 26-2, Revised Guidance on Model Risk Management (Apr 2026); companion OCC Bulletin 2026-13. Supersedes SR 11-7 (2011).
- European Commission, EU AI Act framework; Digital Omnibus deferral analysis: Gibson Dunn, 2026 (provisional at time of writing).
- EIOPA/European Commission, Digital Operational Resilience Act (DORA) — in application since January 2025.
- ISO/IEC, ISO/IEC 42001:2023 — AI management systems.
- OpenTelemetry, GenAI observability and semantic conventions (stabilizing through 2026; pin versions).
- D. Sculley et al., Hidden Technical Debt in Machine Learning Systems, NeurIPS 2015.
- NIST, AI Risk Management Framework — a useful shared vocabulary for the obligations registry.
Regulatory statuses are point-in-time as of mid-2026 and subject to change; verify at decision time. Loss and cost figures are illustrative composites, not client data. Vendor performance claims (e.g., AIOps MTTR reductions) are reported, not audited.