Hosted AI models are now production dependencies for search, document processing, support, code review, and other customer workflows. They need the same ownership and incident planning as payment, identity, and cloud services.
Start with the customer effect
List each AI-backed feature and write down what happens when its provider is unavailable, slow, or returning unusable output. The answers determine the alert route and fallback.
- A failed checkout or account workflow should page the responsible team.
- A delayed background job may need a queue alert and an operator task.
- An optional feature can often show a clear unavailable state without paging anyone.
This inventory is more useful than a generic rule that every AI provider is critical.
Measure the calls your application makes
A provider status page cannot show whether your account, region, model, or quota is affected. Collect application-level measures for each provider and model:
- request count and error rate
- latency percentiles and timeouts
- rate-limit responses
- queue depth and job age
- fallback use
- responses rejected by your own validation
A successful HTTP response does not prove that the output is usable. Where the workflow permits it, validate structure, required fields, and empty or truncated responses. Avoid recording prompts or outputs in monitoring systems unless the data policy explicitly allows it.
Follow provider notices
Provider notices add confirmed scope and recovery information. Subscribe to the providers you use and route the notices to the same incident workflow as your application alerts. ServiceAlert can collect those notices in one account and deliver relevant changes to email or team channels.
Provider notices and application telemetry answer different questions. The first explains what the vendor has acknowledged; the second shows what your customers are experiencing. Keep both.
Define fallbacks before an incident
For each feature, choose an explicit failure behavior:
- show that the feature is temporarily unavailable
- queue work for later processing
- use a simpler model from the same provider
- use an approved secondary provider
- return a non-AI workflow
- stop the transaction when proceeding would be unsafe
A secondary model or provider is not a free failover. Test differences in output, latency, safety controls, data handling, and cost. Confirm that the fallback has enough quota to carry the expected load.
Treat contracts as service-specific
Availability commitments vary by product, region, account tier, and contract. Record the public or contractual target that applies to your deployment, along with the exclusions and claim process. Do not copy a provider-wide percentage into a runbook unless it matches the service you actually buy.
A workable operating model
The monitoring plan does not need a new category of incident process. It needs clear ownership, useful measures, and a tested response.