Your LLM Provider Will Have an Outage: Circuit Breakers, Fallback Chains, and Degraded Modes

On December 11, 2024, OpenAI's API went down at 3:16 PM Pacific and stayed degraded for over four hours: a telemetry rollout overwhelmed the Kubernetes control plane in its largest clusters, which broke data-plane DNS — and DNS caching hid the blast radius for the first twenty minutes^{[OpenAI 2024-12-11 postmortem]}. Every chat feature, agent workflow, and extraction pipeline calling that API was down with it. A retry loop does not bridge a four-hour outage. A plan does.

The Decision Table: Route by Failure Signal

Every signal below is provider-documented^{[OpenAI error codes][Claude API errors][Claude rate limits][Claude streaming]}. Retry-everything-everywhere is how a provider's bad hour becomes your outage too.

The 429 exception: OpenAI's discounted flex tier returns capacity 429s — the documented response is resubmitting at service_tier: "auto", and failed flex attempts aren't charged^{[OpenAI flex processing]}.

The Outage Record Is Public

Neither provider treats this as hypothetical. OpenAI's status page reports rolling aggregate uptime — 99.94% for APIs over May–August 2026^{[OpenAI status]} — which translates to about 26 minutes of unavailability a month, arriving in lumps, not evenly. The December 2024 incident got a full public postmortem^{[OpenAI 2024-12-11 postmortem]}; a June 2025 elevated-error incident ran roughly fifteen hours and its root cause was never publicly disclosed^{[OpenAI 2025-06-10 incident]}. Anthropic's status page listed 25 incidents in the three and a half weeks to mid-August 2026, and exposes machine-readable status endpoints^{[Anthropic status]}.

The deeper signal is in the pricing pages. Anthropic's standard tier is documented as "best-effort availability", and its Priority Tier exists to "minimize 'server overloaded' errors, even during peak times" with a 99.5% uptime target^{[Claude service tiers]}. OpenAI sells enterprise customers a 99.9% uptime SLA through Scale Tier^{[OpenAI Scale Tier]}. A premium product whose pitch is prioritized capacity during overload tells you overload is a normal operating condition of the tier you are probably on.

What makes an LLM provider different from your other dependencies is correlation. A database failover affects one service; when your LLM provider degrades, every AI feature fails at the same moment, across every service — and you cannot fail over to a replica you own. The failure domain is the whole capability — a fact your error budgets have to absorb.

One humility check: in September 2025, Anthropic published a postmortem of three overlapping infrastructure bugs that degraded response quality — not availability — for weeks, peaking at 16% of Sonnet 4 requests in the worst hour^{[Anthropic postmortem, Sept 2025]}. Every one of those requests returned 200. Breakers see errors and latency, not wrongness; that failure mode belongs to output evals and bounds what these patterns can promise.

What "Down" Actually Looks Like

The retry layer you already have

Both official Go SDKs retry the identical error set — connection errors, 408, 409, 429, and ≥500 — twice by default, with exponential backoff, honoring retry-after^{[openai-go SDK][Anthropic Go SDK]}. Two consequences:

Your breaker sees post-retry failures. By the time an error surfaces in your code, the SDK has burned two retries over several seconds — the right input for a breaker, but "first error observed" is already "third attempt failed."

Stacking naive retries multiplies attempts. If your handler retries 3×, the SDK retries 3× underneath, and a gateway in front retries 3×, one user action becomes 27 upstream attempts — the layered-retry amplification the Google SRE book warns about with its 4³ = 64 example^{[SRE Book ch. 22]}. Retry at exactly one layer: let the SDK own transport retries, and spend your layer on classification, the breaker, and the chain. (Backoff and jitter mechanics live in LLM API Integration Patterns; OpenAI's own guidance is jittered backoff^{[OpenAI rate limits]}.) If you hit 429s routinely, add client-side rate limiting rather than more retries.

The timeout defaults deserve a look before an outage, not during one: anthropic-sdk-go applies a 10-minute default to non-streaming Messages calls, while openai-go applies no default timeout at all — a stalled connection hangs until your context says otherwise^{[Anthropic Go SDK][openai-go SDK]}. If your handlers don't set context deadlines, an outage doesn't produce clean errors; it produces goroutines quietly parked on dead sockets.

The 200s that aren't successes

Three documented failure modes never produce an error status code:

Mid-stream overload. Anthropic's streaming docs: during high-traffic periods an SSE error event can arrive after the 200, carrying overloaded_error — "which would normally correspond to an HTTP 529 in a non-streaming context"^{[Claude streaming]}:

event: error
data: {"type": "error", "error": {"type": "overloaded_error", "message": "Overloaded"}}

Silent tier downgrade. OpenAI's Fast mode documents a ramp rate limit: push past roughly 1M tokens/minute with a >50% increase inside 15 minutes and some requests are silently served at standard speed and standard rates — visible only as service_tier: "default" in the response body^{[OpenAI Fast mode]}. Latency degrades; no error is ever returned.

Mid-stream refusals. On the newest Claude models, safety-classifier refusals are a normal 200 with stop_reason: "refusal", and they "can arrive before any output, or mid-stream after partial output" — the docs' instruction is to treat partial output as incomplete and discard it^{[Claude refusals & fallback]}.

The consequence: detection must read response bodies and stream events, not just status codes — a wrapper folding all three into your error taxonomy is the prerequisite for everything below.

A Circuit Breaker That Understands LLM Calls

The textbook breaker — closed, open, half-open — carries over; four assumptions don't.

Failures arrive late. An LLM request legitimately runs 10–120 seconds, so an error-count breaker reacts minutes after the outage began. Set deliberately tight context deadlines and count them as failures — the deadline is your latency threshold, and it is the only latency signal an error-count breaker like gobreaker can see.

Half-open probes cost money. Every probe is a paid API call. Probe deliberately: a max_tokens: 1 request against the cheapest model on that provider answers "is it back?" for a fraction of a cent.

Granularity is provider × model. Anthropic's rate limits are per model class^{[Claude rate limits]}, and incidents are frequently model-scoped ("Elevated error rates for Sonnet 5" is a representative status-page entry^{[Anthropic status]}). One global breaker turns a single-model brownout into a self-inflicted total outage.

Not every error is an outage. A 429 is your quota; a 400 is your bug. Neither says the provider is down, and neither should open the breaker.

With sony/gobreaker v2 the classification is the IsSuccessful hook — everything else is configuration (defaults: 60-second open period, trip after 5 consecutive failures, 1 half-open probe)^{[sony/gobreaker]}. The Provider, CompletionRequest, and APIError types (and Prometheus metric registration) come from the provider abstraction this article builds on:

import (
	"context"
	"errors"
 
	"github.com/sony/gobreaker/v2"
)
 
// isTripSignal reports whether an error indicates provider trouble
// (as opposed to caller trouble, which must not open the breaker).
func isTripSignal(err error) bool {
	var apiErr *APIError
	if errors.As(err, &apiErr) {
		switch apiErr.StatusCode {
		case 500, 502, 503, 504, 529:
			return true
		case 429:
			// Quota, billing, or flex capacity — shed, queue, alert,
			// or resubmit at another service tier. Never a trip.
			return false
		}
		// Mid-stream SSE error events are surfaced by the stream
		// reader as APIError{StatusCode: 529} — see the taxonomy above.
		return false
	}
	// Timeouts are the strongest outage signal an LLM call produces.
	return errors.Is(err, context.DeadlineExceeded)
}
 
func newProviderBreaker(name string) *gobreaker.CircuitBreaker[*CompletionResponse] {
	return gobreaker.NewCircuitBreaker[*CompletionResponse](gobreaker.Settings{
		Name:        name, // "anthropic/claude-sonnet", "openai/gpt-5-mini", …
		MaxRequests: 1,    // one paid probe while half-open
		ReadyToTrip: func(c gobreaker.Counts) bool {
			return c.ConsecutiveFailures >= 5
		},
		IsSuccessful: func(err error) bool {
			return err == nil || !isTripSignal(err)
		},
		OnStateChange: func(name string, from, to gobreaker.State) {
			// Alert on this transition — an open breaker IS the outage
			// alert, minutes before your error-rate dashboard catches up.
			breakerState.WithLabelValues(name).Set(float64(to))
		},
	})
}

OnStateChange is not decoration: "breaker anthropic/claude-sonnet opened" is the on-call alert you actually want — it fires on the decision and names the blast radius.

Fallback Chains: Order by Behavior, Then Cost

When the breaker opens, traffic needs somewhere to go. The ordering principle: prefer the route that changes your product's behavior least.

Rung 1 is often free because the providers built it: Anthropic's service_tier: "auto" uses priority capacity when available and falls back to standard automatically, reporting the served tier in usage.service_tier^{[Claude service tiers]}; OpenAI's flex-tier fallback (the table note above) works the same way^{[OpenAI flex processing]}.

Rung 2 is the one teams underuse. The same Claude models are served through the Claude API, Amazon Bedrock, and Google Vertex AI, with first-party SDK support and near-identical Messages APIs^{[Claude on Bedrock][Claude on Vertex]}, and Bedrock adds its own multi-region routing via inference profiles^{[Bedrock cross-region inference]}. Your prompts, output parsers, and eval baselines survive intact — which is exactly what rung 3 cannot promise. The serving stacks genuinely differ, and that is the point: when Anthropic's 2025 routing bug hit 16% of first-party Sonnet 4 requests at the worst hour, the peak on Bedrock was 0.18%^{[Anthropic postmortem, Sept 2025]}. Microsoft's Azure OpenAI guidance is the same shape: no automatic failover; the reference architecture is a capacity chain behind a circuit-breaking gateway^{[Azure Foundry HA guidance]}.

Rung 3 is a product decision wearing an infrastructure costume. A different model answers differently: formats drift, refusal boundaries move, and your eval suite now has to cover two models to make the fallback trustworthy. Take the rung deliberately or not at all.

The chain itself is a loop over breaker-wrapped targets:

graph LR
    R[request] --> A[breaker A<br/>claude api]
    A -- closed --> S[served]
    A -- open --> B[breaker B<br/>bedrock · same model]
    B -- closed --> S
    B -- open --> C[breaker C<br/>validated cheaper model]
    C -- closed --> S
    C -- open --> D[degraded mode<br/>cache · queue · off]

type Target struct {
	Name     string // metric + log label: "bedrock/claude-sonnet"
	Provider Provider
	Model    string
	Breaker  *gobreaker.CircuitBreaker[*CompletionResponse]
}
 
type FallbackChain struct {
	targets []Target
}
 
func (c *FallbackChain) Complete(ctx context.Context, req *CompletionRequest) (*CompletionResponse, error) {
	var lastErr error
	for _, t := range c.targets {
		reqCopy := *req
		reqCopy.Model = t.Model
		resp, err := t.Breaker.Execute(func() (*CompletionResponse, error) {
			return t.Provider.Complete(ctx, &reqCopy)
		})
		if err == nil {
			fallbackServes.WithLabelValues(t.Name).Inc()
			return resp, nil
		}
		lastErr = err
		// Open breaker returns instantly (gobreaker.ErrOpenState) —
		// the chain advances without paying a timeout. But if the
		// CALLER is gone, stop: walking the chain for a hung-up
		// client wastes every downstream provider's capacity.
		if ctx.Err() != nil {
			return nil, ctx.Err()
		}
	}
	return nil, fmt.Errorf("all fallback targets failed: %w", lastErr)
}

Two disciplines keep the chain from creating new incidents:

Idempotency across providers. A request that timed out on provider A may have executed — and if the model call triggers side effects (an agent placing an order, sending an email), re-running it on provider B is the classic duplicate-effect bug with a new coat of paint. Carry one idempotency key through every target and dedupe at the business layer, as with payment retries.

Cost guards stay in the loop. Fallback routes can be pricier per token, and an outage plus aggressive failover is how a bad day becomes a five-figure invoice — OWASP files this under unbounded consumption^{[OWASP LLM Top 10]}. The cost circuit breaker must wrap the whole chain, not just the primary.

Degraded Modes: What Ships When Nothing Answers

Every breaker can be open at once. What happens next is a product decision; the only wrong answer is improvising it mid-incident. The SRE book's definition of graceful degradation — reduce the work performed per request, such as answering from cache instead of doing the full computation^{[SRE Book ch. 22]} — maps onto LLM features as a ladder, cheapest first:

1. Serve from cache. Repeat and near-repeat queries (support answers, product Q&A) can serve the last good response, labeled as such. This is degradation by the book: the request costs a lookup instead of an inference.
2. Queue and notify. Anything asynchronous — summarization jobs, enrichment pipelines, report generation — accepts the work, persists it durably (the outbox pattern fits), and tells the user when to expect results.
3. Downshift. A smaller or cheaper model you have already validated for the feature — rung 3 of the chain, pre-approved for exactly this moment.
4. Honest feature-off. A feature flag and a plain sentence: "AI assist is temporarily unavailable." Users forgive absence faster than confidently wrong output from an untested fallback.

Recovery is where degraded modes quietly fail. When the provider comes back, every queued job and every waiting client converges on it at once — against capacity that is itself still recovering, and with acceleration limits that will 429 a sharp ramp^{[Claude API errors]}. Drain with a retry budget (the SRE book caps retries near 10% of request volume^{[SRE Book ch. 21]}), pace them through a token bucket so they cannot amplify a still-degraded upstream^{[Brooker, AWS Builders' Library]}, and keep jitter so clients don't wake in sync^{[OpenAI rate limits]}. Half-open probes are your recovery detector; the provider's status feed (Anthropic's is machine-readable^{[Anthropic status]}) is a supplement, not a substitute — status pages trail reality in both directions.

When Single-Provider Is the Right Call

Everything above has a cost. The honest counter-cases:

• Internal tools and batch pipelines don't need a chain. If a human can wait an hour or a job can run tonight, the queue absorbs the entire outage for the price of a table and a cron job.
• Eval integrity can outrank uptime. If your output quality is contractual or regulated, an unevaluated fallback model is a bigger risk than downtime. "Down but never wrong" is a legitimate posture.
• Every fallback hop is a new dependency. It has its own auth, quotas, and failure modes — and it needs its own breaker. Hystrix encoded this a decade ago: a fallback that makes a network call must itself be wrapped in a command^{[Hystrix wiki]}. Three providers means three integrations to keep healthy and an eval matrix that doubled.
• Prompt maintenance multiplies. Rung 3 means every prompt change ships against two models or silently rots on one of them.

A breaker, a queue, and a feature flag — no second provider at all — is a defensible production posture. What is not defensible is shipping the tutorial default: unbounded implicit trust that client.Complete() returns.

Production Checklist

• One breaker per provider × model, tripping on 5xx / 529 / timeouts / mid-stream errors — never on your own 429s
• Alert on breaker state transitions, not just error rates — the open event is the incident notification
• Context deadlines on every call — openai-go has no default timeout; a missing deadline is a goroutine leak with an invoice
• Retries live at exactly one layer — the SDK's; your layer owns classification, the breaker, and the chain
• Stream readers surface SSE error events and refusals into the same error taxonomy as HTTP failures
• Fallback order documented and eval'd — same model elsewhere first; rung 3 pre-approved by product quality owners
• One idempotency key across all targets; side-effectful calls dedupe at the business layer
• Cost breaker wraps the chain, not just the primary
• Each feature has a named degraded mode (cache / queue / downshift / off) and a game day that proves it renders
• Recovery drains through a retry budget with jittered pacing — the outage's second act is self-inflicted load

Frequently Asked Questions

Keep Reading

• Building Resilient Distributed Systems with Go — Circuit breaker internals, bulkheads, and the slow-dependency cascade this article's patterns specialize for LLM calls
• LLM API Integration Patterns — The request path this article assumes: provider abstraction, retry mechanics, streaming backpressure, token budgets, and the cost breaker
• Idempotency Patterns: Building Retry-Safe Distributed Systems — Why retrying across providers without idempotency keys duplicates side effects, and the outbox pattern behind queue-and-notify