Building Resilient Distributed Systems with Go

Picture a database query that slows from 10 ms to 4.8 seconds. Nothing crashes — and that's exactly why it takes everything down. A product-page service calls it on every request, watches its own latency multiply, and burns through its connection pool until it fails. Every dependent service follows.

When One Slow Dependency Takes Down Your Whole System

A recommendation service's query plan degrades from milliseconds to seconds. A downstream product page service, calling it on every request, sees its own latency multiply. At sustained traffic, every connection it allocated stays checked out, waiting. The pool exhausts. Everything depending on it fails. A dependency that fails fast (connection refused, immediate error) releases the caller's connection and goroutine at once; one that is slow or silently unresponsive — including a crashed host that drops packets — holds them until the deadline^⁠[1].

The Quick Start: Pattern Comparison

Circuit Breaker: Stop Hitting a Failing Dependency

When a dependency is failing, a circuit breaker stops calling it — fail fast instead of waiting for timeouts to expire. It has three states: Closed (normal operation), Open (dependency is failing, reject calls immediately), and HalfOpen (dependency may be recovering, let a probe through)^⁠[3].

stateDiagram-v2
    [*] --> Closed
    Closed --> Open: failure rate ≥ threshold
    Open --> HalfOpen: reset timeout elapsed
    HalfOpen --> Closed: probe(s) succeed
    HalfOpen --> Open: any probe fails

    note right of Closed
        Normal ops.
        Count failures.
    end note
    note right of Open
        Reject fast.
        Wait resetTimeout.
    end note
    note right of HalfOpen
        Allow N probes.
        Decide next state.
    end note

Three tunables drive behaviour: the failure threshold (percentage over a window that flips Closed → Open), the reset timeout (how long Open waits before probing), and the probe count in HalfOpen (too few misdiagnoses transient blips as recovered; too many hammer a still-failing dependency).

The Race Condition Most Implementations Get Wrong

The common mistake is upgrading from a read lock to a write lock between reads. Between releasing the read lock and acquiring the write lock, multiple goroutines can slip through and all try to transition to HalfOpen:

// WRONG: race between RUnlock and Lock
func (cb *CircuitBreaker) Execute(fn func() error) error {
    cb.mu.RLock()
    if cb.state == StateOpen {
        cb.mu.RUnlock()           // release read lock
        cb.mu.Lock()              // ← goroutines can slip between here
        cb.state = StateHalfOpen  // ← multiple can reach this
        cb.mu.Unlock()
    } else {
        cb.mu.RUnlock()
    }
    // ...
}

Result: multiple probe requests hit the recovering service simultaneously, causing it to fail again. The fix: use a single mutex for all state transitions, never upgrading locks.

// Circuit breaker states. Code compares cb.state against these constants,
// never against the raw ints, so a stray typo fails to compile instead of
// silently misrouting a transition.
const (
	StateClosed   = iota // 0: normal operation, calls pass through
	StateOpen            // 1: dependency failing, reject calls immediately
	StateHalfOpen        // 2: probing recovery, one call allowed through
)
 
type CircuitBreaker struct {
	mu        sync.Mutex
	state     int
	failCount int
	timeout   time.Duration
	openedAt  time.Time
}
 
func (cb *CircuitBreaker) Execute(fn func() error) error {
	cb.mu.Lock()
 
	switch {
	case cb.state == StateOpen && time.Since(cb.openedAt) <= cb.timeout:
		// Open and still cooling down → fail fast.
		cb.mu.Unlock()
		return fmt.Errorf("circuit breaker open: dependency unavailable")
	case cb.state == StateOpen:
		// Timeout expired: promote to HalfOpen and let THIS request be the probe.
		cb.state = StateHalfOpen
	case cb.state == StateHalfOpen:
		// A probe is already in flight — admit only one at a time, so recovery
		// doesn't send a thundering herd at a dependency that's still fragile.
		cb.mu.Unlock()
		return fmt.Errorf("circuit breaker half-open: probe in progress")
	}
 
	cb.mu.Unlock()
	err := fn() // execute outside the lock to avoid holding it across I/O
	cb.mu.Lock()
	defer cb.mu.Unlock()
 
	if err != nil {
		cb.failCount++
		// A failed probe reopens immediately; otherwise trip at the threshold.
		if cb.state == StateHalfOpen || cb.failCount >= 5 {
			cb.state = StateOpen
			cb.openedAt = time.Now()
		}
	} else {
		cb.failCount = 0
		cb.state = StateClosed // success closes the circuit
	}
	return err
}

This ensures only one transition happens at a time, preventing the thundering herd of probes on recovery.

Bulkhead: Separate Resource Pools Per Dependency

The cascade in the opening scenario happens because all outbound calls — to recommendations, inventory, pricing — share the same goroutine pool and connection pool. When the recommendation service slows, it burns all available connections. Inventory and pricing calls starve.

The shared-pool failure mode versus the bulkhead-isolated fix:

graph TB
    subgraph SharedPool[BEFORE — shared pool]
        S1[100 connections] --> R1[Recommendations<br/>slow — holds 95 conns]
        S1 --> I1[Inventory<br/>cannot get conn — starves]
        S1 --> P1[Pricing<br/>cannot get conn — starves]
    end
    subgraph Bulkheaded[AFTER — bulkhead isolation]
        B1[50 conns —<br/>recommendations] --> RB[Recommendations<br/>slow — only this pool exhausted]
        B2[100 conns —<br/>inventory] --> IB[Inventory<br/>still healthy]
        B3[75 conns —<br/>pricing] --> PB[Pricing<br/>still healthy]
    end
    style R1 fill:#fdd
    style I1 fill:#fdd
    style P1 fill:#fdd
    style RB fill:#fdd
    style IB fill:#dfd
    style PB fill:#dfd

Bulkhead isolation fixes this: each dependency gets its own resource limit. One slow dependency exhausts only its own allocation, leaving others untouched.

type Bulkhead struct {
	sem chan struct{} // buffered channel acts as semaphore
}
 
func NewBulkhead(maxConcurrent int) *Bulkhead {
	return &Bulkhead{sem: make(chan struct{}, maxConcurrent)}
}
 
func (b *Bulkhead) Execute(ctx context.Context, fn func() error) error {
	select {
	case b.sem <- struct{}{}: // acquire one slot
		defer func() { <-b.sem }() // release on exit
		return fn()
	case <-ctx.Done():
		return ctx.Err() // caller already gave up; with default below, select never waits
	default:
		return fmt.Errorf("bulkhead full: %d concurrent limit exceeded", cap(b.sem))
	}
}

Wire separate bulkheads per dependency:

// One bulkhead per dependency, created once at startup
var (
	recommendations = NewBulkhead(50)  // smallest share: least critical
	inventory       = NewBulkhead(100) // largest share: critical for checkout
	pricing         = NewBulkhead(75)  // medium
)
 
// In your handler, each call is isolated
var recs []Recommendation
err := recommendations.Execute(ctx, func() error {
	var err error
	recs, err = svc.getRecommendations(ctx, productID)
	return err
})
if err != nil {
	// Bulkhead full or call failed: recommendations are optional, so
	// render the page without them instead of failing the request.
	slog.WarnContext(ctx, "recommendations skipped", "product", productID, "err", err)
}
// If recommendations is at 50/50, inventory calls don't wait — they have their own pool

Even if recommendations burns all 50 slots with slow calls, inventory and pricing proceed independently.

Timeout Propagation: Respect the Parent Context Deadline

Every downstream call must respect the incoming context's deadline. If the client's timeout is 500ms total and you make two 500ms calls sequentially, you've already violated the contract.^⁠[2]

// WRONG: creates new context, loses parent deadline
func (svc *Service) CallDownstream(ctx context.Context) error {
	newCtx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
	defer cancel()
	return svc.downstream.Call(newCtx)
}
 
// RIGHT: derive from parent, add per-call buffer as safety net
func (svc *Service) CallDownstream(ctx context.Context) error {
	callCtx, cancel := context.WithTimeout(ctx, 2*time.Second)
	defer cancel()
	return svc.downstream.Call(callCtx)
}

Budget timeout across multiple calls:

// Handler has 500ms total budget
ctx, cancel := context.WithTimeout(r.Context(), 500*time.Millisecond)
defer cancel()
 
// First call: capped at 200ms of the budget
productCtx, productCancel := context.WithTimeout(ctx, 200*time.Millisecond)
defer productCancel()
product, err := svc.getProduct(productCtx, id)
if err != nil {
	slog.ErrorContext(ctx, "get product failed", "id", id, "err", err)
	http.Error(w, "product unavailable", http.StatusBadGateway)
	return
}
 
// Second call: uses remaining time from parent deadline (auto-enforced)
recs, err := svc.getRecommendations(ctx, product.ID)
if err != nil {
	// Recommendations are optional: log and render the page without them.
	slog.WarnContext(ctx, "recommendations unavailable", "product", product.ID, "err", err)
}
svc.renderProductPage(w, product, recs)

WithTimeout(parent, duration) never exceeds the parent's deadline — the tighter constraint wins.

Idempotent Retries: Safe Retries Without Duplicate Orders

Retries are essential for transient failures (network hiccups, 502s). But retrying non-idempotent operations — creating orders, charging cards — produces duplicates. The fix: send the same idempotency key on retry, and the server deduplicates.^⁠[4]

// Generate a deterministic key once; use on all retry attempts
func generateIdempotencyKey(userID, productID string, amount float64) string {
	h := sha256.New()
	fmt.Fprintf(h, "%s:%s:%.2f", userID, productID, amount)
	return hex.EncodeToString(h.Sum(nil))
}
 
// isRetryable reports whether a client.Do error is a transient transport
// failure worth another attempt with the same idempotency key. It checks the
// caller's ctx directly: Client.Timeout, ResponseHeaderTimeout and dial
// timeouts all match errors.Is(err, context.DeadlineExceeded) too, so that
// test can't tell "budget spent" from "one attempt was slow". net.Error goes
// last because *url.Error itself implements it, so errors.As stops there.
func isRetryable(ctx context.Context, err error) bool {
	if ctx.Err() != nil {
		return false // caller cancelled or the overall budget is spent
	}
	if errors.Is(err, syscall.ECONNREFUSED) || errors.Is(err, syscall.ECONNRESET) {
		return true // dependency restarting or dropped the connection
	}
	var netErr net.Error
	return errors.As(err, &netErr) && netErr.Timeout() // this attempt was slow
}
 
// On retry, server sees same key and returns original response, not a new order
idempotencyKey := generateIdempotencyKey(userID, productID, amount)
 
var lastErr error
for attempt := 0; attempt < 3; attempt++ {
	if attempt > 0 {
		// Randomized exponential backoff (0–200ms, then 0–400ms): immediate
		// retries hit a restarting dependency before it is back, in lockstep
		// with every other client's retries.
		select {
		case <-time.After(rand.N(100 * time.Millisecond << attempt)):
		case <-ctx.Done():
			return nil, fmt.Errorf("create order: %w (last error: %w)", ctx.Err(), lastErr)
		}
	}
	// payload is the encoded order. Build a fresh reader every attempt:
	// the previous attempt drained the last one.
	req, err := http.NewRequestWithContext(ctx, "POST", orderServiceURL+"/orders", bytes.NewReader(payload))
	if err != nil {
		return nil, err
	}
	req.Header.Set("Idempotency-Key", idempotencyKey) // same key every attempt
	req.Header.Set("Content-Type", "application/json")
 
	resp, err := client.Do(req)
	if err != nil {
		if isRetryable(ctx, err) {
			lastErr = err
			continue // retry transient transport errors only
		}
		return nil, err // permanent error — resp is nil here, never dereference it
	}
	if resp.StatusCode >= 500 {
		resp.Body.Close() // release the connection before retrying
		lastErr = fmt.Errorf("order service returned %s", resp.Status)
		continue // retry on 5xx
	}
	return resp, nil // success or permanent failure (4xx)
}
return nil, fmt.Errorf("create order: 3 attempts failed: %w", lastErr)

See Idempotency Patterns: Building Retry-Safe Distributed Systems for server-side deduplication strategies.

The Opening Incident: How These Patterns Prevent Cascade Failure

Recall the slow recommendation service: 4.8-second queries instead of 10 ms, against a 2-second per-call timeout. With these patterns in place, here's what happens:

1. Circuit breaker detects failure — after 5 consecutive timeouts, it opens; under concurrent traffic those five land within about one timeout window, not one after another. Subsequent calls fail immediately with "circuit breaker open" instead of waiting 2 seconds for the timeout.

2. Bulkhead protects inventory and pricing — even though recommendation's 50-slot bulkhead fills with slow calls, inventory and pricing each have separate limits. They proceed unaffected.

3. Product page degrades gracefully — recommendation calls fail fast, returning an empty list fallback. Product page still renders in 80ms instead of 2+ seconds.

4. Connection pool never exhausts — goroutines are released milliseconds after failing, not held open for full timeout duration.

5. Recommendation service recovers — DBA fixes the query plan. After 30 seconds, circuit transitions to HalfOpen and probes recovery with a single request. If successful, circuit closes and recommendations resume.

Result: users see product pages without recommendations until the fix lands and a probe succeeds. Zero 500 errors. No cascade to dependent services. Every failure is isolated and degrades gracefully.

Production Checklist

For each external dependency:
[ ] Circuit breaker with single mutex (no RLock/Lock races)
[ ] Separate bulkhead (concurrency limit) per dependency
[ ] Idempotency key on all mutating operations (orders, payments)
[ ] Per-call timeout derived from parent context (never context.Background())
[ ] Retry only on retryable errors (503, 502, timeouts — not 400, 404)
 
For your service:
[ ] /healthz (liveness): checks process only, not dependencies
[ ] /readyz (readiness): checks critical dependencies with 3s timeout
[ ] Global request timeout on all handlers (e.g., 500ms for APIs)
[ ] Structured logging for circuit breaker state changes
[ ] Prometheus metrics: circuit open rate, bulkhead rejections, retry attempts

Alerting rules: catch resilience failures before users do

The Prometheus metrics above are useless without alert rules. The three rules below catch the cascade-precursor symptoms — circuit-breaker thrashing, bulkhead saturation, and runaway retry storms — at thresholds that fire minutes before user-visible latency:

# alerts/resilience.yml — paste alongside your service's existing PrometheusRule
groups:
  - name: resilience
    interval: 30s
    rules:
      - alert: CircuitBreakerOpenRateHigh
        expr: |
          (
            sum by (service, dependency) (
              rate(circuit_breaker_state_changes_total{to_state="open"}[5m])
            )
          ) > 0.1
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "{{ $labels.dependency }} breaker opening repeatedly from {{ $labels.service }}"
          description: |
            Circuit breaker for {{ $labels.dependency }} has opened {{ $value }} times/sec
            over the last 5 minutes — usually means the dependency is degraded, not crashed.
            Check downstream latency p99 and connection-pool utilisation.
 
      - alert: BulkheadSaturated
        expr: |
          (
            bulkhead_active_calls{service=~".+"}
            / bulkhead_max_concurrent
          ) > 0.85
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Bulkhead {{ $labels.dependency }} > 85% saturated"
          description: |
            {{ $value | humanizePercentage }} of the bulkhead slots are in use. Either traffic
            is exceeding capacity or the dependency is slow. Check {{ $labels.dependency }}
            latency before raising the bulkhead size.
 
      - alert: RetryStormDetected
        expr: |
          (
            rate(retry_attempts_total{outcome="retry"}[5m])
            / rate(retry_attempts_total[5m])
          ) > 0.3
        for: 3m
        labels:
          severity: critical
        annotations:
          summary: "{{ $labels.service }} is in a retry storm against {{ $labels.dependency }}"
          description: |
            >30% of calls are being retried. Combined with an open circuit breaker this
            is the classic cascade-failure signature — the retry budget is amplifying load
            on a degraded dependency. Verify the breaker is firing and consider tightening
            retry budgets or shortening per-call timeouts.

Tune the 0.1, 0.85, and 0.3 thresholds against your steady-state baselines — every service has a different healthy retry rate. The pattern matters more than the numbers: rate-of-state-change for breakers, ratio-saturation for bulkheads, ratio-of-retries for retry budgets.

Hedged Requests: Trading a Little Load for a Lot of Tail Latency

Circuit breakers and bulkheads protect you from broken dependencies; hedged requests protect you from the long tail of healthy ones. The idea, popularised by Google's "Tail at Scale" paper^⁠[5], is simple: if a request hasn't returned by the dependency's p95 latency, fire a duplicate to a different replica and take whichever response arrives first. The slow request gets cancelled. Waiting for p95 caps the extra load at about 5% while substantially shortening the tail; in the paper's BigTable benchmark, a 10 ms hedge cut 99.9th-percentile latency from 1,800 ms to 74 ms for 2% more requests^⁠[5]. It's useful when downstream replicas are independent (different shards, different pods) and the request is read-only or idempotent.

The two failure modes to design out are duplicate writes and runaway hedging on a globally degraded dependency. The first is solved by hedging only idempotent reads, or pairing the hedge with the same idempotency key from the retry section so the server deduplicates. The second is solved by capping hedges per parent request and gating them through the same circuit breaker: when the breaker is open, hedging is pointless and only adds load.

// HedgedCall fires a primary request immediately and a backup after `delay`.
// Whichever wins is returned; the loser is cancelled via context propagation.
func HedgedCall[T any](
    ctx context.Context,
    delay time.Duration,
    fn func(context.Context) (T, error),
) (T, error) {
    type result struct {
        val T
        err error
    }
    out := make(chan result, 2)
    callCtx, cancel := context.WithCancel(ctx)
    defer cancel() // cancels the loser as soon as we return
 
    go func() {
        v, err := fn(callCtx)
        out <- result{val: v, err: err}
    }()
 
    timer := time.NewTimer(delay)
    defer timer.Stop()
 
    select {
    case r := <-out:
        return r.val, r.err // primary returned before hedge fired
    case <-timer.C:
        go func() {
            v, err := fn(callCtx) // fire the duplicate
            out <- result{val: v, err: err}
        }()
        r := <-out
        return r.val, r.err
    case <-ctx.Done():
        var zero T
        return zero, ctx.Err()
    }
}

Pick delay from histograms — your dependency's measured p95, not a guess. Hedging at p50 duplicates half of all requests for marginal gains; hedging at p99 leaves everything below p99 untouched. The cancellation is doing real work: when the hedge wins, the primary's callCtx is cancelled and a Go HTTP client request built with that context is aborted mid-flight — the HTTP/1.1 connection is closed, an HTTP/2 stream is reset. Skip that, and a "winning" hedge still ties up resources until the original call drains.

Adaptive Concurrency Limits: Self-Tuning Bulkheads

Static bulkhead sizes are a guess. The right answer changes with traffic mix, GC pauses, downstream capacity, and the time of day. Netflix's concurrency-limits^⁠[6] library treats the limit like a TCP congestion window: grow it while latency stays healthy, cut it when queueing or drops appear. It ships delay-based controllers (Vegas, Gradient2) and a loss-based AIMD — additive increase, multiplicative decrease; the sketch below is a simple AIMD variant that also backs off on queueing latency. It works because it's the same problem: many independent senders sharing a finite-capacity bottleneck, with no out-of-band signal of remaining headroom beyond the latency you observe.

The signal is queueing: if measured latency starts climbing above the lowest-observed baseline (a Little's Law proxy for a saturated server), shrink. If it stays close to baseline under load, grow. You never need a hand-tuned value because the limit converges on whatever the dependency can actually serve.

// AdaptiveLimit implements an AIMD controller over the bulkhead semaphore.
// Wrap each call with Acquire/Release; the limit retunes itself.
type AdaptiveLimit struct {
    mu          sync.Mutex
    limit       int           // current concurrency cap
    minLimit    int           // floor — never starve below this
    maxLimit    int           // ceiling — protect downstream
    inFlight    int
    minRTT      time.Duration // best observed latency
    sampleCount int
}
 
func (a *AdaptiveLimit) Acquire(ctx context.Context) (release func(time.Duration, error), err error) {
    a.mu.Lock()
    if a.inFlight >= a.limit {
        a.mu.Unlock()
        return nil, fmt.Errorf("adaptive limit reached: %d", a.limit)
    }
    a.inFlight++
    a.mu.Unlock()
 
    return func(rtt time.Duration, callErr error) {
        a.mu.Lock()
        defer a.mu.Unlock()
        a.inFlight--
        a.sampleCount++
 
        // Track best-case RTT as the no-queue baseline.
        if a.minRTT == 0 || rtt < a.minRTT {
            a.minRTT = rtt
        }
        // Multiplicative decrease on error or queueing latency (>2x baseline).
        if callErr != nil || rtt > 2*a.minRTT {
            a.limit = max(a.minLimit, a.limit/2)
            return
        }
        // Additive increase only when fully utilised — avoid drifting up on idle traffic.
        if a.inFlight+1 >= a.limit && a.limit < a.maxLimit {
            a.limit++
        }
    }, nil
}

In production this looks like a sawtooth that tracks the real bottleneck — peak hours grow the limit toward the ceiling, GC pauses or downstream incidents shrink it within seconds. It removes the worst part of static bulkheads: the team meeting where someone argues for a number that everyone forgets to revisit when traffic doubles.

A 30-Minute Outage Timeline: Where Each Pattern Earns Its Keep

To make the patterns concrete, here's a sketch of a familiar shape — one slow Postgres replica during a checkout-heavy hour. The point isn't the exact wall-clock; it's which patterns flip from "passive observation" to "active load shedding" at each stage.

T+00:00 — degradation begins. A query plan flips on the inventory replica after an autovacuum reshapes statistics. p99 climbs from 12 ms to 480 ms within ninety seconds. No alert fires yet — error rates are zero, the database is "healthy" by every traditional metric. This is the window where bad architectures lose. The adaptive concurrency limiter notices first: rtt has tripled relative to its observed minimum, so the controller halves the inventory bulkhead from ~80 to ~40. Healthy services keep their full quota.

T+02:30 — saturation. Sustained load pushes inventory rtt past the 2-second timeout. The circuit breaker tracks consecutive failures and opens at five — checkout calls now fail in microseconds with circuit breaker open: dependency unavailable, freeing goroutines that would otherwise wait the full timeout. Retried calls cross the retry-storm alert's > 30% threshold; once its for: 3m window passes, PagerDuty wakes someone.

T+05:00 — graceful degradation. With the breaker open, the cart-rendering path falls back to the cached inventory snapshot from Redis. Hedged requests against the secondary replica return in 14 ms; users see a banner ("Stock counts may lag by up to 2 minutes") instead of a 500. The bulkhead, now sized at ~40, protects pricing and recommendations from inheriting the failure — both stay green.

T+12:00 — operator action. The on-call engineer runs ANALYZE inventory_items; on the affected replica. The plan reverts. Replica latency drops to 18 ms. Nothing in the code path changes yet — the breaker is still open, so traffic is still flowing the cached path.

T+12:30 — controlled recovery. The breaker's reset timeout elapses. It transitions to half-open and admits a single probe; the probe succeeds and closes it. The adaptive limiter sees healthy rtt and begins additive growth, walking the inventory bulkhead from 40 back toward 80 over the next four minutes. No human flips a flag — the system retunes itself.

T+30:00 — postmortem. Total user-visible impact: a banner for twelve minutes. Zero 5xx. Zero abandoned carts above baseline. The retro item isn't "build more resilience patterns" — it's "tighten the bulkhead floor and add an alert on adaptive-limit collapse so we get woken at T+00:30 instead of T+05:30."

The pattern that mattered most isn't any single one; it's the layering. Adaptive limits found the degradation before humans did. The breaker turned slow failures into fast ones, releasing resources. Hedging carried the read path on the surviving replica. Idempotency keys meant no duplicate orders during retries. Timeout propagation meant the client's 500 ms budget was respected at every hop. Take any one of these out and the timeline ends with cascade failure and a 30-minute full outage instead of a banner.

Frequently Asked Questions

Keep Reading

• Your LLM Provider Will Have an Outage — These patterns specialized for LLM APIs: breakers that catch failures arriving as HTTP 200, fallback chains across clouds, and degraded modes for AI features
• Idempotency Patterns: Building Retry-Safe Distributed Systems — The full idempotency key pattern, database constraints, and outbox pattern that make the retry logic in this article safe for payments and orders
• Rate Limiter Algorithms: Token Bucket vs Sliding Window — The load shedding section here uses adaptive limits; this covers the five algorithms for request-level rate limiting and distributed coordination
• Event-Driven Microservices in Go: Kafka, Sagas, and the Outbox Pattern — When your resilient services need to coordinate multi-step workflows across boundaries, sagas and the outbox pattern handle the distributed transaction problem