Caching Strategies at Scale

Picture Black Friday peak. A hot key's TTL expires and 40,000 concurrent users miss the cache at once: 800 queries/sec → 40,000 queries/sec. Database CPUs max out, connection pools exhaust, and the site goes down.

For that moment, it is as if there were no cache — against a database sized for 800 queries/sec.

This is the cache stampede: the failure mode of systems that cache correctly. Expiration happens on schedule, thousands of goroutines race to refill, the database — designed for cached traffic — gets hammered instead.

Caching at scale isn't about happy-path latency (cache hit = fast). It's about the failure modes: stampedes when keys expire, stale data when invalidation misses, cold starts that spike database load, and crashes when Redis dies.

Quick Pattern Decision Matrix

Choose cache-aside (most common) for read-heavy workloads. Write-through when you need cache always warm. Write-behind for write-heavy, crash-loss-tolerant data. Read-through when loading logic is complex or shared.

The Four Caching Patterns

Pattern 1: Cache-Aside (Lazy Loading)

App checks cache, handles misses by fetching from DB, populates cache itself. Simplest and most common pattern. The application is responsible for both cache interaction and database fallback.

Trade-offs: Read latency on cache miss (app waits for DB fetch + cache populate), staleness window between writes and TTL expiry, and cold-start overhead when cache is empty.

graph TD
    Client["Client Request"] --> App["Application"]
    App -->|"1. GET key"| Cache["Redis Cache"]
    Cache -->|"2a. Cache HIT"| App
    Cache -->|"2b. Cache MISS"| App
    App -->|"3. SELECT * FROM ..."| DB[(PostgreSQL)]
    DB -->|"4. Row data"| App
    App -->|"5. SET key (TTL)"| Cache
    App -->|"6. Response"| Client

func GetProduct(ctx context.Context, id string) (*Product, error) {
    cacheKey := "product:" + id
 
    // Check cache first
    cached, err := cache.Get(ctx, cacheKey)
    if err == nil {
        var product Product
        if err := json.Unmarshal(cached, &product); err == nil {
            return &product, nil
        }
    }
 
    // Cache miss — fetch from DB
    product, err := db.GetProduct(ctx, id)
    if err != nil {
        return nil, err
    }
 
    // Populate cache for next request. A write failure here shouldn't fail
    // the response — the DB fetch above already succeeded.
    data, err := json.Marshal(product)
    if err != nil {
        slog.WarnContext(ctx, "cache marshal failed", "key", cacheKey, "err", err)
        return product, nil
    }
    if err := cache.Set(ctx, cacheKey, data, 5*time.Minute); err != nil {
        slog.WarnContext(ctx, "cache write failed", "key", cacheKey, "err", err)
    }
    return product, nil
}

On writes, delete the cache key — next read repopulates: cache.Delete(ctx, "product:"+id). This is simple but only safe when staleness is acceptable.

Best for: Read-heavy, staleness-tolerant data (product catalogs, user profiles).

Pattern 2: Write-Through

Writes go to cache and DB synchronously. Cache is always warm. Trade-off: slower writes (the synchronous round-trip to both cache and DB adds latency) and unused cache entries accumulate. Best for mutable data that changes frequently (sessions, shopping carts, config).

Pattern 3: Write-Behind (Write-Back)

Writes go to cache immediately; background worker batches and asynchronously flushes to DB. Fastest write latency but accepts data loss on crash. Mitigate with AOF persistence. Best for write-heavy, crash-loss-tolerant workloads (analytics counters, metrics, ephemeral state).

Pattern 4: Read-Through

Cache layer transparently loads data on misses via a loader function. App only talks to cache; cache handles DB interaction. Encapsulates loading logic but couples cache to data sources and requires app-side write invalidation. Best for complex, shared loading logic.

Preventing Cache Stampedes

Cache stampedes^⁠[1] occur when a popular key expires at peak traffic. All concurrent requests discover the miss simultaneously and race to refill the cache — all querying the database at once. In the opening example, 40,000 req/s means 40,000 database queries per second against a database sized for 800.

The standard defense is request deduplication via singleflight^⁠[2]. If 10,000 goroutines simultaneously discover a cache miss for the same key, only one queries the database. The other 9,999 wait for the result and receive the same response.

sequenceDiagram
    participant R1 as Req 1
    participant R2 as Req 2..N
    participant SF as singleflight
    participant DB as Database

    Note over R1,R2: popular key expires
    R1->>SF: Do("product:42", fetchFn)
    R2->>SF: Do("product:42", fetchFn)
    Note over SF: N-1 callers wait on<br/>the in-flight Do
    SF->>DB: fetchFn() ← only ONE query
    DB-->>SF: row
    SF-->>R1: result
    SF-->>R2: same result (shared=true)

Without singleflight, N concurrent misses = N database queries. With it, N misses = 1 query + N-1 subscribers.

import "golang.org/x/sync/singleflight"
 
var group singleflight.Group
 
func GetProduct(ctx context.Context, id string) (*Product, error) {
    cacheKey := "product:" + id
 
    // Fast path: cache hit
    if val, err := cache.Get(ctx, cacheKey); err == nil {
        var product Product
        if err := json.Unmarshal(val, &product); err == nil {
            return &product, nil
        }
        // Corrupted cache entry — fall through to the DB path below instead
        // of returning a zero-value Product.
    }
 
    // Deduplicate concurrent misses — only one goroutine fetches from DB
    v, err, _ := group.Do(cacheKey, func() (interface{}, error) {
        // Double-check: another goroutine may have populated while we waited
        if val, err := cache.Get(ctx, cacheKey); err == nil {
            var product Product
            if err := json.Unmarshal(val, &product); err == nil {
                return &product, nil
            }
        }
 
        product, err := db.GetProduct(ctx, id)
        if err != nil {
            return nil, err
        }
        data, err := json.Marshal(product)
        if err != nil {
            // The DB fetch above already succeeded; a marshal failure should
            // skip the cache write, not the response.
            slog.WarnContext(ctx, "cache marshal failed", "key", cacheKey, "err", err)
            return product, nil
        }
        if err := cache.Set(ctx, cacheKey, data, 5*time.Minute); err != nil {
            slog.WarnContext(ctx, "cache write failed", "key", cacheKey, "err", err)
        }
        return product, nil
    })
 
    if err != nil {
        return nil, err
    }
    // Do's discarded third result reports whether v was shared with other callers
    return v.(*Product), nil
}

singleflight is process-level: each API pod deduplicates its own concurrent misses. In a fleet of 100 pods, the 40,000 concurrent misses at TTL expiry (~400 per pod) collapse to 1-2 DB queries per pod (~100-200 fleet-wide) instead of 40,000 — a 200-400× reduction.

Invalidation Without Timers

TTL-based invalidation alone is dangerous when data changes frequently. Setting a 5-minute TTL on product prices means customers see stale prices for up to 5 minutes after an admin price change. For critical data, you need invalidation that fires on writes, not timers.

Event-driven invalidation: After every database write, publish an invalidation event. A subscriber (which can be the same service) deletes or updates the affected cache key.

func (s *ProductService) UpdateProduct(ctx context.Context, product *Product) error {
    // Write to DB
    if err := s.db.UpdateProduct(ctx, product); err != nil {
        return err
    }
    // Publish invalidation event — subscriber will delete from cache
    event := map[string]string{"type": "product", "id": product.ID}
    data, err := json.Marshal(event)
    if err != nil {
        // The DB write already succeeded; don't fail the caller over a bad
        // event — TTL still bounds the staleness window as a fallback.
        slog.ErrorContext(ctx, "invalidation event marshal failed", "id", product.ID, "err", err)
        return nil
    }
    if err := s.pubsub.Publish(ctx, "cache.invalidations", data); err != nil {
        slog.ErrorContext(ctx, "invalidation publish failed", "id", product.ID, "err", err)
    }
    return nil
}
 
// Subscriber (runs in background). A failed subscription is returned, not
// dropped — otherwise invalidations silently stop and only TTL bounds staleness.
func (s *InvalidationService) ProcessInvalidations(ctx context.Context) error {
    return s.pubsub.Subscribe(ctx, "cache.invalidations", func(data []byte) error {
        var event map[string]string
        if err := json.Unmarshal(data, &event); err != nil {
            return fmt.Errorf("unmarshal invalidation event: %w", err)
        }
        cacheKey := event["type"] + ":" + event["id"]
        return s.cache.Delete(ctx, cacheKey)
    })
}

Limitation: Brief window between DB write and cache invalidation where readers see stale data. For strict consistency, couple invalidation with the DB transaction using the Outbox pattern or CDC.

CDC-based invalidation (most reliable): Tools like Debezium read the database WAL (write-ahead log) and publish every row change as an event. This catches all DB updates regardless of origin — API writes, direct SQL, migrations, batch processes.

Tag-based invalidation: Store cache keys under tags. When an entity changes, invalidate its tag to delete all related keys at once. Solves "one write, many cache keys" (e.g., updating a product invalidates detail page + inventory + category listing + search results).

Multi-Tier Caching

Single-cache-tier forces a trade-off: in-process caches (L1) skip the network but are limited to one machine's memory and go cold on restart; Redis^⁠[3] (L2) is shared across servers but adds a network round trip. Multi-tier caching eliminates this trade-off: check L1 first (fast), fall through to L2 (shared state), then to database.

L1 absorbs hot-key traffic. L2 provides shared state across instances. Together, they give you both speed and scale.

type TieredCache struct {
	// expirable.LRU (hashicorp/golang-lru/v2) takes the L1 TTL at construction:
	// expirable.NewLRU[string, []byte](10_000, nil, 2*time.Second).
	// A plain lru.Cache never expires entries, only evicts them.
	l1     *expirable.LRU[string, []byte]
	l2     *redis.Client
	warmed atomic.Bool // set by WarmFromHotKeys; gates the readiness probe
}
 
func (tc *TieredCache) Get(ctx context.Context, key string) ([]byte, error) {
	// L1: in-process (fast, but not shared)
	if val, ok := tc.l1.Get(key); ok {
		return val, nil
	}
 
	// L2: Redis (shared, but slower)
	val, err := tc.l2.Get(ctx, key).Bytes()
	if err == redis.Nil {
		return nil, ErrCacheMiss
	}
	if err != nil {
		return nil, err
	}
 
	// Promote to L1 (next request will hit L1)
	tc.l1.Add(key, val)
	return val, nil
}
 
func (tc *TieredCache) Set(ctx context.Context, key string, value []byte, l2TTL time.Duration) error {
	// Write to both tiers
	tc.l1.Add(key, value)
	return tc.l2.Set(ctx, key, value, l2TTL).Err()
}
 
// On invalidation, delete from both tiers. Other pods' L1 copies expire on the L1 TTL.
func (tc *TieredCache) Delete(ctx context.Context, key string) error {
	tc.l1.Remove(key)
	return tc.l2.Del(ctx, key).Err()
}

Critical: L1 TTL must be much shorter than L2 TTL (e.g., 2 seconds vs 5 minutes). Short L1 bounds staleness: when L2 is invalidated, L1 expires within seconds, visible to readers on the next cache miss.

When NOT to Cache

Caching the wrong data is worse than no cache. Avoid caching in these scenarios:

Write-heavy, read-rare data. If a key is written 100 times/sec and read once/sec, you pay invalidation overhead 100x for every 1x benefit. Examples: last-activity timestamps (write on every request), inventory counts (write on every sale, read occasionally), auction bids. The cost of cache coherence exceeds the read speedup.

Strong-consistency data. If showing a stale payment method or expired permission causes a security or financial failure, don't cache it. Money, authorization tokens, and critical state must be fresh. If you must cache, use write-through with zero TTL (synchronous cache + DB), not eventual consistency.

High-cardinality requests. Queries with unique parameters (generated request IDs, coordinates, full-text search queries) generate infinite distinct keys. Hit rate approaches zero; overhead of cache management (memory, eviction, network) exceeds any latency gain.

Cheap-to-compute data. In front of a fast indexed query, a Redis round trip saves little on a hit, adds latency on every miss, and still costs you invalidation logic. Cache queries whose cost dwarfs the round trip and whose hit rate stays high.

Unencrypted sensitive data. Credit card numbers, SSNs, health records in Redis require app-level encryption. Redis has no field-level encryption; a breach exposes everything at once. Encrypt before writing, decrypt after reading, or skip caching entirely for high-sensitivity data.

Production Checklist

• Stampede defense in place: singleflight or probabilistic early expiration for high-traffic keys
• Invalidation strategy chosen: Event-driven, CDC, TTL, or tag-based (not just TTL alone for mutable data)
• Key schema versioned: Embed version in key or value to handle deployment mismatches
• Multi-tier if needed: L1 LRU + L2 Redis for hot keys if L2 alone hits latency budget
• Monitoring active: Hit rate, eviction rate, command latency p99 tracked
• Memory capacity planned: Know when you'll hit Redis memory limit and have scaling plan
• Redis Cluster configured (if >200K ops/sec): Hash tags for multi-key operations, no KEYS commands
• Lifeboat/circuit breaker: Graceful degradation when Redis fails (serve stale from local LRU)
• Compression evaluated: Large values (> 1KB) may benefit from gzip before caching
• Security review done: No unencrypted CCNs, SSNs, health data in Redis

The four production patterns every cache layer needs

Singleflight in Go — collapse N concurrent misses on the same key into a single backend fetch, the canonical defence against cache stampedes that doesn't require probabilistic refresh:

import "golang.org/x/sync/singleflight"
 
var sfg singleflight.Group
 
func (c *Cache) Get(ctx context.Context, key string) ([]byte, error) {
    if v, ok := c.local.Get(key); ok { return v, nil }
 
    // singleflight: any concurrent Get(key) waits on the in-flight fetch.
    val, err, _ := sfg.Do(key, func() (any, error) {
        return c.fetchFromOrigin(ctx, key)
    })
    if err != nil { return nil, err }
    c.local.Set(key, val.([]byte))
    return val.([]byte), nil
}

XFetch (probabilistic early expiration) — preempts the deterministic miss-storm at TTL boundary by giving each concurrent reader a small probability of refreshing slightly early:

import "math"
import "math/rand/v2"
 
// xFetchShouldRefresh returns true if the caller should pre-emptively refresh.
// delta is the typical time the origin fetch takes; beta typically 1.0.
func xFetchShouldRefresh(ttlRemaining, delta time.Duration, beta float64) bool {
    rand := rand.Float64()
    threshold := delta.Seconds() * beta * (-math.Log(rand))
    return ttlRemaining.Seconds() < threshold
}

Cache-key namespace convention — version-prefixed so a deploy that changes the cached struct doesn't poison the cache, with a test that fails when the struct changes without a version bump:

{service}:{entity}:v{schema}:{id}      # canonical
billing:invoice:v3:42                  # invoice schema v3, id=42
search:results:v1:hash:{queryHash}     # search-results table, content-hashed key

// Pin the cache-key version to the struct's declared schema version so a deploy
// that reshapes Invoice without bumping the version can be caught in a test.
const InvoiceSchemaVersion = 3
 
// Invoice carries its own schema version; bump it in lockstep with the struct.
type Invoice struct {
    SchemaVersion int    `json:"_schema"` // always == InvoiceSchemaVersion
    ID            string `json:"id"`
    // ... fields that, if changed, require bumping InvoiceSchemaVersion ...
}
 
// InvoiceCacheKey builds "billing:invoice:v{N}:{id}" from the single source of truth.
func InvoiceCacheKey(id string) string {
    return fmt.Sprintf("billing:invoice:v%d:%s", InvoiceSchemaVersion, id)
}

// Test guard: fingerprint Invoice's fields (name, type, JSON tag) per schema
// version. A deploy that changes the struct without bumping the version fails
// here instead of silently poisoning the cache with mismatched payloads.
var invoiceShapes = map[int]string{
    3: "SchemaVersion:int:_schema,ID:string:id", // record a new entry on every bump
}
 
func TestInvoiceCacheVersionPinned(t *testing.T) {
    typ := reflect.TypeOf(Invoice{})
    fields := make([]string, 0, typ.NumField())
    for i := 0; i < typ.NumField(); i++ {
        f := typ.Field(i)
        fields = append(fields, f.Name+":"+f.Type.String()+":"+f.Tag.Get("json"))
    }
    got := strings.Join(fields, ",")
    if want, ok := invoiceShapes[InvoiceSchemaVersion]; !ok || got != want {
        t.Fatalf("Invoice shape %q does not match v%d's recorded shape %q — bump InvoiceSchemaVersion and record the new shape",
            got, InvoiceSchemaVersion, want)
    }
}

Circuit breaker around the cache itself — when Redis is unreachable, serve stale from local LRU instead of forwarding the miss storm to your origin database:

// GetOrStale wraps the Redis read in a circuit breaker. On a Redis error or
// open circuit, it returns the stale local copy (if any) instead of hitting
// the origin — the origin is what the cache exists to protect.
func (c *Cache) GetOrStale(ctx context.Context, key string) ([]byte, error) {
    val, err := c.cb.Execute(func() (any, error) {
        b, err := c.redis.Get(ctx, key).Bytes()
        if errors.Is(err, redis.Nil) {
            return nil, nil // a miss means Redis is healthy; don't count it as a failure
        }
        return b, err
    })
    if err == nil {
        if b, ok := val.([]byte); ok {
            return b, nil
        }
        return nil, ErrCacheMiss // healthy miss: caller fetches from origin
    }
    if stale, ok := c.staleLocal.Get(key); ok {
        c.metrics.staleServed.Inc()
        return stale, nil
    }
    return nil, err
}

The pattern: only fall back to origin when stale-local is also empty. Otherwise a Redis blip becomes a thundering herd against the database — the exact failure mode the cache exists to prevent. Keep ordinary misses out of the breaker's failure count: by default gobreaker treats every non-nil error as a failure and trips after more than five in a row^⁠[4], so returning redis.Nil would open the circuit on a healthy Redis.

Cache Warming: Surviving Cold Starts

A freshly deployed pod or a flushed Redis cluster has a hit rate of zero. The first wave of traffic punches through to the database, and on a busy service that wave is the same shape as a stampede. Three strategies cover the real cases.

Backfill from a known hot-key list. Maintain a rolling list of the top N keys by hit count (a Redis sorted set works, refreshed by a background sampler). On boot, the new pod fetches the list and populates its L1 LRU — but only if L1's TTL outlives the readiness gap; at 2 seconds it usually doesn't, so a fresh pod's real protection is L2 already being warm.

Lazy-load-on-deploy with traffic shifting. Pair backfill with a load balancer that ramps traffic to the new pod instead of sending a full share immediately — for example, 10% for the first 30 seconds, then 50%, then 100%. The pod warms organically without any single replica taking the full miss-storm. This is the cheapest option when you already run a service mesh or weighted routing.

Purge-on-promote for write-through tiers. Redis replication is asynchronous, so a replica promoted to primary (failover, maintenance) can be missing the last writes the old primary acknowledged^⁠[5], and pods' L1 copies can disagree with it. Keyspace notifications can't reconcile the two: Redis Pub/Sub is fire-and-forget, so events sent while a subscriber is disconnected are lost^⁠[6]. Refilling L1 from that primary just copies its stale values — repopulate the affected keys from the database instead, since it holds the newer writes. Critical for write-through tiers where staleness causes correctness bugs, not just slow pages.

// WarmFromHotKeys fetches the top-N hot keys from Redis and populates L1.
// Called from the readiness probe — pod doesn't accept traffic until warm.
func (tc *TieredCache) WarmFromHotKeys(ctx context.Context, n int) error {
    keys, err := tc.l2.ZRevRange(ctx, "hot:keys", 0, int64(n-1)).Result()
    if err != nil {
        return fmt.Errorf("fetch hot key list: %w", err)
    }
    pipe := tc.l2.Pipeline()
    cmds := make(map[string]*redis.StringCmd, len(keys))
    for _, k := range keys {
        cmds[k] = pipe.Get(ctx, k)
    }
    if _, err := pipe.Exec(ctx); err != nil && err != redis.Nil {
        return fmt.Errorf("pipeline get: %w", err)
    }
    for k, cmd := range cmds {
        if val, err := cmd.Bytes(); err == nil {
            tc.l1.Add(k, val)
        }
    }
    tc.warmed.Store(true)
    return nil
}

Singleflight in Anger: A Stampede Walkthrough

Picture a travel-pricing service running a 30-second TTL on its quote:{route}:{date} keys to keep prices fresh. Under normal load (~6,000 req/s) the quote engine absorbs the predictable miss wave at each TTL boundary without trouble. Now picture a viral promo: traffic spikes to 38,000 req/s, nearly all of it on the same 12 hot route:date keys. At the next TTL flip those keys go cold together, and every request that arrives before a refill lands misses and calls the engine itself. The engine is capped at 4,000 concurrent computations, so calls queue, quote latency climbs from a 40ms p99 into seconds, refills land later, and the miss window keeps widening until the pricing pool exhausts and checkout starts returning 503s.

The fix is one deploy: wrap the quote lookup in singleflight keyed by route:date, and add negative-cache entries for routes the engine returns no-availability for. The negative entry uses a short, jittered TTL — long enough to absorb retry storms from the same client, short enough that legitimate availability changes propagate quickly.

// QuoteOrAbsent returns a quote, the sentinel ErrNoAvailability, or a real error.
// Concurrent callers for the same key share one origin call via singleflight.
func (s *PricingCache) QuoteOrAbsent(ctx context.Context, route, date string) (*Quote, error) {
    key := fmt.Sprintf("quote:%s:%s", route, date)
 
    if val, err := s.cache.Get(ctx, key); err == nil {
        if bytes.Equal(val, sentinelAbsent) {
            return nil, ErrNoAvailability
        }
        var q Quote
        if err := json.Unmarshal(val, &q); err == nil {
            return &q, nil
        }
    }
 
    v, err, _ := s.sfg.Do(key, func() (any, error) {
        q, err := s.engine.Quote(ctx, route, date)
        if errors.Is(err, ErrNoAvailability) {
            // Negative cache: jitter prevents synchronized re-fetch.
            ttl := 20*time.Second + time.Duration(rand.Int63n(int64(10*time.Second)))
            if setErr := s.cache.Set(ctx, key, sentinelAbsent, ttl); setErr != nil {
                slog.WarnContext(ctx, "negative cache write failed", "key", key, "err", setErr)
            }
            return nil, ErrNoAvailability
        }
        if err != nil {
            return nil, err
        }
        data, err := json.Marshal(q)
        if err != nil {
            slog.WarnContext(ctx, "cache marshal failed", "key", key, "err", err)
            return q, nil
        }
        if err := s.cache.Set(ctx, key, data, 30*time.Second); err != nil {
            slog.WarnContext(ctx, "cache write failed", "key", key, "err", err)
        }
        return q, nil
    })
 
    if err != nil {
        return nil, err
    }
    return v.(*Quote), nil
}

With singleflight in place, each pod sends at most one in-flight quote-engine call per hot key, whatever the request rate: on a 90-pod fleet that same spike costs about 1,100 calls per TTL flip (90 × 12), and quote latency stays near its normal p99 instead of climbing into seconds.

Negative Caching: TTL Math for Absent Data

Negative caching means storing the answer "this does not exist" with a deliberate TTL. It defends against repeated lookups for missing rows — bot traffic probing for /users/{random-uuid}, expired session IDs, deleted SKU pages — that would otherwise reach the database on every request.

The TTL math is sharper than for positive entries. A positive cache entry trades freshness for hit rate; a negative entry trades latency on first appearance for hit rate. If a real user creates the resource that the cache currently says is absent, they wait until the negative TTL expires before seeing their own data.

Three rules: size the negative TTL by how long a newly created row may stay invisible (seconds to a minute), delete the negative entry on the create path right after the INSERT commits (a Redis delete can't join the database transaction), and always jitter to avoid synchronized expiry across clients. Without jitter, negative caching turns a steady miss stream into a synchronized stampede every TTL window — exactly the failure mode it was meant to prevent.

Frequently Asked Questions

Keep Reading

• Scaling Redis for High-Throughput Systems — Cluster topology, hot key mitigation, Lua scripts for atomicity, 500K+ ops/sec
• Rate Limiter Algorithms: Token Bucket vs Sliding Window — Rate limiters interact with cache misses; separate buckets prevent cascading failures
• Database Indexing Strategies — Proper indexing eliminates cache need for fast queries; cache only expensive ones