TCP vs UDP vs QUIC: Protocol Selection Under Production Load

A single dropped packet freezes an entire price feed until TCP retransmits it. A trading or telemetry feed runs over WebSocket (TCP). Under high loss — congested mobile networks, market-volatility traffic spikes — TCP's head-of-line blocking surfaces: one lost packet stalls every subsequent packet on the same connection until retransmission completes, so every following price arrives stale.

The fix is surgical: move time-sensitive feeds off TCP. For a price feed, a lost packet means "skip to the next one" — not "retransmit and delay everything else." Move to UDP^⁠[1] with application-level sequence numbers (drop out-of-order, no retransmit), or to QUIC^⁠[2] which keeps reliability per-stream so loss on one stream doesn't block others.

The quick start

Choose your transport based on what the data requires, not what feels default:

TCP: Correctness at the Cost of Speed

TCP's five guarantees — ordered delivery, reliability, flow control, congestion control, checksums — come with latency overhead. You get a promise: every byte you send will arrive exactly once, in order, and the sender will slow down if the network can't keep up.^⁠[3]

Setup overhead: The three-way handshake (SYN, SYN-ACK, ACK) burns one RTT before data moves. Add TLS 1.3 and you're at 2 RTTs (40ms on a 20ms cross-datacenter link). For file transfers and databases, this cost is negligible. For real-time feeds, it's fatal.

sequenceDiagram
    participant C as Client
    participant S as Server

    Note over C,S: TCP — 1 RTT before data flows
    C->>S: SYN
    S-->>C: SYN-ACK
    C->>S: ACK + data
    S-->>C: response

    Note over C,S: UDP — 0 RTT, fire and forget
    C->>S: datagram
    Note right of S: no ack
    C->>S: datagram
    C->>S: datagram
    Note right of S: packet 2 lost?<br/>App decides what to do.

The gap matters on every connection: TCP spends the first RTT negotiating; UDP sends the first byte immediately. For a ticker feeding 100 updates/sec, the one-RTT setup is noise. For a connection handshake that's a one-shot, TCP's overhead amortises over millions of packets. For real-time voice, gaming, or telemetry, UDP wins because you'd rather lose a packet than wait for its retransmit.

Head-of-line blocking: TCP buffers out-of-order packets until the missing one arrives. If packet B is lost and C, D arrive, TCP holds them until B retransmits — even though C and D are already there. For a price feed sending 100 ticks/sec, a 50ms retransmit delay stalls 5 fresh prices. By the time they deliver, they're worthless.

Congestion control tuning matters: Linux ships with CUBIC (default) and BBR. CUBIC is loss-based — it ramps the congestion window until a packet drops, then cuts its sending rate sharply and regrows it slowly. BBR is model-based — it estimates bottleneck bandwidth and minimum RTT, then paces packets to match. The IETF's BBRv3 draft reports substantially higher throughput than CUBIC with shallow buffers or random loss^⁠[4]; mainline Linux's bbr module is the original BBRv1^⁠[5]⁠[4]. For internal datacenter traffic with high bandwidth/latency (cross-region replication), enable BBR:

sudo sysctl -w net.ipv4.tcp_congestion_control=bbr

When to use TCP: APIs, chat, payments, order execution, state mutations. Any message where a dropped packet means a broken guarantee. The latency cost is the price of correctness.

For the full request lifecycle with TCP handshake and TLS negotiation, see What Happens When You Type a URL.

UDP: Speed Over Reliability

UDP is the opposite bargain: send a packet to an IP:port, that's it. No connection, no ACKs, no retransmissions, no ordering. If it arrives, fine. If it gets lost in a congested router buffer, gone. The application is responsible for everything.^⁠[1]

What you gain:

• 0-RTT startup: No handshake before data flows
• No blocking on loss: Each datagram is independent — lose packet 5, and packet 6 arrives instantly
• You control send rate: No congestion window slowing you down
• Minimal overhead: 8-byte header vs TCP's 20+ bytes

What you lose:

• No retransmission: Lost packets are gone
• No ordering guarantee: Datagrams may arrive out of order
• No flow control: You can overwhelm a slow receiver
• You implement reliability: Sequence numbers, ACKs, and congestion control are your problem

The key insight: UDP is right when stale data is worse than no data. A game state update from 5ms ago is useless; you want the latest. A price tick from 50ms ago is worthless; you want now. For VoIP, multiplayer games, and real-time telemetry, UDP fits.

Here's a production UDP price-feed server in Go with sequence-based deduplication to discard stale packets:

package main
 
import (
    "bytes"
    "encoding/binary"
    "math"
    "net"
    "sync"
    "sync/atomic"
    "time"
)
 
type PriceUpdate struct {
    Sequence    uint64
    Symbol      [8]byte
    Price       float64
    TimestampMs int64
}
 
func (s *PriceFeedServer) PublishPrice(symbol string, price float64) {
    seq := atomic.AddUint64(&s.seq, 1)
    update := PriceUpdate{
        Sequence:    seq,
        Price:       price,
        TimestampMs: time.Now().UnixMilli(),
    }
    copy(update.Symbol[:], symbol)
    buf := make([]byte, 32)
    binary.BigEndian.PutUint64(buf[0:8], update.Sequence)
    copy(buf[8:16], update.Symbol[:])
    binary.BigEndian.PutUint64(buf[16:24], math.Float64bits(update.Price))
    binary.BigEndian.PutUint64(buf[24:32], uint64(update.TimestampMs))
    s.clients.Range(func(_, value any) bool {
        s.conn.WriteToUDP(buf, value.(*net.UDPAddr))
        return true
    })
}
 
func (c *PriceFeedClient) Receive(handler func(string, float64, int64)) {
    buf := make([]byte, 32)
    for {
        c.conn.SetReadDeadline(time.Now().Add(5 * time.Second))
        n, err := c.conn.Read(buf)
        if err != nil || n < 32 {
            continue
        }
        seq := binary.BigEndian.Uint64(buf[0:8])
        symbol := string(bytes.TrimRight(buf[8:16], "\x00"))
        lastSeq, _ := c.lastSeq.LoadOrStore(symbol, uint64(0))
        if seq <= lastSeq.(uint64) {
            continue
        }
        c.lastSeq.Store(symbol, seq)
        price := math.Float64frombits(binary.BigEndian.Uint64(buf[16:24]))
        ageMs := time.Now().UnixMilli() - int64(binary.BigEndian.Uint64(buf[24:32]))
        handler(symbol, price, ageMs)
    }
}

What it doesn't do: retransmissions, ACKs, or ordering guarantees. Dropped packets mean skipped ticks. For order execution or any message you cannot afford to lose, use TCP or a durable queue like Kafka with TCP backing.

TCP Production Tuning

Connection pooling amortizes handshake cost. Key settings:

transport := &http.Transport{
    MaxIdleConnsPerHost:  100,      // Default: 2
    IdleConnTimeout:      90 * time.Second,
    DialContext: (&net.Dialer{
        Timeout:   5 * time.Second,
        KeepAlive: 30 * time.Second,
    }).DialContext,
    TLSHandshakeTimeout:    5 * time.Second,
    ResponseHeaderTimeout:  10 * time.Second,
}
client := &http.Client{Transport: transport, Timeout: 30 * time.Second}

Server-side kernel tuning for high connection counts:

sudo sysctl -w net.core.somaxconn=65535
sudo sysctl -w net.ipv4.tcp_fastopen=3           # 0-RTT on returns
sudo sysctl -w net.core.rmem_max=16777216        # 16MB socket buffers
sudo sysctl -w net.ipv4.tcp_tw_reuse=1           # Reuse TIME_WAIT sockets
sudo sysctl -w net.ipv4.ip_local_port_range="1024 65535"

QUIC and HTTP/3: The Best of Both

QUIC (RFC 9000) is UDP with TCP's reliability features, but without head-of-line blocking. Each stream is independent — lose a packet on stream 1, and stream 2 keeps flowing unaffected. QUIC was built for the internet's actual conditions: high latency, mobile handoffs (IP changes), and packet loss.

The head-of-line-blocking comparison in one picture — the entire reason QUIC exists:

graph LR
    subgraph TCP[TCP — head-of-line blocking]
        direction TB
        T1[stream A: packets 1, 2, 3] --> Tlost[packet 2 lost]
        Tlost --> Tstall[ALL streams stall<br/>waiting for retransmit]
        T2[stream B: packets 1, 2, 3] -.->|blocked too —<br/>shares the connection| Tstall
        Tstall --> Trecv[receive in order<br/>after retransmit]
    end
    subgraph QUIC[QUIC — stream-independent reliability]
        direction TB
        Q1[stream A: packets 1, 2, 3] --> Qlost[packet 2 lost on stream A]
        Qlost --> Qa[stream A retransmits<br/>independently]
        Q2[stream B: packets 1, 2, 3] -->|unaffected| Qb[stream B keeps<br/>flowing]
    end
    style Tstall fill:#fdd
    style Qb fill:#dfd
    style Qa fill:#ffd

The diagram captures why, under packet loss, HTTP/3 pulls ahead of HTTP/2 on concurrent requests: TCP serialises everything behind the lost packet; QUIC lets unaffected streams continue.

Key wins over TCP:

• Handshake: 1-RTT initial (vs TCP 1-RTT + TLS 1-RTT = 2 RTTs)
• Return connections: 0-RTT with session resumption
• Connection migration: Phone switches WiFi to LTE without breaking the connection (QUIC uses Connection ID, not 4-tuple)
• Stream independence: Packet loss on one stream doesn't block others^⁠[2]
• Built-in encryption: TLS 1.3 is integral, not layered on top

The practical play: enable HTTP/3 at your CDN and let it handle QUIC termination and multiplexing. Building a QUIC server from scratch is rarely justified unless you're a CDN or VPN provider managing high-concurrency public traffic.

Debugging and Observability

When things go wrong, understand where:

# Show all listening sockets with protocol
ss -tuln
 
# Spot TCP connection state issues (TIME_WAIT buildup)
ss -tan | awk '{print $1}' | sort | uniq -c | sort -rn
 
# Capture TCP handshakes in real time
tcpdump -i eth0 'tcp[tcpflags] & (tcp-syn) != 0' -n
 
# Capture UDP traffic on port
tcpdump -i eth0 udp port 9000
 
# Check interface packet loss
ip -s link show eth0  # Look for RX errors, dropped, overrun
 
# Measure latency to a host
ping -c 100 example.com | tail -1
 
# Test TCP throughput
iperf3 -s                    # server
iperf3 -c server_ip -t 30    # client
 
# Test UDP throughput
iperf3 -c server_ip -u -b 100M -t 30
 
# Simulate packet loss to test protocol behavior
sudo tc qdisc add dev eth0 root netem loss 2%
sudo tc qdisc del dev eth0 root  # Clean up
``` <Cite id="iproute2-docs" n="6" />
 
To check if HTTP/3 negotiation is working:
 
```bash
curl -v --http3 https://example.com 2>&1 | grep -E "alt-svc|HTTP/"
# Look for: < alt-svc: h3=":443"; ma=86400
# And:      Using HTTP/3

Production Checklist

When architecting a transport layer, work through this decision tree:

Protocol selection:

• Use TCP for payments, orders, authentication, state mutations — any workload where a dropped or reordered packet breaks the guarantee
• Use UDP for real-time feeds, gaming, VoIP, sensor telemetry — where stale data is worse than no data, and you can tolerate loss
• Use QUIC/HTTP/3 for mobile-heavy web traffic with concurrent requests — enable at your CDN and let it handle QUIC termination
• Hybrid approach: Use both (e.g., UDP for price feeds, TCP for order execution as in the opening example)

TCP tuning (if applicable):

• Client-side: Raise MaxIdleConnsPerHost to match concurrency (default 2 is often too low)
• Server-side: Enable tcp_fastopen=3 for 0-RTT on returning clients
• Kernel: Increase socket buffers (rmem_max, wmem_max to 16MB+) for high-throughput datacenter transfers
• Kernel: Set tcp_tw_reuse=1 to avoid TIME_WAIT socket exhaustion on servers with many short-lived connections

UDP safety:

• Implement sequence numbers if you need to discard stale or duplicate packets
• Implement your own congestion control — pacing at the application level — to avoid saturating shared links
• Monitor packet loss under load — use tc qdisc to simulate loss and verify graceful degradation

Observability:

• Monitor TCP connection states — TIME_WAIT accumulation indicates connection churn
• Profile handshake overhead — measure cumulative time spent on 3-way handshakes under realistic load
• Test protocol behavior under loss — use synthetic packet loss to validate choices before they break in production

The diagnostic toolkit: kernel knobs, loss simulation, latency probe

A sysctl block that captures the production-tested defaults for high-RPS TCP servers — paste it into your AMI bake or DaemonSet so kernel tuning travels with the deploy:

# /etc/sysctl.d/99-tcp-tuning.conf — production defaults
# Apply with: sysctl -p /etc/sysctl.d/99-tcp-tuning.conf
 
# Larger socket buffers for high-bandwidth × latency datacenter transfers.
# Raises BDP ceiling — a 10 Gbit link with 1ms RTT = 1.25 MB BDP.
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
 
# TIME_WAIT sockets eat ephemeral ports under high connection churn.
# tcp_tw_reuse=1 lets the kernel reuse TIME_WAIT slots SAFELY (RFC 6191).
# Do NOT enable tcp_tw_recycle — it breaks NAT and was removed in 4.12.
net.ipv4.tcp_tw_reuse = 1
net.ipv4.ip_local_port_range = 1024 65535
 
# 0-RTT for returning clients (fastopen=3 = both client and server).
# Carries data in the SYN on warm reconnects, saving up to one RTT.
net.ipv4.tcp_fastopen = 3
 
# BBR (mainline Linux = BBRv1): paces from a bandwidth/RTT model, not loss.
# Flip with caution; some middleboxes punish unrecognised cwnd patterns.
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr

Loss simulation with tc qdisc — adds 5% packet loss + 50ms latency to a single interface so a UDP-based service can be tested under real-world WAN conditions: ^⁠[6]

# Inject loss + latency on eth0 outbound. Run inside a netns or test pod.
tc qdisc add dev eth0 root netem loss 5% delay 50ms 10ms distribution normal
 
# Verify with iperf3 or ping:
ping -c 100 8.8.8.8 | tail -3
# rtt min/avg/max/mdev = 51.2/57.8/89.4/8.1 ms — packet loss surfaces in mdev
 
# Clean up:
tc qdisc del dev eth0 root

A Go latency-probe that sweeps TCP vs UDP vs QUIC against the same target — paste-ready for an SRE dashboard that validates the protocol-choice assumption holds under your network conditions:

package probe
 
import (
    "context"
    "net"
    "time"
)
 
// MeasureRoundTrip returns the wire RTT for one of {tcp, udp} to addr:port.
// For QUIC, use http3.RoundTrip; the API surface is similar.
func MeasureRoundTrip(ctx context.Context, network, addr string) (time.Duration, error) {
    var d net.Dialer
    start := time.Now()
    conn, err := d.DialContext(ctx, network, addr)
    if err != nil { return 0, err }
    rtt := time.Since(start)
 
    // For UDP we have no handshake — measure send→reply round trip explicitly.
    if network == "udp" {
        _, _ = conn.Write([]byte("ping"))
        buf := make([]byte, 64)
        rttStart := time.Now()
        _ = conn.SetReadDeadline(time.Now().Add(3 * time.Second))
        _, _ = conn.Read(buf)
        rtt = time.Since(rttStart)
    }
    _ = conn.Close()
    return rtt, nil
}

Pair the probe with a 1-minute Prometheus recording rule so the protocol RTT becomes a graphable signal — when "TCP feels slower than UDP" turns into a real customer report, you have data instead of guesses:

# probe-rules.yml — recording rules for transport-layer latency dashboards
groups:
  - name: transport-probes
    interval: 60s
    rules:
      - record: probe_rtt_seconds:tcp:p95
        expr: histogram_quantile(0.95, sum(rate(probe_rtt_seconds_bucket{network="tcp"}[5m])) by (le, target))
      - record: probe_rtt_seconds:udp:p95
        expr: histogram_quantile(0.95, sum(rate(probe_rtt_seconds_bucket{network="udp"}[5m])) by (le, target))

The ratio tcp:p95 / udp:p95 is a quick "is TCP overhead worth it on this path?" check — pick thresholds, e.g. under 1.3 as negligible overhead and over 2 as a packet-loss problem amplifying retransmits, where a UDP-based protocol could be measurably faster.

QUIC Connection Migration: Surviving the IP Change

TCP identifies a connection by the 4-tuple of source IP, source port, destination IP, destination port. Change any one of those — and a phone walking out of a cafe does that the moment it switches from WiFi to LTE — and the kernel tears the socket down. Every in-flight HTTP request fails, the application gets a connection reset, and the user sees a spinner while the browser reopens TCP, redoes TLS, and replays the request. A handoff costs a TCP handshake plus a TLS 1.3 handshake, full or resumed^⁠[7] — two round trips — before the replay.

QUIC sidesteps the 4-tuple entirely. Each endpoint issues its peer opaque Connection IDs (up to 20 bytes in QUIC v1). Packets are routed by Connection ID, not by IP — so when the client's IP changes, the server still recognises the connection from the ID and continues delivering data on the existing streams. The handoff cost drops from a full handshake-and-retry to a single round-trip path validation.

Walk through the IP-change scenario step by step. The user is mid-stream watching a video on HTTP/3 when they leave WiFi:

T0: WiFi connection established — Connection ID = 0xa3f7...
    Client IP = 192.168.1.42, Server IP = 203.0.113.10
    Streams 1, 3, 5 carrying video segments; QPACK table warm.
 
T1: WiFi signal drops — radio switches to LTE.
    Client IP changes: 192.168.1.42 -> 100.64.7.81 (carrier NAT).
    Client OS surfaces an "IP changed" event to the QUIC stack.
 
T2: Client sends from 100.64.7.81 using a fresh Connection ID
    (0x5c21..., issued earlier via NEW_CONNECTION_ID).
 
T3: Server maps 0x5c21... to the same connection, but treats
    the new address as UNVALIDATED — sends it at most 3x the
    bytes received from it (anti-amplification, RFC 9000 §8).
 
T4: Server sends PATH_CHALLENGE (8 random bytes) to the new
    address; the client echoes them in PATH_RESPONSE.
    Round trip complete — the new address is validated.
 
T5: Server lifts the amplification cap; congestion control
    restarts on the new path. Streams 1, 3, 5 keep flowing —
    no streams reset, no HTTP requests replayed, no TLS
    handshake. The stall is bounded by path validation,
    not a new TCP+TLS handshake.

The anti-amplification rule at T3 limits how far attackers can use QUIC servers as DDoS reflectors — without it, a spoofed source IP could trick the server into firing a flood at an unsuspecting victim. The 3x cap limits a spoofed reflection to three times the bytes the attacker sent. ^⁠[2]

Connection IDs are also rotated. Both peers can issue NEW_CONNECTION_ID frames to introduce fresh IDs, then retire old ones with RETIRE_CONNECTION_ID. This prevents a passive observer who saw the original handshake from linking a user's WiFi traffic to their LTE traffic — the Connection ID changes across the migration, so to a network observer, the LTE flow looks unrelated. Privacy by construction.

To watch migration in production, capture a QUIC trace and grep for path frames:

# Capture QUIC traffic on the wire (UDP/443 by default).
sudo tcpdump -i any -w quic.pcap 'udp port 443'
 
# Decode with Wireshark — needs SSLKEYLOGFILE export from the client
# so Wireshark can decrypt the QUIC payloads.
SSLKEYLOGFILE=/tmp/quic.keys curl --http3 https://example.com/
wireshark -o tls.keylog_file:/tmp/quic.keys quic.pcap
 
# Filter in Wireshark for migration events:
#   quic.frame_type == 0x1a   -- PATH_CHALLENGE
#   quic.frame_type == 0x1b   -- PATH_RESPONSE
#   quic.frame_type == 0x18   -- NEW_CONNECTION_ID

The quic-go library exposes a Tracer hook so server-side code can record migrations directly without packet capture — log the event with the new remote address and increment a Prometheus counter so the dashboard reflects how often clients are actually migrating in your traffic mix; every migration is a stall HTTP/2 over TCP would have suffered and HTTP/3 over QUIC absorbed silently.

Frequently Asked Questions

Keep Reading

• What Happens When You Type a URL — The full request lifecycle from DNS resolution through TCP/TLS handshake to application response, with debugging tools for each layer
• HTTP Protocol Evolution Guide — HTTP/1.1 keep-alive through HTTP/2 multiplexing to HTTP/3 over QUIC, and what each version changed at the transport layer
• DNS Records Production Guide — A, AAAA, CNAME, SRV, and the TTL tradeoffs that affect every TCP connection before it starts