Kafka Producer Tuning Cheat Sheet: Throughput, Latency & Durability

Picture a broker failing over at peak: every message the leader acknowledged but hadn't yet replicated vanishes — no error, no retry. The config: acks=1, linger.ms=0, retries=0, enable.idempotence=false — it passes staging fine. But acks=1 only confirms the leader, not the followers.

That config predates Kafka 3.0, which made the Java producer safe-by-default (enable.idempotence=true, acks=all — KIP-679). You reach the failure mode today through legacy configs, explicit overrides, or non-Java clients with weaker defaults — verify every value below rather than trusting an inherited config.

Kafka Producer Tuning Essentials

graph LR
    App["Application<br/>producer.send()"] --> RA["RecordAccumulator<br/>buffer per partition"]
    RA -->|"batch.size or<br/>linger.ms expires"| ST["Sender Thread"]
    ST -->|"max.in.flight<br/>batches"| Broker["Kafka Broker"]
    Broker -->|"acks=all"| ST
    ST -->|"callback"| App

The Quick Start

Durability: Choosing acks

acks is the kill switch. acks=0 can lose data on restart. acks=1 can silently lose writes the leader acknowledged but hadn't replicated when it fails over. Use acks=all everywhere except metrics.^⁠[1]

Pair it with broker-side min.insync.replicas=2 on a 3-replica cluster, so a majority persists every write and one broker failure loses nothing^⁠[2].

Throughput: Batching + Linger

Three knobs move throughput: batch.size, linger.ms, and compression.type. batch.size is a ceiling per partition (default 16384 bytes). linger.ms decides how full batches get — Kafka 4.0 raised its default from 0 to 5ms because larger batches "typically result in similar or lower producer latency despite the increased linger" ^⁠[1]. Use lz4 (good ratio/CPU) unless you're network-bound; zstd achieves better compression ratios but costs more CPU.

Measure fill ratio: batch-size-avg / batch.size; well below 1 means raise linger.ms.

Buffer + sender thread lifecycle

send() appends to an in-memory RecordAccumulator; a background Sender thread drains per-partition batches when they fill OR linger.ms expires.

sequenceDiagram
    participant App
    participant Acc as Accumulator
    participant Sender
    participant Broker

    App->>Acc: send(record)
    App->>Acc: send(record)
    App->>Acc: send(record)
    Note over Acc: batch fills OR<br/>linger.ms expires
    Acc->>Sender: flush batch
    Sender->>Broker: Produce (acks=all)
    Broker-->>Sender: ack (ISR copied)
    Sender-->>App: onCompletion

If buffer.memory fills before the Sender can drain, send() blocks up to max.block.ms. This is the hidden backpressure path — misconfigured linger.ms + slow broker can stall the application thread.

Idempotence & Ordering

Enable enable.idempotence=true unconditionally. It eliminates duplicates on retry at no per-record byte cost: producer ID, epoch and base sequence are fixed fields of every v2 batch header ^⁠[3]. With idempotence, max.in.flight.requests.per.connection stays at 5; without it, drop to 1 for ordering.^⁠[1]

Order is guaranteed within a partition. Key records by entity ID for strict per-entity ordering:

producer.send(new ProducerRecord<>("orders", order.customerId(), order));

Don't use transactions for ordering — they're for atomic multi-partition writes and exactly-once semantics, and every commit adds coordinator work. ^⁠[4]

Tune by symptom

Route by the symptom you measured, not the knob you want to tweak:^⁠[1]

graph LR
    Sym{What is<br/>broken?} -->|Data loss on failover| Loss[acks=all<br/>+ min.insync.replicas=2<br/>+ enable.idempotence=true]
    Sym -->|Low throughput| Tp[linger.ms 5 to 20<br/>+ batch.size 65536<br/>+ compression.type=lz4]
    Sym -->|High p99 latency| Lat[Measure first: larger batches often<br/>keep latency similar or lower<br/>then try linger.ms 0 to 1<br/>+ acks=1 if data-loss tolerable]
    Sym -->|Duplicates downstream| Dup[enable.idempotence=true<br/>+ transactional.id for exactly-once]
    Sym -->|Out-of-order messages| Ord[max.in.flight.requests=1<br/>or enable.idempotence=true<br/>which keeps order with five in-flight]
    Sym -->|Producer blocks on send| Buf[buffer.memory e.g. 256 MiB<br/>+ check broker backpressure]
    Sym -->|Network-blip outages| Net[retries=MAX_INT<br/>+ delivery.timeout.ms 120 seconds<br/>+ enable.idempotence=true]
    style Loss fill:#fdd
    style Dup fill:#ffd
    style Ord fill:#ffd
    style Buf fill:#fdd
    style Net fill:#dfd
    style Tp fill:#dfd
    style Lat fill:#dfd

Classify the symptom first; never tune in isolation.

Common Gotchas

• acks=1 + min.insync.replicas=1: Can silently lose acknowledged-but-unreplicated writes on failover. Use acks=all with min.insync.replicas=2.
• retries=0: Drops on network blips. Use enable.idempotence=true + retries=Integer.MAX_VALUE.
• 32 MiB buffer on high-throughput: Fills when brokers slow down, then blocks send(). Raise it (e.g. 256 MiB).
• linger.ms=0 on pre-4.0 clients: That was the default until Kafka 4.0 (now 5). Set 5–20ms on non-latency-critical producers.
• Per-message KafkaProducer: The client is thread-safe. Create one per app, never per-request; causes metadata fetch storms.

Producer Metrics Worth Alerting On

Five JMX/MBean metrics that make production producer issues visible before they become incidents^⁠[5]:

Export via the JMX exporter or the Java client's producer.metrics() map. Burn-rate alert on record-error-rate; saturation alert on buffer-available-bytes.

Sizing Worker Threads in Java Apps

The producer is non-blocking by default — send() returns a Future immediately and the I/O thread handles delivery. Application threads should NOT wait on the future synchronously; use a callback or accumulate batches:

// Anti-pattern: blocks application thread per message
producer.send(record).get();  // synchronous — defeats batching
 
// Production pattern: callback + structured error handling
producer.send(record, (metadata, exception) -> {
    if (exception != null) {
        log.error("send failed: topic={}, partition={}", record.topic(), record.partition(), exception);
        deadLetterQueue.offer(record);  // application-side fallback
    }
});

In Go (sarama/franz-go): async-producer with a goroutine draining Successes and Errors — never silently drop the error channel.

Production Tuning Checklist

Apply in this order — each step depends on the prior:

1. Durability first: acks=all, min.insync.replicas=2 on a 3-replica topic. Confirm with kafka-configs.sh --describe.
2. Idempotence: enable.idempotence=true. Default-on unless a conflicting setting (e.g. acks=1) disables it — set it explicitly first.
3. Throughput second: linger.ms=10, batch.size=65536, compression.type=lz4. Measure record-send-rate before/after.
4. Buffer + delivery timeouts: buffer.memory (e.g. 268435456 = 256 MiB), delivery.timeout.ms=120000. Delays send() blocking; a long stall still blocks max.block.ms, then fails.
5. Per-tenant quotas (multi-tenant clusters): Configure broker-side quotas via kafka-configs.sh --add-config 'producer_byte_rate=...'. Producer-side enforcement is fragile.
6. Observability: Wire the 5 metrics above to Prometheus + alerting. Verify with jconsole or an equivalent for non-JVM producers.
7. Schema evolution: Avro/Protobuf + schema registry from day one — retrofitting from raw JSON is a migration.
8. Dead-letter topics: route send() callback failures to a local DLQ so retries are bounded; consumer DLQs get retention.ms=2592000000 (30 days).
9. Partition key strategy: high-cardinality key (order_id, not customer_id) to avoid hot partitions.
10. Cluster topology: 3 brokers minimum for min.insync.replicas=2 to survive one broker loss; cross-AZ replication in cloud.

Idempotent Producers and Transactional Writes

Two different layers, often conflated: idempotence dedupes retries within one producer session (broker tracks a per-partition sequence number); transactions add atomicity across partitions and sessions. Transactions earn their keep in consume-process-produce — read from topic A, write to topic B, commit the consumer offset atomically. Without them, a crash between produce and offset-commit either loses the downstream record or duplicates it on restart.

Properties props = new Properties();
props.put("bootstrap.servers", "broker1:9092,broker2:9092,broker3:9092");
props.put("transactional.id", "order-processor-instance-7");
props.put("enable.idempotence", "true");
props.put("acks", "all");
props.put("retries", Integer.MAX_VALUE);
props.put("max.in.flight.requests.per.connection", "5");
props.put("delivery.timeout.ms", "120000");
props.put("transaction.timeout.ms", "60000");
 
KafkaProducer<String, OrderEvent> producer = new KafkaProducer<>(props);
producer.initTransactions();  // fences any prior producer with the same transactional.id
 
while (running) {
    ConsumerRecords<String, RawOrder> batch = consumer.poll(Duration.ofMillis(500));
    if (batch.isEmpty()) continue;
 
    producer.beginTransaction();
    try {
        for (ConsumerRecord<String, RawOrder> in : batch) {
            OrderEvent out = transform(in.value());
            producer.send(new ProducerRecord<>("orders.normalized", out.orderId(), out));
        }
        producer.sendOffsetsToTransaction(
            currentOffsets(batch),
            consumer.groupMetadata()
        );
        producer.commitTransaction();
    } catch (ProducerFencedException | OutOfOrderSequenceException fatal) {
        producer.close();
        throw fatal;  // a newer instance has taken over; do not restart this one
    } catch (KafkaException e) {
        log.warn("transaction aborted, rewinding batch", e);
        producer.abortTransaction();
        // poll() already advanced the consumer: seek back or this batch is skipped
        for (TopicPartition tp : batch.partitions()) {
            consumer.seek(tp, batch.records(tp).get(0).offset());
        }
    }
}

Two rules: keep transactional.id stable across restarts (hostname + stable ordinal — random UUIDs defeat zombie-fencing), and batch a poll loop's worth of records per transaction, since each commitTransaction() is a blocking coordinator round-trip plus a marker write per partition. Confluent measured a 3% throughput cost versus an in-order acks=all producer (1 KB messages, 100 ms transactions). ^⁠[4]

Partition Assignment Strategies

The default producer partitioner is sticky for keyless records: it keeps sending to one partition until at least batch.size bytes have gone there, then switches — fuller batches than per-record round-robin. ^⁠[1] Keyed records hash with murmur2 mod partition count — deterministic for a fixed count; adding partitions later breaks the mapping, which is why re-partitioning at scale is non-trivial.

For consumers, the assignment strategy is set via partition.assignment.strategy:

# Cooperative sticky — recommended default since Kafka 2.4
partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor
 
# Range — assigns contiguous partition ranges per topic; risks skew with multiple topics
# partition.assignment.strategy=org.apache.kafka.clients.consumer.RangeAssignor
 
# Round-robin — distributes evenly but reshuffles all partitions on every rebalance
# partition.assignment.strategy=org.apache.kafka.clients.consumer.RoundRobinAssignor
 
# Sticky (legacy, eager) — minimises movement but stops the world during rebalance
# partition.assignment.strategy=org.apache.kafka.clients.consumer.StickyAssignor
 
session.timeout.ms=45000
heartbeat.interval.ms=3000
max.poll.interval.ms=300000
group.instance.id=consumer-pod-2  # static membership — survives short pod restarts without rebalance

CooperativeStickyAssignor rebalances incrementally — only moving partitions pause, where eager protocols stall the whole group on any pod restart. Custom producer partitioners (tenant isolation, hot-key splitting) implement Partitioner via partitioner.class; keep partition() O(1) and lock-free — it runs on the application thread.

ProducerInterceptor for Auditing

Interceptors attach cross-cutting concerns (audit, schema enforcement, header injection) without touching business code. Both hot-path methods must be fast and non-blocking — onSend() runs on the application thread, onAcknowledgement() on the I/O thread; a slow interceptor stalls every producer in the JVM.

public class AuditingInterceptor implements ProducerInterceptor<String, byte[]> {
 
    private static final Logger AUDIT = LoggerFactory.getLogger("kafka.audit");
    private final AtomicLong sent = new AtomicLong();
    private final AtomicLong failed = new AtomicLong();
 
    @Override
    public ProducerRecord<String, byte[]> onSend(ProducerRecord<String, byte[]> record) {
        record.headers()
            .add("audit.producer", System.getenv("HOSTNAME").getBytes(StandardCharsets.UTF_8))
            .add("audit.timestamp", Long.toString(System.currentTimeMillis()).getBytes())
            .add("audit.trace-id", currentTraceId().getBytes(StandardCharsets.UTF_8));
        return record;
    }
 
    @Override
    public void onAcknowledgement(RecordMetadata metadata, Exception exception) {
        if (exception == null) {
            sent.incrementAndGet();
        } else {
            failed.incrementAndGet();
            AUDIT.warn("send failed: topic={} partition={} cause={}",
                metadata != null ? metadata.topic() : "unknown",
                metadata != null ? metadata.partition() : -1,
                exception.getClass().getSimpleName());
        }
    }
 
    @Override public void close() { /* flush metrics */ }
    @Override public void configure(Map<String, ?> configs) { /* read interceptor config */ }
}

Wire via interceptor.classes (declaration order on send, reverse on ack). Benchmark anything that allocates per record — a 10 µs regression in onSend() is the bottleneck at 100k records/sec.

Kafka 4.0 KRaft-Mode Operational Changes

Kafka 4.0 (GA March 2025) is the first major release to run entirely without ZooKeeper — KRaft only. ^⁠[6] Two changes reach producers: old protocol API versions are gone (KIP-896), so Java clients and brokers must both be on 2.1 or newer before either side moves to 4.0, and the Java producer's linger.ms default rose from 0 to 5ms (KIP-1030). ^⁠[1]

Frequently Asked Questions

Keep Reading

• Event-Driven Microservices with Go and Kafka
• Building Resilient Distributed Systems with Go — circuit breakers and retries on the consumer side complement producer idempotence.
• Idempotency Patterns in Distributed Systems — how enable.idempotence=true fits the broader idempotency story.
• Go Concurrency Best Practices — passing context.Context to producer.send() so callers can cancel.
• Microservices Architecture Patterns — when Kafka is the integration layer, producer config defaults the entire system's durability.