Kafka Consumer Lag Is Lying to You - How to Measure and Fix It Properly

Every Kafka incident review I have sat in has the same slide near the end: a graph of consumer lag that stayed flat and green for the entire outage. Lag was 200 messages. It never moved. Meanwhile customers waited eleven minutes for an order confirmation email, a fraud check ran on data that was already stale, and the on-call engineer had no idea because the dashboard said everything was fine.

The number was not wrong. It was answering a different question than the one you actually care about. kafka-consumer-groups.sh --describe tells you how many messages a consumer has not committed yet. That is not the same as how far behind you are in time, and on a partitioned topic with uneven load it is not even close.

This post is about the gap between that number and reality. We will start with the naive lag check everyone runs, show the three specific ways it lies, then build up to a measurement that actually predicts SLA breaches, and the Prometheus setup to alert on it.

The naive version: message lag from the CLI

Here is what almost everyone starts with.

kafka-consumer-groups.sh \
  --bootstrap-server broker:9092 \
  --group order-processor \
  --describe

GROUP           TOPIC   PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG
order-processor orders  0          1048200         1048250         50
order-processor orders  1          2094100         2094180         80
order-processor orders  2          770010          770040          30
order-processor orders  3          9950120         9958900         8780

Lag here is a simple subtraction per partition: LOG-END-OFFSET - CURRENT-OFFSET. It is the count of messages that have been produced but not yet committed by the consumer group. People sum it, put it on a dashboard, and alert when the total crosses some threshold like 10,000.

For a toy topic with one partition and steady traffic, this is fine. For a real production topic it fails in three ways that all matter.

Lie #1: total lag hides the partition that is on fire

Look at partition 3 in the output above. It has 8,780 messages of lag while every other partition sits under 100. Sum them and you get about 8,940, which is under a 10,000 alert threshold, so nothing fires. But partition 3 is stuck. Maybe one consumer in the group is wedged on a slow downstream call, maybe a poison message is being retried, maybe the key distribution is skewed and that partition just gets more traffic.

Summing lag across partitions is the original sin. Kafka processes partitions independently. A single partition falling behind is a real, user-visible failure for every key that hashes to it, and the sum drowns it out. The moment your topic has more than a couple of partitions, you have to look at the worst partition, not the total.

   Total lag = 8940 (looks healthy under a 10k threshold)

   p0  |=                     50
   p1  |==                    80
   p2  |=                     30
   p3  |=======================================  8780   <- this is the incident

The metric you want is max lag across partitions, not sum. A better version of the same query, per partition, alerting on any single partition crossing a line.

Lie #2: lag is in messages, but your SLA is in time

Your SLA is never “fewer than 10,000 messages behind.” It is “the confirmation email goes out within 30 seconds.” Message lag cannot answer that, because the conversion between messages and seconds depends entirely on throughput, which changes constantly.

Consider two partitions, both showing 5,000 messages of lag.

Same message lag. One is a rounding error, the other has blown your 30-second SLA by more than 3x. A fixed message-count threshold treats them identically. During a traffic spike, message lag balloons even though the consumer is keeping up in wall-clock terms; during a quiet night, a small message lag can mean minutes of delay. Message lag is throughput-dependent, and that is exactly the variable you are trying to control for.

What you actually want is time lag: how old is the oldest un-consumed message, in seconds. That is the number that maps directly to an SLA.

Lie #3: committed offset is not the same as “done processing”

This is the subtle one, and it is the reason lag can look great during an outage.

Kafka lag is computed from the committed offset. But when a consumer commits depends on your code. With the common pattern of enable.auto.commit=true, the client commits offsets on a timer (every 5 seconds by default) for whatever it has polled, not for what it has finished processing. So this sequence is entirely possible:

1. Consumer polls a batch of 500 messages. Offset advances in memory.
2. Auto-commit fires and commits those offsets to Kafka.
3. Your handler is still churning through message #12 of that batch, blocked on a slow API.

At that instant, Kafka reports near-zero lag. The offset is committed. But 488 messages are sitting in an in-memory buffer that nobody has processed, and if the pod dies they will be redelivered, and in the meantime the actual processing delay is climbing. The lag metric measures fetch progress, not work progress.

The fix in the consumer is to commit only after processing, which also makes redelivery correct:

consumer = KafkaConsumer(
    "orders",
    bootstrap_servers="broker:9092",
    group_id="order-processor",
    enable_auto_commit=False,          # commit means "done", not "fetched"
    max_poll_records=100,
)

for message in consumer:
    process(message)                    # do the actual work first
    consumer.commit()                   # then advance the committed offset

Committing per message is slow; in practice you commit per batch after the whole batch is processed. The point is that “committed” now means “processed,” so the lag number finally corresponds to real work left to do. If you cannot change the commit behavior, you must measure delay a different way, which is the next section.

The measurement that actually works: end-to-end time lag

The honest metric is measured from the consumer’s own point of view, not from the broker’s offset arithmetic. Stamp each message when it is produced, and when the consumer finishes processing, record the difference.

import time
from prometheus_client import Histogram, Gauge

E2E_LATENCY = Histogram(
    "kafka_e2e_latency_seconds",
    "Time from produce to end of processing",
    ["topic", "partition"],
    buckets=[0.05, 0.1, 0.5, 1, 2, 5, 10, 30, 60, 120],
)
OLDEST_UNPROCESSED = Gauge(
    "kafka_oldest_unprocessed_seconds",
    "Age of the oldest message still being processed",
    ["topic", "partition"],
)

for message in consumer:
    # message.timestamp is the producer timestamp, in ms
    produced_at = message.timestamp / 1000.0
    OLDEST_UNPROCESSED.labels("orders", message.partition).set(
        time.time() - produced_at
    )

    process(message)

    E2E_LATENCY.labels("orders", message.partition).observe(
        time.time() - produced_at
    )
    consumer.commit()

kafka_e2e_latency_seconds is the ground truth: it includes broker time, fetch time, queue time inside your consumer, and processing time. This is the number your SLA is written against. The histogram lets you alert on p99 instead of an average that hides the tail.

Two things make this reliable. First, use the producer timestamp (message.timestamp with CreateTime), not the broker append time, so the clock starts when the event actually happened. Second, watch for clock skew between producer and consumer hosts; if your p50 latency is negative you have a skew problem, not a fast consumer. In practice, ensure NTP is running everywhere and treat sub-zero readings as a monitoring bug.

There is a catch: this only emits a data point when a message is actually processed. If a partition is completely stalled, no message flows, so no latency is observed, and a naive p99 alert can go quiet exactly when things are worst. That is why you pair it with a broker-side lag exporter that keeps reporting even when the consumer is frozen.

The Prometheus setup: exporter plus e2e, together

The standard tool is Kafka Lag Exporter (or Burrow, or the kafka_exporter project). It polls the broker for LOG-END-OFFSET and the group’s committed offset per partition, and, crucially, it interpolates a time lag by tracking how offsets map to timestamps. That gives you a per-partition time lag that keeps reporting even when the consumer is stuck.

Here is the shape of the whole thing.

   producers                                      Prometheus
      |                                          /    |     \
      v                                         /     |      \
   [ Kafka topic: orders, 4 partitions ]  <----+      |       scrapes
      |                    ^                    (offsets)      |
      v                    |                                   v
   consumer group     Kafka Lag Exporter                app /metrics
   (order-processor)  (broker-side, per partition)     (e2e histogram)
      |                                                       |
      +----------------- emits e2e latency ------------------+
                                |
                                v
                          Alertmanager --> pager

You run two sources of truth on purpose:

• The lag exporter never stops reporting, because it reads the broker, so it catches a fully stalled consumer. But its time lag is an interpolation and it cannot see in-flight processing.
• The app e2e histogram is exact and includes processing time, but goes silent on a hard stall.

Together they cover each other’s blind spots. The exporter scrape config is ordinary:

scrape_configs:
  - job_name: kafka-lag-exporter
    static_configs:
      - targets: ["lag-exporter:9999"]
  - job_name: order-processor
    static_configs:
      - targets: ["order-processor:8000"]   # app /metrics with the histogram

Now the alerts. This is where the earlier lessons turn into rules that actually fire at the right time.

groups:
  - name: kafka-consumer-lag
    rules:
      # Lie #1 + #2: alert on the WORST partition, measured in TIME
      - alert: PartitionTimeLagHigh
        expr: max by (group, topic) (kafka_consumergroup_group_lag_seconds) > 30
        for: 2m
        labels: { severity: page }
        annotations:
          summary: "{{ $labels.group }} worst-partition time lag > 30s"

      # Lie #3: real processing delay from the consumer's own histogram
      - alert: E2ELatencyP99High
        expr: |
          histogram_quantile(0.99,
            sum by (le, topic) (rate(kafka_e2e_latency_seconds_bucket[5m]))
          ) > 30
        for: 3m
        labels: { severity: page }

      # Catch the silent stall: consumer stopped committing entirely
      - alert: ConsumerStalled
        expr: |
          max by (group, topic) (kafka_consumergroup_group_lag_seconds) > 10
          and
          rate(kafka_e2e_latency_seconds_count[5m]) == 0
        for: 2m
        labels: { severity: page }

The three rules map exactly onto the three lies. PartitionTimeLagHigh uses max by (not sum) and _seconds (not messages). E2ELatencyP99High measures real processing delay from the consumer side. ConsumerStalled is the AND condition that only fires when the broker still sees lag but the consumer has emitted zero processed messages, which is the exact signature of a wedged partition that a p99 alert would miss.

Diagnosing which failure you have

Once these three fire independently, the combination tells you what is actually wrong without SSHing into anything.

That fourth row is worth calling out. If broker lag looks fine but e2e latency is high, the delay is happening before the message reached the point you are measuring, or your clocks disagree. Kafka is not your problem in that case, and chasing consumer scaling will waste an afternoon.

Fixing the common causes

Measurement tells you which problem you have; here is what actually moves the number.

• Hot partition (one partition lagging). Your partition key is skewed. If you keyed by country and 60% of traffic is one country, that partition is permanently hotter. Re-key by something higher-cardinality (like user_id), or if ordering per key does not truly matter, key by nothing and let Kafka round-robin.
• All partitions lagging. You are simply not consuming fast enough. Add consumer instances up to the partition count (you cannot have more useful consumers in a group than partitions), and if you are already at the partition count, increase partitions or make the handler faster. A blocking downstream call in the hot path is the usual culprit; batch it or make it async.
• Silent stall. Almost always a single message the handler cannot process, retried forever, holding the partition. Add a retry ceiling and a dead-letter topic so one bad message does not freeze everything behind it.
• Rebalance storms. If lag spikes every few minutes in sync with consumers joining and leaving, your max.poll.interval.ms is shorter than your worst-case batch processing time, so Kafka thinks the consumer is dead and rebalances. Lower max.poll.records or raise the interval.

What works, what does not, what to actually do

What does not work: a single summed message-lag number with a fixed threshold. It hides hot partitions, it is throughput-dependent so the threshold is meaningless, and it reads the committed offset which can lie about real progress. If that is your only Kafka alert today, you have a dashboard that turns green during outages.

What works, in order of effort:

1. Stop summing. Alert on max lag per partition, not the total. This one change catches most hot-partition incidents and costs you a query edit.
2. Convert to time. Deploy a lag exporter that reports per-partition lag in seconds, and set thresholds against your actual SLA in seconds. Now your alert threshold means something.
3. Add e2e latency from the consumer. A produce-timestamp histogram gives you the real processing delay including in-flight work, and it is the number your SLA is literally written against.
4. Wire the AND rule for stalls. The one alert that fires when broker lag is rising but the consumer has processed nothing is the one that catches the worst, quietest failures.

What to actually do if you only have an afternoon: change the alert from sum to max, and deploy Kafka Lag Exporter so you get time lag per partition. That alone moves you from “green during the outage” to “paged before the SLA breaks,” which is the entire point. The e2e histogram is the correct long-term answer, but the two-line query change buys you most of the safety today.

The lag number was never lying to you on purpose. You were just reading an answer to a question you were not asking. Ask it in seconds, ask it per partition, and ask it from the consumer that does the work.