At 3 AM an invoice-generation consumer starts failing. A single message in its queue has a GST number with an unexpected format, the parser throws, the message is not acknowledged, and the broker delivers it again. The consumer fails again. And again. On a Kafka topic, every invoice behind that message waits, because the consumer never moves past it. By the time someone looks, 40,000 invoices are stuck behind one bad record. That record is a poison message: a message that will fail every time it is processed, no matter how often you try.

Not all failures are the same

The first design decision is to separate two kinds of failure, because they need opposite treatment.

Transient failures come from the environment: a database connection timed out, a downstream API returned 503, a lock could not be acquired. The message is fine. Retrying later, after the dependency recovers, will probably succeed.

Permanent failures come from the message itself or from code that cannot handle it: malformed JSON, a missing required field, a reference to a customer that does not exist, a bug triggered by an unusual value. Retrying will fail identically, forever. Every retry wastes resources and, worse, may block other work.

Good consumers classify errors explicitly. Validation errors, deserialisation errors and known business-rule violations are marked as non-retryable and go straight to the side. Network errors, timeouts and throttling responses are retried. Unknown exceptions are usually retried a few times, then treated as permanent.

The dead-letter queue

A dead-letter queue (DLQ) is where messages go when they cannot be processed. Moving a poison message there and acknowledging the original lets the consumer carry on with the next message. The failed message is not lost; it is parked for inspection and later reprocessing.

Brokers support this differently. Amazon SQS has a redrive policy: after a message has been received a configured number of times without being deleted (the maxReceiveCount), SQS moves it to the designated DLQ. RabbitMQ lets a queue name a dead-letter exchange, to which messages are routed when rejected without requeue, when they expire, or when the queue overflows; quorum queues can also enforce a delivery limit. Apache Kafka’s broker has no built-in DLQ, because consumers simply read a log at an offset. Dead-lettering in Kafka is done by the consumer, by publishing the failed record to a separate topic. Frameworks such as Spring for Apache Kafka and Kafka Connect provide this as configuration.

A dead-lettered message should carry enough context to be useful later: the original topic or queue, partition and offset where relevant, the error message and exception type, the number of attempts, the time of failure, and the consumer version. A DLQ full of payloads with no explanation turns into a guessing game.

Retrying without blocking the line

For transient failures, the naive approach is to retry in place: sleep and try again. On a queue like SQS that is acceptable, because other consumers keep pulling other messages. On a Kafka partition it is head-of-line blocking: while one consumer retries one message for two minutes, every later message in that partition waits.

A common Kafka pattern is a ladder of retry topics. A failed message is published to a retry topic with a delay, for example one for 30 seconds, one for 5 minutes, one for 30 minutes, and the main consumer moves on. A retry consumer reads each retry topic and processes messages only once their delay has passed. After the last rung, the message goes to the DLQ. This keeps the main flow moving and spaces out retries so a struggling dependency gets breathing room.

The cost is ordering. Once a message leaves the main partition for a retry topic, later messages for the same key can be processed before it. For events where order matters, such as an account’s balance updates, a retry ladder can produce wrong results. Options include pausing the specific key until its failed message succeeds, or designing consumers to ignore events older than what they have already applied, using a version number in the event.

A DLQ nobody reads is a data-loss bug

The most common failure of DLQ design is organisational. Messages flow into the DLQ, consumers look healthy, dashboards are green, and three months later someone finds 12,000 unprocessed refunds. A DLQ needs an owner and an operating routine.

  • Alert on DLQ depth and inflow rate. Any message arriving in a DLQ for a payment or order flow probably deserves attention the same day. A sudden spike usually means a deployment or an upstream change broke something for every message, not just one.
  • Build a redrive path. Once the bug is fixed or the data corrected, you need a safe way to move messages back to the main queue. SQS offers DLQ redrive natively; for Kafka it is usually a small tool or job. Redriven messages may be duplicates of partly processed work, so consumers must be idempotent.
  • Set retention deliberately. SQS retains messages for a maximum of 14 days. A DLQ with short retention silently deletes evidence. Choose retention longer than the time it would realistically take your team to notice and fix a problem.
  • Handle sensitive payloads carefully. DLQs often hold full customer data. Access to them, and to any tool that displays their contents, should follow the same rules as the main data store.

Preventing poison in the first place

Many poison messages come from producers and consumers disagreeing about the shape of data. A schema registry with compatibility checks, as used with Avro or Protobuf in Kafka ecosystems, rejects incompatible changes before they are published. Contract tests between producer and consumer teams catch the rest earlier. Validating on the producer side, so a malformed GST number never enters the queue, is cheaper than cleaning up after it on the consumer side.

The overall shape is simple to state: retry what might succeed, park what will not, never let one message hold up the rest, and treat the parking lot as a queue of work for humans rather than a place where messages go to be forgotten.

Written by the RP Strativa Team. Message counts, delays and the invoice scenario are illustrative.

Leave a Reply

Trending