A payments team publishes three events for each transaction: Initiated, Authorised, Settled. A downstream ledger service occasionally receives Settled before Authorised and rejects it as invalid. The team’s first instinct is that Kafka is unreliable. It is not. Kafka made a specific, narrow promise about ordering, and the producer’s setup did not line up with it. Knowing exactly what that promise is solves a large share of Kafka design questions.
Topics are split into partitions, and partitions are logs
A Kafka topic is divided into partitions. Each partition is an append-only log: messages are written to the end and given a sequential number called an offset. Within one partition, a consumer reads messages in offset order, which is the order they were written. Across partitions there is no ordering at all. Partition 0 and partition 5 are independent logs, possibly on different brokers, and a consumer reading both may see them interleaved in any way.
So the guarantee is: messages in the same partition are delivered in order. That is it. Partitions exist for scale. A topic with 12 partitions can be written to and read from in parallel by up to 12 consumers in a group, and spread across many brokers. Ordering is traded away between partitions to get that parallelism.
Keys decide which partition a message lands in
When a producer sends a message with a key, the default partitioner hashes the key and maps it to a partition. The same key always lands in the same partition, as long as the partition count does not change. That is how you get ordering where you need it: give all events that must stay in order the same key.
In the payments example, the producer was sending without a key. Keyless messages are spread across partitions (modern clients use a “sticky” strategy that fills a batch for one partition before moving on), so Initiated, Authorised and Settled for one transaction could land in three different partitions and be consumed in any order. Using the transaction id as the key puts all three in one partition, and the ledger sees them in sequence. Choosing the key is a decision about what needs to be ordered relative to what: per transaction, per account, per customer. A coarser key gives broader ordering but less even distribution, and the hot-key problems familiar from database sharding apply here too.
Two producer-side details can still break order inside a partition. If the producer retries a failed batch while a later batch has already succeeded, the retry can land after it. The idempotent producer, enabled by default since Kafka 3.0, uses sequence numbers so the broker keeps batches in order even with retries and up to five in-flight requests per connection. And if two different producer processes send events for the same key, Kafka orders them by arrival at the broker, not by when your application thinks they happened.
Consumer groups: dividing the partitions
Consumers that share a group id form a consumer group. Kafka assigns each partition to exactly one consumer in the group, so every message is processed by one member of the group. One consumer may own several partitions, but a partition is never split between two consumers of the same group. This is what preserves per-partition ordering during parallel consumption.
A consequence follows directly: the number of partitions caps the useful parallelism of a group. With 12 partitions and 15 consumers, three consumers sit idle. Different groups are independent. The ledger service and an analytics service can each have their own group and each receive every message, each tracking its own position.
That position is the committed offset. Consumers periodically commit the offset of the next message to read for each partition, stored by Kafka in an internal topic. After a restart, a consumer resumes from the committed offset. If it processed messages but crashed before committing, those messages are delivered again, which is why consumers must handle duplicates.
Rebalancing: when ownership moves
Whenever a consumer joins or leaves a group, or is judged dead, partitions are reassigned. That is a rebalance. A consumer is judged dead if it stops sending heartbeats or if it fails to poll for new messages within max.poll.interval.ms, five minutes by default. A slow batch of work that exceeds that interval gets the consumer kicked out, its partitions handed to someone else, and its uncommitted messages reprocessed, sometimes in a loop if the same slow messages keep causing the same timeout.
In the older “eager” protocol, every consumer gave up all its partitions at the start of a rebalance, so the whole group paused. Kafka later added cooperative incremental rebalancing, where only the partitions that need to move are revoked, and static group membership, which lets a consumer restart with the same identity without triggering a rebalance at all, useful for rolling deployments. More recent Kafka versions also introduced a new, broker-coordinated consumer group protocol designed to make rebalances faster and less disruptive. Which options you have depends on your Kafka and client versions.
Ways ordering still breaks in practice
- Parallel processing inside a consumer. A consumer that reads a batch from one partition and hands messages to a thread pool has thrown away ordering itself. If you need concurrency within a partition, route messages by key to dedicated workers, so each key still has one processing lane.
- Adding partitions. Going from 12 to 24 partitions changes the mapping of keys to partitions. New events for a key may land in a different partition from its older, not-yet-consumed events. Plan partition counts with growth in mind, or migrate deliberately.
- Retrying by republishing. Sending a failed message to a retry topic and moving on lets later messages for the same key overtake it. If order matters, either block the key until the retry succeeds or design consumers to tolerate out-of-order arrival.
- Relying on order you do not need. Sometimes the better fix is a version number or timestamp in each event, so consumers can ignore stale updates regardless of arrival order. That makes the system robust to all of the above.
The compact version: pick a key that groups everything needing order, keep one consumer lane per key, size partitions for future parallelism, and make consumers safe against duplicates. Kafka does the rest.
Written by the RP Strativa Team. The payments events and partition counts are illustrative.
Leave a Reply