An order service saves a new order to its database and then publishes an “OrderPlaced” event to Kafka. The warehouse service listens for that event to start picking, the notification service to send an SMS, the loyalty service to add points. It works in testing. In production, a few orders a week get stuck: the customer paid, the order exists in the database, but the warehouse never heard about it. Somewhere between the database commit and the publish, the process died, a network call timed out, or the broker was briefly unavailable.
This is the dual-write problem. Whenever code writes to two independent systems, there is no single moment at which both writes become true together. One can succeed while the other fails.
Why the obvious fixes fail
Publish first, then save. Now the failure flips: the warehouse may start picking an order that never got saved, because the database insert failed a validation check after the event went out. That is worse.
Publish inside the transaction, before commit. The event goes out, then the commit fails or is rolled back. Consumers have acted on an order that does not exist. Also, the database transaction is now held open while waiting on a network call to the broker.
Retry the publish on failure. Helps with brief broker hiccups, but not with the process crashing between commit and publish. The retry logic dies with the process.
Distributed transactions. Two-phase commit across a database and a message broker exists in theory (XA), but most modern brokers and cloud services do not support it, and where they do, it adds latency and operational fragility. Most teams rule it out.
The outbox: one write, then a relay
The transactional outbox removes the dual write by turning it into a single write. Alongside the business tables, the service has an outbox table. When an order is placed, the same database transaction inserts the order row and inserts an outbox row describing the event: an id, the event type, the aggregate it belongs to (the order id), the payload, and a creation time. Either both rows commit or neither does. The database guarantees that, because it is one local transaction.
A separate component, the relay, reads outbox rows that have not yet been published, publishes them to the broker, and marks them as sent (or deletes them). If the relay crashes after publishing but before marking, it will publish the same event again when it restarts. If the broker is down, events wait safely in the table until it returns. Nothing is lost; at worst something is delayed or duplicated.
Two ways to build the relay
Polling. A background job queries the outbox for unsent rows every second or so, in creation order, publishes them and updates their status. It is easy to build and understand. With several relay instances, use row locks with skip-locked semantics so they do not publish the same rows at once, or run a single active relay with a standby. Polling adds a little database load and up to one polling interval of latency.
Log tailing with change data capture. Instead of querying, a CDC tool reads the database’s own replication log (the write-ahead log in PostgreSQL, the binlog in MySQL) and streams every new outbox row to the broker. Debezium is the best-known open-source tool for this and ships an outbox event router specifically for the pattern. Latency is low and there is no polling load, at the cost of running and monitoring another piece of infrastructure.
What the pattern guarantees, and what it does not
The outbox gives you at-least-once publication: every committed business change produces its event eventually, possibly more than once. It does not give exactly-once delivery. So consumers must be idempotent. The usual approach is to include a unique event id in every message and have each consumer record the ids it has processed, ideally in the same transaction as its own state change, skipping any it has already seen. A loyalty service that adds points twice for one order is the bug the outbox would otherwise introduce.
Ordering needs thought. If events for the same order (placed, paid, shipped) must be processed in sequence, publish them with the order id as the message key so Kafka places them in the same partition, and have the relay publish in commit order for each key. A polling relay that publishes in parallel across rows can reorder events unless it is careful.
The outbox table also needs housekeeping. A busy service can write lakhs of outbox rows a day. Delete or archive published rows on a schedule, or the table becomes a slow, bloated liability. In PostgreSQL, a table with constant inserts and deletes is a classic candidate for careful autovacuum tuning.
The mirror image: the inbox
The consumer side has its own dual-write problem. A consumer reads a message, updates its database, and acknowledges the message to the broker. If it crashes after updating but before acknowledging, the message comes back. An inbox table, holding ids of processed messages written in the same transaction as the business update, lets the consumer recognise and skip the redelivery. Outbox on the producer, inbox on the consumer, and idempotent handling everywhere: together they give effectively-once processing on top of at-least-once infrastructure.
When you do not need it
If losing an occasional event is acceptable, for example an analytics click event, a direct publish with retries is simpler. If the downstream system can periodically reconcile against your database, such as a nightly job that finds orders without a shipment, that reconciliation may be enough. The outbox earns its keep where a missing event means real harm: a paid order never shipped, a refund never processed, a KYC status that never reaches the lending system.
Written by the RP Strativa Team. The order, warehouse and loyalty services are illustrative.
Leave a Reply