Most testing instincts come from synchronous systems. You send a request, the system does the work, a response comes back, you check it. In an event-driven system that last step is missing. A food delivery app places an order and immediately returns “order received”. Behind the scenes an event goes onto a broker such as Kafka or RabbitMQ; a payment service picks it up, then a restaurant service, then a delivery-assignment service, then a notification service. Each reacts on its own schedule. The response the test receives says almost nothing about whether the order actually worked.
Testing these systems well means accepting that outcomes are delayed, may arrive more than once, may arrive out of order, and may fail silently. Each of those properties calls for a specific technique.
Delay: wait for a condition, not for a duration
The tempting fix for delay is a fixed pause: place the order, wait five seconds, check the status. This is wrong in both directions. On a fast day the test wastes time; on a slow day, when the consumer is lagging, it fails even though the system is fine. Multiply by hundreds of tests and you get a suite that is both slow and flaky.
The better pattern is polling with a timeout: check the condition repeatedly at a short interval until it becomes true or a deadline passes. Libraries exist for this in most ecosystems, such as Awaitility in Java, and Playwright’s and Cypress’s assertions retry automatically. The test then finishes as soon as the order reaches “assigned”, and fails with a clear message (“order still PAYMENT_PENDING after 30 seconds”) only when something is genuinely stuck.
The timeout itself is a product question worth asking. If the business expects a delivery partner to be assigned within 60 seconds, a test that waits up to 60 seconds is checking a real requirement. If nobody knows how long it should take, that is a gap in the requirements, not just in the test.
Duplicates: test idempotency deliberately
Most message systems in practice guarantee at-least-once delivery: a message will arrive, but it may arrive twice, for example if a consumer processes it and crashes before acknowledging. Exactly-once processing is possible in some configurations but requires careful design end to end, and many teams do not have it.
So consumers must be idempotent: processing the same event twice should have the same effect as processing it once. A payment-captured event delivered twice must not charge the customer twice or credit the restaurant twice. This is easy to state and easy to get wrong, and it rarely shows up in normal test runs because duplicates are rare under light load.
Test it on purpose. Publish the same event, with the same event ID, twice, and verify the end state. Restart a consumer in the middle of processing and check what happens on redelivery. These are some of the highest-value tests in an event-driven system because the failure they catch, double-charging, is both likely and expensive.
Ordering: know what is guaranteed
Kafka guarantees order only within a partition. If all events for one order use the order ID as the key, they land in the same partition and stay in sequence. Events for different orders, or events keyed differently, have no ordering guarantee relative to each other. RabbitMQ preserves order within a single queue in simple cases but redelivery and multiple consumers can change what a consumer sees.
Testers should find out what ordering the design relies on, then challenge it. What happens if “order cancelled” arrives before “order created” because they took different paths? What if a “price updated” event from an hour ago arrives after a newer one? Robust consumers handle this with version numbers or timestamps and ignore stale updates. Tests can publish events out of order directly to the broker to see whether the consumer copes.
Silent failure: test the unhappy path of messaging
In a synchronous API a failure returns an error to the caller. In an event-driven flow, a consumer that cannot process a message might log an error and move on, retry forever and block everything behind it, or move the message to a dead-letter queue. The customer just sees an order that never progresses.
- Publish a malformed or schema-incompatible event and check it lands in the dead-letter queue with enough context to investigate, rather than crashing or blocking the consumer.
- Make a downstream dependency unavailable and verify retries back off sensibly and the system recovers when the dependency returns.
- Check that someone would notice: an alert on dead-letter queue depth or consumer lag is part of the system’s correctness, not an operational extra.
Where to test what
Full end-to-end tests through every service are slow and fragile, and when they fail it is hard to tell which hop broke. A more effective spread:
- Consumer component tests: run one service against a real broker in a container (Testcontainers is the usual tool), publish events to its input topic, and assert on its output events and database. This covers idempotency, ordering and error handling cheaply.
- Schema and contract checks: a schema registry with compatibility rules (for Avro, Protobuf or JSON Schema) or consumer-driven contract tests stop a producer from changing an event shape that consumers depend on.
- A few end-to-end journeys: one or two flows such as “order placed to delivery assigned” confirm the wiring, using polling and correlation IDs to trace the event across services.
Observability makes these tests possible
An end-to-end test of an asynchronous flow needs a way to see where an event went. A correlation ID carried in every event header lets a test, or a person, find every log line and span related to one order across six services. Without it, a failed test produces “order not delivered” and nothing else. Asking for correlation IDs, trace propagation and queryable consumer lag during design is one of the most useful things a tester can do for an event-driven product, because it serves both testing and production support.
Written by the RP Strativa Team. The food delivery flow in the examples is illustrative.
Leave a Reply