When someone first says “we test in production”, you probably wince, and fair enough. It sounds like carelessness. Let me walk you through why serious teams do it deliberately, and what keeps it from being reckless.
Here is the uncomfortable fact. Your test environment is a model of production, and every model leaves things out. Real traffic patterns, the actual size of the data, the strange browsers, the third-party systems that behave differently under live load, the customer who has 4,000 items in a wishlist. Some defects will only show up out there. So the question is not whether production gets tested, since users will do it anyway, but whether you do it in a controlled way or let customers do it for you by accident.
Canary release: the small first batch
The name comes from canaries once carried into coal mines as an early warning. You deploy the new version to a small slice of production, maybe one server out of twenty, or 2% of traffic, while everyone else stays on the old version. Then you watch. If error rates, latency or business numbers such as payment success look worse than the old version’s, you route traffic back. If they look fine, you widen: 2%, then 10%, then 50%, then all.
Notice what carries the weight here: the watching. A canary is only as good as the comparison you make. Define in advance which signals you will look at, how long you wait at each step, and what counts as bad. “Looks okay to me” at 11 p.m. is not a criterion. Many teams automate it: a tool compares the canary’s metrics against the stable version and rolls back on its own when a threshold is crossed. Tools such as Argo Rollouts and Flagger for Kubernetes do this, among others.
One catch. If the canary shares a database with the old version, and the new code changes how data is written, rolling back the code does not roll back the data. Plan schema changes so that old and new versions can both work against the same database for a while. That is a design habit, not a test technique, but a tester who asks about it early saves everyone a bad night.
Feature flags: separating deploy from release
A feature flag is a switch in configuration that turns a feature on or off without a new deployment. Code for the new coupon engine can be deployed to everyone but remain invisible until the flag is turned on for, say, internal staff first, then 1% of customers, then customers in one city, then everyone. Deployment becomes a technical event; release becomes a business decision you can undo in seconds.
Testers should care for two reasons. First, flags multiply your test paths. Ten independent flags are in principle 1,024 combinations. You cannot test them all, so test each flag on and off against the default state of the others, and test the specific combinations that are known to interact. Second, old flags rot. A flag that stayed on for a year is now just code with an extra if-statement and a hidden failure mode if someone switches it off. Treat removal as part of the feature’s definition of done.
Dark launches and shadow traffic
A dark launch runs new code against real production input without showing the result to users. Suppose you have rewritten the delivery-fee calculation. You send each real request to both the old and the new calculators, return the old answer to the customer, and log any differences. After a week you will have thousands of real comparisons, and every mismatch is either a bug in the new code or a case the old code handled by accident. This is the oracle problem solved by using the existing system as the reference.
Be careful with shadow calls that have side effects. A shadow copy of the payment call that really charges the card is a disaster. Shadow only read paths, or point the shadow at a sandbox that cannot change real state.
Rules that make it safe
- Limit the blast radius: small percentage, small region, internal users first.
- Have a rollback that you have actually rehearsed, and know how long it takes.
- Watch business metrics as well as technical ones. A release can keep error rates flat while checkout conversion quietly falls because a button moved.
- Keep synthetic test accounts clearly marked so test orders never reach real warehouses or finance reports.
None of this replaces pre-release testing. It adds a second line of defence for the things pre-release testing cannot see. Banks, payment systems and anything regulated need extra care about which customers see experimental behaviour, so involve compliance before running experiments on real accounts.
Figures and names in the examples are illustrative.
Leave a Reply