Every automation framework offers a retry option: if a test fails, run it again, and count it as passed if the second go works. Teams switch it on the day the pipeline starts going red for no good reason. Whether that is wise depends on arithmetic you can do on the back of an envelope. Let us do it with one suite from start to finish.

The suite

A team in Bhubaneswar has 400 end-to-end tests that gate every merge. Most are well behaved. But network blips, a slow staging server and the odd timing issue mean each test fails spuriously about 1 time in 100, for reasons unrelated to the product. A 1 percent failure rate sounds negligible.

The chance that all 400 tests pass in a given run is 0.99 multiplied by itself 400 times, which comes to about 1.8 percent. Put the other way, roughly 98 percent of runs have at least one red test. Nobody trusts a pipeline that is red 98 times in 100, so people start re-running the whole thing until it goes green, or merging anyway.

Turning on one retry

Now let the framework retry each failed test once. A spurious failure must now happen twice in a row, so if the two attempts are independent, the per-test spurious failure drops to 1 percent of 1 percent, which is 0.01 percent. Over 400 tests, the chance of at least one false red is about 3.9 percent. A red build now usually means something. This is the mercy side, and it is real: the retry has rescued a pipeline.

The same arithmetic, reversed

Here is the part that gets skipped. Suppose a different test fails not because of the environment but because of a real race condition in the product, and it triggers on 30 percent of runs. A customer would meet it three times in ten. With two retries allowed, three attempts in all, the test goes green unless all three fail: 0.3 cubed, which is 2.7 percent. So 97.3 percent of the time, a genuine product defect that bites nearly a third of users is reported as a pass.

The retry cannot tell the two situations apart. It sees only “failed, then passed”, and both a bad network and a buggy product produce that pattern. This is why I call it a cover-up risk, and why intermittent failures deserve to be investigated and not just absorbed.

Making retries honest

The answer is not to ban retries. It is to keep the information they would otherwise destroy. Some ways to do that:

  • Report “passed on retry” as a separate state, such as flaky, in the results. It should never look identical to a clean pass.
  • Count it. Track each test’s retry rate over time. A test that needed a retry on 20 of the last 100 runs is telling you something, and a dashboard that lists the worst ten is more useful than any single run.
  • Keep the logs, screenshots and traces of the failed first attempt, not only the final successful one. That evidence is the whole point.
  • Set a limit. One retry is defensible; three is hiding a problem. Apply retries only to the test layers and failure types where flakiness is expected, for example UI tests, not unit tests, which should never be flaky.
  • Agree a rule for repeat offenders: a test flaky past a threshold is quarantined and given an owner and a deadline, instead of being quietly retried forever.

What should be retried, exactly?

There is a finer choice. You can retry a whole test, or just the step that touched an unreliable dependency, such as the call to a staging payment sandbox. Narrow retries, with a short wait between attempts, mask less. You can also retry only on failures that look like infrastructure problems, a connection reset or a timeout, and refuse to retry on an assertion failure. If the page loaded fine and the total was wrong, running it again should not rescue it. An assertion failing then passing is a more serious sign than a timeout then passing.

One caution about independence. My arithmetic assumed the attempts are independent, and often they are not. If staging is down for ten minutes, the retry fails too. If a test leaves dirty data behind, the second attempt may fail for a new reason. Retries also assume the test is safe to repeat: re-running something that creates a real order may place two orders.

My position

I side with keeping a single, visible retry on top-level tests as a pragmatic tool, while treating the retry count as a quality metric in its own right. Retries buy you time to fix flakiness; they are a loan, and the interest is paid in missed real bugs. A team that turns retries on and never looks at the flaky report has not solved anything. It has just moved the red from the pipeline to the customer’s phone.

Figures and names in the examples are illustrative.

Leave a Reply

Trending