A flaky test is one that passes and fails on the same code without anything changing. Run the pipeline, it goes red; run it again, it goes green. The natural reaction is to hit “re-run” and move on. Do that often enough and the team stops trusting the suite, starts ignoring failures, and eventually misses a real bug that looked like just another flake.
Flakiness almost always has a cause, and the cause usually tells you something useful about the test, the environment or the application itself.
Timing: the most common culprit in UI tests
Web pages load in pieces. A button may exist in the page before it is clickable, or a table may render before its data arrives. A Selenium script that clicks immediately will work on a fast machine and fail on a slow CI agent.
The tempting fix is a fixed sleep of a few seconds. It hides the problem, slows the whole suite, and still fails on the day the page takes a little longer. The better fix is an explicit wait for a specific condition, such as Selenium’s WebDriverWait waiting until an element is clickable. Playwright and Cypress build automatic waiting into most actions, which is one reason teams moving to them often see flakiness drop, though neither removes the need to wait for the right thing, such as a network response rather than just an element appearing.
Shared state and test order
If test A creates a user called “testuser1” and test B expects that user not to exist, the result depends on which runs first. When suites run in parallel, order becomes unpredictable and these hidden dependencies surface as random failures.
The cure is isolation. Each test should create the data it needs, ideally with unique identifiers such as a timestamp or random suffix, and clean up after itself or run against a fresh dataset. A simple diagnostic is to run the suite in random order. Most frameworks support this, and tests that fail only in certain orders are leaking state.
Unstable environments and dependencies
Tests that call a shared QA environment, a third-party payment sandbox or an SMS gateway inherit all of their instability. If the sandbox is slow on a Monday morning, your tests fail on Monday mornings. Options include:
- stubbing or mocking external services in most tests, with a smaller set of contract or integration tests against the real thing;
- using containers, for example through Testcontainers, to start a clean database for each run;
- monitoring environment health separately, so a failure can be attributed quickly.
Time, randomness and concurrency in the code itself
Some flakiness comes from the application. A report that groups by “today” fails just after midnight UTC, which is 5:30 am in India and can catch teams off guard. A function that uses random numbers produces different output each run. A race condition between two threads appears only under certain timing. In these cases the flaky test has found a genuine bug, and treating it as noise would be the wrong call. Injecting a fixed clock or a seeded random generator into the code makes such behaviour testable and repeatable.
Fragile locators
A long absolute XPath that walks down from the top of the page, counting nested divs, breaks the moment a designer adds a banner. Locators tied to dedicated test attributes, such as a data-testid set to “submit-order”, or to accessible roles and labels, survive layout changes far better. Agreeing with developers to add stable test IDs is one of the cheapest improvements a QA team can make.
Managing flakiness as a team
- Measure it. Track which tests fail and then pass on retry. Many CI tools and reporting dashboards can show this.
- Quarantine, don’t ignore. Move known flaky tests to a separate job so they do not block releases, but give each one an owner and a deadline.
- Be careful with automatic retries. Retrying once can keep a pipeline usable, but if retries are silent, flakiness grows unnoticed.
- Fix the cause, then return the test. A test that stays in quarantine forever is just a deleted test with extra steps.
A reliable suite of 300 tests is worth far more than an unreliable suite of 3,000. Trust is the whole value of automation, and flakiness is what erodes it.
Written by the RP Strativa Team.
Leave a Reply