Code coverage tells you which lines your tests executed. It does not tell you whether the tests would notice if those lines were wrong. A test that calls a function and checks nothing about the result gives 100 percent coverage of that function and zero protection. Mutation testing measures the thing coverage cannot: whether your tests actually detect faults.

The idea

A mutation testing tool makes a small change to your code, such as flipping a comparison, changing a constant or deleting a statement. Each altered copy is called a mutant. The tool then runs your test suite against the mutant.

  • If at least one test fails, the mutant is killed. Good: your tests noticed the bug.
  • If every test still passes, the mutant survived. Your tests could not tell the broken code from the correct code.

The mutation score is the percentage of mutants killed. Each surviving mutant points to a specific, concrete weakness in the suite, which makes the technique unusually actionable.

A small worked example

Suppose a bank has a function that decides whether a customer qualifies for a senior citizen interest rate. Its logic is a single line: return true if age >= 60. A developer writes two tests: age 70 should return true, and age 30 should return false. Both pass. Line coverage is 100 percent and so is branch coverage, because both the true and false outcomes have been exercised.

Now a mutation tool generates five mutants:

MutantChangeAge 70 testAge 30 testResult
M1>= becomes >passpassSurvived
M2>= becomes <=failfailKilled
M360 becomes 61passpassSurvived
M4Always return truepassfailKilled
M5Always return falsefailpassKilled

Three of five killed: a mutation score of 60 percent, despite perfect coverage. The two survivors tell you exactly what is missing. M1 survives because nobody tested age 60 itself. M3 survives for the same reason. These are boundary bugs, the classic off-by-one, and the suite would have let either reach production.

Add one test, age 60 returns true, and both survivors die: under M1, 60 is not greater than 60, so it returns false; under M3, 60 is not at least 61. Add a second test, age 59 returns false, and you would also kill a mutant that changed 60 to 59. The tool has effectively taught boundary value analysis by pointing at the exact gap.

The kinds of mutations tools make

Tools apply a set of mutation operators, each modelling a common programming mistake:

  • Relational: swapping < and <=, or == and !=.
  • Arithmetic: swapping plus and minus, multiply and divide.
  • Logical: replacing AND with OR, negating a condition.
  • Return values: returning null, zero, an empty collection, or a constant true or false.
  • Statement removal: deleting a method call, such as the one that writes an audit log or sends a notification.

The last category is revealing. If deleting the call that saves a record to the database does not fail any test, your tests are checking what the function returns but not what it does.

The approach rests on two assumptions from the research literature: that programmers mostly write code that is nearly correct (the “competent programmer” hypothesis), and that tests which catch simple faults tend to catch more complex ones too (the “coupling effect”). Neither is a guarantee, but both have reasonable empirical support.

Equivalent mutants: the awkward part

Some mutants cannot be killed by any test, because the change does not alter behaviour. If a loop counts from 0 while the index is less than 10, and a mutant changes that to “not equal to 10”, the loop behaves identically for normal execution. Such equivalent mutants survive forever and drag the score down unfairly. Detecting them automatically is, in general, undecidable, so a person has to look and mark them. This is one reason teams rarely chase a 100 percent mutation score.

Cost, and how teams keep it manageable

Mutation testing is expensive. A codebase can produce thousands of mutants, and in principle each needs a test run. Modern tools reduce this substantially: they run only the tests that cover the mutated line, stop at the first failing test, and reuse compiled code where possible. Even so, running mutation analysis across a large codebase on every commit is usually impractical.

Practical patterns:

  • Run it on changed code only, as part of pull request checks.
  • Target critical modules: interest calculations, pricing, tax, eligibility rules, authorisation checks.
  • Run full analysis nightly or weekly rather than on every build.
  • Track the trend rather than an absolute target.

Widely used tools include PIT (often called PITest) for Java and other JVM languages, Stryker for JavaScript and TypeScript, C# and Scala, and mutmut for Python. Each integrates with standard build tools.

What it does not tell you

Mutation testing measures how well your tests check the code that exists. It says nothing about code that should exist and does not. If the requirement says senior citizens over 80 get an extra premium and nobody implemented it, there is no code to mutate and no mutant to survive. That gap belongs to requirement coverage and review, not to mutation analysis.

It also works mainly at the unit level. Mutating code and re-running a 40-minute UI suite for each mutant is rarely sensible. The technique is best seen as a quality check on developer-level tests, and a very good one: few other methods show so directly whether a green test run means anything.

Written by the RP Strativa Team. The senior citizen rate function is illustrative.

Leave a Reply

Trending