Field notes. Opinions marked as opinions.

A dashboard says 100% line coverage. Management is happy. Do not be. That number says every line ran at least once during the tests. It does not say anything checked the result, and it does not say the lines ran under the conditions that matter. Coverage is a measure of what you have not touched. Used in that direction it is useful. Used as a score of quality it is nearly worthless.

One small function, three answers

Take a function that decides whether an order ships free. The rule: free shipping if the order total is at least ₹500 and the customer is a member. The body is one decision with two conditions joined by AND, and one statement that returns “free” when the decision is true, and one that returns “paid” otherwise. Call the conditions A (total at least 500) and B (is member).

Statement coverage asks: did every statement execute? One test with A true and B true runs the “free” statement. A second test with A false runs the “paid” statement. Two tests, 100%. If the code were written so the “paid” result were the default after the decision, a single test might touch every line except it, but you get the idea: the bar is low.

Branch (decision) coverage asks: did every decision go both ways, true and false? Our two tests already achieve that. The decision was true once and false once. Still two tests, still satisfied. And notice what we never did: we never tried a non-member with a big order. If the developer typed OR instead of AND, both tests still pass. Branch coverage says 100%, and the bug ships.

Condition coverage asks that each individual condition takes both true and false values. Test one: A true, B false. Test two: A false, B true. Both conditions have been true and false, but the decision as a whole was false both times. So you can satisfy condition coverage without ever exercising the “free” path. Funny, and a known weakness.

MC/DC: each condition must matter

Modified condition/decision coverage fixes this. For every condition you must show a pair of tests where only that condition changes and the decision outcome changes with it. For our AND, three tests are enough: both true (outcome true), A true with B false (outcome false), A false with B true (outcome false). Compare the first and second: only B changed, outcome flipped, so B is shown to matter. Compare the first and third: A alone flips it. Now an OR typo fails, because the second test would return true instead of false.

The general rule is that a decision with n conditions needs at least n + 1 tests for MC/DC, rather than the 2 to the power n that exhaustive testing of every combination would demand. With 6 conditions that is 7 tests instead of 64. That linear growth is why aviation accepted it: the DO-178C software standard requires MC/DC for the highest criticality level of airborne software. For an e-commerce checkout you almost certainly do not need to go that far, and I would say so plainly: my opinion is that applying MC/DC across an ordinary business application is a waste of effort, while applying the idea to the five or six decisions that move money is smart.

What your tool is actually counting

Check this before you quote a figure. Popular tools such as JaCoCo for Java report line, instruction and branch coverage; the “branch” there counts the outcomes of conditional jumps in the compiled bytecode, which is close to, but not identical with, the source-level notion. Python’s coverage.py can measure branches only if you switch that on. Most mainstream tools do not report MC/DC at all. So when a team says “we have 85% coverage”, the first question is: 85% of what?

Also, short-circuit evaluation muddies things. In most languages AND stops as soon as A is false and never looks at B. A test with A false therefore says nothing about B, and a coverage tool may report the second condition as unevaluated. That is accurate, not a tool bug.

What I would do with coverage numbers

  • Open the report on new or changed code and read the red lines. Each uncovered branch is a question: is this error handling, dead code, or a case nobody thought of?
  • Prefer branch coverage over line coverage as the number you track. It costs nothing extra and exposes the untested else.
  • Never set a company-wide target of 90% and then reward hitting it. People will write tests with no assertions to reach the figure. A test that calls the code and asserts nothing counts exactly the same to the tool.

The deeper check on test quality is whether the tests would fail if the code were wrong. Coverage cannot tell you that. Mutation testing, which plants small bugs and sees if tests notice, is the more honest instrument, and it is a separate subject.

One last practical point: when a decision has more than three conditions, ask whether the code should have been split first. If you need a table of 8 tests to cover one if-statement, the if-statement is the problem.

Figures and names in the examples are illustrative.

Leave a Reply

Trending