There is a table that circulates in every QA training deck. Smoke testing: wide and shallow, done on a new build, by developers or testers, scripted. Sanity testing: narrow and deep, done after a small fix, unscripted. Regression testing: everything, to check old features still work. It looks tidy, and people repeat it as though it were settled law.
It is not settled. Pick up three books and two glossaries and you will find the terms swapped, merged, or contradicted. The table is a useful memory aid and a poor description of how real teams talk. Let us take the common beliefs apart one at a time.
Belief one: these are three official, distinct test types
Regression testing has a stable meaning: checking that a change did not break something that used to work. That much is consistent everywhere. Smoke testing is also reasonably stable: a quick set of checks that tells you whether a build is healthy enough to be worth testing further. The name comes from hardware, where you switch on a new board and see whether smoke comes out.
“Sanity testing” is the slippery one. Some sources treat it as a narrow check after a bug fix or minor change. Others treat it as a rough synonym for smoke testing. Others use it for a quick reasonableness check of the output, as in “does this total look sane?”. Standards bodies do not agree strongly enough for you to win an argument by quoting one. So when a manager says “do a sanity pass”, you have learned nothing until you ask what they want covered and how long you have.
Belief two: smoke tests are manual, regression tests are automated
Backwards, if anything. A smoke suite is the best candidate for automation because it runs on every build, many times a day. Ten or fifteen checks: the application starts, you can log in, the main page loads, a core transaction such as placing an order in a test account completes, the database connection works. If this suite runs in the pipeline right after deployment to a test environment and fails, nobody wastes the afternoon testing a build that cannot even log in.
Regression testing is a mix. Some of it is automated, some is manual, and a good part is chosen per release based on what changed. Calling regression “the automated suite” confuses a tool with a purpose.
Belief three: smoke happens once per build, sanity once per fix, regression once per release
Frequency follows cost and risk, not vocabulary. Consider a fictional payments team with a pipeline that deploys to a test environment after each merge. A smoke suite of about 3 minutes runs on every deployment. A larger “core regression” suite of about 40 minutes runs nightly. A full regression of several hours runs before a production release. Same underlying idea, three layers by time budget.
Inside that setup, where does the sanity check sit? It is what a tester does by hand when a developer says “fixed the refund rounding bug, build 214 is ready”: try the refund scenario, try one or two neighbours such as partial refund and refund on a discounted order, and decide whether the fix is worth a deeper look. Call it sanity, call it a fix verification, call it confirmation testing. The ISTQB term for re-running the failed test after a fix is confirmation testing (also called retesting), and it is distinct from regression because it targets the defect itself, while regression looks for side effects elsewhere.
Belief four: a passing smoke test means the build is good
A smoke test is a gate in one direction only. A failure means stop. A pass means continue, nothing more. Teams occasionally forget this and quote “smoke passed” as a quality statement in a release email. It tells you the build is not catastrophically broken. It says nothing about the 90% of features it never touched.
The reverse risk is a smoke suite that grows until it is a slow regression suite with a friendly name. Once it takes 25 minutes, developers stop waiting for it, and the gate no longer gates. Set a time limit and hold it. If a new check wants in, something else goes out or moves to the nightly run.
What to say instead of quoting the table
Replace vocabulary with three questions, and agree answers with your team in writing:
- What decision does this run support? (Is the build testable? Is the fix real? Is the release safe?)
- How long may it take, and who waits for it?
- What triggers it, and what happens on failure? (Block the merge, page someone, or just report?)
Write down your team’s meaning of “smoke” and “sanity” in a short wiki page. Five lines. New joiners from other companies arrive with different definitions, and a shared page ends the confusion faster than a debate. When you move to another organisation, ask for theirs in the first week instead of assuming yours transfers.
Figures and names in the examples are illustrative.
Leave a Reply