Notes from watching pipelines go wrong. These are opinions, flagged as such.

A quality gate is an automated decision point in a pipeline: if the condition holds, the change moves on; if not, it stops. That is all. The difficulty is not the tooling. Jenkins, GitHub Actions, GitLab CI and SonarQube (which uses the actual term “quality gate”) all let you set conditions in an afternoon. The difficulty is choosing conditions that people will respect for two years.

A gate has to be trusted before it can be strict

Rule one. A check may block only if a failure almost always means a real problem. If it fails for no good reason one run in ten, developers learn to click “re-run”, then learn to ask for an override, then learn to ignore it. Once the team believes that red is noise, even true failures get waved through. So flaky tests never belong in a blocking gate. Quarantine them, track them, fix them, and let them rejoin only when they are stable. Opinion: one untrustworthy blocking check does more damage than having no check.

Rule two. Every blocker should be explainable in one sentence to a developer at 6 p.m. on a Friday: “This fails because the new code lets an unauthenticated user read other people’s orders.” If the sentence is “because the number is below 80”, you probably have a target, not a gate.

Candidates for blocking

  • The build compiles and the unit and component tests pass. Fast, deterministic, cheap.
  • Contract checks against the services you talk to. A broken contract is a broken deployment waiting to happen.
  • No new critical or high-severity vulnerabilities in your code or dependencies, scoped to what this change introduced, with a known exceptions list.
  • No secrets (keys, passwords, tokens) committed to the repository.
  • A short smoke test after deployment to the next environment: can the app start, log in and complete its main transaction?
  • Database migration scripts apply cleanly on a copy and can be reversed or are explicitly marked as one-way.

Notice what these share: each is objective, quick, and tied to a specific harm.

Candidates for a warning only

Style and lint findings that are not bugs. A small dip in a performance benchmark within normal variation. Code complexity growing. Documentation missing. A slightly lower coverage figure. These deserve visibility, a comment on the pull request or a trend chart, and a human who looks at them periodically. They do not deserve to stop a release, because the cost of a false stop is real and the harm per instance is small.

Coverage deserves a special note. A gate that says “overall coverage must be 80 percent” invites people to write assertion-free tests that execute lines, which raises the number and proves nothing. A more useful variant is coverage on new or changed code, and better still, combine it with a review of whether those tests assert anything. Opinion: use coverage to find untested areas, not to certify tested ones.

Tighten gradually: the ratchet

If a legacy codebase has 4,000 lint warnings and a vulnerability backlog, switching on a strict gate today means nobody can ship. Instead, gate on “no new issues”. The existing ones are recorded as the baseline and cleaned over time, while every new change must leave things no worse. This is sometimes called a ratchet, and it works because it asks for something every developer can actually do today.

Different stages, different gates

Think in stages, and put the quickest checks first so feedback arrives in minutes.

  1. On commit or pull request: compile, unit tests, static analysis, secret scan, dependency check. Target a few minutes.
  2. After merge, in a test environment: API and integration tests, contract verification, smoke tests.
  3. Before production: selected end-to-end flows, performance check against a baseline, and, for regulated services, evidence collected for the change record.
  4. After production deployment: health checks, a canary period with automatic rollback if error rates or latency cross thresholds.

For something like a bank’s payment service, stage three is where it gets stricter: reconciliation checks, security test results, approvals from named roles, and a rollback that has actually been rehearsed. RBI-regulated organisations often need audit evidence of these steps, and a pipeline that records them automatically is far easier to defend than a spreadsheet filled in afterwards. Whatever your sector, check the specific rules that apply to you rather than assuming.

Overrides, because they will be needed

Sooner or later there will be a critical fix that must go out while one gate is red for a reason unrelated to it. Design for that. An override should need a named approver, a written reason and an expiry, and it should appear on a dashboard. Count overrides per month. A rising count is information: either a gate is too strict or the team is under pressure it has not admitted. A gate that cannot be overridden gets bypassed by hacking the pipeline itself, which is worse.

Review the gates quarterly. Ask of each blocker how many real defects it stopped, and how many false stops it caused. Remove or demote anything that cannot justify its place, because a short list of respected gates beats a long list of resented ones.

Leave a Reply

Trending