DORA, the DevOps Research and Assessment programme now run under Google Cloud, has studied software delivery performance across thousands of organisations since the mid-2010s. Its findings were popularised in the 2018 book Accelerate by Nicole Forsgren, Jez Humble and Gene Kim. The headline contribution was a small set of delivery metrics, usually called the four key metrics, and the finding that high-performing teams tend to do well on speed and stability together rather than trading one for the other.

Testers sometimes treat DORA as a DevOps concern that belongs to platform teams. That is a mistake. Two of the four metrics are direct measures of quality as users experience it, and the other two are heavily shaped by how testing is done.

The four metrics in plain terms

MetricWhat it measuresType
Deployment frequencyHow often the team deploys to productionThroughput
Lead time for changesTime from code committed to code running in productionThroughput
Change failure rateShare of deployments that cause a failure needing remediationStability
Time to restore serviceHow long it takes to recover from a failure in productionStability

DORA has refined names and definitions over the years; recent reports, for instance, describe the last one as failed deployment recovery time and have explored an additional rework measure. The underlying idea is unchanged: two metrics for how fast changes flow, two for how safely.

Change failure rate is a quality metric

Suppose a team made 40 production deployments in a month, and 6 of them led to a rollback, a hotfix or an incident. The change failure rate is 6 divided by 40, or 15%. That number captures what escaped all testing, reviews and checks before release, measured per change rather than per bug.

This framing has an advantage over a traditional “defects leaked” count. It is tied to deployments, which are concrete events everyone agrees happened, and it counts failures by their consequence (something had to be fixed in production) rather than by classification debates about whether something was really a bug. It also scales sensibly: a team deploying twice a day and one deploying twice a month can be compared on the same percentage.

The definition needs care. Teams must agree what counts as a failure: a rollback, a hotfix, a feature flag switched off, a P1 or P2 incident linked to the change. If the definition is loose, the metric becomes a negotiation. If failures are only counted when someone raises a formal incident, a culture of quietly patching will hide them.

Time to restore reflects testability and observability

When something breaks in production, recovery time depends on how quickly the problem is detected, diagnosed and fixed or reverted. Testing contributes to every stage. Good synthetic checks and post-deployment smoke tests detect breakage before customers report it. A fast, reliable regression suite means a fix can be verified and shipped in an hour rather than waiting for a two-day manual cycle. The ability to roll back safely, which is itself something that should be tested, can make recovery a matter of minutes.

Teams that need a full manual regression before any release, including hotfixes, often face an uncomfortable choice during an incident: ship an untested fix or leave customers broken for longer. Automated checks that can run in minutes remove that dilemma.

How testing shapes the throughput metrics

Lead time for changes includes every wait between commit and production. In many Indian services and product teams, the biggest single wait is a testing phase: code is done on Monday, deployed to QA on Wednesday, regression runs until the following Tuesday, release happens on Thursday. Testing work itself might be a fraction of that time; the rest is queueing for environments, builds and sign-offs.

Shorter lead times come from shifting checks earlier and making them faster: unit and API tests running on every pull request, ephemeral test environments created per branch, and risk-based selection of which slow end-to-end tests are needed for a given change. Deployment frequency follows naturally. If each release requires a week of manual regression, a team cannot deploy more than about once a week however good its developers are.

The trade-off that is not supposed to exist

A common intuition says that releasing more often must mean more failures. DORA’s research consistently found the opposite pattern among strong performers: frequent, small deployments with low failure rates and quick recovery. The mechanism is not mysterious. A deployment containing three small changes is easier to test, easier to diagnose when it fails, and easier to roll back than one containing three months of work.

For quality engineering this reframes the job. The goal is not to slow releases down until confidence is complete, which never happens. It is to build the checks, environments and monitoring that let a team release small changes often and catch problems fast.

Using DORA metrics without spoiling them

  • Measure all four together. Deployment frequency alone rewards reckless shipping. Change failure rate alone rewards never shipping. The pairing is the point.
  • Compare a team with itself over time. DORA publishes performance clusters from its surveys, which are useful context, but different products and regulatory settings legitimately differ. A core banking system and a marketing website should not chase the same deployment frequency.
  • Collect automatically where possible. Deployment events from the pipeline and incident links from the ticketing system are more reliable than spreadsheets filled in at month end.
  • Do not use them to rank individuals. They describe a delivery system, and they distort quickly if they become personal targets.

What DORA does not tell you

The metrics say nothing direct about whether the right thing was built, whether the product is usable or accessible, or how slow it feels. A team can have excellent DORA numbers while shipping features nobody wants, or while degrading performance by a few percent each release in ways that never trigger a failure. Quality engineering needs DORA plus customer-facing signals: support ticket trends, app store ratings, performance budgets and user research. DORA tells you whether delivery is healthy; it does not tell you whether the product is.

Written by the RP Strativa Team. Deployment counts and timelines in the examples are illustrative.

Leave a Reply

Trending