Picture a fictional team at a Pune-based travel booking startup, finishing a load test the evening before a festival sale. The report goes to the product head: average response time for the search API is 0.59 seconds, target is one second. Green tick. Everyone goes home.

The next afternoon the support inbox fills with messages saying search “hangs”. Nothing crashed. The test was not wrong about the average. The average was simply the wrong thing to look at, and the arithmetic behind it is worth seeing once.

Where 0.59 came from

Suppose the load test sent 100 search requests. Ninety-five came back in 0.2 seconds. Five took 8 seconds each, perhaps because they missed the cache and queued behind a slow database query. Add it up: 95 times 0.2 is 19 seconds, 5 times 8 is 40 seconds, total 59 seconds over 100 requests. Average: 0.59 seconds, comfortably under target.

Now think about who experienced what. Ninety-five customers saw a snappy site. Five waited eight seconds, long enough to give up and book elsewhere. Nobody experienced 0.59 seconds. The average describes a user who does not exist.

Percentiles say who is being left behind

A percentile answers a different question: below what time did a given share of requests finish? Sort all 100 timings from fastest to slowest. The p50, or median, is the middle one, here 0.2 seconds. The p95 is the value that 95% of requests were at or under, which in our data is still 0.2 seconds because the 95th request in sorted order is the last fast one. The p99 is 8 seconds. One extra slow request in the sample and the p95 would jump too.

So the honest report reads: median 0.2 s, p95 0.2 s, p99 8 s. That single line says there is a small, serious tail. The team would have caught it the night before. Note how thin the line between p95 and p99 can be in small samples: with only 100 requests, “p99” is essentially the second-slowest request. Percentiles near the top need many samples to be stable, which is a reason to run long enough and to send thousands of requests, not hundreds.

Different tools also compute percentiles slightly differently when the value falls between two samples (some interpolate, some pick the nearest rank). The difference is small on big samples, but do not compare a p95 from one tool with a p95 from another to the millisecond.

Why a rare slow call becomes a common slow page

Back to our startup. The search results page is not one call. Say it makes 20 backend calls to render: availability, prices, offers, user profile, and so on. Suppose each call is slow just 1% of the time. The chance that at least one of the 20 is slow is one minus 0.99 raised to the 20th power, which works out to about 18%. Nearly one page in five is dragged down by some slow call, even though each individual service looks fine at p99.

This is the reason experienced performance engineers watch the tail more than the middle. The more services a page touches, the more the tail of each service shows up in the user’s experience. This assumes the calls are independent, which real systems often are not (a shared slow database makes slowness correlated), but the direction of the effect holds.

Setting targets that mean something

A service-level objective phrased as “average under one second” is weak. Phrase it as “95% of search requests complete within 800 ms and 99% within 2 seconds, measured over a rolling day”. It states the percentile, the threshold and the window. Teams that publish such objectives can then argue sensibly about whether a release meets them.

A few practical points when you set up your own measurements:

  • Report the median, p95 and p99 together, plus the maximum. The shape tells you more than any one figure.
  • Never average percentiles across machines or time slices. The average of two p95 values is not the p95 of the combined data. Aggregate the raw timings or use histograms, which most monitoring systems support.
  • Look at the distribution as a histogram at least once. Two humps, one fast and one slow, usually mean two code paths such as cache hit and cache miss, and that is where to investigate.
  • Exclude warm-up requests if cold start is not what you are testing, but keep a separate cold-start measurement because real users hit it too.

The average still has a use. Multiply it by request count and you get total work, which is handy for capacity sums. For describing what users feel, leave it out of the headline.

Figures and names in the examples are illustrative.

Leave a Reply

Trending