A typical voice process handles thousands of calls a month and a quality team can listen to only a small slice of them. So every quality score you see on a dashboard is an estimate built from a sample. The question that decides whether the estimate means anything is simple: how many interactions did you audit, and how were they picked? This article walks through the statistics in plain terms, using small numbers you can check yourself.

Two different questions hiding inside “sample size”

Quality teams usually need samples for two separate purposes, and mixing them up causes most of the confusion.

  • Process-level estimation: “What is the accuracy of the whole process this month?” This is a question about one large population, and statistics answers it well with a few hundred audits.
  • Agent-level judgement: “Is Priya performing better than Arjun?” This is a question about many small populations, one per agent, and each needs its own adequate sample. This is where most floors fall short.

A third purpose, finding problems you did not know existed, is not really an estimation question at all. It is better served by targeted sampling, which we come to later.

Confidence level and margin of error, without the jargon

Suppose you audit a random set of calls and find 90% of them error-free. The true accuracy across all calls is probably close to 90%, but not exactly. The margin of error is how far off you might reasonably be, say plus or minus 3 percentage points. The confidence level is how often this method of sampling would produce a range that contains the true figure. A 95% confidence level means that if you repeated the whole exercise many times, about 95 out of 100 such ranges would capture the real accuracy.

For a proportion such as “percentage of calls without errors”, the standard formula for the sample needed is n = z² × p × (1 − p) ÷ e², where z is 1.96 for 95% confidence, p is the expected proportion and e is the margin of error you can tolerate. When you have no idea what p will be, you use 0.5, because that gives the largest, safest sample.

Working through the numbers

Take a process with 20,000 calls a month. With 95% confidence, a margin of 5 points and p of 0.5, the formula gives 1.96 × 1.96 × 0.25 ÷ 0.0025, which is about 384.2, so roughly 385 calls.

Because the population is finite, you can apply a correction: divide that figure by 1 + (384.2 − 1) ÷ 20,000. The answer drops only slightly, to about 377. The correction matters more for small processes. For a back-office queue with 2,000 transactions a month, the corrected figure is about 323; for 500 transactions it is about 218. Once the population is large, the sample you need barely changes, which surprises people: auditing 385 calls gives the same precision whether the month had 20,000 calls or 2 lakh.

Monthly volumeSample for ±5 points at 95% confidence
500about 218
2,000about 323
20,000about 377
Very largeabout 385

Tightening the margin is expensive. Moving from ±5 to ±3 points with p at 0.5 pushes the uncapped sample to about 1,068. Precision improves with the square root of sample size, so halving the margin needs roughly four times the audits. If your accuracy is consistently high, say around 90%, using p = 0.9 brings the requirement down: for ±3 points the formula gives about 385 instead of 1,068. That is legitimate, but only if the 90% assumption is based on recent data.

Why five calls per agent cannot rank anyone

Many floors audit four to eight calls per agent per month and then publish agent rankings to one decimal place. Run the arithmetic. If an agent’s true accuracy is 90% and you audit five calls, the margin of error using the same formula is about ±26 points. Even ten calls gives about ±19 points, and thirty calls gives about ±11. At those small sizes the normal approximation behind the formula is itself shaky, so the real uncertainty is, if anything, messier.

In practical terms, an agent who scored 80% on five audits and one who scored 100% may not be different at all. One bad call swings a five-call score by 20 points. This is why agents often feel their QA scores are a lottery, and they are not entirely wrong. Sensible responses include:

  • Judging agents on rolling windows, such as the last 60 or 90 days of audits, rather than a single month.
  • Showing bands (meets, below, well below) instead of precise ranks when samples are small.
  • Never basing a formal performance action on a handful of audits; adding more targeted audits first.
  • Using fatal-error counts and behaviour patterns, which are clearer signals than a single percentage.

Random, stratified and targeted samples

The formulas above assume the calls were picked at random. In reality, auditors often pick whatever is convenient: short calls that are quick to review, calls from the morning shift, or calls the team leader flagged. Each of these introduces bias, and no amount of extra volume fixes a biased sample.

Stratified random sampling is a better default. You split calls into groups that behave differently, such as call type, shift, tenure band or language, and draw randomly within each group. Night shift and new joiners then get represented in proportion, or deliberately over-represented if they carry more risk.

Targeted sampling deliberately picks risky interactions: very long or very short calls, repeat callers within 48 hours, cancellations, refunds above a threshold, or calls flagged by speech analytics. Targeted audits are excellent for finding problems and coaching, but their scores must not be blended into the headline accuracy figure. A targeted sample is designed to find errors, so it will show worse numbers than reality. Keep two reports: a random sample for measurement and a targeted sample for discovery.

When the client contract decides for you

Many outsourcing contracts specify audit volumes, for example a fixed number per agent per week or a percentage of transactions. These are commercial agreements, not statistical ones, and they vary widely by client and industry. A fixed percentage, say 2% of volume, gives far more audits than needed in a large process and too few in a small one. If the contract’s figure gives a margin of error so wide that monthly movements are noise, it is worth raising this with the client and proposing a statistically grounded design, perhaps combined with a quarterly review of the per-agent rule.

Finally, sample size says nothing about whether auditors score consistently. Two evaluators listening to the same 385 calls can still disagree. That is the job of calibration, which sits alongside sampling as the second pillar of a quality score anyone can trust.

Written by the RP Strativa Team. Figures and names in the examples are illustrative.

Leave a Reply

Trending