Hand the same recorded call to five quality analysts and ask each to score it on the process’s quality form. In many contact centres the scores will come back something like 88, 92, 76, 90 and 84. Same call, same form, a 16-point spread. Nothing about the agent changed. The difference is entirely in the people holding the form.
Calibration is the routine that shrinks that spread. It is one of the least glamorous activities in a quality team and one of the most important, because without it every quality score carries a hidden “depends who audited you” footnote.
What calibration actually is
Calibration is a structured exercise in which everyone who evaluates interactions scores the same sample independently, compares the results, discusses the differences and agrees on how each parameter should be interpreted. The outcome is not just a “correct” score for those few calls. It is a shared understanding of the rules, written down so it applies to the thousands of calls nobody discussed.
Who attends varies by organisation. A typical session includes quality analysts, team leaders (who coach to these scores), sometimes trainers and subject matter experts, and in many outsourcing setups the client’s own quality team. Including the client is valuable, because a gap between vendor and client interpretation is one of the commonest reasons a process looks healthy internally while failing the client’s audits.
Running a session, step by step
- Choose the calls deliberately. Two to four interactions is usually enough for an hour. Pick ones that test grey areas: a partially resolved query, a borderline compliance statement, a customer who was rude. Easy, perfect calls teach nothing.
- Score independently first. Everyone submits scores before any discussion. If people score together in the room, the most senior voice anchors everyone else and you never learn the real variance.
- Compare at parameter level, not just totals. Two auditors can both reach 86% by marking down entirely different things. The useful conversation is about each line of the form.
- Discuss the disagreements and settle the interpretation. The facilitator, usually a quality lead, drives to a decision, with reasons tied to the client’s policy or the form’s definitions.
- Record the decision. Every resolved grey area goes into a calibration log or the form’s guidelines, so next month’s new auditor inherits the decision.
Measuring how far apart auditors are
Go back to the five scores of 88, 92, 76, 90 and 84, and suppose the agreed “master” score after discussion is 86. Each auditor’s deviation is then +2, +6, −10, +4 and −2. Many quality teams set a tolerance band, often something like ±3 or ±5 points from the master score; the exact figure is a local choice, not an industry standard. With a ±5 band, auditors B (+6) and C (−10) are outside tolerance.
Total-score variance can hide problems, so stronger programmes also track parameter agreement: on a 12-parameter form, if auditor C matched the master marking on 9 parameters, their agreement is 75%. A pattern of low agreement on the same parameter across auditors tells you the form is ambiguous. Low agreement for one auditor across parameters tells you that person needs coaching.
Over months, tracking each auditor’s average deviation reveals habits. Some auditors are consistently strict, some consistently lenient. A strict auditor assigned to one team and a lenient one to another will produce a ranking of team leaders that says more about the auditors than the teams.
Where disagreement really comes from
Most calibration gaps fall into a few recognisable buckets:
- Vague parameter wording. “Showed empathy” or “used effective probing” invite personal taste. Rewriting them as observable behaviours, such as “acknowledged the customer’s stated problem before giving a solution”, removes much of the debate.
- Unclear partial credit. If the agent verified two of three security questions, is that a pass, a partial or a fatal error? If the form does not say, auditors will each decide.
- Outdated knowledge. An auditor who missed last week’s policy update marks the agent down for following the new process correctly.
- Outcome bias. If the customer was happy at the end, auditors tend to forgive process misses. If the customer was angry, they find more faults.
- Halo effect. Knowing the agent is a top performer quietly nudges scores upward. Blind scoring, where the agent’s name is hidden during calibration, helps.
The connection to QA disputes
When an agent says an evaluation was unfair, the instinct is to treat it as a personal complaint. More often it is a calibration signal. A healthy dispute process works like this: the agent raises the dispute within a fixed window, a different evaluator or the quality lead re-scores the call blind, and the decision references the written guideline. If the guideline did not cover the situation, the outcome is a new calibration entry, not just a verdict on one call.
Tracking dispute outcomes is revealing. If a large share of disputes against one auditor are upheld, that auditor is out of calibration. If disputes cluster around one parameter, the form needs rework. If almost no disputes are ever upheld, agents may have stopped bothering, which is its own warning about trust.
How often, and what good looks like
Weekly sessions are common in new or volatile processes; fortnightly or monthly is typical in stable ones. Extra sessions are worth running whenever a policy changes, a new auditor joins, or the client’s audit results drift away from internal scores.
Calibration is working when deviation from the master score shrinks over time, parameter agreement rises, internal and client scores move together, and agents stop asking “who audited me?” before reading their feedback. It is failing when sessions become a formality where people adjust their scores to match the lead before submitting, which produces perfect agreement on paper and none on the floor.
Written by the RP Strativa Team. Scores and examples are illustrative.
Leave a Reply