Field notes on security metrics. I will mark opinions as opinions and keep this short.

The definitions first

MTTD, mean time to detect: the average gap between an incident starting and your team noticing. MTTR: usually mean time to respond or to recover, and here is the first problem. Some teams measure from detection to containment, some to full recovery, and some even from the start of the attack. If two companies quote MTTR, check which clock they mean before comparing anything. Some also split it further: time to acknowledge, time to contain, time to eradicate.

Another early catch. “Start of the incident” is often known only after the investigation. The attacker’s first foothold may be weeks before the first alert. So MTTD is usually calculated after the fact, and its quality depends on how thorough the investigation was.

Why the mean misleads

Take eight incidents with detection times in hours: 1, 2, 2, 3, 4, 5, 6 and 700. The last one was a quiet compromise found a month later. The mean is 723 divided by 8, about 90 hours. The median is 3.5 hours. Which describes your team? Both, in different ways. The mean says something is badly wrong somewhere; the median says typical alerts are handled fast. Reporting only one hides the other.

My opinion: report the median and a high percentile (the 90th, say) side by side, and list the worst few cases by name. Averages of skewed data are fine for spreadsheets and bad for decisions.

Metrics that can be gamed

Goodhart’s law is the principle that when a measure becomes a target, it stops being a good measure. Security is full of examples:

  • Reward a short time-to-close and analysts close tickets early or mark real issues as benign to stop the clock.
  • Reward “alerts handled” and the team tunes detection to produce easy, low-value alerts.
  • Reward “phishing click rate under 3 per cent” and the awareness team makes the tests easier.
  • Count “vulnerabilities fixed” and teams fix hundreds of trivial ones while a few internet-facing, exploitable ones wait.

The defence is to pair every speed metric with a quality check. Time to close alongside reopen rate. Alerts handled alongside the share confirmed as true positives. Sample closed tickets and have a second person re-review a few each week.

Numbers I actually like

  1. Detection coverage against known techniques. Map your detections to MITRE ATT&CK and see where the gaps are. It is crude, since having a rule is not the same as it working, but it points at blind spots.
  2. Asset coverage. What fraction of your endpoints and servers actually send logs and run the agent? If you believe 98 per cent and the real figure is 80, every other metric is based on a fiction. Unknown assets are the biggest hole in most dashboards.
  3. Time to remediate by severity and exposure. Days to fix critical, internet-facing vulnerabilities, shown as a distribution, not an average, plus the count that breach the agreed deadline.
  4. Exercise results. How long did your team take to detect a simulated intrusion in a purple team test? This measures reality instead of recording history.
  5. Recovery tested. The proportion of critical systems whose restore procedure was actually exercised in the last year, and whether recovery met the stated target time.
  6. Alert precision for each detection rule. True positives divided by all alerts. It tells you which rules deserve tuning or retirement.

Different audiences, different views

The SOC manager needs operational detail: queue ageing, rule precision, workload per shift. The CISO needs trends and risk: top exposures, remediation against deadlines, incidents by cause. The board needs a handful of business-linked statements: are critical services recoverable within target, what is the loss exposure of our most serious risks, and are we meeting obligations such as CERT-In reporting windows and the DPDP Act’s expectations. Showing a board a chart of blocked attacks per month says almost nothing, since that number rises with internet noise, not with risk.

Leading and lagging

Lagging indicators, such as incidents and breaches, tell you what already happened, and for a small firm there may be only one a year, too few to draw a trend. Leading indicators, like patch latency, MFA coverage and the share of privileged accounts reviewed, show conditions that make incidents likelier. Use both, and be candid that a quiet year can mean good security or good luck. Absence of incidents is weak evidence.

One last opinion. Every metric on a dashboard should come with the decision it is meant to inform. If no one can name the decision, remove the metric.

Figures and names in the examples are illustrative.

Leave a Reply

Trending