Picture a mid-sized online grocery business in Hyderabad. It is Monday evening, and the operations head glances at the dashboard: 3,150 orders so far for the day. Is that bad? Last week’s total was higher, but last week was a different day. Nobody can say for sure. They wait until Tuesday’s review. By then, a failing payment gateway integration on one Android app version has been losing orders for twenty-four hours.
This is the problem anomaly detection solves, and it does not need machine learning to start. It needs a good definition of “normal” and a rule for “too far from normal”.
Define normal by comparing like with like
The first mistake in the grocery story is comparing Monday with Sunday. Most businesses have a weekly rhythm, so the sensible baseline for a Monday is previous Mondays. Suppose the last eight Mondays gave 4,120, 4,260, 3,980, 4,180, 4,300, 4,050, 4,210 and 4,190 orders. The mean is about 4,161 and the standard deviation about 107.
Today’s 3,150 is roughly 1,010 below the mean, a drop of about 24%. Divide by the standard deviation and you get a z-score near minus 9.5. In plain words: this Monday is nine and a half “typical wobbles” below normal. Ordinary variation almost never produces that.
The same logic extends. Compare against the same hour of the same weekday if you monitor intraday. Remove known events, such as a sale day or a bank holiday, from the baseline or give them their own baseline. If your numbers swing around festivals, compare with the same festival window a year ago, as far as the history allows.
Make the baseline hard to fool
Mean and standard deviation get dragged by past anomalies. If one of those eight Mondays had been a 2,000-order outage, it would drag the mean down and inflate the spread, and tomorrow’s smaller problem would hide. A sturdier approach uses the median and the median absolute deviation (MAD). For the eight Mondays above, the median is 4,185 and the MAD is 70. Scaling MAD by 1.4826 makes it comparable to a standard deviation for roughly bell-shaped data, and today’s score becomes about minus 10. Same verdict here, but the method would not have been fooled by a polluted history.
With a short history, say eight data points, treat these numbers as rough guidance. Thresholds from a handful of points are noisy.
The threshold is a business decision
If you alert at three standard deviations on a bell-shaped metric, about 0.3% of normal days will trigger a false alarm, roughly one in 370. Monitor 200 metrics each day and you will see about half a false alarm a day from that alone. In practice, business metrics are not perfectly bell-shaped, so false alarms are more common than the textbook figure.
Too many alerts and people mute the channel. Too few and the gateway failure runs for a day. A few ways teams tune this:
- Alert only on metrics that tie to money or customer promises: orders, payment success rate, delivery SLA.
- Require the breach to persist for two consecutive checks, trading a little speed for fewer false alarms.
- Use two tiers: a warning at about 2.5 deviations posted quietly, a page at 4 or more.
- Ignore tiny absolute changes. A 40% drop in a metric that normally has five events a day is noise.
From “something is off” to “this is why”
An alert that says “orders down 24%” starts a hunt. The useful systems attach a breakdown automatically: orders by app version, by city, by payment method, by new versus returning customer. In the Hyderabad story, that breakdown would show one app version carrying almost all of the shortfall, and the investigation shrinks from a day to an hour. If your tooling cannot do that, even a pre-built dashboard page linked from the alert is a big improvement.
Remember also that not every anomaly is bad news. A sudden spike could be a viral post, or a duplicate-event bug in tracking that is double counting orders. Check the pipeline first when numbers look too good.
When simple rules stop working
Rules like this struggle with metrics that have strong trend and several overlapping seasonal patterns, or with thousands of series where nobody can tune thresholds by hand. That is where forecasting-based approaches come in: predict the expected value with an interval, and flag points outside it. The idea is identical, normal first and surprise second. Start with the Monday rule anyway, because you can explain it to anyone in the room, and an alert people understand is an alert people act on.
Figures and names in the examples are illustrative.
Leave a Reply