The belief goes like this: outliers are bad data, so a careful analyst removes them before doing anything else. It sounds rigorous. It is also how teams delete their best customers, their fraud cases and their one honest signal that a system is broken.

An outlier is just a value far from the rest. That is a statement about distance, not about truth. Whether the value is a mistake, a rare but real event, or something the business urgently needs to see is a separate question, and no formula answers it.

Two common ways to flag one

Take twelve annual salaries, in lakh rupees, from a small team: 6, 7, 7, 8, 8, 9, 9, 10, 11, 12, 14 and 95. The last one is the founder, or a typo for 9.5, or a one-time bonus included by mistake.

The interquartile range method looks at the middle half of the data. Here the first quartile is 7.75 and the third is 11.25 (using the common linear interpolation method; other tools may compute quartiles slightly differently). The IQR is 3.5. The usual fences sit 1.5 times the IQR beyond the quartiles: from 2.5 to 16.5. The value 95 is far outside, so it is flagged. Note that 14 is inside. The fence is tight because the middle of the data is tight.

The z-score method measures how many standard deviations a point sits from the mean. The mean here is about 16.3 and the standard deviation about 24.9, so 95 scores only about 3.2. Many analysts use 3 as the cut-off, so it just scrapes through, but with a slightly bigger spread it would have escaped. This is the masking problem: the outlier itself inflates the mean and the standard deviation, which makes it look less extreme. With small samples, the IQR approach or median-based scores are safer.

Flagged is where the work starts

Before touching a flagged value, ask what produced it. Broadly there are three answers, and each leads somewhere different.

  • A data error: a misplaced decimal, a currency unit mix-up (paise stored as rupees), a test transaction, a duplicated row. Correct it at the source if possible, otherwise exclude it and write down why.
  • A real but unrepresentative event: a corporate bulk order of 5,000 units on a platform where the usual order is two. It is true, and it should not drive the average order value you report for ordinary shoppers. Report it separately.
  • A real and important event: a spike in refund claims from one pincode, an unusually large card transaction at 3 a.m. For fraud and risk teams, the outlier is the whole point of the dataset.

Treating all three the same way, by deleting, throws away information in two of them.

Options short of deleting

Use a robust summary. In the salary list the median is 9 with or without 95, while the mean jumps from about 9.2 to about 16.3. Reporting the median sidesteps the dispute entirely.

Cap, also called winsorising. Replace values above, say, the 99th percentile with the 99th percentile value. You keep the row and its direction (this customer is a heavy spender) while limiting its leverage. This is common in model-building, but record the cap, since it changes results.

Transform. Revenue per customer is usually heavily right-skewed, and taking logarithms pulls the long tail in. A tenfold difference becomes a constant step, which suits many models.

Analyse with and without. If a conclusion survives removal of the extreme rows, say so. If it flips, that is the finding, and the honest report is that the result depends on a handful of cases.

A rule I would defend

Never remove a value only because a rule flagged it. Remove it because you can name the reason it does not belong, and note that reason somewhere another person can find it. A threshold like 1.5 times the IQR is a convention that happens to work on bell-shaped data; on naturally skewed things like income, order values or session times, it will flag a steady stream of perfectly normal big values. In such data, the question is rarely “which points are outliers” and more often “does my chosen measure suit data with a long tail”.

Figures and names in the examples are illustrative.

Leave a Reply

Trending