“We surveyed 50,000 customers, so the result must be reliable.” It sounds airtight. It isn’t, and the reason is the most important idea in sampling: size and fairness are two separate properties. A huge sample can be badly skewed. A small one can be perfectly fair.

What a large sample actually buys

Size reduces random error, the wobble you get because you happened to pick these people and not those. For a yes/no question where about half say yes, the 95% margin of error is roughly 1.96 times the square root of (0.5 times 0.5 divided by n). With 1,000 responses that is about 3.1 percentage points. With only 20 responses it balloons to about 21.9 points. I ran the numbers, and they are why a poll of 1,000 can speak for a country while a poll of 20 cannot speak for a class.

But the formula assumes every member of the population had a known chance of being picked. That assumption is where the 50,000-person survey falls apart.

Where bias comes from

Say a payments app wants to know how satisfied users are with its new bill-pay feature, and shows a rating pop-up inside the app. Fifty thousand people respond. But the pop-up reaches only people who opened the app that week, and rating it takes effort that mostly the delighted and the furious bother with. Lapsed users, the ones who may be unhappy enough to have left, are missing completely.

Adding another 50,000 responses of the same kind would make the answer more precise and just as wrong. Errors from who is included do not shrink with volume. This is convenience sampling, taking whoever is easy to reach, and it is behind a great many internal “insights”. Survivorship bias and non-response bias are cousins in the same family.

Simple random sampling

The standard fix is to start from a complete list of the population, called the sampling frame, and draw members by chance, so that each has the same probability. If you have a customer table, this is easy: assign a random number to each row and take the lowest 1,000. The hard part is the frame itself. If it omits people (customers without email, say), the sample inherits the omission.

Even with a perfect frame, random draws can stumble over small groups. Suppose 2% of a 100,000-customer base lives in a rural segment you care about. A simple random sample of 1,000 should contain about 20 of them, but the actual count wobbles, with a standard deviation of about 4.4. Some draws will contain 12, others 28. And 20 people, as above, give a margin of error near 22 points. You can’t say much about that segment from so few.

Stratified sampling

Stratifying means splitting the population into groups first (strata), such as metro, tier-2 and rural, and then sampling inside each group. Two flavours:

  • Proportional: each stratum is represented in the sample in the same proportion as in the population. This guarantees the 2% group gets exactly its share, with no wobble from the draw.
  • Disproportional: you deliberately oversample a small stratum, say 200 of the 1,000 from rural, so that you can read it with a margin of error nearer 7 points. When you then add the groups back together for an overall figure, you weight each response by how much of the population it represents. Without that weighting, rural opinions would count ten times too much.

Strata should be chosen because the people inside them are alike and the groups differ on what you measure. Stratifying on something unrelated, like the first letter of a surname, gains nothing.

Other designs worth recognising

Systematic sampling takes every kth record from an ordered list, which is simple but risky if the list has a hidden rhythm (every seventh row is a Sunday). Cluster sampling picks whole units, such as 30 branches and everyone in them, which is cheaper for fieldwork but usually less precise than the same number of individuals chosen at random. Audits often use these; internal controls testing of invoices is a familiar case where an auditor samples by value, since big invoices matter more than small ones.

When sampling is the wrong move

If you already hold every transaction in a table, there is often no reason to sample for a total: just count them all. Sampling earns its place when measurement is costly (surveys, call audits, physical inspection) or when you want a quick approximation from a very large table. Be careful with rare events. A 1% sample of transactions will contain few fraud cases, and any analysis of them will be thin. For those, you keep all the rare cases and sample down the common ones, remembering to weight afterwards.

A better question than “how big is my sample?” is “how did people get into it?” Whenever you read a survey result, find that out first. If the answer is “they chose to”, treat the precision figures with real scepticism, however many zeros are in the count.

Figures and names in the examples are illustrative.

Leave a Reply

Trending