Statistics courses present tests as a long list of names, which makes choosing one feel like memorisation. In practice the choice comes down to a short chain of questions about your data. Answer them in order and the test usually picks itself.

Question 1: what kind of outcome did you measure?

There are two broad kinds. A numeric outcome is a measurement on a scale: order value, time to resolve a ticket, monthly data usage, test score. A categorical outcome is a label: converted or not, payment method, plan chosen, region. Means and averages belong to numeric outcomes; proportions and counts belong to categorical ones.

Question 2: how many groups, and are they independent?

Comparing two groups is different from comparing three or more. And it matters whether the groups contain different people (independent samples) or the same people measured twice (paired samples), such as each agent’s handle time before and after a training programme.

With those two answers, the common cases fall out like this.

SituationTypical testIf assumptions look shaky
Mean of a numeric outcome, two independent groupsTwo-sample t-test (Welch’s version)Mann-Whitney U test
Mean of a numeric outcome, same subjects before and afterPaired t-testWilcoxon signed-rank test
Mean of a numeric outcome, three or more groupsOne-way ANOVAKruskal-Wallis test
Proportion, two groupsTwo-proportion z-test (or chi-square on a 2×2 table)Fisher’s exact test for small counts
Relationship between two categorical variablesChi-square test of independenceFisher’s exact test for small 2×2 tables
Relationship between two numeric variablesTest on Pearson correlation, or regressionSpearman rank correlation

Comparing two means: the t-test

A t-test compares the difference between two group means with the amount of noise in the data. A large difference relative to the noise gives a small p-value. The classic Student’s t-test assumes the two groups have equal variances. Welch’s t-test does not, and it loses very little when variances are equal, which is why many statisticians recommend it as the default. In Python, SciPy’s ttest_ind uses Welch’s version when you pass equal_var=False.

The t-test also assumes the sample means are roughly normally distributed. With moderate samples of reasonably well-behaved data, the Central Limit Theorem takes care of this. With small samples of heavily skewed data, such as revenue per customer where a few customers dominate, consider the Mann-Whitney U test, a rank-based alternative. Be aware that Mann-Whitney does not test means; it tests whether values in one group tend to be larger than in the other. If the business question is specifically about average revenue, bootstrapping the difference in means is often a better fallback.

When the same subjects are measured twice, use the paired version. It works on each subject’s change, which removes the large differences between subjects. Running an independent-samples test on paired data throws away that advantage and often misses a real effect.

Comparing proportions and categories: chi-square

Suppose a D2C brand wants to know whether payment preference differs between metro and non-metro customers. In a sample of 700 orders:

UPICash on deliveryTotal
Metro300100400
Non-metro180120300

Metro customers pay by UPI 75% of the time, non-metro 60%. The chi-square test of independence asks what counts would be expected in each cell if payment method were unrelated to location, then measures how far the actual counts are from those expectations. If there were no relationship, we would expect about 274 metro UPI orders rather than 300. Across all four cells the chi-square statistic comes to about 17.9 with one degree of freedom, giving a p-value of roughly 0.00002. The relationship is very unlikely to be chance.

The test needs reasonable expected counts. A common rule of thumb is that expected counts should be at least 5 in each cell. When they are smaller, use Fisher’s exact test. Also note that chi-square only says the variables are related; it says nothing about which direction or how strongly. Report the percentages alongside it.

Three or more groups: ANOVA

To compare average delivery time across four warehouses, it is tempting to run t-tests on every pair. With four groups that is six tests, and the chance of at least one false alarm climbs well above 5%. Even with three independent tests at the 5% level, the chance is about 14%.

One-way ANOVA tests all groups at once, asking whether the variation between group means is large compared with the variation within groups. A significant result says “at least one group differs”, not which. Follow it with a post-hoc procedure designed for multiple comparisons, such as Tukey’s HSD, to identify the pairs. When group variances differ a lot, Welch’s ANOVA is the safer variant.

Before any test

Every one of these tests assumes the observations are independent, and that assumption is violated far more often than normality. Repeated orders from the same customer, tickets handled by the same agent, or daily figures that follow a weekly cycle all reduce the effective sample size. If one customer contributes fifty orders, either analyse at the customer level or use methods that account for the clustering.

And the test is the last step, not the first. Plot the data, look at group sizes and spreads, and decide what difference would matter commercially. The test then tells you whether the difference you see is distinguishable from noise, which is a narrower question than whether it matters.

Written by the RP Strativa Team. The payment data and other figures are illustrative.

Leave a Reply

Trending