Statistics & AB Testing
36 of 68 Completed
So far, all of our tests have been parametric, meaning they assume that they make assumptions about the sample distribution. We assumed that the sample followed a normal distribution in the , , and tests. In the proportions and tests, we assume that the samples follow a binomial or multinomial distribution, respectively.
But sometimes, particularly with small samples, it is not far to make these assumptions. For this reason, we have non-parametric tests that do not make any assumptions about the distribution of the sample.
While there are many, many non-parametric tests. We will go over two of the most popular ones: test and the paired signed-rank test.
Please note that there are ways to calculate -values for the test statistics of non-parametric tests, but we don’t describe how to do it here due to their esoteric nature.
- Description: Tests if the median of two independent samples (say and ) are different/more than/less than the median of another sample
- Statistic: (difference of medians)
- Sidedness: Either
- Null Hypothesis: (one-sided), (two-sided)
- Alternative Hypothesis: (one-sided), (two-sided)
- Test Statistic: where
The idea behind this test is that and are proxies for . In fact, the hypotheses of the -test can be restated as Since the median is just defined as such that
As stated before, there is a way to calculate a cdf for and test it against a significance level , but it is beyond this course’s scope and better left to software.
Paired Signed-ranked Test
Description: Tests if the sample median of a sample at one point in time () is different/more than/less than the median of a sample at a different point in time ()
Statistic: (difference of medians)
Null Hypothesis: (one-sided), (two-sided)
Alternative Hypothesis: (one-sided), (two-sided)
The function in the -statistic is called the rank function. It returns the index of when is sorted in ascending order. For example, if , then sorted that would be , so .
The function (read “sign”) in takes the “sign” of its input. It is defined as:
So contains information about the relative ranks of the difference between the observations at the time of and .