### Statistics & AB Testing

36 of 68 Completed

## Non-parametric Tests

So far, all of our tests have been *parametric*, meaning they assume that they make assumptions about the sample distribution. We assumed that the sample followed a normal distribution in the $Z$, $t$, and $F$ tests. In the proportions and $\chi^2$ tests, we assume that the samples follow a binomial or multinomial distribution, respectively.

But sometimes, particularly with small samples, it is not far to make these assumptions. For this reason, we have non-parametric tests that *do not* make any assumptions about the distribution of the sample.

While there are many, *many* non-parametric tests. We will go over two of the most popular ones: $U$ test and the paired signed-rank test.

Please note that there are ways to calculate $p$-values for the test statistics of non-parametric tests, but we don’t describe how to do it here due to their esoteric nature.

# $U$ Test

## Cheat Sheet

**Description**: Tests if the median of two*independent*samples (say $\vec{x}_1$ and $\vec{x}_2$) are different/more than/less than the median of another sample**Statistic**: $m_1-m_2$ (difference of medians)**Sidedness**: Either**Null Hypothesis:**$H_0: m_1=m_2$ (one-sided), $m_1\leq m_2,m_1\geq m_2$ (two-sided)**Alternative Hypothesis:**$H_a: m_1\neq m_2$ (one-sided), $m_1\gt m_2,m_1\lt m_2$ (two-sided)**Test Statistic**: $U=\min(U_1,U_2)$ where $U_1 = \sum_{i=1}^{n_1}\sum_{j=1}^{n_2}S(x_{1,i},x_{2,j}),\quad S(x,y)=\begin{cases}1,\quad\text{if} \ x\gt y\\\\ 0.5,\quad\text{if} \ x=y\\\\ 0,\quad\text{if} \ x\lt y\end{cases}$

$U_2 = \sum_{i=1}^{n_1}\sum_{j=1}^{n_2}S_{-1}(x_{1,i},x_{2,j}),\quad S _{-1}(x,y)=\begin{cases}0,\quad\text{if} \ x\gt y\\\\ 0.5,\quad\text{if} \ x=y\\\\ 1,\quad\text{if} \ x\lt y\end{cases}$

## Description

The idea behind this test is that $U_1$ and $U_2$ are proxies for $\mathbb{P}(X_1\gt X_2)$. In fact, the hypotheses of the $U$-test can be restated as $H_0:\mathbb{P}(X_1\gt X_2)=\mathbb{P}(X_1\lt X_2)$ $H_a:\mathbb{P}(X_1\gt X_2)\neq\mathbb{P}(X_1\lt X_2)$ Since the median is just defined as $m=x$ such that $\mathbb{P}(X\lt x)=0.5$

As stated before, there is a way to calculate a cdf for $U$ and test it against a significance level $\alpha$, but it is beyond this course’s scope and better left to software.

# Paired Signed-ranked Test

## Cheat Sheet

**Description**: Tests if the sample median of a sample at one point in time ($\vec{x}$) is different/more than/less than the median of a sample at a different point in time ($\vec{x}'$)**Statistic**: $m-m'$ (difference of medians)**Sidedness**: Either**Null Hypothesis:**$H_0: m=m'$ (one-sided), $m\leq m',m\geq m'$ (two-sided)**Alternative Hypothesis:**$H_a: m\neq m'$ (one-sided), $m\gt m',m\lt m'$ (two-sided)**Test Statistic**: $W=\sum_{i=1}^n\text{sgn}(\Delta x_i)R_{\vec{x}-\vec{x}'}(\Delta x_i)$## Description

The function $R$ in the $W$-statistic is called the

*rank function*. It returns the index of $x\in\vec{x}$ when $\vec{x}$ is sorted in ascending order. For example, if $\vec{x}=[5,3,8]$, then sorted that would be $[3,5,8]$, so $R_{\vec{x}}(5)=2$.

The $\text{sgn}$ function (read “sign”) in $W$ takes the “sign” of its input. It is defined as:

$\text{sgn}(x)=\begin{cases}1\ \text{if}\ x\gt 0\\\\ -1\ \text{if} \ x\lt 0\\\\ 0\ \text{if} \ x=0 \end{cases}$ So $W$ contains information about the relative ranks of the difference between the observations at the time of $\vec{x}$ and $\vec{x}'$.