In this lesson, we're going to go over problems you might face in interviews focused on A/B testing and statistics.

Statistics & AB Testing Interview

Introduction

Why do A/B testing and statistics questions get asked?

Introduction to Statistics and A/B Testing

Hypothesis testing covers the fundamental theory and background behind A/B Testing. In this course we'll cover Z and T test, multiple hypothesis testing, and the different type errors.

Fundamentals of Hypothesis Testing

The Z and t test

Z and t Tests

Two-sample Tests

Proportion Testing

Different Parcel Effectiveness

Analysis of Variance (ANOVA)

Hundreds of Hypotheses

Non-parametric Tests

Non-Normal AB Testing

Hypothesis Testing

Let's start with a general framework for A/B testing. In practice, an A/B testing and experimentation all follow a step by step process of setting metrics and designing experiments.

General Framework

Before Testing

Setting Metrics

Experiment Design

Randomization

Analyze Results

Button AB Test

Experiment Validity

P-value to a Layman

New UI Effect

A/B Testing & Experiment Design

Confidence intervals help us deal with this imprecision by giving us a way to talk about a range of values with some certainty where the true value of the statistic is contained in.

Introduction to Confidence Intervals

Confidence Interval Fundamentals

Confidence Intervals for Statistics

Confidence Interval Explanation

Bounds for Coffee and Tea Drinkers

Reducing Error Margin

Confidence Intervals

The next couple of chapters will cover common scenarios and concepts involved in A/B testing. As A/B testing involves statistical concepts, there may be terms that you need refreshing on.

Overview

Multiple Comparisons

Debugging A/B Tests

Holdback Effects

Cannibalization

Non-Normal A/B Testing

Random Bucketing

Sample Size Bias

Twenty Variants

A/B Testing Common Scenarios

There are scenarios where A/B testing is not necessarily the best course of action. Often, there are technical, infrastructure, or practical concerns that come up while planning an A/B test.

When you shouldn't A/B test

Before and After Testing

Island Testing

Common Pitfalls

Network Experiment Design

A/B Test Ties

A/B Testing Tradeoffs

This is a refresher on some important statistical concepts that will help us with A/B testing and beyond. While by no means a comprehensive guide, this chapter will go over some important basics about statistical testing and probability distributions.

Statistical Concepts

Probability Distributions

Advanced Concepts

2X - Y

Estimating D

Location Feature Sharing

Significance Time Series

Time on FB Distribution

Survey Response Randomness

Incorrect Packets

Statistics

In this course we’ll go over the core concepts of causality, significance, and analyzing data. This is meant as a quick refresher and a high level overview of causal inference basics to eventually apply them in data analytics problems.

What is Causal Inference?

Determining Causality

Causal Inference Case Study

Advanced Causal Inference Techniques

Causal Inference Tradeoffs

Boarding Times Bias

Email Discount Effectiveness 

Approval Drop

Data Analytics Fundamentals: Causal Inference

Regression models are used to predict the value of a dependent variable from one or more independent variables.

Introduction to GLMs

Linear Regression

Lasso vs Ridge

Assumptions of Linear Regression

Using R Squared

Logistic Regression

Coefficients of Logistic Regression

Perfectly Separable

Softmax vs Logistic

Time Series Model

Why Do We Need Time Series Models?

AR and MA

Generalized Linear Models and Regression

Statistics and A/B testing are the underlying parts of the data science knowledge base. Knowing how to run an A/B or multivariate test is table stakes.

Refresh your knowledge on statistics, distributions, and applying A/B testing

Say you are given a dataset of perfectly linearly separable data.

What would happen when you run logistic regression?

What happens when logistic regression is run on a dataset of perfectly linearly separable data?

What happens when logistic regression is run on a dataset of perfectly linearly separable data?

The model generates a flat decision boundary

Say you are given a dataset of perfectly linearly separable data. What would happen when you run logistic regression?

Machine Learning

Statistics & AB Testing Interview

Perfectly Separable
Go to question details page

54%

Statistics & AB Testing Interview

Perfectly SeparableGo to question details page

54%

Perfectly Separable
Go to question details page