In this lesson, we're going to go over problems you might face in interviews focused on A/B testing and statistics.

Statistics & AB Testing Interview

Introduction

Why do A/B testing and statistics questions get asked?

Introduction to Statistics and A/B Testing

Hypothesis testing covers the fundamental theory and background behind A/B Testing. In this course we'll cover Z and T test, multiple hypothesis testing, and the different type errors.

Fundamentals of Hypothesis Testing

The Z and t test

Z and t Tests

Two-sample Tests

Proportion Testing

Different Parcel Effectiveness

Analysis of Variance (ANOVA)

Hundreds of Hypotheses

Non-parametric Tests

Non-Normal AB Testing

Hypothesis Testing

Let's start with a general framework for A/B testing. In practice, an A/B testing and experimentation all follow a step by step process of setting metrics and designing experiments.

General Framework

Before Testing

Setting Metrics

Experiment Design

Randomization

Analyze Results

Button AB Test

Experiment Validity

P-value to a Layman

New UI Effect

A/B Testing & Experiment Design

Confidence intervals help us deal with this imprecision by giving us a way to talk about a range of values with some certainty where the true value of the statistic is contained in.

Introduction to Confidence Intervals

Confidence Interval Fundamentals

Confidence Intervals for Statistics

Confidence Interval Explanation

Bounds for Coffee and Tea Drinkers

Reducing Error Margin

Confidence Intervals

The next couple of chapters will cover common scenarios and concepts involved in A/B testing. As A/B testing involves statistical concepts, there may be terms that you need refreshing on.

Overview

Multiple Comparisons

Debugging A/B Tests

Holdback Effects

Cannibalization

Non-Normal A/B Testing

Random Bucketing

Sample Size Bias

Twenty Variants

A/B Testing Common Scenarios

There are scenarios where A/B testing is not necessarily the best course of action. Often, there are technical, infrastructure, or practical concerns that come up while planning an A/B test.

When you shouldn't A/B test

Before and After Testing

Island Testing

Common Pitfalls

Network Experiment Design

A/B Test Ties

A/B Testing Tradeoffs

This is a refresher on some important statistical concepts that will help us with A/B testing and beyond. While by no means a comprehensive guide, this chapter will go over some important basics about statistical testing and probability distributions.

Statistical Concepts

Probability Distributions

Advanced Concepts

2X - Y

Estimating D

Location Feature Sharing

Significance Time Series

Time on FB Distribution

Survey Response Randomness

Incorrect Packets

Statistics

In this course we’ll go over the core concepts of causality, significance, and analyzing data. This is meant as a quick refresher and a high level overview of causal inference basics to eventually apply them in data analytics problems.

What is Causal Inference?

Determining Causality

Causal Inference Case Study

Advanced Causal Inference Techniques

Causal Inference Tradeoffs

Boarding Times Bias

Email Discount Effectiveness 

Approval Drop

Data Analytics Fundamentals: Causal Inference

Regression models are used to predict the value of a dependent variable from one or more independent variables.

Introduction to GLMs

Linear Regression

Lasso vs Ridge

Assumptions of Linear Regression

Using R Squared

Logistic Regression

Coefficients of Logistic Regression

Perfectly Separable

Softmax vs Logistic

Time Series Model

Why Do We Need Time Series Models?

AR and MA

Generalized Linear Models and Regression

Statistics and A/B testing are the underlying parts of the data science knowledge base. Knowing how to run an A/B or multivariate test is table stakes.

Refresh your knowledge on statistics, distributions, and applying A/B testing

You are testing hundreds of hypotheses with many t-tests. What considerations should be made?

What is a key disadvantage of the F-test when compared to multiple t-tests?

What is a key disadvantage of the F-test when compared to multiple t-tests?

The F-test cannot provide specific details such as how much the groups differ or which specific groups differ

The F-test has a higher likelihood of returning false positives

The F-test cannot detect differences in mean between two groups

The F-test is more computationally expensive

The F-test has a higher likelihood of returning false negatives

You are testing hundreds of hypotheses with many t-tests. What considerations should be made?

Statistics & AB Testing Interview

Hundreds of Hypotheses
Go to question details page

54%

Statistics & AB Testing Interview

Hundreds of HypothesesGo to question details page

54%

Hundreds of Hypotheses
Go to question details page