In this lesson, we're going to go over problems you might face in interviews focused on A/B testing and statistics.

Statistics & AB Testing Interview

Introduction

Why do A/B testing and statistics questions get asked?

Introduction to Statistics and A/B Testing

Hypothesis testing covers the fundamental theory and background behind A/B Testing. In this course we'll cover Z and T test, multiple hypothesis testing, and the different type errors.

Fundamentals of Hypothesis Testing

The Z and t test

Z and t Tests

Two-sample Tests

Proportion Testing

Different Parcel Effectiveness

Analysis of Variance (ANOVA)

Hundreds of Hypotheses

Non-parametric Tests

Non-Normal AB Testing

Hypothesis Testing

Let's start with a general framework for A/B testing. In practice, an A/B testing and experimentation all follow a step by step process of setting metrics and designing experiments.

General Framework

Before Testing

Setting Metrics

Experiment Design

Randomization

Analyze Results

Button AB Test

Experiment Validity

P-value to a Layman

New UI Effect

A/B Testing & Experiment Design

Confidence intervals help us deal with this imprecision by giving us a way to talk about a range of values with some certainty where the true value of the statistic is contained in.

Introduction to Confidence Intervals

Confidence Interval Fundamentals

Confidence Intervals for Statistics

Confidence Interval Explanation

Bounds for Coffee and Tea Drinkers

Reducing Error Margin

Confidence Intervals

The next couple of chapters will cover common scenarios and concepts involved in A/B testing. As A/B testing involves statistical concepts, there may be terms that you need refreshing on.

Overview

Multiple Comparisons

Debugging A/B Tests

Holdback Effects

Cannibalization

Non-Normal A/B Testing

Random Bucketing

Sample Size Bias

Twenty Variants

A/B Testing Common Scenarios

There are scenarios where A/B testing is not necessarily the best course of action. Often, there are technical, infrastructure, or practical concerns that come up while planning an A/B test.

When you shouldn't A/B test

Before and After Testing

Island Testing

Common Pitfalls

Network Experiment Design

A/B Test Ties

A/B Testing Tradeoffs

This is a refresher on some important statistical concepts that will help us with A/B testing and beyond. While by no means a comprehensive guide, this chapter will go over some important basics about statistical testing and probability distributions.

Statistical Concepts

Probability Distributions

Advanced Concepts

2X - Y

Estimating D

Location Feature Sharing

Significance Time Series

Time on FB Distribution

Survey Response Randomness

Incorrect Packets

Statistics

In this course we’ll go over the core concepts of causality, significance, and analyzing data. This is meant as a quick refresher and a high level overview of causal inference basics to eventually apply them in data analytics problems.

What is Causal Inference?

Determining Causality

Causal Inference Case Study

Advanced Causal Inference Techniques

Causal Inference Tradeoffs

Boarding Times Bias

Email Discount Effectiveness 

Approval Drop

Data Analytics Fundamentals: Causal Inference

Regression models are used to predict the value of a dependent variable from one or more independent variables.

Introduction to GLMs

Linear Regression

Lasso vs Ridge

Assumptions of Linear Regression

Using R Squared

Logistic Regression

Coefficients of Logistic Regression

Perfectly Separable

Softmax vs Logistic

Time Series Model

Why Do We Need Time Series Models?

AR and MA

Generalized Linear Models and Regression

Statistics and A/B testing are the underlying parts of the data science knowledge base. Knowing how to run an A/B or multivariate test is table stakes.

Refresh your knowledge on statistics, distributions, and applying A/B testing

Let’s say you want to test the close friends feature on Instagram Stories.

How would you make a control group and test group to account for network effects?

What could be a potential risk when Facebook segments the metrics of the user interface change by market or demographic groups?

What could be a potential risk when Facebook segments the metrics of the user interface change by market or demographic groups?

Network Experiment Design I

A Simpson's paradox scenario might occur where the result in most markets differs from the aggregate result

Specific markets might reject the change, skewing overall results

Different demographics may use the interface in different ways, creating inconsistent data

Segmented data may reveal privacy issues, resulting in legal implications

It may highlight unfavorable differences between markets, leading to user dissatisfaction

What is the primary metric that Facebook should monitor if it decides to make the user interface of its posting feature more like Instagram's?

What is the primary metric that Facebook should monitor if it decides to make the user interface of its posting feature more like Instagram's?

Network Experiment Design II

Increase in the number of photo posts per active user

Increase in total time spent by users on the platform

Increase in the number of likes and shares per post

Decrease in the number of posts containing links

If Facebook changes its user interface to mimic Instagram's, which adverse effect should they anticipate and monitor?

If Facebook changes its user interface to mimic Instagram's, which adverse effect should they anticipate and monitor?

Network Experiment Design III

A decrease in status updates, geo-location shares, and items for sale

Increase in data storage requirements due to more photo uploads

Increase in user interface load time due to graphical elements

Decrease in the number of likes and comments due to interface confusion

Decrease in the number of active users due to change resistance

Statistics & AB Testing Interview

Network Experiment Design
Go to question details page

54%

Statistics & AB Testing Interview

Network Experiment DesignGo to question details page

54%

Network Experiment Design
Go to question details page