Let’s say your manager is running A/B tests on a website. You find out that your manager has run an A/B test with 20 different variants and come back to you that one of the variants is significant.

Would you think there was anything fishy about the results?

Your manager reconfigured an A/B test from 2 to 20 different variants, while keeping all other parameters unchanged. They come back to you that one of the variants is significant.

Is this conclusion likely to be correct? Why or why not?

Your manager reconfigured an A/B test from 2 to 20 different variants, while keeping all other parameters unchanged. They come back to you that one of the variants is significant.

Is this conclusion likely to be correct? Why or why not?

Twenty Variant Test

Yes, because the number of variants doesn't affect the conclusions of hypothesis tests

No, because doing 20 tests significantly increases the chance of a false positive

Yes, because doing 20 tests more than enough to draw conclusions

No, because doing 20 tests is way too low and the tests should be repeated

Twenty Variants

Comments