Let’s say you work at Robinhood.

The company ran an experiment where they sent a push notification to users every morning at market opening to remind them that the market was opening.

They sent these notifications to 1,000,000 members to reduce risk and only sent the notifications to current active users (who had installed the app at least four days prior to the experiment).

The experiment returned the following data:

Metric	Impact	P-Value
D1_TradingRev/User	+0.12%	0.1723
D1_OtherRev/User	+0.20%	0.2992
D1_Revenue/User	+0.32%	0.0475
Daily_Sessions/User	+1.98%	0.0022
D14_NetPromoterScore	-0.22%	0.2021
D1_Retention	+0.03%	0.0495
D7_Retention	+0.01%	0.1023
D14_Retention	-0.02%	0.0819
D1_TimeSpent/ActiveUser	+0.32%	0.1456
D7_TimeSpent/ActiveUser	+0.64%	0.0921
D14_TimeSpent/ActiveUser	+0.92%	0.0433
D1_TimeSpentPerDay/ActiveUser	+0.91%	0.0644

Which metrics do you interpret as significant? Please describe your criteria.
Do you think Robinhood should roll out these push notifications to their entire user base? Why or why not?

Market Opening Experiment

Comments