Market Opening Experiment
Let’s say you work at Robinhood.
The company ran an experiment where they sent a push notification to users every morning at market opening to remind them that the market was opening.
They sent these notifications to 1,000,000 members to reduce risk and only sent the notifications to current active users (who had installed the app at least four days prior to the experiment).
The experiment returned the following data:
| Metric | Impact | P-Value |
|---|---|---|
| D1_TradingRev/User | +0.12% | 0.1723 |
| D1_OtherRev/User | +0.20% | 0.2992 |
| D1_Revenue/User | +0.32% | 0.0475 |
| Daily_Sessions/User | +1.98% | 0.0022 |
| D14_NetPromoterScore | -0.22% | 0.2021 |
| D1_Retention | +0.03% | 0.0495 |
| D7_Retention | +0.01% | 0.1023 |
| D14_Retention | -0.02% | 0.0819 |
| D1_TimeSpent/ActiveUser | +0.32% | 0.1456 |
| D7_TimeSpent/ActiveUser | +0.64% | 0.0921 |
| D14_TimeSpent/ActiveUser | +0.92% | 0.0433 |
| D1_TimeSpentPerDay/ActiveUser | +0.91% | 0.0644 |
Which metrics do you interpret as significant? Please describe your criteria.
Do you think Robinhood should roll out these push notifications to their entire user base? Why or why not?
.....
Loading comments