Practical Guide to Controlled Experiments on the Web: Listen to Your Customers not to the HiPPO
Experiment!
- Run A/A tests — make sure the variation you are seeing isn’t just natural variation
- Beware Coupon Codes — too prominent and people abandon shopping carts to go look for that discount code
- Two step feedback requests result in better feedback than the stars and the textarea presented all at once.
- Need to experiment across every user to get meaningful data
- Multiple experiments rarely interact — so run multiple experiments (even on same page)
- Drilldown on data — could discover that one browser is having a javascript bug