HOW TO USE DATA AND A/B TESTING TO IMPROVE YOUR BUSINESS
@KYLERUSH
KYLERUSH.NET
Head of Optimization, Optimizely
$1 billion expected
$1.1 billion actual
$690 million online
$115 million
1.5 million users
500 experiments
weeks of user testing
668,720,213 GA events
+49% DONATE PAGE
+161% EMAIL sign up
experimentation
observation
data gathering
1. identify goals
Top of the funnel vs. bottom of the funnel
http://www.evanmiller.org/ab-testing/sample-size.html
sample size best practices:
Use 5% significance (95% confidence level)
use 80%+ statistical power
More on sample size:
kylerush.net
first test
control variation
+ 36% revenue
second TEST
control variation
statistical tie
Third TEST
control variation
statistical tie
control
variation
more direct copy resulted in:
+21% conversions
SAVE YOUR PAYMENT INFO FOR NEXT TIME
VS.
NOW, SAVE YOUR PAYMENT INFO
control
More situational imagery resulted in:
+19% conversions
what we started with
5-7 second load time
No CDN
No caching
80% faster time-to-paint
63% reduction in page weight
52% reduction in requests
80% faster resulted in:
+14% conversions
OR
$32 million
Gradual incline instead of steep slope resulted in:
+5% conversion
on an already optimized paged
experimentation
best practices
2. always have
a test running
3. don't be afraid
to fail
placing the up-sell on the conversion page instead of after resulted in:
-44% conversions
668,720,213
Google analytics custom events
Right: 82% of clicks
_gaq.push(['_trackEvent', 'one million infographic', 'name slides right']);
adding field hints resulted in:
63% error reduction
How to use data and a/b testing to improve your business
@KYLERUSH
kylerush.net
Head of Optimization, Optimizely