How to build

Kickass Tests...

on day one 


Tai Rattigan


Partnerships EMEA

tai@optimizely.com

linkedin.com/tairattigan

Today:


4 steps before every experiment 

Your first experiment ideas

QA
 
Is it done yet?

Let’s go back and
look at the basics…


4 Steps before every experiment


  1. Define your goals
    • Form Submits?
    • Purchases?
  2. Outline your hypothesis
  3. Define your test subjects / target audience
  4. Calculate your sample size

Ideas… keep ’em coming!

 

Do you have a list / backlog of testing ideas?

Collect centrally!!

Don't have ideas already?

Analytics 

optiverse.com

Usability testing / hallway tests

Heatmaps

Optimizely Partners


Go big



 


Chickity Check ya test 

before you

wreck your test


Pre-Launch Checklist

  • Variation Code - is it appearing and working correctly?
  • Timing - Does your variation show up when it should, without flickering?
  • Targeting - Is your variation on the right page, for the right audience?
  • Goals - Are your goals triggering when they should?

Is it Done yet?


This team has data ,
not opinions.

Ryan Meerstein, Targeted Victory

 Sample Size 

 Statistical 
 Significance 

 Statistical 
 Power 

The right 

approach

1

Collect

Baseline

Just an Original, No variation

w/ Main Goal

Run for 7 / 14 / 30 Days


Goal:  Estimate sample size potential, get baseline conversion

2

Calculate

Sample Size

Use Sample Size Calculator

Enter data from Baseline Experiment

Pick a realistic Sample Size


Goal:  Define sample size for future experiments

3

Start Tests

& Wait ;)

Build and start Experiments

Look at Experiments when Sample Size
reached, not before


Goal:  Get actionable insights that aren't false positives

4

Congrats!

You've collected results
that you can be confident about!

BUT WAIT....

How long do i run the test for?




(Sample size x 2) /  Average visitors per day = Estimate no. of days to run test.

Q&A


Thank you


Slides will be available on slideshare

For any feedback/questions/resources contact me:

tai@optimizely.com

linkedin.com/tairattigan

Brighton CRO Presentation

By tairattigan

Brighton CRO Presentation

More from tairattigan