Demystifying
the
Dreaded A/B Test
It's not that scary
trust me...
Chris DeMars
Developer Advocate
@saltnburnem
- International Speaker
- Love spooky stuff
- Tattoo collector
About Me
What is A/B testing?
What is A/B testing?
A/B testing, otherwise known as split testing, is the process of testing two different versions of a web page or product feature against the original, existing version.
- Talia Nassi
Username
Password
Login
provider
Create Account
A
Username
Password
Login
provider
Create Account
Username
Password
Login
provider
Create
A
B
Why?
⬇️ Answers ⬇️
- Awesome user experience
- Increase revenue
- Data correlation for better business decisions
- Fine grained control of code exposure
How?
Well set up an experiment of course!
That is what we call a moron.
That is not an experiment.
First
First
Gather that data!
Scientific Method
Scientific Method
- Ask a question
- Do research
- Construct a hypothesis
- Experiment
- Draw conclusion
Scientific Method
- Construct a hypothesis
- Experiment
If we ship this new button, our retention will increase by 20%.
Hypothesis
Experiment time!
Username
Password
Login
provider
Create Account
Username
Password
Login
provider
Create
A
B
Control
Treatment
Feature Flags
Segment Users
Username
Password
Login
provider
Create Account
Username
Password
Login
provider
Create
A
B
Control
Treatment
50%
50%
Flex that UI muscle!
You can be a CSS dev and use feature flags!
CSS...like CSS
IYKYK
It's a happy cry!
Go forth and test in production!
Chris DeMars
@saltnburnem