How to run an A/B test

Step one: write hypothesis with specifics

Vague hypothesis fails

Not: changing checkout improves conversion.

Specific hypothesis wins

Better: we think simplifying checkout from five-step process to one-page form will increase purchase completion by at least three percent because customers abandon when form feels long.

Hypothesis should state. What you are changing. Why you think it will help. What metric you expect to improve. By how much.

Vague hypothesis leads to vague results. Specific hypothesis is testable.

Step two: determine sample size before starting

The math matters

How many visitors do you need to see real difference.

Formula. You need minimum five hundred conversions per variation. So one thousand total if testing two versions.

Calculate your test duration

If you convert at two percent and get two thousand visitors per month, you get forty conversions per month. Reaching five hundred conversions takes twelve months. Test will not finish in time.

If you convert at ten percent and get five thousand visitors per month, you get five hundred conversions per month. Reaching one thousand takes two months. Test finishes quickly.

Know your timeline upfront

Low conversion rate websites need longer tests. High conversion websites finish faster.

Use sample size calculator. Input your conversion rate and traffic. Calculator tells you how long test takes.

If test takes longer than three months, consider. Is the change worth waiting three months. Or should we test something else that finishes faster.

Step three: pick test traffic allocation

Fifty-fifty is standard

Allocate fifty percent to control. Fifty percent to treatment.

Cautious allocation slows results

Some people allocate ninety-ten. Ninety percent to current good version, ten percent to new version they are unsure about. This is cautious but slows test.

Fifty-fifty is standard. Fastest results.

Step four: determine test duration before running

Commit to timeline

One week minimum. Two weeks standard. One month for very high-traffic sites.

Do not decide duration during test. Decide before. Write it down. Commit to it.

Early peeking ruins tests

Early peeking ruins tests. You check results after one week. Looks like treatment is winning. You stop test. You implement. Later you realize results were not real. You implemented something that did not actually work.

Run full duration.

Step five: make treatment change

One change only

One change only. Not multiple changes.

If testing checkout simplification, simplify form. Do not change color, copy, layout at same time. One variable at a time.

Make it obvious

Change should be obvious. Visitors should notice difference. If change is so subtle visitors do not see it, test will not work.

Step six: set up test in tool

Choose your platform

Use testing platform. . . .

Install and configure

Connect to website. Install tracking code. This usually one line of code or plugin install.

Create versions and metrics

Create control version. This is current version.

Create treatment version. Implement your change.

Set traffic split. Fifty percent each.

Set duration. Two weeks or however long you determined.

Set success metric. What are you measuring. Checkout completion. Form submission. Page scroll.

Step seven: let it run without peeking

Resist the urge

This is hard. Resist urge to check results.

Results after few days are meaningless. Random variation is huge with small sample. You will see results that disappear later.

Set a calendar reminder

Check results exactly on end date. Not before.

Set calendar reminder for test end date. Check then.

Step eight: analyze results correctly

Look for statistical significance

Test finishes. You look at results. Treatment converted at eleven percent. Control at ten percent. Treatment is better. Right.

But was it statistically significant. Is this real or luck.

Look for confidence interval or p-value. Should be above ninety-five percent confidence or below zero point zero five p-value.

What statistical significance means

If statistical significance says yes, result is real. Implement treatment.

If statistical significance says no, result unclear. Could be luck. Could be real but needed more time. Run test longer or try different change.

Step nine: implement and document

If treatment wins

If treatment wins, implement permanently.

Document what you learned. Why you thought it would work. What happened. What you will test next.

Build playbook. Over time, patterns emerge. You learn what works for your audience.

If control wins

If control wins, revert change.

Document why control won. Maybe your hypothesis was wrong. Maybe change was not noticeable enough. Maybe audience did not like change.

Losing test is still learning.

Common first test mistakes

Running test too short

After one week you are impatient. You stop. Results are not real yet.

Testing multiple changes

You changed form and button and copy. Form wins but was it form or button. Now you do not know.

No significance requirement

Treatment is one percent better. Could be luck. You implement. Later it does not hold up.

Testing on everyone

You are nervous about change. You test on ninety percent current, ten percent new. Test takes forever.

Not committing to implementation

Test finishes. Treatment wins. You say we will implement later. You never do. Test created no value.

What separates winners from everyone else

Not smarter testing

Not smarter testing. Consistent testing.

The winning pattern

Winner companies run test. Results come in. They implement immediately. They run next test.

They commit to testing program. Not one-off tests. Ongoing testing. Month after month.

They build internal knowledge. After fifty tests they know their audience. Know what works. Know what does not. They move faster than competitors.

Compound growth

Test once, nothing happens. Test fifty times, everything changes.

Frequently asked questions