A/B testing best practices

Running test too short (peeking bias)

The mistake

Biggest mistake: checking results after one week when test was planned for two weeks.

Why it is a mistake

One week gives four hundred conversions per variation. Two weeks gives eight hundred. With four hundred, random noise is higher. You see differences that disappear with more data.

Example: week one shows treatment nine percent conversion. Control eight percent. One percent win looks real. You peek and get excited. Week two: treatment drops to eight point five percent. Control stays at eight percent. Half percent win. Still statistically significant but much smaller than week one showed.

If you stopped at week one, you would have over-estimated treatment benefit by fifty percent.

How to avoid

Commit to duration before starting test. Two weeks standard. Write it down. Do not peek. Set calendar reminder for end date.

Testing multiple changes at once

The mistake

You change headline. You change button color. You change button text. You change form fields. Four changes simultaneously.

Why it is a mistake

Treatment converts at twelve percent. Control at ten percent. Two percent win. You implement all four.

Month later: conversion drops to nine percent. Something broke. But which change. Headline. Button. Form. You changed four things and do not know which one hurt. You cannot learn which change was beneficial and which was harmful.

How to avoid

One change only. Change headline only. Test it. Implement if it wins. Then test button color. Then test form. One variable at a time.

Tracking errors and broken tests

The mistake

Test runs for two weeks. Results show treatment winning by five percent. You implement. Month later you realize tracking was broken. Conversion pixel did not fire for ninety percent of visitors.

Why it is a mistake

Results are garbage. Implementation was based on bad data. You spent engineering resources implementing something based on invalid results.

How to avoid

Spot-check test setup. Run a simple manual flow. Does event fire. Check tracking logs. Verify pixel is firing. Audit tracking before declaring winner.

Inadequate sample size

The mistake

You run test with only five hundred total conversions. Two hundred fifty per variation. With sample this small, random noise is huge. Five percent differences are common. Ten percent differences are not shocking.

You see treatment five percent higher than control. No statistical significance. You declare test inconclusive.

Why it is a mistake

You were under-powered from start. You should have known test needed two weeks, not one week. Under-powered tests waste time and produce inconclusive results.

How to avoid

Calculate sample size before starting. How many conversions do you need. How long will test take to get there. If test takes three months, consider testing something else with faster results.

Not controlling for external factors

The mistake

You run test Tuesday through Thursday. Monday results before test. Friday results after test. You compare Friday to Monday.

But Friday is different traffic than Monday. Friday is weekend casual browsing. Monday is weekday work browsing. Different audiences. Different conversion rates. You attribute difference to test when it is really composition difference.

Why it is a mistake

You implemented change based on traffic composition difference, not treatment effectiveness. When new traffic composition arrives, results do not hold.

How to avoid

Run test long enough to balance traffic composition. Two weeks includes weekdays and weekends. One week might be skewed.

Changing test hypothesis mid-test

The mistake

You start test with hypothesis: shorter form increases submissions.

Halfway through, you get new data from support. Customers want more field options. You change hypothesis: more fields might actually help. You now have test designed for one hypothesis being analyzed as different hypothesis.

Why it is a mistake

Confuses results. You cannot tell which hypothesis the test was actually testing. You might implement based on confused interpretation.

How to avoid

Write hypothesis before starting test. Commit to it. If you get new information and want to test different hypothesis, finish current test. Then run new test.

Not implementing winners

The mistake

Test finishes. Treatment wins. You say "we will implement later." Later never comes. Test created no value. You wasted time.

Why it is a mistake

Winners only create value if implemented. Testing program only compounds if you implement winners consistently. One unimplemented winner kills momentum and team belief in testing.

How to avoid

Commit to implementation before running test. If treatment wins, implement within one week. If control wins, revert immediately. Do not let winners languish.

Not segmenting results

The mistake

Overall results show treatment wins. You implement globally.

But segmenting shows: desktop treatment wins. Mobile treatment loses. International treatment has no impact. Global implementation hurts mobile and international visitors.

Why it is a mistake

You optimized for overall average but hurt specific segments. Mobile traffic might grow faster than desktop. By implementing treatment, you hurt future growth segment.

How to avoid

Always check segment results before declaring global winner. Winner might not be universal. Implement for segments where it wins. Skip for segments where it does not.

Confusing winner with causation direction

The mistake

Test shows treatment beats control. You assume treatment is better. But what if control is just broken. Control button does not work. Everyone is confused. Treatment button works fine. Treatment wins because control is broken, not because treatment is inherently better.

Why it is a mistake

You learn wrong lesson. You think button style wins when actually you learned "working button beats broken button." Different implications for future testing.

How to avoid

Understand why treatment won. Was it because treatment is better or control is worse. Different implications.

Repeating same test looking for different result

The mistake

Test one: treatment loses to control. You try again with slight variation. Test two: same result. Control wins again. You try again. Test three: same pattern. Control keeps winning.

You are testing same thing repeatedly and getting same result.

Why it is a mistake

You are wasting testing budget on approach that does not work. Better to pivot to different approach than to keep testing same losing strategy.

How to avoid

Stop testing same thing. Different strategy needed. Current approach is not working. Pivot to different treatment.

Not documenting results and learning

The mistake

Test finishes. Results show winner. You implement. Then forget about it.

Six months later: new team member asks "what did we learn about headlines." You have no documentation. Start from scratch.

Why it is a mistake

You lose institutional knowledge. Every test should teach you something. Without documentation, knowledge disappears when team members leave.

How to avoid

Document every test. Hypothesis. Results. Learning. Share with team. Build institutional knowledge.

Frequently asked questions