You will have come across A/B page testing. Bonus point to you if you know what A/A page testing is for, answer at the bottom) For A/B testing you do need a significant amount of visitors. How many visitors make a test valid?
– For modest conversion improvements, you may need 5,000–20,000+ visitors PER version before confidence levels become trust worthy
– Lower conversion rates require even larger sample sizes
[sources below]
Panel Data
Watching real people attempt real tasks on your website can reveal issues that analytics alone often miss. Ideally you would have 5 – 10 people in your panel. You would also record their screens.
Give the panel a number of tasks (and if you have a test credit card, let them purchase)
Potential Frictions
- users cannot find the CTA
- messaging is not focused (don’t make people think)
- trust signals are weak
- navigation causes confusion
- forms create friction
PLUS - mobile layouts break the experience
- people misunderstand the offer
Why the Panel?
– Panel testing introduces fresh eyes and realistic behaviour patterns.
– Uncover major usability problems because many friction points repeat themselves quickly.
Points to Remember
User testing and A/B testing are NOT the same thing.
– User testing helps explain behaviour.
– A/B testing helps validate performance changes statistically.
This matters because many businesses run A/B tests with nowhere near enough traffic to produce reliable conclusions. A test result is only useful if the sample size is statistically meaningful.
As a rough guide:
- For modest conversion improvements, you may need 5,000–20,000+ visitors PER version before confidence levels become trustworthy
- Lower conversion rates require even larger sample sizes
- Without sufficient volume, businesses can end up making decisions based on noise rather than evidence. This will lead to incorrect conclusions.
Conclusion
Analytics tells you WHAT happened.
Panel testing and behavioural observation help explain WHY it happened.
A/B testing then validates whether changes genuinely improve outcomes at scale.
The strongest conversion optimisation strategies rarely come from guessing.
They come from:
- observation
- behavioural psychology
- user friction analysis
- data interpretation
- iterative testing
Let me know if we can help further.
Sources
– Optimizely: Offers a Sample Size Calculator
– VWO (Visual Website Optimizer): Industry data from VWO confirms that typical web experiments require thousands of visitors per variation to confidently rule out random chance.
[A/A Page testing the same page against itself. This tests the testing software]
About Conversion Detectives
Conversion Detectives is a full-service digital marketing agency in Hertfordshire.
We find the best quality traffic for your website and convert more of that traffic into sales or sales leads.
Services
Digital Marketing Strategy | Creative Content Marketing | Email Marketing |
Lead Generation | Content Strategy | Social Media Marketing |
Video Marketing | SEO (Basics) | SEM | Shopify Marketing |
Podcast scripting
About Post Author
