Myth Buster

Psychology, behavioural economics, market research, website conversion and more by Neal Cole @northresearch
Recent Tweets @northresearch
Who I Follow
Posts tagged "cro experiments measure ecommerce conversion"


As online businesses have matured and competition for customers has intensified, conversion optimisation has become an increasingly important method for improving the customer experience and ultimately conversion. Developing a program of online experiments using split (A/B) testing and Multivariate Testing (MVT) enables organizations to identify which new elements of a customer experience have the desired impact on visitor behaviour. This normally follows at least three stages:

  • Understand current visitor behaviour by using web analytics to identify conversion funnels. This helps you to prioritize pages and journeys to review. Start with the low hanging fruit by focusing on pages with the highest levels of traffic and the highest conversion rates.
  • Align webpage objectives with the business goals. This process can help identify low priority objectives that can either be removed or  their prominence reduced  to allow the focus to be on the high priority objectives.  
  • Review priority web-pages and customer journeys to develop test hypothesis to increase conversion (or whatever the desired behaviour is). It is important to segment customers (e.g. new vs existing) as you will need to understand how changes to your website affect different key groups. 

Up to this point I find everyone appears to be in agreement about what needs to be done. However, when it comes to developing  hypothesis for online experiments I have found there are a number of myths that some marketeers have about conversion optimisation. When I originally published this post I outlined 5 such myths, but I have now added a further 2 myths of conversion optimisation:

1. You should follow the rules and principles of conversion optimization.

One such rule I’ve heard is that the fewer clicks to conversion the better. Taking this particular myth first, if more clicks allows the visitor to build trust and engagement with the site, then the opposite may be true. Context is so important in improving the customer experience. There are no rules in conversion optimization, just hypothesis that need testing. Always seek to challenge existing thinking and don’t make assumptions about the customer journey unless you have evidence to back it up.   

2. Apply best practice User Experience (UX) design.

By definition best practice is already out of date and the online world is rapidly changing. I don’t dispute that elements of best practice generate useful ideas, but if we always considered best practice websites would rarely change. Good UX design is an important driver of hypothesis but this doesn’t always align with current best practice. Be bold, be different, test new ideas as otherwise testing will have very limited benefits.

3. Do what customer and usability research tells us to do.

Usability testing tells us if customers can complete a task or transaction, but it can’t tell us how visitors will behave in a real purchase situation, when they might be searching for a dress to impress at a party and their own money is at risk. Real life is very different from a usability test. During research observe what people do and listen to the language they use as this is often more illuminating than their opinions of a particular webpage. Behavioural economics tells us that people are poor at predicting how they will adapt to change and generally are resistant to and dislike change. One of the easiest mistakes you can make is taking what customers say and applying it in a literal way. Listen, observe, and interpret according to a recognized framework of consumer behaviour. 

4. Rely on gurus and online tips.

Anyone who needs to tell you that they are a guru probably isn’t one. Whilst online tips and advice from ‘experts’ can generate ideas, every website is unique. Your visitors and their motivations should reflect your value proposition. You also need to consider your business model and how you generate income. Companies that are serious about online experiments, such as Amazon and, don’t allow their conversion team members to write blogs or tweet about their experiments. They  don’t want to give away a competitive advantage. So, ensure your experiments are tailored to your website and your visitors to ensure they have the highest chance of success. I have some recommendations for additional reading at the end of this post.

5. Focus on the design of a single competitor or best in class website.

It’s good practice to browse other websites as you will see design features, interfaces, tools etc that  generate ideas for testing. However, never fall into the trap of focusing or modelling your site on a single website. This is dangerous as their value proposition, and visitor profile is likely to be different from yours. This means they have good reason to present information and interact with visitors differently from you. Their customers probably also have very different expectations to your visitors. Further, there is no guarantee that they have optimized their site unless you have evidence to support this assumption.  Focus your efforts on better meeting the needs of your own customers and seek to create a user experience that reflects their aspirations and motivations.  

6. If we didn’t experience a fall in sales last time we made a change to our proposition/offer it is not necessary to test the impact of a further change!

There is still a tendency among some marketeers to want to change elements of the offer without first testing. They tell you that they didn’t see a drop in sales last time they made a significant change so they don’t expect one this time. This misses the point as many factors affect sales and unless there is a control it is impossible to isolate how sales might have moved if the offer had not been changed. Further, it is critical to understand how different segments respond to  changes to really understand its impact on the business. Inertia caused by factors such as brand loyalty, perceived risks of switching suppliers and habit formation mean that existing customer behaviour may not change immediately. The danger is that existing customer behaviour (i.e. no change) will hide changes in new customer behaviour as most sites are  dominated by repeat visitors.  Potential customers may be put off from signing up to your offer and you will never be aware of it unless you conduct a controlled online experiment. 

7. If a small change is made to just one element of a customer journey we don’t need to consider conversion for the journey as a whole, just the element we are changing.

Websites are ecosystems. Every element is linked to other parts of the system through their interaction and influence on visitor behaviour. However, people who have been tasked with changing one element of a journey can be prone to focusing on their change in isolation of the customer journey as a whole. This can have serious consequences if not challenged as even a minor change early on in a journey can significantly influence visitor expectations and behaviour. This can result in a reduction in overall conversion that is not anticipated by those working in silos. To avoid such situations ensure that marketeers appreciate the importance of measuring all the relevant conversion rates, including the overall journey conversion rate. 

Myths can develop for all kinds of reasons, but unless they are challenged they can become adopted and soon become embedded in an organisation’s culture. You may have come across other myths relating to conversion optimization, so please drop me a line if you have other examples. Thank you for reading and I hope you found the blog useful.

  • The author: Neal Cole has over 20 years’ experience of working in market research and website optimization for some of the UK’s largest financial services providers and online retailers. Neal is based near Chester, and is currently working in London as a conversion specialist for a large online gaming group. He is a regular contributor to the GreenBook Blog market research website. You can follow Neal on Twitter @northresearch and view his LinkedIn profile

Recommended reading: Influence by Robert B. Cialdini, PHD; Predictably Irrational by Dan Ariely (@danariely); the Upside of irrationality by Dan Ariely; The Wisdom of Crowds by James Surowiecki; Nudge by Richard Thaler (@R_Thaler).