I recently attended a conference on conversion optimisation. One of the speakers talked about how they had used focus groups to provide feedback on alternative designs for their new home page. One of the designs received much more favourable feedback than any of the other variants. This gave the client confidence that this design would win over their visitors. However, when they used A/B testing to measure the performance of each variant the preferred design completely bombed. They were relieved that they had decided to test the design before changing their home page.
The outcome of the test did not surprise me. How we articulate what we think about a user interface design is very different from how we are likely to behave in reality. When we browse the internet we are normally seeking to complete a task that takes us closer to achieving a personal goal. This is not a group experience.
Further, neuroscience indicates that most of our decisions are taken intuitively, often with little conscious awareness. We use behavioural short-cuts to minimize cognitive load and our emotions can over-ride our rational thought processes.
How often do you browse the internet with a group of people you have never met whilst being observed through a one way mirror? Website navigation and task completion is more often than not undertaken by an individual without any group interaction. This may be changing to an extent with mobile devices, but in most cases these are still people we know.
Context is critical in how we respond to any product or service. And yet focus groups are often conducted in a completely alien and artificial environment which does not match the reality of website browsing. Viewing facilities exasperate this problem as they create a laboratory atmosphere.
Some times it is suggested that focus groups are used before designs are drawn up to understand user’s wants, needs and likes. I would also caution against this as focus groups are problematic on a number of counts:
- A single person can dominate the discussion and respondents will sometimes change their opinion to conform to the majority or because they mistake confidence for knowledge.
- People over-analyse and rationalize the topic when we are looking for their emotional and intuitive response to achieving a goal. People will say one thing but behave in a totally different way in reality. This is because people are extremely poor at predicting future behaviour as we are prone to the influence of short-term emotions and cognitive bias.
- Presentation is critical. Despite explaining to people that a design is a mock-up or not fully complete you can guarantee that the stage of development of a design will influence how people respond to it.
- People dislike uncertainty and when it does exist people look to the actions of others to guide them. ‘Group think’ may also set in if a group is too homogeneous and we seek a consensus of opinion rather than having a show of hands. This is particularly worrying as participants to focus groups are normally recruited on the basis of being aligned to agreed demographic characteristics.
As a result I would certainly avoid using focus groups for evaluating new website pages or journeys. All research methods have their limitations but focus groups are especially problematic and inappropriate in the context of website design.
Individual usability interviews can provide much more meaningful insights about how a website or web page works or not. Learning to listen and observe rather than asking direct questions is likely to provide more useful feedback on how to improve your site. There are also other online research tools that can provide relevant sources of insight. These include:
Eye tracking: AttentionWizard.com - Uses algorithms to predict what page elements visitors are likely to look at before the page goes live.
Feedback from usability & conversion experts: ConceptFeedback.com
Thank you for reading my post and good luck with improving your website’s performance.
- The author: Neal Cole has over 20 years’ experience of working in market research and website optimization for some of the UK’s largest financial services providers and online retailers. Neal is based near Chester, but is currently working in London as a conversion specialist for a large online gaming group. He is a regular contributor to the GreenBook Blog market research website. You can follow Neal on Twitter @northresearch and view his LinkedIn profile.