Are both qualitative and quantitative research part of your marketing methodology? They should be.
As search marketers, we want to know as much as possible about our target audience so we can deliver the best user experience (UX). To accomplish this, we address the following questions:
What are people searching for? (keywords/labels, file type)
Where are people conducting their searches? (location, Web search, site search)
When are people conducting searches? (date, time)
Who is using the commercial Web search engines? (target audience)
How are people searching? (desktop/tablet/mobile, query/browse/ask)
Why are people conducting searches? (goals, intention, motivation)
To better understand our target audience, researchers use a variety of resources — keyword research tools, Web analytics data, advertising data, and others. However, for our conclusions to be accurate, we should also understand the data and resources in context.
Both qualitative data and quantitative data are critical for understanding our target audience and the impact they have on our businesses.
Quantitative Research Methods
Quantitative research has to do with numerical data and statistics. Many people prefer quantitative research because it often seems simple and straightforward: 40% of usability test participants clicked on the orange button; 27% clicked on the green button. Let’s use the orange button!
Simple and straightforward, right?
Many people feel that the “hard numbers” in quantitative research are somehow more scientific or credible than insights gained from qualitative research. Quantitative research can answer how many and how much types of questions. These numbers can help us prioritize resources.
For example, if a particular blog post goes viral, we can usually see that spike in traffic and social mentions in our Web analytics reports. For future blog posts, writers can focus on this topic and related topics that have a large impact on user engagement and revenue.
One quantitative research method is a large-scale log analysis. Log file data provides a large amount of information within a specified period of time. Web analytics data and keyword research data are types of log analyses. Log file data can tell us what people search for (keywords) and how people search.
We can even learn how searchers interact with search engine results pages (SERPS). Which search listing generated the most clicks — the ad or the organic listing? Which search listing and corresponding landing page had the highest bounce rate? Insights gained from search engine advertising can be applied to organic SEO, and vice versa.
Below are some qualitative metrics that businesses commonly measure:
Amount of traffic generated per resource (such as a Web search engine)
Unique visitors per month (new and repeat visitors)
Time on site
Bounce rate
Number of subscribers (if used)
Number of acquired links
Source of acquired links
Anchor text analysis (on links)
Number of social shares
Social shares per visit
Number of call-to-action clicks
Call-to-action click percentage
Conversion percentages
Number of leads
Cost per lead (CPL)
Number of sales
Revenue generated
Cost per acquisition (CPA)
Unfortunately, the commonly held belief in the “hard numbers” in quantitative research is often misleading.
Quantitative research can tell us how many, how much, and the order in which users complete tasks. But these numbers don’t give us the full picture. These numbers don’t tell us why people search (their goals and intentions behind their keyword selection) or why people do not look at a specific area on a Web page.
Therefore, to keep numbers in proper context, we should rely on qualitative research as well.
Qualitative Research Methods
Qualitative research methods are used for answering questions about why or how to fix a problem. Usability testing, field research, and diary studies are types of qualitative research.
Field research or fieldwork is the collection of information outside of a laboratory or workplace setting. Diary studies are a methodology in which participants record the dates, times, location and context of search tasks. Usability testing measures:
Success rate — i.e., whether or not test participants complete their desired tasks
How efficiently they were able to complete tasks (task time, number of steps, keystrokes, etc.)
Number and types of roadblocks encountered
Error rate
Possible workarounds (error prevention and handling)
Users’ subjective satisfaction
“Usability testing is (or at least should be) a mixed-method approach: both qualitative and quantitative data are collected,” author and usability expert Jeff Sauro wrote in a blog post.
Some people dismiss the validity of usability tests due to low base numbers. How can testing an interface with only five users generate statistically accurate data?
The goal of qualitative research methods is to generate insights and solutions. A minimum of five test participants is needed to identify approximately 85% of the problems in an interface, given that the probability a user would encounter a problem is about 31%.
Over-recruiting is common in usability testing. We often conduct pilot tests to ensure that our questions and scenarios are clear to test participants. Single-user problems can be accepted, rejected or reported as outliers.
Furthermore, many well-intentioned search marketers constantly misinterpret usability expert Jakob Nielsen’s recommendation of testing with five users.
Card sort studies require at least 15 users. Eyetracking studies require at least 39 users. Quantitative tests require at least 20 users. For details, please read How Many Test Users in a Usability Study? by Jakob Nielsen and Why You Only Need To Test With Five Users (Explained) by Jeff Sauro.
Qualitative research helps us put many numbers into the right context. For example, I commonly observe high bounce rates on websites. These bounce rates can be negative (users do not see what they want) or positive (Web page answers their question right away).
If the bounce rate is a negative one, I want to know why. Is perceived download time the problem? Maybe the desired information is on the page, but the information scent goes unnoticed because the link does not look clickable or tappable.
I understand the appeal of Web analytics reports and search metrics. But a number without context is just a number. By keeping customer data in the right context, we can all build better, more effective websites.
Comments