The digital marketing landscape is moving faster than ever before, with dozens of new tools, platforms, and advertising media coming into existence every year. When making a decision to adopt one tool over another or making an entrance into a new market, the case study oftentimes becomes an important factor in whether the trigger is pulled.
While a proven tool or medium is certainly less risky to adopt compared to one that is completely unproven, it’s worth remembering that case studies are generally crafted to make the object look as good as possible. The results presented may be distortions of reality, and any single study must be taken with a grain of salt.
In this article, I hope to shed some light on how to effectively review a case study — including what questions to ask yourself or others while reading them.
Case Study Result Metrics
Some case studies contain little more than the fact that a particular marketer was happy with the results of using the certain tool or market. Though such case studies help in a manner similar to product reviews on retail websites, marketers are increasingly demanding hard numbers to quantify the potential gain.
The most commonly cited quantitative results are the following:
Lower CPC or CPM (cost per click/mille)
Higher CTR (click-through rate)
Improved conversion rate
Reduction in CPA (cost per action)
Improvement in ROI/ROAS (return on investment/advertising spend)
Increase in amount of conversions, revenue, or other KPI
While all of these are positive results, some are more important than others, and some can be misleading. The questions below will help cut through smoke and mirrors to help determine how applicable the case study is to a marketer’s business goals.
The questions are broadly classified into three categories: high-level applicability questions, experimental design questions, and results validation.
Applicability
1. Does the subject of the case study have a similar business model and KPI as mine?
This is an obvious but important question. A tool or marketplace that works for one industry may not work for another, depending on the target audience and creative format. If the subject of the case study is in a similar vertical to yours with similar business goals, then the relevance of the study will improve.
2. When was the case study written, and is it still applicable now?
In today’s rapidly evolving digital landscape, case studies that are even two or three years old may become outdated, since the tool features, or market user base and competition, may have changed significantly since then. This is particularly applicable to relatively new markets such as social or mobile.
Experimental Design
3. Was this a simultaneous or sequential test?
Simultaneous tests are generally better than sequential, because seasonality or market changes may have affected results during the test period. However, simultaneous tests also have the challenge of selecting campaigns that are sufficiently similar to each other (an example of control factors, detailed in #6).
4. What was the sample size (number of impressions/clicks/conversions)?
If the sample size of the comparison is too small, the results do not mean much. For example, if 2 conversions occurred for one scenario and 3 for the other, this would be a 50% increase in conversions on paper, but it could have easily happened purely by chance alone.
5. How long did the test run for, and what periods were used in the comparison?
Similar to sample size, a test that is too short has greater potential for noise. However, a test that runs for too long introduces potential for various confounding factors that may have influenced test results (especially in a sequential test).
6. What kind of control factors (variables held constant) were used in the test?
For the comparison to be as fair as possible, it is vital for there to be controls put into place. Spend, for example, is one factor that absolutely needs to have been controlled for in the test, since it is easy to get higher efficiency with a lower budget.
7. Was the result statistically significant?
Related to #4, if the case study was based on too little data, a seemingly good result can be obtained just by chance. Statistical significance will put this doubt to rest; if the subject of the case study can respond to this question with a definitive yes, then there is greater reason to believe that the study was grounded on good data and science.
Results Validation
8. How about the results for [insert metric]?
I rarely see A/B tests where all metrics point towards a positive result. For instance, CPA may have improved, but conversion rate may have gone down (which means the CPA improvement was driven by CPC/CPM decrease). These negative or neutral components of the result are sometimes withheld from print. Particularly concerning is when results of higher-funnel metrics such as CPC or CPM are given, while lower-funnel metrics such as CPA and ROI are not.
9. How do I know that the CPC/CPM will not rise in the future?
Since price is influenced by competitive landscape, there is risk that an early-stage market will become increasingly more expensive in the future. Early adoption of new markets may pay off while the market is still fresh, but results may become stagnant with competition increase.
Conclusions And Tips For Marketers
With data-driven marketing becoming the new norm, case studies are also become increasingly quantified. While there are many metrics that can be presented to seemingly show a positive result, only a small subset may actually be applicable to any single marketer.
The above questions should help shed light on how much to value the results of a case study for a tool, platform, or marketplace.
Comments